Google Reviews ScraperGoogleReviewsScraper

The scraper targets Maps rather than Search, which is far more permissive about automated access. The paid alternative is a reviews API, and for one list of businesses that is a subscription bought for an afternoon of work. The other alternative is the Business Profile API, which only helps if you own the listing. What was left was a scraper a person can run for free and leave running. That framing is what pushed the design toward state and away from parsing.

grscraper init creates data/scraper.db with three tables. businesses carries the input value, its classified type, the resolved place metadata, and a status beside retry_count and status_reason. A UNIQUE (place_fingerprint) constraint means the same place ingested twice, once by name and once by URL, lands once. reviews uses Google’s data-review-id as its primary key, so a rerun writes nothing when nothing changed. Each row keeps the reviewer, their counts, the rating, the relative date, the text, photo URLs, any owner reply, and the scraper version that produced it. scrape_runs records each run with its worker count and totals. Because status lives on the row, a crash resumes where it stopped.

browser.py drives Playwright Chromium and selectors.py is the single chokepoint for Google’s DOM. Every class name lives there: div.jftiEf[data-review-id] for a review card, .wiI7pd for its text, .rsqaWe for the relative date, h1.DUwDvf for the place title. When Google rotates classes there is exactly one file to fix. The reviews pane opens through its tab, then scrolls with a randomised pause of 1500 to 2500 milliseconds. Google often withholds that pane from automated clients without ever showing a CAPTCHA, so stealth.py masks the usual signals. GRS_BROWSER_CHANNEL, GRS_HEADED and GRS_TIMEZONE exist so a deployment can test what actually helps.

• Stop on the block page rather than retry. Finding /sorry/index in the URL halts the run, exits with code 2 and marks the business blocked. Pushing harder while blocked makes the block worse. One bad target also ends a batch that might have finished.
• SQLite instead of a queue or a database service. It is a file, it survives a crash, and a container only has to mount /data. Writes stay single threaded, which is why raising --workers is not really the point.
• Two runtime dependencies, Playwright and BeautifulSoup. The S3 and GCS sinks import their SDKs lazily and declare them as extras, so the base image stays small. Anyone using those sinks has to remember the extra.
• No pinned user agent by default. A pinned string goes stale as the bundled browser updates, and an agent string that disagrees with the engine is a fingerprinting signal of its own. GRS_USER_AGENT is there for whoever disagrees.

Five commands take a name to a CSV.

grscraper init
echo "value" > inputs/test.csv && echo "acme coffee roasters" >> inputs/test.csv
grscraper ingest inputs/test.csv
grscraper -v run
grscraper export reviews --format csv

The export lands in data/exports/ with a UTC timestamp in the filename. grscraper sync collapses those steps into one idempotent command and hands the result to every configured sink, which is what the container runs by default. A sink is a URI: stdout, a file path, an HTTPS endpoint, s3:// or gs://. Exit codes carry the outcome, where 0 is clean, 2 is a CAPTCHA and 3 means the scrape worked and a sink did not.

Proxy rotation, distributed workers and mid-business resume are all deliberately absent. Every run re-scrapes a business in full, and deduplication makes that a no-op rather than an incremental fetch. Two things consumers get wrong. The Maps surface carries no absolute review dates, only relative ones, and the advertised review_count is not the number of reviews you actually collected. Tests run under pytest and ruff on Python 3.11, 3.12 and 3.13 in CI. The suite is offline by design: fixture HTML, a temporary SQLite file and a loopback server, with no browser installed. It is MIT licensed.