About our crawler

Who we are

Coffee Near Me is an Australian price-discovery service for coffee. We help locals find good-value coffee nearby. To do that, we read publicly available cafe menus and pair the prices we find with community submissions.

What we crawl

First-party cafe websites only. We follow robots.txt on every domain. We don't scrape Instagram, Facebook, Linktree, or third-party ordering platforms like Toast or Mr Yum. If a menu is only published on social media, we skip it.

How we identify ourselves

Every request from our crawler carries this User-Agent header:

CoffeeNearMe/1.0 (+https://coffeenearme.au)

We fetch your robots.txt first, cache it for 24 hours per domain, and honour every Disallow rule we find.

Politeness

• No more than 2 concurrent requests per domain
• Minimum 1.5 seconds between requests to the same domain
• 20-second timeout per request
• We honour Retry-After on 429 and 503 responses

How to opt out

Email hey@coffeenearme.au with the cafe name or domain you want excluded. We'll add it to our block-list and stop fetching within a day. You can also block us at the server level — our User-Agent above is stable and unique.

How we present prices

Prices on Coffee Near Me come from two sources: community submissions and menu-extracted prices. Both are labelled on each cafe page so visitors can see where a price came from. When both exist for the same drink, the community price wins.