If the data is on a website, it can be in your spreadsheet. Usually within the week, always for a price agreed before anything starts.
Off-the-shelf scraping tools work beautifully until the site does something interesting — an infinite scroll, a login wall, a price that only appears after a JavaScript call, a layout that differs on every fifth page. That is where point-and-click products stop and custom work starts, and it is where most of the sites worth scraping actually live.
A custom build is written against your specific target. It handles that site's pagination, that site's defences, that site's inconsistencies, and it produces exactly the columns you asked for rather than a generic dump you have to clean before it is useful.
Almost always one of a handful of jobs, wearing different clothes depending on the industry.
Cloudflare, DataDome, PerimeterX, fingerprinting, rate limits, CAPTCHA challenges, login walls, and single-page apps that render nothing in the raw HTML. These are the normal conditions of the job rather than exceptions to it.
The tooling that gets through is a real browser engine driven properly, request pacing that looks like traffic rather than an attack, residential exits where the site demands them, and session handling that persists. Where a login is involved, it is your credentials, on your account, doing what you are entitled to do.
Extraction is the first half. The half that determines whether you can use the output is normalisation: prices parsed to numbers with currency separated, dates in one format, addresses split into components, duplicates collapsed, missing fields marked as missing rather than as an empty string that looks like a zero.
Every delivered dataset gets validated before it reaches you — row counts against expectations, null-rate checks on required fields, and range checks that catch a price column quietly returning empty because a selector moved. Silent data corruption is worse than an outright failure, because you act on it.
You send the URLs and the fields you want, in plain English — no technical spec required. Within 24 hours you get a scope, a fixed price and a delivery date. Most builds take 2 to 5 days, and you see sample output early so a misunderstanding about what a field means costs an hour rather than the whole project.
Then you either get a file, or a running system handed over with scheduling and alerting so it keeps delivering without you thinking about it. Either way you are talking to the person writing the code, from the first email to whatever the site does six months later.
Amazon, eBay, Walmart, Etsy, Zillow, Airbnb, Redfin, Google Maps, LinkedIn, Yelp, Facebook
Python, Playwright, Selenium, Scrapy, BeautifulSoup, Pandas, PostgreSQL, Proxy rotation, FastAPI
Most one-off jobs land between $150 and $800 depending on how many sites, how many fields and how much anti-bot protection is involved. Recurring systems that run daily are quoted as a build fee plus a small monthly amount. The price is fixed before work starts — never an hourly meter.
Scraping publicly accessible data is generally legal in the United States, and US courts have repeatedly upheld that. What matters is what you collect and how you use it — I stay away from personal data behind logins, respect terms where they bind, and will tell you plainly if a request looks like a problem rather than take the money.
Most builds are delivered in 2 to 5 days. You get a scope, a fixed price and a delivery date within 24 hours of describing what you need, and sample output arrives early enough to correct course.
Excel, CSV, JSON, Google Sheets, a direct write into MySQL or Postgres, or a REST endpoint your own systems can query. Most clients take Excel or a Google Sheet that refreshes on a schedule.
No. You describe the website and the details you want in plain English and you get back a finished file, or a system that emails you a fresh one every morning. You never open a terminal.
Sites change and scrapers break — anyone who tells you otherwise is selling something. Delivered systems include alerting so you know immediately rather than discovering it through stale data, and I fix breakages on systems I built.