Data and web scraping A broken collector reports zero, not an error.

There is no single answer, which is why the honest framing is a set of questions rather than a yes or a no. Collecting information a site publishes openly, at a considerate rate, without getting past any access control, is a very different activity from circumventing a login or gathering people's contact details. The questions that decide it are: what does the site's own terms permit, is any of it personal information, is the material itself protected as a database or a copyrighted work, and would the collection burden the source. We answer those in writing per source before building, and we are not lawyers — where a project sits near a line, we will tell you it needs your own legal advice rather than pretend it does not.

Not as a list to market to, no. Australian privacy obligations attach to personal information regardless of how publicly it was displayed, so a dataset of individuals assembled because their details happened to be visible is a liability sitting on your server. Where a project genuinely requires personal information — and some legitimately do — it is scoped with a stated purpose, a retention period and a written basis for holding it, and everything not needed for that purpose is discarded at collection rather than kept in case.

Then we use it, and we look for it first. Advertising platforms, e-commerce systems, accounting tools, property and business registers and most modern services publish a supported route, and it beats collection from a page on every measure that matters: it does not break when somebody redesigns the site, it usually exposes fields the page never shows, and it is permitted rather than tolerated. A meaningful share of the projects that arrive here described as scraping turn out to be integration work, and that is a better outcome for everybody.

Four things, and none of them is negotiable. We will not get past a login, a paywall or an anti-bot control. We will not build a collection of personal information for marketing. We will not ignore a site's published terms or its robots directives, or run at a rate that burdens the source. And we will not lift somebody else's structured database and hand it to you with a new header row. Those refusals cost coverage occasionally, and they are the reason the feeds we build are still running a long time after they were switched on.

It fails loudly, which is the single most important design decision in this kind of work. The default behaviour of a naive collector is to find nothing and record zero, and a zero looks exactly like a quiet week — so businesses make decisions for months on data that stopped arriving. Ours treat an empty result as a fault until proven otherwise, detect structural change rather than absorbing it, and alert a named person the same day. Repairing a source is routine; discovering six weeks late that it broke is not.

In your environment, under your accounts, with access control and a history — not in a tool of ours that you rent by the month. Where the requirement includes data residency, that is a hosting decision made deliberately rather than inherited from whichever platform was convenient. The practical test we hold ourselves to is that removing us should be an inconvenience rather than a migration, and that is only true if the storage and the code were yours from the first week.

Usually, and the answer depends on the source rather than on the intent. Publicly listed prices on a site that permits automated access, collected at a considerate rate, is ordinary market research and businesses have done it by hand for a century. What changes the answer is a site whose terms prohibit it, a page behind an account, or a request pattern that would burden the operator. We read the position per source and tell you which of your targets are in and which are out, rather than quietly collecting everything and hoping.

Because somebody checks it against reality on a schedule and writes down what they found. A fixed sample is opened at the source and compared with what the pipeline recorded, monthly, by a person, with the discrepancies recorded rather than silently corrected. Pipelines produce output with a confidence entirely unrelated to whether it is true, and the only reliable defence is a human reading a small sample regularly. If a supplier cannot tell you when their data was last verified by hand, it has not been.

Yes, and it is often the most valuable version of this work. Your CRM, your booking system, your accounting package and your advertising accounts each hold part of the picture and none of them holds all of it, so the job is joining them on an agreed key and deciding what one customer means across all four. That is your own data, accessed with your own credentials through supported routes, and none of the collection questions on this page apply to it.

A maintained feed rather than a file, and the machinery behind it. That means the collection code in your repository, the written source rules, the record definitions and matching logic, the schedule, the alerting, the run history, and a runbook a competent person on your side can follow without calling us. Where the output is meant to be read rather than queried, it lands in a dashboard or a warehouse table you own. What we do not hand over is a spreadsheet, because a spreadsheet is how these projects die.

A single well-behaved source with an official route can be standing within a fortnight. Several sources with no supported access, real matching problems and a residency requirement is a longer engagement, because the work is in the definitions rather than in the collecting. The fee scales with the number of sources, how hostile they are to being read, and how much cleaning and matching sits between raw and usable — quoted after we have looked at the sources rather than before. The first conversation, including the honest read on whether you need this at all, is free.

The published platform work is the closest reference and it is linked from this page: an operations system handling quoting, invoicing and dispatch, and a marketplace operations dashboard where the data model rather than the interface was the difficult part. What we deliberately do not publish is a row count, an uptime percentage or a saving figure for a collection project, because no such figure exists on this site and borrowing one from elsewhere would be exactly the kind of confidently wrong number this page is about.