The brief said scraping
A product team at a UK savings provider needed a daily view of competitor rates. Their market is priced against a handful of savings marketplaces, and if a rival moves a headline rate, the commercial response is measured in days. So every working day someone opened around a dozen pages across four comparison platforms, read off the best rates for each term, and typed them into ten ranked tables: easy access, six fixed terms, then the same again for cash ISAs. Around two and a half hours a day, every working day.
The request was to automate that. Written down, it looks like one job: fetch some pages, pull some numbers out, send an email. That framing is where most reporting automation projects go wrong, because the scraping turned out to be the smallest part of the work. The full client story is on the competitor rate monitoring case study; this post is about what the job actually consisted of, because the same pattern shows up in every manual report I have replaced.
Four problems wearing the same coat
Four platforms, four completely different collection problems.
The first rendered its rates server side, which is the easy case, but paginated its listings behind query parameters. Reading the first page returned about a fifth of its products. A collector that stopped there would have produced a plausible, confidently wrong top ten every single morning.
The second rendered nothing useful in its page source. Rates arrived after load, and the full inventory sat behind a "Load more" button that had to be exhausted, not scrolled past. It also priced by deposit size, so a single query missed products at both ends of the range.
The third published structured markup on some pages and a client-rendered table on others, and split its listings by account type in the URL. Two collection strategies for one site, plus a URL map.
The fourth served different content to an automated client than it served a browser. Its own freshness timestamp was current, so this was not caching. It withheld a category of third-party rates that it republishes under licence.
The lesson is not that comparison sites are awkward. It is that "collect the data" is never one task. Each source needs to be tested against the question "did I get all of it?", and the source that looks simplest is often the one quietly hiding four fifths of its inventory.
The one you do not engineer around
That fourth platform is the finding worth dwelling on. It would have been possible to make an automated client look more like a browser. I did not, and the reason matters more than the technique.
Those rates originate with a market data provider that licenses them commercially. The platform withholding them from automated clients is not a bug to be routed around; it is the licence being enforced. The right route to that data is a licence, so the report now says on its face which category of rates it does not include and why, instead of quietly under-reporting the market and letting the team believe they were seeing all of it.
A report that names its own gaps is more useful than one that hides them. The team knows exactly which slice of the market the automated view covers, and can source the rest the proper way. A cleverer scraper would have given them a number they could not defend.
Normalisation is where the value is
Once the data is in, the actual work starts. Four platforms meant four naming conventions for the same product. "1 Year", "12 months" and "Fixed 12 month" all had to land in the same table, along with every other term the team tracked, and a normalisation rule that misfiles one of them produces a ranked table with a hole in it that nobody notices.
Minimum deposit mattered just as much. A rate nobody can open without a large sum is not the best rate available to a typical saver, and a table that ranks it first is misleading. Capturing the minimum deposit per product and showing it alongside the rate turned a list of numbers into something the team could actually price against.
Then provenance. Every ranked figure carries its bank, its source platform, the term exactly as that platform worded it, and its minimum deposit. Around 460 products a run across twelve source URLs, each one traceable. When a number looks wrong, it takes seconds to see where it came from and what the platform called it, instead of a conversation about who typed what.
The output format was not redesigned. The team already read ten tables in a fixed order, and the automated version is the same ten tables in the same order, with per-platform breakdowns alongside the pooled view. A report that arrives looking like the one it replaces gets trusted on the second morning.
A report has to report on itself
Any automated report that quietly serves stale figures is worse than no report at all, because people act on it without checking. A manual process fails visibly: the email does not arrive, someone asks. An automated one that breaks tends to keep sending yesterday's numbers with today's date, and it can do that for weeks.
So the system reports on itself. Every run writes a health block naming each source and how many products it returned. A platform that silently starts returning nothing, because it redesigned a page or changed a URL, is visible in the next email rather than three weeks later when a rate looks suspiciously static.
The send step also checks the collection date before it does anything. If the figures are not from that morning, the subject line says so, and the email opens with a warning naming the date it actually holds. Nobody has to remember to check freshness, because the report refuses to pretend.
Reliability was engineered rather than assumed. The collector is triggered from two independent schedulers, retries three times within each run, and finishes hours before the report is due, so a single failure has several chances to correct itself before anyone sees it. The team can also refresh and resend on demand from a phone, which matters on the mornings a competitor moves early.
What this does not do
It does not get the team data they are not entitled to; the licensed rates are named as a gap, not filled in. It does not decide what to do when a rival moves; it gives the team the market at nine instead of eleven, with figures they can trace. And it is not immune to change: when a platform redesigns, that source's collector will return less, and the health block will say so the same morning. That is the design working, not failing. The fix is a small one-off job on one source, which is the trade you make for never typing the tables again.
Where to start with your own
If someone on your team builds a report by hand at a fixed time, the questions to ask before automating it are not about tooling. Write down every source and check whether the page you read is the whole inventory or the first page of it. Note every place two sources name the same thing differently. Decide which fields decide whether a figure is real for you, the way minimum deposit did here. Then insist the automated version can tell you, in its subject line, when it should not be trusted.
The scraping is a few hours. The rest is the job, and in this case the whole build took two weeks from brief to live. More on how I scope this work on the competitor price monitoring and reporting automation pages, and the same "one repeatable run instead of a manual routine" thinking applied to keyword research in why I stopped using the Keyword Planner UI.