Competitor Price Monitoring Case Study: A Daily Savings Rate Report, Automated

The short version

A product team at a UK savings provider needed a daily view of competitor rates. Someone was building it by hand every morning from around a dozen pages across four comparison platforms, typing the results into ten ranked tables. It took around two and a half hours a day.

I replaced that with a collector that runs before the team starts work, reads every listing across all four platforms, normalises the results into the exact ten tables the team already used, and emails them. Around 460 products per run, every figure carrying its bank, its source platform, the term exactly as that platform worded it, and its minimum deposit.

The part I would point to first is not the collection. It is that the report checks itself: every run states what each source returned, and if the figures are not from that morning the email says so in the subject line.

The problem

The client's market is priced against a handful of savings marketplaces. If a rival moves a headline rate, the commercial response is measured in days rather than weeks, so the team needs to see the market every morning, not every month.

The existing process was manual. Someone opened around a dozen pages across four comparison platforms, read off the best rates for each term, and typed them into a fixed format: a top ten for easy access, then top tens for six, nine, twelve, twenty-four, thirty-six and sixty month fixed terms, then the same again for cash ISAs. Ten ranked tables, every working day, from four sites that each present their data differently.

It took around two and a half hours a day. It was also fragile in the way manual collection always is: rates get misread, a term gets skipped when someone is busy, and nobody can reconstruct where last Tuesday's figure came from.

What I found

The brief looked like a scraping job. It was really four separate problems wearing the same coat.

Platform one rendered its rates server side, but paginated its listings behind query parameters. Reading the first page returned about a fifth of its products.

Platform two rendered nothing useful in its page source. Its rates arrived after load, and its full inventory sat behind a "Load more" button that had to be exhausted, not scrolled past. It also priced by deposit size, so a single query missed products at both ends of the range.

Platform three published its products in structured markup on some pages and in a client-rendered table on others, and split its listings by account type in the URL.

Platform four served different content to an automated client than it served a browser. Its own freshness timestamp was current, so the page was not cached. It simply withheld a category of third-party rates that it republishes under licence.

What I built

A collector that runs every morning before the team starts work, reads every listing across all four platforms, normalises the results into the ten tables the team already used, and emails them.

• Around 460 products collected per run, across twelve source URLs.
• Terms normalised across four different naming conventions, so "1 Year", "12 months" and "Fixed 12 month" all land in the same table.
• Minimum deposit captured per product, so a rate nobody can actually open is visible as such rather than silently ranked first.
• Per-platform breakdowns alongside the pooled market view, so the team can see both the best rate available anywhere and each platform's own best.
• An audit trail: every ranked figure carries its bank, source platform, the term exactly as that platform worded it, and its minimum deposit.

The output format was not redesigned. The team already knew how to read ten tables in a particular order, and a report that arrives looking like the one it replaces gets used from day one.

The part that matters more than the scraping

Any automated report that quietly serves stale figures is worse than no report at all, because people act on it without checking.

So the system reports on itself. Every run writes a health block naming each source and how many products it returned, so a platform that silently starts returning nothing is visible immediately rather than three weeks later. The email checks the collection date before sending, and if the figures are not from that morning it says so in the subject line and opens with a warning naming the date it actually has.

Reliability was engineered rather than assumed. The collector is triggered from two independent schedulers, retries three times within each run, and finishes hours before the report is due, so a single failure has several chances to correct itself before anyone sees it.

Result

The ten tables arrive daily without anyone opening a browser. The team can also refresh and resend on demand from their phone, which matters on the mornings when a competitor moves early.

More usefully, the figures are now traceable. When a number looks wrong, it takes seconds to see which platform it came from and what that platform called it, instead of a conversation about who typed what.

And the report is honest about its own limits. The licensed rates it cannot see are named as such on the page, so the team knows exactly which slice of the market the automated view does not cover and can source it the right way.

What I would tell someone with a similar report

Replicate the format first, then improve it. The team had ten tables in a fixed order for a reason. Matching that exactly meant the report was usable on the first morning and trusted by the second.

Treat each source as its own problem. Four platforms meant four collection strategies. The one that looked simplest was hiding four fifths of its products behind pagination.

Normalisation is where the value is. Collecting rates is a few hours. Making "1 Year" and "Fixed 12 month" land in the same table, with the minimum deposit that decides whether the rate is real for you, is the work.

Build the report to report on itself. A health block per source and a date check before sending cost very little and are the difference between a system that fails loudly and one that fails into a spreadsheet someone acts on.

Know where the boundary is. If a platform withholds licensed data from automated clients, the answer is a licence and a clear note in the report, not a cleverer scraper.

Scope and outcome

Delivered as a fixed-scope project, two weeks from brief to live: discovery across the four platforms, the collector and normalisation layer, the ten-table email report with per-platform breakdowns, the health and freshness checks, dual scheduling with retries, and on-demand refresh. Running daily, unattended.

This is the worked example behind the competitor price monitoring service, part of the wider marketing data and reporting automation work. The general lessons, including why the scraping is the easy part, are written up on the blog: Automating a Daily Report: The Scraping Is the Easy Part.