How to build a competitor website monitoring workflow

Most website-monitoring automations are very good at noticing that a page changed. That is the easy part. A new footer, a revised cookie banner and a pricing change can all produce the same alert.

The workflow becomes useful after detection: preserve the source, show the exact change and give an analyst enough evidence to judge it. AI can absorb some of the reading and sorting. The commercial decision still belongs to someone who knows why the page was being watched.

Choose the question before the tool

Start with one recurring question, such as:

• Did a named competitor change its pricing, packaging or contract terms?
• Did its product documentation add a capability relevant to an active position?
• Did its website move toward a new audience, use case or category claim?
• Did several small edits together change the competitive story?

The output should be a cited brief for a named owner. If nobody is waiting for the answer, the monitoring will become another news feed.

When to use this workflow

Use it when a small, known set of public pages can inform a real pricing, product, positioning or field decision. Do not use it to crawl the open web without a question, bypass access controls, imitate another user agent or collect information the organisation is not permitted to use.

robots.txt is a crawler-coordination mechanism, not an access-control system. Public availability also does not remove contractual, copyright, privacy or rate-limit obligations. The Robots Exclusion Protocol explicitly separates crawler rules from security and authorisation.

Write down the operating rules

Before opening an automation tool, settle five things:

1. The decision. What could change because of this evidence, and who can make that call?
2. The scope. Which competitors, products and exact URLs are covered?
3. The collection rule. What access method and cadence are permitted?
4. Materiality. Which facts deserve review, and which cosmetic edits should be ignored?
5. Retention and tests. How long will captures remain available, and which known changes must the workflow handle?

If the team cannot name the decision owner or the material-change rule, it is not ready to automate the monitoring.

The workflow, step by step

1. Register each source

Give every monitored source a stable record containing the company, page purpose, URL, expected content, permitted collection method, cadence, owner and last successful capture. A spreadsheet can work for a first pilot; a database becomes useful when versions and relationships grow.

2. Fetch politely and record the run

Use an identifiable user agent, respect applicable crawler directives and terms, limit concurrency and back off after errors. Record the final URL, response status, timestamp and a hash of the captured content. Never turn a blocked fetch into an invitation to disguise the crawler.

3. Preserve a readable snapshot

Keep the original response where permitted, but also extract the meaningful page content into a stable representation. Remove recurring navigation, consent banners and timestamps only through documented rules. Otherwise the same template edit can create hundreds of false changes.

4. Run deterministic comparison first

Compare hashes and structured fields before paying for model analysis. If nothing meaningful changed, close the run. If content changed, create a before-and-after record with the exact affected passages.

5. Give the model a narrow reading job

Give the model the relevant old and new passages, not an entire unlabelled website, and request a structured output:

• What factual statement changed?
• Which product, audience, price, claim or policy does it affect?
• Is the evidence explicit or inferred?
• Which source passages support the answer?
• Which facts remain unknown?
• Does the change match one of the material-change rules?

Reject outputs that do not include source passages or that introduce facts absent from the captured material.

6. Group changes before creating alerts

Several pages may reflect the same release. Group by entity, time window and extracted subject so the analyst sees one candidate finding with several sources, not five alerts. Keep every underlying record accessible.

7. Review the candidate and record the response

Exact duplicates and known template edits can close automatically. Everything material or uncertain should reach an analyst with both versions in view. The analyst can accept the finding, reject it, ask for more evidence or flag the classification rule for repair.

An accepted finding then needs a commercial owner and a status. Perhaps the response is to investigate, update seller guidance, revisit a claim or do nothing for now. Record that choice. Otherwise the workflow measures alerts sent, which says little about whether the monitoring helped.

Give AI the reading, not the authority

The model can extract comparable facts, group duplicate edits and prepare a brief. It should work only from approved captures and quote the passages behind its conclusion. Source permission, materiality and the company response are human decisions. When the evidence is contradictory or incomplete, the correct model output is an unresolved question.

Failure and recovery paths

Design these before launch:

• Blocked or rate-limited source: stop, log the reason and ask the owner to change the approved method or remove the source.
• Page structure changed: keep the raw capture, mark extraction as failed and prevent an empty page from becoming a “deleted product” conclusion.
• Dynamic content missing: compare the captured evidence with a browser-rendered check before changing the collector.
• Repeated cosmetic changes: update the deterministic cleaning rule and replay the stored test cases.
• Model output lacks evidence: return it to the queue; do not publish the summary.
• Contradictory sources: present both and route to a specialist rather than selecting the more convenient one.
• No decision owner responds: escalate or pause that signal class. Unowned intelligence becomes noise.

Test the failure paths before launch

A pilot should prove that:

• A known material change is detected and includes the correct before-and-after evidence.
• A navigation edit is ignored, while a missing page is held for inspection rather than labelled as a discontinued product.
• The model cannot create a finding without attached passages, and uncertain results wait for review.
• The analyst can replay the run, record the response and pause one source without stopping the rest.

Choosing the automation layer

Both code-first and visual workflow tools can run this pattern. Choose based on the team that will operate it, the required hosting and data controls, the complexity of state and recovery, and the visibility reviewers need. The companion n8n vs Make assessment tests this exact workflow rather than comparing feature lists in the abstract.

If you need the source register, review queue, test set and handover built around your actual market decision, see Continuous Market Intelligence.