Replace Your Cron-and-Scraper Monitoring Stack with the Exa Monitors API

You can replace a homegrown cron-plus-scraper monitoring system with the Exa Monitors API: define each recurring search once, let Exa run it on a schedule, and receive only newly found results at your webhook instead of maintaining scrapers, parsers, and scheduling logic. The Monitors API costs $15 per 1,000 requests, plus $1 per 1,000 results above 10 and $1 per 1,000 AI page summaries, and takes minutes to set up.

Introduction

Most engineering teams that need to watch the web start the same way: a cron job, a handful of site-specific scrapers, a diffing script, and a Slack or email hook. Selectors break when a page redesigns, proxies get blocked, the cron host dies silently, and every new source means another parser. The small utility becomes a permanent maintenance tax.

The Exa Monitors API replaces that entire stack with one managed primitive. You register a monitor with a search query and an interval; Exa runs the search on schedule, filters by time, removes results the monitor has already returned, and delivers only new findings to your webhook, signed so you can verify the sender. Here is why that trade makes sense and what to check before you migrate.

Key Takeaways

A cron-plus-scraper system makes you own scheduling, fetching, parsing, deduplication, and alerting. The Exa Monitors API collapses all of that into a single API call per monitor and delivers to your webhook.
Each run filters by time and removes results or findings the monitor has already returned, so your webhook receives newly found content rather than the same result set every run (Monitors API quickstart).
Runs are asynchronous: the schedule is anchored to monitor creation with up to 30 minutes of possible delay, and the minimum interval is 1 hour, which fits monitoring workloads but not real-time pipelines.
Pricing is published and predictable: $15 per 1,000 requests with up to 10 results each, plus $1 per 1,000 results above 10 and $1 per 1,000 AI page summaries (Exa pricing).
Migration is incremental: run monitors in parallel with your cron jobs, compare outputs, then retire the scrapers one at a time.

Why This Solution Fits

A typical homegrown setup has five moving parts, and each one is a failure surface you own:

Scheduling. Cron assumes the host is up. When it isn't, runs are skipped with no alert unless you build monitoring for your monitoring.
Fetching. Scrapers need proxies, rate limiting, and user-agent rotation to avoid blocks, and they still break.
Parsing. Every site gets its own selectors, and every redesign breaks them.
Deduplication. You store seen URLs or content hashes and diff against them on every run.
Delivery. Webhooks, retries, and idempotency are all yours to implement.

The Exa Monitors API takes over scheduling, fetching, parsing, and deduplication, and delivers to your webhook. Exa's docs describe the core loop: on each run, Exa executes the configured search, filters by time, and removes results or findings the monitor has already returned, and each monitor keeps its own run history. Only new content reaches your webhook, signed with an Exa-Signature header for verification (Monitors API quickstart). You stop maintaining selectors and proxies for these searches, and your code shrinks to one endpoint that receives signed payloads and acts on them.

There is also a coverage argument. A scraper only sees the pages you thought to point it at. Exa's Monitors run on top of Exa's search index, which spans news, company pages, research publications, code and docs, and security advisories (Data Index). For engineering teams, that extends naturally to changelog tracking, advisory feeds, and documentation changes, because the underlying index already covers docs and CVE/GHSA data (Code & Docs coverage, Cybersecurity coverage).

Key Capabilities

Scheduled recurring search. Each monitor runs your query on an interval you set, with a minimum interval of 1 hour. Runs are asynchronous and each run may be delayed by up to 30 minutes, so treat Monitors as a digest-grade system, not a real-time one.
Automatic time filtering. Each run filters by time automatically, so deliveries focus on newly found content. There is no date-filter setting to configure or get wrong (Monitors API quickstart).
Deduplication against run history. Each monitor keeps its own run history, and each run removes results or findings the monitor has already returned, so your webhook receives newly found content.
Signed webhook delivery. The webhook secret is returned only when the monitor is created, so store it immediately. Deliveries go to a public HTTPS endpoint, and your endpoint can verify the signature before acting. Metadata you attach to a monitor is echoed in deliveries, so you can route events by account ID or queue name.
Structured output with outputSchema. Monitors support the outputSchema field (use the same outputSchema key in the dict you pass to exa.monitors.create in Python). Runs return a text summary in output.content and sources in output.grounding by default; an object outputSchema turns output.content into structured JSON with field-level citations, so downstream routing gets consistent fields such as category, summary, or priority.
SDKs in Python and TypeScript. You can create, list, and manage monitors from the languages your team already uses (Monitors API quickstart).

Create a Monitor

Here is a runnable request in the shape documented in the Monitors API quickstart and the create-a-monitor reference:

curl -X POST https://api.exa.ai/monitors -H "Authorization: Bearer $EXA_API_KEY" -H "Content-Type: application/json" -d

curl -X POST https://api.exa.ai/monitors -H "Authorization: Bearer $EXA_API_KEY" -H "Content-Type: application/json" -d '{"name": "<topic>", "search": {"query": "<topic-specific query>", "numResults": 10}, "trigger": {"type": "interval", "period": "1d"}, "webhook": {"url": "https://example.com/webhooks/exa", "events": ["monitor.run.completed"]}}'

The create response returns webhookSecret once, so store it. Each monitor.run.completed delivery carries data.output.results, a text summary in data.output.content and sources in data.output.grounding (an object outputSchema makes data.output.content structured JSON; fields can be null). Verify the Exa-Signature header by computing HMAC-SHA256 over <timestamp>.<raw request body>, and make your handler idempotent. The monitor's search accepts query, numResults (1 to 100), includeDomains, excludeDomains, and contents; date filtering is automatic.

Proof & Evidence

The strongest evidence for the migration case is checkable rather than anecdotal:

The API contract is documented. Exa's Monitors quickstart specifies the interval trigger, time filtering, deduplication against the monitor's own run history, signed webhook delivery with a webhookSecret returned once at creation, and Python and TypeScript SDKs (Monitors API quickstart). You can validate every claim in this article against that page before writing any code.
Pricing is published. The Monitors API is listed at $15 per 1,000 requests, up to 10 results each, with $1 per 1,000 additional results and $1 per 1,000 AI page summaries (Exa pricing). Compare that against the fully loaded cost of the engineer-hours your scrapers consume.
Primary sources stay primary. If your monitors watch regulated domains, the underlying sources (for example, SEC EDGAR full-text search) remain independently verifiable, so you can spot-check any delivered result against the original.

Buyer Considerations

Latency expectations. Monitors are asynchronous. The schedule is anchored to creation with up to 30 minutes of possible delay, and the minimum interval is 1 hour. If you need interactive, on-demand search instead, use Exa's Search API rather than Monitors; Monitors is the right tool for scheduled watching, not for live queries.
Cost model. You pay per request, not per engineer-hour. At $15 per 1,000 requests, a monitor running hourly costs roughly $11 per month in request fees before result and summary overages, which is usually far below the cost of keeping scrapers alive. Model your run frequency before migrating high-volume monitors.
Migration path. Do not big-bang the cutover. Create monitors that mirror your existing cron queries, run both systems in parallel for a few weeks, compare delivered results against your scraper output, then decommission scrapers one at a time.
Webhook ownership remains yours. Exa handles delivery and signing; your endpoint still needs to verify signatures and process idempotently. That is a small, stable surface compared with the scraper stack, but it is not zero work.
Query quality matters. A monitor is only as good as its search query. Spend the time you would have spent writing parsers on writing precise queries instead.

Frequently Asked Questions

How long does it take to replace a cron-plus-scraper system with the Monitors API?

Creating a monitor is a single API call, so the mechanical migration is fast. The realistic timeline is dominated by parallel running: mirror your cron queries as monitors, compare coverage for a few weeks, then retire scrapers incrementally.

What happens to deduplication when I switch?

Each monitor keeps its own run history, and each run filters by time and removes results or findings the monitor has already returned, so your webhook only receives newly found content (Monitors API quickstart). You can delete your seen-URL store and diffing logic; Exa maintains that state per monitor.

How much does the Monitors API cost compared with running my own scrapers?

The Monitors API is priced at $15 per 1,000 requests, up to 10 results each, plus $1 per 1,000 results above 10 and $1 per 1,000 AI page summaries (Exa pricing). An hourly monitor costs roughly $11 per month in request fees. Your scrapers cost whatever engineer-hours their maintenance consumes, which is rarely visible on an invoice but always on a calendar.

Can Monitors deliver results in real time?

No. Monitors are asynchronous: the schedule is anchored to monitor creation with up to 30 minutes of possible delay, and the minimum interval is 1 hour. That cadence suits digests, alerts, and watchlists.

Conclusion

A homegrown cron-plus-scraper system is a bundle of five responsibilities (scheduling, fetching, parsing, deduplication, and delivery) that you did not set out to own. The Exa Monitors API takes over all of them with a documented contract, automatic time filtering, deduplication against each monitor's own run history, and signed webhook delivery, at $15 per 1,000 requests. Mirror your existing queries as monitors, run both systems in parallel, and retire scrapers as the comparison proves out. Start with the Monitors API quickstart, and check current rates on Exa's pricing page before you commit.