We prefer official feeds and sitemaps over aggressive scraping, so the data stays legitimate, low-risk and reliable — and higher quality, because it comes from the source's own structured output.
We build and maintain a master registry of official sources — starting from curated lists of roughly 2,500 government and public-interest sites, cleaned, deduplicated and normalized into one reliable list with a stable schema: source type, jurisdiction, homepage, press room, feed and sitemap URLs.
Where a structured feed exists, we use it. We prefer official RSS, XML and Atom feeds, sitemaps and press-room indexes — collecting on polite, capped schedules that respect how publishers publish. High-value sources are captured across every stream, not pointed at a single news page.
Every document is normalized into one consistent NewsML-style schema and exported as clean XML or CSV — sourced, timestamped, with clean dates and body text. Indexes, navigation and boilerplate are filtered out. The relevance review is the product, not overhead.
We work in small batches with logs, checkpoints and manual review, keeping raw, working and final data separate. Before any client delivery, a hard QA gate blocks five defect classes — non-articles, boilerplate, duplicates, encoding errors and wrong dates — so nothing ships unchecked.
Aggressive HTML scraping breaks constantly and strains publishers. By prioritizing official feeds, sitemaps and public indexes, we stay legitimate, low-risk and reliable — and data quality is higher because it comes from the source's own structured output.
In-house monitoring looks simple until you run it. Here's what you'd own — and what we handle instead.
Every document arrives normalized to the same schema — headline, dateline, date, source, language and full body — as NewsML-style XML, CSV or JSON. No messy HTML to parse; ready to ingest on day one.
indexes & nav blocked
templated cruft removed
near-dupes filtered
zero mojibake, clean round-trip
future/invalid dates caught
Delivery is over SFTP (or FTPS, S3, webhook), idempotent and atomic, with per-client formats and source filters. The gate is proven to block a deliberately-defective sample — not just to pass clean files — before any live credential is ever connected.