Contact us
Errata
Media Cloud is a large data system, built across multiple generations of software code and storage systems over more than a decade. This has inevitably led to some idiosyncrasies, quirks, and errors. This page documents some of those.
Articles Mis-attributed to Google.com
About 2-3% of our legacy system stories (~100 million) are mis-attributed to google.com as the publishing source. This impacts stories before 2023, when we launched our revised system architecture. This error is the result of how we re-ingested legacy stories after our database crashed in that era. This means we have stories published by various outlets, but are incorrectly associating them with the google.com source as the publisher. If we had technical staff available we could reingest and associate some large percentage of those stories with the appropriate publisher. We undertook some work to characterize the impact and found that:
- In a sample of 3,700 impacted stories, 77% had an HTML canonical url metadata tag that could be used to associate them with the appropriate source. Top actual publishers identified in that sample included mlive.com, mcall.com, rosbalt.ru, time.com, ettoday.net, sun-sentinel.com, pjmedia.com.
- In a sample of 500, 46% had titles that were already in our index in the same time period.
Articles in which headlines from related/other stories are captured as body text
In a small number of publications, text promoting another story to read from the publication is captured and processed as body text of the current article. For example: Mr. Jones had no further comment on today’s city council vote. READ MORE: Climate crisis affecting more countries than previously thought. The city council will meet again next month. This type of error will result in the example article returning when searching for “climate crisis,” even if the main article about the city council does not mention the term. This is bounded by publication, so if you notice this error in your results, you can remove the specific publication in question from your returned results. You can also mitigate this error by using proximity search syntax, requiring that your selected keywords appear more than once in your return results.
Take caution with source-level research
Media Cloud was built to support ecosystem style research, across lots of sources as pulled together in collections. This implied a set of assumptions and design decisions that have carried forward through the decade-plus of work on this project, across multiple generations of technology development. That said, over time we’ve found ourselves and others doing more searches against one source, or a small set of them, more often. If you’re doing that kind of research, take caution with the idiosyncrasies you might run into. When sites change URLs we add alternative domains or create child sources, but can’t capture every change over time. Sometimes sources just stop working and then come back to life. We advise checking out each source in depth if you’re taking this approach, because it isn’t the kind of work the system was originally designed for.