News & Media Data

News and media data can be organized around individual articles or content items. Each content item can be associated with its publisher or source, its headline, its dates, its author byline, and its section, category and topics, as the source publicly displays them.

The same article can appear at different URLs, and articles can be updated after they are first published. Matching the same article across sources and URLs, and keeping timestamps for publication, update and collection, helps keep a news dataset consistent.

Request a Data Collection Project

News and media data is relevant to organizations that need a structured view of published content, including:

• News organizations studying publishing output
• Media companies researching publishers and topics
• Researchers working with article and topic data
• Digital platforms building structured content datasets

The unit of data is the article or content item. The publisher is the source it comes from, and the headline, author byline, dates, section, topics and media references are attributes of that item. The dataset is built from publicly displayed article metadata and displayed summaries.

News and media data is easier to use when it is clear what each record represents. The article or content item is the primary record, and the other information attaches to it. Deciding what a record represents, and how the same article is identified across sources, is one of the first choices in a news data project.

An article or content item is the primary record. The publisher is the source it comes from, recorded as an attribute of the item: the publisher’s name and website URL where displayed. The publisher is not treated as a business profile. Each item has an article URL and, where a source displays one, a content type.

The same article can appear at different URLs, for example when it is republished or reachable through more than one path. Where a source displays a canonical URL, it helps identify the article. When a project draws on more than one source, matching decides, where possible, whether two records describe the same article. The fields available for matching depend on what each source displays.

Each item carries a headline or article title. The author is recorded as the byline, the name as the source displays it, and author data does not extend beyond the byline.

Dates matter as much as content in news data. The publication date shows when an item was published and, where a source displays an update date, it shows when the item was updated. A collection timestamp records when the data was collected, so that publication, update and collection times stay distinct.

Publications organize their content in their own way. Where displayed, an item can carry a publication section, a category, a topic and subtopic, keywords, tags and related content. These labels are attributes of the item.

Taxonomies differ by publisher, so sections, categories and topics from different sources may need to be aligned when data comes from more than one publisher.

Sources also display supporting metadata alongside each item. Where displayed, this can include a displayed summary, an image URL, video or media information and the content type, together with the canonical URL. Image URLs and media references are recorded as references only.

Article body or full text is not part of the defined scope. The dataset consists of article metadata and displayed summaries.

The data types below reflect what news and media sources commonly display. Which of them apply depends on the project and on what each selected source publicly shows.

Article and publication data describes a content item as a source displays it. Where publicly displayed, it may include:

• The article title or headline
• The article URL
• The canonical URL, where displayed
• The publication date
• The update date, where displayed
• The displayed summary

Topic and category data describes how a publisher classifies an item. Where publicly displayed, it may include:

• Categories
• Publication sections
• Topics and subtopics
• Keywords
• Tags
• Related content

Author and publisher data is limited to attribution. Where publicly displayed, it may include:

• The author byline, as displayed
• The publisher name
• The publisher website URL
• Publication information, where displayed

It does not include author contact details, social profiles, personal email addresses or personal biographical information.

Media metadata describes the media that accompany a content item. Where publicly displayed, it may include:

• The image URL
• Video or media information
• The content type
• The canonical URL
• The update date

News and media data can come from a range of publicly accessible source types, including:

• News websites
• Media outlets
• Online magazines
• Blogs
• Other public sources designated for the project

Collected content remains subject to the source’s availability, access conditions and permitted use requirements. The scope can be limited to metadata and displayed summaries where appropriate. Collection from selected sources is covered by our news content data collection service, with the publishers, sections and fields defined for each project. Not every source type applies to every project, and each source structures its content differently.

News and media data projects usually start from a question about articles, publishers or topics. What the data describes is limited to what the selected sources publicly display.

Building a structured view of published content from selected sources: which articles appear, which publishers they come from, which topics they cover and what publication information the sources display.

Comparing publishers, topics and publishing activity across selected sources: which publishers cover which sections and topics, and what publishing activity is visible in the collected article dates. Results describe the collected data, not a publisher’s full output.

Looking at the topics, tags and keywords that appear across collected articles, and how they are spread across sections and categories. Because taxonomies differ by publisher, labels may be aligned before topics are compared.

Building structured article datasets from selected sources: article metadata, publisher and content datasets, and datasets of displayed summaries and media references, with each article kept as one record.

News and media projects are defined around the sources, the publishers and the article records to be collected. The main points to settle are:

• Source websites
• Publishers in scope
• Sections
• Topics
• Date range, as a project requirement
• Record level: article or content item
• Required metadata fields
• Handling of updated articles
• Canonical URLs
• Duplicate and matching requirements
• Media references, such as image URLs
• Author scope, limited to the byline as displayed
• One-time or recurring collection

Article body or full text is not part of the defined scope. The data consists of article metadata and displayed summaries.

Author contact details, social profiles, user comments and unrelated personal information about individuals are outside the scope of this data.

A one-time collection captures article and content datasets from selected sources at a single point and can support:

• News and media research
• Publisher and content research
• Topic research
• News database and content dataset creation

Recurring collection repeats the same scope as timestamped snapshots of newly published or updated articles for defined sources, sections or topics. Each snapshot records the articles a source displayed when it was collected, with publication and update dates where displayed.

How often data is collected is determined by the project requirements.

A clear requirement makes it easier to choose the right sources and record structure. A project can be scoped around the sources, the publishers, the sections and topics, the date range, the metadata fields, the author scope and whether collection is one-time or recurring. Once these are defined, we can review the project scope and confirm an approach.

Request a Data Collection Project