Most of Starling Lab’s work involves trusted timestamping. Normally timestamping is very easy: you simply add  a timestamp field to your metadata and call a function such as time.Now(). Trusted timestamping, by contrast, is concerned with proving that the provided timestamp is actually correct, and wasn’t forged after the fact.

This is simple to do in one direction, in order to prove some data existed after a certain point in time. That can be easily achieved by including unguessable current information, such as news headlines and stock prices. A classic pop culture example is kidnappers proving a hostage is still alive by having them pose with that day’s newspaper.

But trusted timestamping is concerned with the opposite direction: proving data existed before a certain time. This is not so easily done, but it is possible, and the use cases are quite important.

Purpose

There are three main reasons for doing trusted timestamping.

1. Verifying old cryptographic signatures
2. Data integrity
3. Proof of technology ✨

In addition to these there is the obvious one: authenticity. Trusted timestamping can prove that a document that claims to have been made on a certain date was actually created on that date.

Verifying Signatures

The mainstream industry use case is verifying that a signature was added on a given date. For example, Apple requires timestamping as part of its code-signing process. The reasoning is simple: sometimes private keys get leaked or revoked. This scenario can break the guarantees signatures provide, namely that you can be sure of the author of the signed data.

Once a key is leaked, can we validate any previous signatures? They could have been made by a third party using the leaked key. But if the signature was timestamped, we can know whether the signature was created before the key was made invalid. This is a real problem that trusted timestamping solves.

Data Integrity

A basic method of ensuring data integrity is storing hashes. But what if those hashes are tampered with? Using signatures instead allows us to verify that no one but a person/org we trust has modified the data. But what if we don’t want to trust that person, or their ability to guard their keys? Using signatures adds an extra moving part and security risk that in some cases may not even be needed. If we can agree that the data was valid and unmodified at a certain point in time, trusted timestamping will provide data integrity forever. An example use case is backups, or storing org-internal archival data like legal agreements.

Proof of Technology

This is a new, modern use case. As AI-generated content gets better and better, being able to prove that a piece of media was generated before certain AI software even existed will become extremely important. Trusted timestamping has the ability to end all these new uncomfortable arguments that such-and-such picture was created whole-cloth by some inscrutable algorithm, and instead bring us back into old school discussions about whether something looks “PhotoShopped”.

Of course, it can be that AI content generation has advanced too far for trusted timestamping to be useful. But the technology is not perfect, nor has it peaked. As the saying goes, the best time to plant a tree timestamp data was 20 years ago, the second best time is now.

Software

There currently exist two viable trusted timestamping methods I am aware of. The first is OpenTimestamps, and the second is the Time-Stamp Protocol, or RFC 3161. The former timestamps using Bitcoin (but efficiently, not one timestamp per block) and the latter uses the signature of a trusted service (Time Stamp Authority).

Here is a quick comparison:

You can learn more about OpenTimestamps on the website. Beyond the actual RFC linked above, you can learn more about RFC 3161 by playing around with OpenSSL, use the CLI and type the command man openssl-ts

I’m not going to dive any deeper into these tools than the above table, but my general recommendation is to use RFC 3161 with a good third-party unless not needing to trust anyone is actually a requirement for your project. And you could always do both!

Usage

OpenTimestamps has a CLI tool and libraries in several programming languages, see here. It also has a GUI on the website.

RFC 3161 is supported by OpenSSL as a CLI tool with the openssl ts command. To make RFC 3161 timestamping on the CLI easier (as it’s not just one OpenSSL command), I’ve created a bash script each for timestamping and verifying. You can get them here.

The only GUI option for RFC 3161 I’m aware of is on the freeTSA.org website, although personally I have no reason to trust freeTSA. Better than nothing though, for anyone who doesn’t use the command line.

For trusted third-parties there are various options, including reputable companies such as DigiCert.

Conclusion

It’s time for trusted timestamping! The cost of this is basically zero (storing a few kilobytes of proof at most), but the advantage can be very large. It’s one of those things that you’ll wish you had done at the start.

Many groups can act on this by adding timestamps for their documents and media. In increasing order of importance:

• General public: timestamp your important documents and media to prove their authenticity. You can do this today on your phone with ProofMode, and on the desktop using CLI or GUI tools as discussed above.
• Archivists: timestamp everything you archive, for integrity and authenticity. For redundancy consider using both methods mentioned above and many RFC 3161 timestamp authorities.

Developers: if you work on software where data or signature integrity is at all important, add trusted timestamping to your application. And if you use Git signing, you can easily start timestamping all those signatures, see here.

We are thrilled to announce our participation in an extensive archiving initiative supporting the Atlantic Council’s Digital Forensic Research Lab (DFRLab). This significant project involves preserving 10,000 web pages as part of the research for the “Narrative Warfare” report, providing a robust resource for understanding the complex landscape of digital misinformation and information warfare.

The “Narrative Warfare” report, published by the DFRLab, delves into the intricacies of how narratives are weaponized in the digital age. The report uncovers how state and non-state actors manipulate narratives to influence public perception and destabilize societies. 

The dataset of 10,000 web pages represents pro-Kremlin news publications from the 70 days prior to the ground invasion (Dec 16, 2021 to Feb 24, 2022). The team at the DFRLab identified and tracked five primary narratives pushed in support and ahead of the invasion of Ukraine. 

Starling Lab, renowned for its pioneering work in data integrity and digital preservation, played a crucial role in this initiative. By leveraging advanced technologies and methodologies, Starling Lab ensured that the archived web pages were meticulously preserved and accessible for future research and analysis. This collaboration underscores the Lab’s commitment to safeguarding digital content and promoting transparency in the digital realm.

Given the risk to the material once the report would be published, we preserved and durably stored the material on the request of the DFRLab. Director Andy Carvin’s team shared a spreadsheet containing, for each URL, original research metadata such as the associated broad narrative, the source pushing this narrative, and the article’s reach on social media.

Each page was crawled individually, meaning that all its content (media, code, style) were downloaded in a package permitting to “replay” it in the future and offline, independently of whether the original content would be deleted. Archives were prepared and packaged up using the cloud tools provided by WebRecorder, and all the content cryptographically signed by Starling Lab’s SSL certificate. For more details on these authenticated web archives, read our Dispatch on WACZ files.

The resulting dataset, totalling 400GB, can be accessed by members of the “Dokaz” initiative on request. While each page can be individually browsed as it was captured at the time, regardless of whether it went away, we are looking for support to produce an explorable, searchable interface to facilitate access for researchers. 

In conjunction with the DFRLab’s “Dokaz” initiative, which focuses on preserving and verifying evidence in conflict zones, this archiving effort marks a significant step forward in the fight against digital misinformation. The “Dokaz” initiative, supported by Starling Lab, aims to provide reliable and verifiable digital evidence that can withstand scrutiny and support investigative journalism and research.

This collaborative effort between Starling Lab and DFRLab highlights the power of interdisciplinary partnerships in addressing the challenges posed by digital misinformation. By combining expertise in data integrity, digital forensics, and narrative analysis, this project not only enhances our understanding of narrative warfare but also strengthens our ability to counteract its harmful effects.

For more information, visit the Atlantic Council’s Narrative Warfare Report.

In the autumn of 2022, we convened two pivotal workshops to focus on the evolving role of archiving web pages and social media in the context of international justice, particularly concerning Russia’s war against Ukraine. These workshops, one technical and the other legal, aimed to explore how recent advances in web archiving could support the collection, storage, authentication, and utilization of digital evidence in accountability proceedings for victims of the conflict.

The discussions formed the basis of a whitepaper, authored by Scott Martin (Global Justice Advisors) and Basile Simon (Starling Lab) and set of best practices that outline the ideal characteristics of a web archive for use in court, drawing on the requirements of the Berkeley Protocol on Digital Open Source Investigations.

Best Practices for Web Archiving

According to this whitepaper, the ideal web archive demonstrates the following properties:

• It can be produced by anyone, notably by individual actors with tools they can grasp and control (as opposed to using a commercial service or being granted access to a platform). This is correlated, to an extent, with the use of open-source and local software.
• It is of high fidelity, meaning it was carried out by a tool that preserved most, if not all, of the original material.
• It includes the content itself, its surrounding metadata, the metadata of the web scraping software. This includes cryptographic hashes of all website assets and the signature of these hashes authenticating it to the author.
  • Furthermore, cryptographic hashes and signatures must be preserved, that is to say, stored securely and made available for the long term, as would the content itself.

Establishing Clear Methodologies

To maximize the admissibility of a web archive as evidence, archivists and legal professionals must establish clear, detailed methodologies. These methodologies should document the provenance of the digital evidence — detailing where it comes from, how it was procured, who procured it, when it was procured, and the process followed. This includes documenting the chain of custody and demonstrating that the webpage has not been altered during archiving.

Key points include:

• Detailed Record-Keeping: Identify the person conducting the archiving, their qualifications, and the web collection protocols observed. Describe the hardware and software used, and explain the process for selecting and assessing websites and articles for credibility and resistance to manipulation.
• Storage Protocols: Describe measures against corruption, hacking, and other risks to ensure the integrity of the archives over time. This should be recorded in a chain of custody that tracks who has handled the document.

Background on Workshops

Technical Workshop: Enhancing the Integrity of Web Archives

On August 25, Starling brought together experts in web archiving to discuss methods to preserve information for accountability purposes in Ukraine. The workshop delved into various collection, authentication, and preservation strategies, emphasizing the technical aspects that ensure the integrity of recorded web pages and other digital materials.

Participants first examined existing web archiving practices and their operation on a technical level, then discussed the potential risks to these archives that could threaten their integrity. A significant focus was on the vulnerabilities of storing web archives using traditional archival models. The discussion highlighted how a shift towards more distributed and decentralized models could offer improved long-term resilience and availability, essential for maintaining the integrity of the archives in unpredictable environments.

We thank the following participants for their contributions:

• Mark Graham, from the Internet Archive;
• Ilya Kreymer, from WebRecorder;
• Michael Nelson, from the Old Dominion University;
• Nicholas Taylor, expert witness in the Internet Archive Wayback Machine;
• Ed Summers, from the Stanford Libraries;
• And Cade Diehm, from the New Design Congress.

Legal Workshop: Web Archives in the Courtroom

Following the technical discussions, a roundtable of legal experts convened on September 27 to explore the legal dimensions of web archiving practices. This group included lawyers specializing in war crimes and legal professionals experienced with digital evidence. The goal was to identify potential legal vulnerabilities in current archiving practices and determine how such materials could be admitted into evidence in courtrooms, particularly in war crimes and other international criminal proceedings.

The legal experts articulated best practices to ensure that web archive data are preserved, produced, and authenticated in ways that maintain their integrity. This enhances their reliability, utility, and probative value as evidence in a judicial context. The roundtable discussed the characteristics and challenges of various web archiving practices and presented a framework to assess these methods.

We thank the following participants for their contributions:

• Scott Martin, from Global Justice Advisors;
• Melissa Bender, from Ropes and Gray LLC;
• Tim Parker, from Blackstone Barristers;
• Cari Spivack, from the Internet Archive;
• Karolina Aklamitowska, from Tallinn University;
• Clare Stanton, from Harvard Law School;
• Bastiaan van der Laaken, from the UN IIIM Syria.

Next Steps: Call for Contributions on Witness Servers

Finally, to improve on the process of entering web archives into evidence, Starling are formalizing the concept of “Witness Servers” as an additional layer of self-corroboration for web archives. A Witness Server is a service, hosted and run by an institution, which carries out web crawls on-demand on behalf of individuals conducting web archiving activities.

Participating institutions, e.g. the Stanford Libraries, WebRecorder, or the Harvard Library Innovation Lab, bestow the individuals or team they accept to witness with the trust that might be placed in the institutions themselves. The roundtable findings identified the reliance on the social trust placed in institutions as particularly supportive of strengthening the work of potentially vulnerable investigators and archivists.

Several Witness Servers act in concert on the instruction of a web archivist and simultaneously capture the same web page. Such an approach addresses the possibility of a webpage having slight variations depending on locale (and many other potential anomalies) and works to otherwise corroborate the contents of a website through a replication process that validates the contents of a web archive from several different locations and actors.To learn more, participate as an institution or a researcher, read the Call for Contributions.

Criminal investigations depend on evidence gathered in the field. Unfortunately, these scenes can be difficult to access (ex: war zones, disaster areas) and individuals on the ground may not be accessible to courts (ex: threats to safety, loss of contact). Emerging technologies could help overcome such challenges, providing novel solutions to weigh probative value and establish authenticity.

In the summer of 2021, and as part of the Human Rights and International Justice Policy Lab at Stanford Law School, the Starling Lab and Hala Systems convened a group of international experts on digital evidence and war crimes prosecutions to solicit feedback on their proposed model of Supervisory Testimony.

The gathering was chaired by Beth Van Schaack (then visiting professor at Stanford), assisted by Mackenzie D Austin (then Lab associate). 

Data (including multimedia assets like photo and video)  that is collected by field observers can present a number of evidentiary and admissibility issues when that data is submitted in accountability settings. Because of the ad hoc nature of evidence collection in conflict zones, the chain of custody of individual pieces of digital evidence can be especially hard to trace. Furthermore, a variety of custodians might have handled evidence before it reaches a host organization or tribunal, which can be difficult to track as well. The extended period of time between evidence collection, storage, and admission into evidence also presents the possibility that many primary observers and custodians may no longer be reachable by the time that accountability processes take place. Thus, those individuals would not be able to provide critical affidavits or testimony as to the veracity and authenticity of the digital evidence at the time of trial. All of these factors threaten the admission of digital evidence in a court of law. 

Supervisory testimony, or the testimony of a primary observer’s supervisor (who may participate remotely), could help bridge the evidentiary gap. Rather than requiring a different witness to account for each individual link in the chain of custody, a single supervisor could account for the entire chain of custody. Supervisory testimony would consolidate the institutional knowledge of the chain of custody of specific pieces of digital evidence in a single person.

For example, a supervisor would be tasked with training and overseeing a set of field observers. During a years-long conflict, the supervisor would keep track of the provenance of the swaths of digital evidence submitted by their cohort of observers. Once an accountability mechanism (e.g. a tribunal) is initiated, that single supervisor would present a consolidated bundle of data (e.g. photographs) obtained by their field observers. As a witness, the supervisor would testify as to the chain of custody of specific pieces of evidence, having been present as a supervisor during the collection and storage process. That supervisor would also explain the general process of capture and transfer of all recorded data functions, including the training of field observers. Most importantly, that supervisor would provide a blanket certification of authenticity of the digital evidence. Ultimately, a single supervisory witness could stand in for the dozens of field observers who collected the data over an extended conflict and account for the entire lifecycle of digital evidence from the moment of capture to its presentation in court. This strikes an important balance, streamlining several obstacles to evidence admission while still allowing a defendant an appropriate party to cross-examine.

Technological solutions like cryptographic hashes and distributed ledger entries have been pitched as solutions for chain of custody concerns, serving a bit like a notary system. However, those solutions may be insufficient to adequately account for the human observers whose testimony may still be necessary to authenticate digital evidence. Instead, the combination of technological authenticity markers and supervisory testimony would help shore up any gaps in the chain of custody and enhance authentication for accountability purposes. While technological solutions will be discussed, the workshop has primarily explored how both human and technological protocols are essential. Together, these robust protocols could be a new frontier in authentication.

• Asynchronous supervision: A supervisor would train new cohorts of field observers with an established protocol to ensure the authenticity of the collected data and reliability of the collection methods. This could include corroborating data, like photos or videos of the field process itself. A supervisor could also conduct debriefs with their observers to affirm that the collection protocol was followed at the time that specific evidence was collected. 
• Direct supervision synchronous with the moment of capture: For example, a supervisor may text back and forth with a field observer at the moment they collect evidence. Alternatively, the field observer may livestream the evidence collection with their supervisor.
• Development of uniform collection protocols: Hosting organizations that employ supervisors must develop uniform collection protocols to be used in the field. As part of these protocols, organizations should also contemplate ways to maintain impartiality and neutrality in the collection of their data, and consider common or likely rules of evidence.