Law

Context
Framework Technology Learnings Archive

Context

Our search for a project narrative and partner was one of compromises and of navigating constraints – the most important one being an access to the field we could all feel comfortable with.
From a technical perspective, the idea was to demonstrate the capture of a scene for photogrammetric reconstruction, and to do so using consumer hardware in a minimal amount of time.

In the end, the team decided to rely on a location known to Artem: a group of residential buildings in Borodyanka (Бородянка, a suburb in the Kyiv Oblast, northwest of the capital), which received artillery fire at the onset of the ground invasion, and notably around the time fighting occurred around Bucha. Following the withdrawal of the Russian troops, several international artists produced artworks on the destroyed buildings. Among them were two stencil artists: Banksy, of international fame, and C215, from Paris. The latter, real name Christian Guémy, has produced several artworks of this kind in Ukraine over trips between March and April 2022.

These artworks were documented by Artem and the Murals team, and the locations in the region are well known to them, as Artem and his team have carried out numerous trips to photograph the pieces, ruins, and surroundings. The familiarity with the location helped support our risk assessment in reducing both time spent in the field and in unknown locations – a notable challenge when considering wandering around destroyed buildings and rubble.

Artem’s goal was to capture photographs of the ruined apartment hosting the stencil murals from C215, in order for the team to produce a photogrammetric / 3D model of the scene later. The photos Artem would produce would be strongly-authenticated (see Technology section below): they would be accompanied with strong guarantees of integrity and provenance.
The 3D model created from these photographs would itself be viewable through a virtual reality (VR) headset permitting to navigate inside the ruins, as one would look around an apartment, and to look at the whole or details of the scene, rendered photorealistically.

http://starlinglab.org/wp-content/uploads/2024/09/Photogrammetry-0-initial-render.mp4

Contents

Framework
Technology Learnings Archive

Framework

The Challenge

The missing link in the extended reality trust chain is a byproduct of presenting a 3D model to the viewer through the VR headset. This model is the result of a photogrammetric computation, essentially matching features (e.g. cracks in the wall, furniture edges, etc.) across several photographs.

This 3D model does not account for the extra information we have about the strong original photographs. Nowhere can the metadata be presented, and there are no mechanisms to compare the work of the rendering computer against the actual original photographs.

In the VR space, the original photographs are presented as anchors. With a press of the trigger controller, information about their capture and provenance is displayed over the 3D space.

Contents

Framework
Technology Learnings Archive

The Prototype

Until now. To bridge this gap between strong, contextualized, metadata-rich photographs and a complex compute, we produced a user interface study and prototype surfacing the integrity data we have available for the original photos.

The prototype draws from our cross-disciplinary research in producing and leveraging authenticated data, best summarized by the three-step “Capture, Store, Verify” framework. The latter step, Verify, relates to permitting some verifications and inspections of the material, but also to communicating the value of having authentication data about a piece of digital media: What does it mean to me? To which extent does it affect the trust I might have in the material presented to me?

In this prototype, and inside the VR experience, the viewer is presented with means to explore not only the 3D space but also the original photographs, as well as relevant integrity data, in support of our research questions.

From authenticated photographs of the apartment ruins and murals in Ukraine, we produced a photogrammetric model, which we presented inside a Meta Quest VR headset through a Unity render. Inside this 3D space, we gave exploration tools to the viewer accessible from the physical controls linked to the headset.

Within this photorealistic reconstruction, viewers now have a virtual heads-up display UI permitting them to trace all the way back to the original photographs, and to inspect relevant integrity data.

Following a press of the trigger controller, the integrity data can be explored – above, for example, the various cryptographic registrations and timestamps associated with the asset.

Contents

Context
Framework
Technology
Learnings Archive

Technology

Capture

Artem captured detailed photographs of the site using a Samsung Galaxy Note 20 Ultra, an Android smartphone equipped with a high-end camera lens. The photos were processed through Starling Capture, an early iteration of what would later evolve into the Numbers Capture app. The app, built to be open source and available for public scrutiny, handled both the secure transmission and the cryptographic signing of the images, using a robust flow available on GitHub.

The Starling Capture app collected rich contextual metadata along with each photograph, including timestamp and GPS coordinates. The app then computed a unique cryptographic hash for each image and bundled this metadata for digital signing. The cryptographic signature, which tied the data to a secure, private key on Artem’s device, ensured both the authenticity and the integrity of the photographs. These signed data bundles were then securely transmitted to the Starling server infrastructure for further processing.

A high level diagram of how the Starling Framework for Data Integrity is applied to photo assets.

Store

Upon receipt at Starling, the data bundles were subjected to rigorous validation procedures. The validated images, along with their corresponding cryptographic metadata, were registered on multiple independent cryptographic ledgers. These included the Numbers blockchain, which provided fast and efficient registration; Avalanche, for its flexibility and speed in handling integrity proofs; and LikeCoin, where records were aligned with the International Standard Content Number (ISCN) specification.

The authenticated and signed bundles were then encrypted and stored on Starling’s secure storage pools. These storage servers feature state-of-the-art encryption at rest, protecting the data against unauthorized access or tampering. While the prototype user interface referenced an IPFS hash value—indicating the potential for decentralized storage on the IPFS network—the data was not uploaded to IPFS at this stage.

Verify

Verification of the images involved validating the original cryptographic signatures and associating the photographs with a C2PA stamp. This stamp, issued under the Starling certificate authority, served as an additional layer of authentication, clearly marking the photographs as unaltered and verified by a trusted source. The verified images were then employed to create a photogrammetric model of the scanned location.

For presentation purposes, the photogrammetric model was rendered using the Unity open-source 3D engine. A demonstration of this model was showcased through a Meta Quest VR headset, with side-loading used to deploy the model into the VR environment. This immersive setup allowed viewers to explore the rendered location while ensuring that all elements of the model could be traced back to their verified, original sources.

To enhance the credibility of the presented data, the user interface aggregated and displayed various forms of integrity information. The C2PA manifests, cryptographic hashes, and corresponding Content Identifiers (CIDs) were all made available for inspection. Additionally, the UI provided references to the Avalanche, Numbers, and ISCN records, alongside a hash of the Filecoin deal, which documented the agreement for decentralized storage of the original photographs. This multi-layered verification system ensured transparency and the ability to independently confirm the authenticity and integrity of the data.

Contents

Context
Framework
Technology
LearningsArchive

Learnings

There are constraints to the use of photogrammetry

In the realm of 3D modeling and mapping, photogrammetry, while powerful, faces several limitations. The technique is inherently time-consuming, often requiring extended periods for data capture and processing. Accessibility to specific locations for on-site imaging can pose challenges, particularly when rapid documentation is critical. Furthermore, photogrammetry struggles with certain types of objects: those with minimal distinctive features, such as white painted walls, reflective surfaces, scenes with extremes of dark and bright areas, or moving targets, often yield unsatisfactory results.

These constraints, however, are being significantly addressed by the advent of Neural Radiance Fields (NeRF) models and Gaussian Splatting (3DGS), which promise drastic improvements in ease of capture both in efficiency and versatility in 3D imaging.

Photogrammetry uniquely benefits from quality photographs, and phones aren’t the best

The quality of source photographs is crucial, and smartphones, while convenient, often fall short compared to dedicated cameras. Successful photogrammetry requires sharp, minimally processed, low-noise photos, something that smartphone cameras sometimes struggle to deliver. Using a consumer-grade smartphone involves trade-offs: while accessible, a phone yields slower processing, lower quality results, and requires more effort than a professional camera.

Photorealism and heads-up display

The exceptional fidelity and clarity of the 3D models have drawn comparisons to an augmented reality (AR) experience, where a virtual user interface seamlessly integrates with the physical environment. This photorealistic rendering, often seen in heads-up displays, highlights the advanced capabilities of modern 3D modeling, offering a blend of digital and real-world elements that enhances user interaction and perception, a noteworthy achievement in both technical and legal contexts for its immersive quality.

Display of the cryptographic proofs

Even in VR, the presentation of cryptographic proofs still very much relies on a user's understanding of complex concepts, yet there is a noticeable gap in user research and feedback on these designs. This lack of engagement in the user experience and research process raises concerns about effectiveness and accessibility. For a technical and legal audience, it highlights the necessity of bridging this knowledge gap to ensure that users can confidently interact with and understand the security and authenticity mechanisms in place.

On the generation of the 3D models

In the generation of 3D models, there exists a notable lack of clarity regarding their deterministic properties. This ambiguity raises concerns in the legal and forensic spheres, as these properties are crucial in establishing the authenticity and provenance of digital evidence. Current content and metadata authentication systems do not fully capture these nuances, potentially leading to a gap in the chain of provenance. This oversight underscores the need for more robust protocols in the verification and validation of 3D model data in legal and forensic investigations.

Further work

A continuation of this project aims to interrogate the benefits and limitations of this type of reconstruction in legal settings. The use of scene capture is of interest to both law enforcement and first responders for initial rapid documentation, as well as to investigators and trial teams for potential examination and analysis. As the reconstruction work implies computations and, for new algorithms, an element of machine learning and non-deterministic processes, we argue the use of anchors and explicit surfacing of the original, strongly-authenticated material provides guarantees and clues of explainability and support the verification of the model.

Also by this author

Event Roundup: Evidence & International Justice In a Generative AI World

Sep 1, 2024

Law

Artificial intelligence presents new challenges and opportunities in upholding accountability and human rights on the global stage. On September 20, the Starling Lab and Hala Systems convened an engaging group of experts – and developed a number of policy recommendations – to address these emerging intersections of international justice, digital evidence, and generative AI. The roundtable was hosted at Georgetown University’s new Capitol Campus thanks to their Tech & Public Policy program at the McCourt School of Public Policy.

United States Ambassador-at-Large for Global Criminal Justice Beth Van Schaack opened the forum with compelling remarks on how generative AI and synthetic media are reshaping the landscape of international justice. She emphasized the dual-edged nature of AI—its potential to bolster forensic verification and reconstruct crime scenes versus its capacity to amplify misinformation and complicate evidence authentication.

Key Themes and Insights

The “Liar’s Dividend” and the Crisis of Trust: Participants unpacked the risks generative AI poses, including its ability to cast doubt on authentic evidence by fostering plausible deniability. A recurring theme was the urgency of establishing robust protocols to distinguish genuine digital content from falsified media.
Provenance and Digital Evidence Standards: The event highlighted cutting-edge technical standards for provenance, such as the Coalition for Content Provenance and Authenticity (C2PA), as transformative tools for ensuring the integrity of digital evidence. Experts shared innovative methods for embedding authenticity markers into multimedia assets to secure their credibility in judicial proceedings.
AI-Driven Forensic Tools: From automating data analysis to 3D crime scene reconstructions, the symposium showcased AI’s ability to streamline evidence collection and processing. Such tools are proving potentially valuable in addressing the complex demands of war crime investigations and international tribunals.
Human Rights and Metadata Protections: With frontline documentarians and human rights defenders facing increased risks, participants underscored the need for metadata privacy safeguards and opt-in approaches to protect user safety.
Collaborative Futures: A forward-looking session explored how civil society, government institutions, and the private sector could democratize access to AI tools while fostering inclusivity and ethical AI use.

Policy Recommendations

The discussions at the intersection of international justice and AI led to a range of recommendations aimed at fortifying the global justice ecosystem. These have been categorized based on their intended audience: civil society organizations and the Office of Global Criminal Justice (GCJ).

Recommendations for Civil Society

Collaboration and Democratization: Promote deeper collaborations between the private and international justice sectors to democratize AI access. Decision-makers should support efforts ensuring all civil society members benefit equally and address the risk of overlooking underrepresented voices in the justice ecosystem. Shared access to knowledge and technology can ensure these partnerships’ benefits extend beyond a select few, fostering inclusive growth and broader societal impact.
Open Technical Standards: Consider adopting open-source, interoperable digital provenance standards poised to improve the reliability of potential evidence and foster greater trust within court systems. The design of these standards should be informed by cross-disciplinary research and common legal practices. One such example is the Coalition for Content Provenance and Authenticity (C2PA) standard, which addresses the prevalence of misleading information online by certifying the source and history of digital content.
Digital Evidence Protocols: Engage with the development and implementation of protocols for high-quality digital evidence. As a prominent example, the Hala Protocol for the Collection, Processing, and Transfer of Audio Data will outline standards for the use of audio data, bridging a gap between the technical aspects of the collection and the potential evidentiary admissibility and value of audio data.
Protection of Privacy and Expression: Support authentication technologies that prioritize the protection of human rights defenders by requiring explicit opt-in mechanisms to metadata collection, and empowering them to control when and how their metadata is shared. These protective design principles must be embedded at every stage of the information workflow.

Recommendations for the Office of Global Criminal Justice (GCJ)

Training: Hands-on training, including technical trial advocacy skills, in the well-established principles of provenance, authenticity and verification – and methods of applying them to digital content. This will empower practitioners with an effective means of combating mis- and disinformation and malicious synthetic media.
Report Card: The creation of a “consumer report card” for AI models and tools in the investigative and justice fields, to promote greater access and improve digital literacy. Using a standardized grading scale, the report card would evaluate each tool based on transparency, accuracy, and security. It would also emphasize the need for maintaining sustainability and open access, even as technologies evolve or are discontinued. This transparency would empower all to better understand AI’s potential and dangers, while incentivizing tool-makers to prioritize ethical standards, inclusivity, and sustainability.

The Starling Lab Perspective

Consistent with our mission, the Starling Lab participants emphasized practical applications for securing justice in the digital age. Our focus remains on implementing tools for capturing, preserving, and verifying digital evidence while fostering transparency and resilience in justice systems worldwide.

“This is more than a technological challenge—it’s a societal imperative,” said Basile Simon of Starling Lab during the event, adding:

“As generative AI technologies become more accessible, their implications for truth, trust, and accountability demand immediate and sustained attention. Our mission is to ensure that AI supports, rather than undermines, justice.”

Acknowledgments and Next Steps
The symposium featured contributions from global leaders in technology, law, and human rights, including representatives from the State Department’s Office of Global Criminal Justice and LOCATE team, the Prosecutor-General’s Office of Ukraine, the Atrocity Crimes Advisory Group for Ukraine, the Department of Homeland Security, the Georgetown McCourt School of Public Policy, the Georgetown Center for Security and Emerging Technology, the University of Colorado Boulder, Duke School of Law, the Guardian Project, Harvard’s Library Innovation Lab, HRDAG, the Security Force Monitor, EqualAI, Palantir, the Case Matrix Network, Microsoft’s Digital Crimes Unit, WilmerHale, the War Crimes Research Office, and Georgetown Law’s International Criminal Justice Program, WITNESS, and Hala Systems.

We thank all the guests for their generous contributions and time.

Starling Lab is especially grateful to the Tech & Public Policy program at the Georgetown University McCourt School of Public Policy for so kindly hosting us in the center of Washington, DC, for this conversation.

Event roundup: "Shall We Press Play?" – Navigating AI and Evidence in the Digital Age

Jul 17, 2024

Law

A warm thank you for joining us yesterday (Tuesday 16 July) for our enlightening event, “Shall We Press Play? – Evidence Considerations in an Age of AI.” We were thrilled to see so many of you interested in this critical discussion about the authenticity and verification of digital open-source imagery (OSI) in today’s AI-driven world.

Event Recap

Our expert panel, featuring Prof. Yvonne McDermott Rees from Swansea University and Jonathan W. Hak KC PhD, a seasoned Crown Prosecutor from Canada, led us through a comprehensive exploration of the challenges and opportunities presented by generative AI in the realm of digital evidence.

Introduction and Context: We began with a discussion on the rise of generative AI and its broad accessibility to all. With AI tools now available for as little as $20/month, the creation of photorealistic but inauthentic evidence is no longer limited to advanced disinformation campaigns by nation-states. This democratization of technology has significant implications for the authenticity of OSI, as evidence already subjected to intense scrutiny due to the unavailability of the original author to testify is now further cast into doubt by potential reflexive calls of fakery.

Core Topics:

Admissibility vs. Weight: Considering our mixed audience including non-legal professionals, our panelists began by debunking the notion that admissibility of digital evidence is the primary issue, emphasizing instead the importance of weight and probative value of the evidence, most notably in international jurisdictions. Both emphasized that instead of aiming to “make it” to admissible evidence, investigators ought to “aim as high as possible” to not be caught out. They referred to the recent Al Hassan ICC judgment (26 June 2024) to highlight how little guidance practitioners might receive from the court with regards to admissibility and weighting. Best come prepared!
Prejudice and Authenticity: The discussion then moved to the potential prejudicial effects of inauthentic material on judicial processes. Our panelists emphasized the concept of “seeing is believing,” highlighting how visual evidence can heavily influence jurors and judges. The power of images to create lasting impressions makes it difficult for fact-finders to “unsee” evidence, even if its authenticity is later questioned. This challenge is compounded by the strong emotional responses visual materials can evoke. To mitigate these prejudicial effects, rigorous technical interrogation and verification of digital evidence are essential before presenting it in court, helping protect proceedings from being compromised by misleading visuals–which are by now too likely.
Technical Interrogation: Further to the point above, we explored the necessity of comprehensive technical interrogation before viewing evidence, discussing its legal and ethical implications. Our panelists referenced the Berkeley Protocol on Open Source Investigations, which notes the importance of technical analysis in assessing the authenticity of digital evidence. We believe this analysis should include verifying the provenance and integrity markers (the objects of our research at Starling), supporting the verification of digital materials.
However, while rigorous technical interrogation might help prevent prejudice by filtering out falsified evidence, it also risks discounting authentic evidence that may lack sophisticated markers due to technical limitations or resource constraints. Balancing the thoroughness of technical verification with the inclusivity of all relevant evidence is crucial in maintaining the integrity of judicial processes.

Audience Q&A

The event concluded with an engaging Q&A session where we tackled questions about:

The need for legal education and training to adapt to these new challenges for all involved in proceedings, be they judges and fact-finders but also experts and investigators.
The role of the Confrontation Clause in U.S. settings, as guaranteeing a defendant’s right to confront their accuser, which could (should?) include the possibility to challenge the authenticity and reliability of digital evidence presented against them.

We sincerely hope that you found the discussion as insightful and thought-provoking as we did. Your participation and questions enriched the conversation and highlighted the importance of community engagement in navigating these complex issues.

Thank you once again for joining us. We look forward to seeing you at our future events as we continue to explore the intersection of technology and justice. I have several such events planned and hope to convene us again in August and September.

Until then,
Basile

Navigating AI’s Challenges in Courtrooms: New Insights from Starling Lab & TRUE Project in Opinio Juris

Jul 1, 2024

Law

Earlier this month, a joint byline including Starling Lab’s own Law program published an article in Opinio Juris titled “Old Doubts, New Doubts: Evaluating Digital Open Source Imagery in the Courtroom.”

Authored by Yvonne McDermott Rees, Stephen Sharp Queener, and Basile Simon, this piece delves into the complexities of authenticating digital open-source imagery within legal proceedings, especially in the context of advancements in artificial intelligence. The authors emphasize the necessity for rigorous verification methodologies to ensure the reliability of such evidence.

We at Starling Lab are delighted to have collaborated with Yvonne alongside her groundbreaking TRUE Project in the UK. A Professor of Law at Swansea University, she is a leading expert in international law and human rights, with a particular focus on the evidentiary challenges posed by digital technologies in legal contexts. The TRUE project, funded by the UK’s Economic and Social Research Council (ESRC), explores the dynamics of international criminal justice through a unique interdisciplinary lens, investigating the role of technology, evidence, and storytelling in shaping accountability mechanisms. This collaboration exemplifies our shared mission of advancing justice and integrity in an increasingly complex digital landscape.

This is Stephen Sharp Queener’s first publication with Starling Lab. He has supported the Lab for several years while an International Relations student at Stanford. He is presently an associate of the Law program and a Fulbright scholar. He will head back to campus later this year to finish his Master’s degree, ahead of law school.

Opinio Juris is a leading platform for international law discussions, renowned for its insightful analysis and contributions from scholars, practitioners, and thought leaders. Managed by a dedicated team of legal experts, including Managing Editor Jessica Dorsey of Utrecht University School of Law and Deputy Managing Editor Sarah Zarmsky of Queen’s University Belfast, the publication fosters diverse perspectives on critical global issues. It combines academic rigor with practical relevance, making it a trusted resource for the legal community worldwide.

Navigating the Tidal Wave of Gen AI: Challenges and Solutions for Verifying Digital Evidence

Generative AI poses a significant challenge to legal proceedings due to the ease of creating convincing fake evidence. Traditional verification methods are becoming inadequate, necessitating a shift towards trust-based frameworks leveraging cryptographic signatures and decentralized provenance systems.

While essential, raising standards for digital evidence should not exclude genuine evidence lacking advanced verification markers. Over-reliance on technical indicators can create a “reverse CSI effect,” disproportionately impacting marginalized groups and undermining the democratizing potential of open-source intelligence.

The focus should be on “raising the baseline” for verification and reliability by embedding robust systems into digital content creation and management. This ensures traceable and trustworthy evidence, enabling the legal system to adapt to the challenges of generative AI and manipulated content while maintaining the integrity of justice in the digital age.

We encourage you to head to OpinioJuris to read the full article and to share with your colleagues.

Proposing An 'Index For Accountability' To UN Human Rights Chief

Mar 9, 2024

Law

The work of documenting human rights abuses is inherently decentralized, reflecting the diversity of civil society organizations and monitors working tirelessly in conflict zones and beyond. This natural fragmentation is not a shortcoming but a testament to the resilience and independence of these groups. However, within this diverse ecosystem lies the potential to amplify impact through better coordination and interoperability. By respecting the unique contributions of each actor while fostering collaboration, we can explore ways to transform these decentralized efforts into a cohesive force for justice and accountability.

The Starling Lab for Data Integrity, in collaboration with Hala Systems, have jointly submitted a comprehensive proposal to the United Nations Office of the High Commissioner for Human Rights (OHCHR). This document outlines a new vision for using digital technology in the administration of justice, particularly focusing on enhancing the management and utilization of digital evidence in conflict zones like Ukraine.

We believe our proposed design and prototype software, called Index for Accountability, can enhance the authenticity and reliability of digital evidence, making it a powerful tool in the pursuit of justice.

The Index is underpinned by Starling’s research and prototypes for associating digital assets and their metadata, which we call Authenticated Attributes. Our proposal to the United Nations also rests on our principles of “Authenticity by Design” which place provenance and integrity data front and center in the architecture of information-gathering tools.

Starling and Hala had previously made submissions to the UN with documentation of attacks against schools in both Ukraine and Syria (with Russia among the alleged perpetrators in both conflicts), along with recommendations for the long-term preservation and authentication of that evidence. Like the new submission, the prior ones leveraged a series of advanced technologies including cryptography, and decentralized systems. The undertaking is part of an ongoing effort to help the international legal community hold perpetrators to account for acts that deprived children of their right to education.

Supporting Justice with Digital Technology

An original collaboration and consortium including Bellingcat, Hala Systems, Ukraine War Archive, Benetech, Airwars, and Mnemonic, the Index for Accountability seeks to acknowledge the fragmentation of digital evidence archives while offering a secure framework for evidence management. This can turn traditional challenges into strengths for the administration of justice.

The original goals of the tool go back some years, and have been supported by, among others, a concept note and proof of concept from Benetech, and initial user research by Interstitial – both circulated to members of the consortium.

In essence, the Index is a software-driven collaboration designed to simplify accessing and sharing digital evidence. It is intended for participants seeking to support atrocity crimes investigations in conflict zones. The tool is designed to facilitate the sharing of evidence collected by civil society and monitoring organizations, including with UN mechanisms, while ensuring that archive creators and holders retain control over their specific evidence and the ability to manage their consent for participation in networks and engagement with new entrants.

Our prototype of the Index has been integrated with the digital asset management system Uwazi, which is popular among investigators of human rights violations.

The approach of the Index, which protects the files and evidence itself with state-of-the-art provenance and custody markers, enables collaborative workflows whereby teams, or separate organizations, can effortlessly share some parts of their data – or rather let others effortlessly discover some of the data they have created and curated.

In the following example, two organizations who trust each other share metadata they create (description of the item, geolocation of a photograph, etc.) in a system, making the data discoverable by the other organization – provided the other holds the exact same photograph in their database:

Transformative Benefits and Implementation

The Index for Accountability (IfA) is designed to deliver substantial benefits to justice systems:

Streamlines Justice Processes: The IfA simplifies the collection, sharing, and analysis of digital evidence, optimizing resource use.
Enhances Evidence Integrity: By improving the reliability and consistency of evidence, the Index ensures a fairer administration of justice.
Encourages Global Collaboration: The framework fosters respectful international collaborations, strengthening global justice efforts.
Supports Broader Justice Goals: As a neutral and authoritative initiative, the IfA aids in truth-seeking and the establishment of collective memory.

The Index prioritizes privacy and incorporates Authenticity By Design, safeguarding data integrity and confidentiality to uphold human rights. It ensures equal access and participation, supporting fairness and non-discrimination. The IfA also includes measures for accountability and remedies for violations related to digital technology use in justice administration.

Earlier last year, Starling Lab and Hala Systems submitted related materials to prosecutors at the International Criminal Court as evidence of potential war crimes.

Basile Simon, accountability lead at Starling Lab, said:

“Once again we are privileged to work with Hala Systems, and to present our work to the UN. Our research into usable and portable authenticated data overlaps with the OHCHR’s public call for contribution about the administration of justice. We believe that a decentralized architecture uniquely fits the priorities and concerns of civil society organizations seeking to efficiently communicate their evidence with the United Nations. We call on the UN and its Mechanisms to consider implementing and being a leader of this project.”

Starling’s Call to Action for the UN Commission of Inquiry on Ukraine

Jan 25, 2024

Law

The United Nations Human Rights Council creates unique opportunities for citizens and experts to speak out and share their evidence with the world. In 2022, they formed the United Nations Independent International Commission of Inquiry on Ukraine (or “Ukraine COI”) to investigate aggression by Russia.

In 2023, the Commission put out a call for submissions, specifically seeking information about Russian mistreatment of children and attacks on civilian infrastructure in Ukraine. Starling Lab, Hala Systems, Global Justice Advisors, and NORSAR responded with a comprehensive submission. Not merely a report, it is also a vital call to action – shedding light on the severe human rights violations committed by the Russian Federation and affiliated paramilitary groups against Ukrainian citizens.

Highlights from the Submission:

Our document is a fusion of advanced technological analysis and exhaustive research. Key findings include details of multiple alleged crimes:

Critical Situation of Displaced Ukrainian Children: The submission draws on a Conflict Observatory report on the removal of Ukrainian children from temporarily-occupied territories of Ukraine. We combined the report with original investigations by Hala to identify 91 sites that potentially housed or otherwise facilitated the deportation of these children. While the resulting evidence base is being kept confidential for safety reasons, the documentation by the team falls under the broad category of open-source intelligence (OSINT) and related novel approaches.
Targeted Attacks on Civilian Infrastructure: The submission also includes an original dataset assembled by nuclear test treaty (CTBT) monitors at NORSAR, a Norwegian research foundation acting on behalf of the government of Norway in this mission. Their team published an innovative methodology using ground seismic data as well as infrasound signals to derive a geo-, yield- and time-series datasets of explosions in Ukraine – a uniquely detailed picture of the ground invasion and related bombardment campaign.
Legal Implications and Potential Violations: Based on the above, the submission thoroughly examines possible infringements under international laws like the Rome Statute, Geneva Conventions, and the UN Convention on the Rights of the Child, suggesting potential crimes against humanity and war crimes.

Emphasis on Strategic Recommendations

Another significant aspect of this submission lies in our recommendations to the Commission and the HRC more broadly:

In-depth Investigations into Child Transfers and Civilian Attacks: The submission advocates for comprehensive investigations into the forced transfer of children and the attacks on civilian infrastructure, utilizing datasets such as GPS mobility data, as well as seismic and acoustic wave monitoring.
Adoption of Advanced Cryptographic Methods: We propose the implementation of cutting-edge cryptographic methods to verify the integrity and authenticity of digital evidence, crucial in the current digital landscape to ensure data integrity and overcome potential denialism.
Creation of an Inclusive Evidence Repository: The recommendation to develop a broad repository of evidence, inclusive of civil society actors, is a step towards ensuring that the evidence base is both comprehensive and diverse. We believe the UN is uniquely positioned to federate around this effort.
Encouraging Interdisciplinary Collaboration and Training: Finally, the submission calls for the collaboration of legal experts, digital forensics specialists, and OSINT analysts – highlighting the need for a multifaceted approach to handling digital evidence.

What’s Next?

The Commission is now considering the submissions received, and will report their findings to the Human Rights Council (HRC) ahead of the Council’s next session running February to April 2024. This report will be tabled under reference A/HRC/55/66. [Editor’s note: An initial version of their report is published here, with a comprehensive report to the Human Rights Council expected at its 58th session in March 2025.]

A joint submission from Starling and Hala was previously recognised by the Council’s 53rd session.

"New Frontiers in Evidence" OSINT event at the Inner Temple, London

Dec 1, 2023

Law

Attending the “New Frontiers in Evidence” conference at the Inner Temple on November 25, 2023, was an enlightening experience for us at Starling Lab. The event brought together preeminent legal minds to discuss the challenges and opportunities presented by user-generated evidence in modern legal contexts.

Her Excellency Judge Joanna Korner CMG KC opened the day with insightful remarks, setting the stage for a day rich with expert discussions. Eliot Higgins of Bellingcat delivered the keynote speech on the accountability and disinformation challenges inherent in user-generated evidence, highlighting methodologies grounded in the Berkeley Protocol and innovative tools like Hunchly and Mnemonic.

Session I focused on the admission and evaluation of user-generated evidence, with Dr. Jonathan Hak KC emphasizing the risks of miscarriages of justice and the difficulties in authenticating digital evidence. Dr. Alice Liefgreen and Prof. Yvonne McDermott provided crucial insights into the impacts of deepfakes, while Dr. Micheál Ó Floinn critiqued antiquated legal presumptions around digital evidence.

Ethics and practicalities in evidence collection were explored in Session II. Raquel Vazquez Llorente from WITNESS and Andrew Finkelman of Meta discussed the challenges of data retention and access under Article 40 DSA, highlighting the need for bridging gaps between current practices and future international mechanisms.

Interactive workshops followed, delving into the future of synthetic media, advocacy, and verification techniques. In the afternoon, Session III on investigative methodologies featured experts like Ben Strick and Dr. Alexa Koenig, who shared strategies for preserving and validating remote information.

The day concluded with discussions on the presentation of evidence in court and reflections on mock trial hearings. Notably, Her Excellency Judge Solomy Balungi Bossa and other legal luminaries underscored the critical need for training judges and ensuring the reliability and authenticity of digital evidence.

This conference was a profound reminder of the evolving landscape of evidence in the digital age, reaffirming our commitment to staying at the forefront of these pivotal discussions.

Starling-Hala joint legal methodology recognised by UN Human Rights Council

Jul 15, 2023

Law

A new report, presented to the United Nations Human Rights Council, cites work of the Starling Lab for Data Integrity and Hala Systems as an emerging good practice to protect the universal right to education.

The two organizations had previously submitted documentation of attacks against schools in both Ukraine and Syria (a conflict also involving Russia), along with recommendations for the long-term preservation and authentication of that evidence. They implemented a series of advanced technologies – including cryptography, decentralized systems, and blockchains – as part of an ongoing effort to help the international legal community defend the right to education.

Ms. Farida Shaheed, the UN Special Rapporteur on the right to education, presented her review of the UN’s mission to protect the right to education during the 53rd session of the Human Rights Council, which concluded July 14, 2023. In her report, global leaders reflect on achievements that ensure children can learn — as well as ongoing crises that continue to deprive millions of that opportunity.

Some of the most chilling examples stem from conflicts where armed forces have bombarded the most inappropriate of targets. “In 2020 and 2021, over 5,000 reported attacks and instances of military use of educational facilities were documented in 84 countries, with systematic patterns of such use in 28 countries,” according to Shaheed’s review.

Starling Lab and Hala Systems’ earlier submission to the Special Rapporteur:

Demonstrates how digital evidence — including social media posts and photos of the damaged schools — can be authenticated and preserved with open-source tools like Webrecorder, ProofMode and Signal. Using advanced cryptography, investigators were able to include strong proofs of critical metadata like date, time, and GPS coordinates of digital media depicting attacks.
Illustrates how all relevant data might be preserved for the very-long term, in order to ensure that children are able to access the information and seize justice at a later date. In Ukraine, the underlying media assets and metadata were preserved using decentralized Web3 storage systems and registered on multiple blockchains, providing evidence of their continued integrity.
Recommends investigations of such attacks by all means available and not limited to traditional criminal accountability mechanisms.

The new report, titled “Securing the right to education: advances and critical challenges,” is Shaheed’s first since being appointed last year. It addresses the need for universal acceptance of the right to education. The documentation by Starling Lab and Hala Systems indicates that some regimes are operating in defiance of these international norms. In many cases, sanctuaries of learning were turned into literal war zones with apparent deliberate attacks — despite being far from any militarily significant location.

Earlier last year, Starling Lab and Hala Systems submitted related materials to prosecutors at the International Criminal Court as evidence of potential war crimes.

Basile Simon, accountability lead at Starling Lab, said:

“We’re delighted that the Council should recognise that ensuring and protecting the right to education means empowering those deprived of this opportunity. These schools can’t be rebuilt fast enough, but when they are, the physical evidence will disappear. It can take decades to prosecute a war crime, and we want to ensure the first victims, the children, can trust the digital evidence, have an opportunity to know the truth, and be supported in their choice of the kind of justice they want.”

The techniques used for establishing trust in digital assets are taken from the Starling Framework, which focuses on three stages of data’s lifecycle: Capture, Store and Verify. This provides both experts and novices with a guide to establish and evaluate data integrity.

Authenticated Web Archives

New Evidence Techniques Document Bombed Ukrainian Schools For ICC

Empowering justice through evidence: Starling Lab leverages cryptographic tools to document attacks on Ukrainian schools for criminal accountability at the International Criminal Court.

Basile Simon

Reading Time: 5min

law-icon

Law

Prototypes

Timestamp Verification

ProofMode Authentication

Contents

Background
Context
Framework
Technology Learnings Archive

More about Project Dokaz →

Background

The large-scale invasion of Ukraine by Russia in February 2022 triggered widespread international outrage. This was accompanied by a surge of interest in the conflict from many in the West: not only did the conflict feel closer geographically than what many of us had experienced in our lifetimes, the information coming out of Ukraine was also of extraordinary quality (and impact). Smartphones seemed to be in every pocket, and strong 3G and 4G networks around the country added to the sense of immediacy and rawness of this new open conflict on the eastern edge of Europe, which marked a dramatic escalation of a conflict that had been simmering since Russia’s initial intervention in Ukraine in 2014.

The ongoing conflict has severely impacted Ukraine’s education system, with over 3,790 educational facilities damaged or destroyed since 2022. This destruction has left more than five million children struggling to access education, compounding previous disruptions caused by the COVID-19 pandemic and ongoing conflict in Eastern Ukraine since 2014. Many schools have allegedly been used for military purposes, turning them into legitimate targets and further endangering students’ lives. As a result, Ukrainian children are facing significant learning losses, particularly in foundational subjects like language and mathematics. International efforts are focused on supporting Ukraine in rebuilding its education infrastructure and ensuring that students have access to safe learning environments, whether through physical reconstruction or online alternatives. Furthermore, the disruption emphasizes the need for transitional justice, as documenting these attacks is essential to empower the affected generation with the evidence needed to pursue future accountability and justice for war crimes against educational facilities.

Concerned with allegations of harm in Ukraine, the Lab’s law program made international criminal law a key focus of our research. We started Project Dokaz (“Доказ”, Ukrainian for proof) in partnership with Hala Systems, a social enterprise we have historical connections with from consultations on their projects related to early-warning systems in Syria.

This coalition was led by the Lab’s newly-appointed accountability lead, Basile Simon, and Hala’s director of accountability, Ashley Jordana. Basile joined from past work on OSINT methodologies and investigations, including developing an open source suite of tools aiming to meet evidentiary standards. Ashley, a Canadian-qualified barrister, had joined Hala Systems with a wealth of international trials experience, including strategic litigation covering notably Ukraine.

We were motivated by the words of the International Criminal Court Prosecutor, Karim A.A. Khan KC, who four days after Russia invaded Ukraine and escalated the conflict simmering since 2014, said he had tasked his team with “explor[ing] all evidence preservation opportunities.” We set out to demonstrate how our methodology could apply to an ICC set of standards.

The first research partner of Project Dokaz was the Digital Forensic Research Lab at the Atlantic Council, a Washington D.C. think-tank. The DFRLab team focuses on implementing OSINT methodologies and monitoring for the purpose of studying disinformation campaigns and documenting human rights abuses.

Michael J. Sheldon, a researcher at the DFRLab at the time, compiled a list of social messaging material allegedly posted by Russian entities about the invasion of Kharkiv. Sheldon’s systematic monitoring focused on March 2-16, 2022, representing the approach of the city by Russian armed forces. The dataset was shared with the Lab and Project Dokaz, and served as the basis of the investigation.

Another early-day partner of the project was Scott Martin from Global Justice Advisors, a Washington DC-licensed attorney based in Europe. Scott has spent decades investigating international crimes and working in the international criminal tribunal system. His experience putting together these investigations, as well as preparing submissions to the ICC, was instrumental in our ability to translate what this evidence base could mean in an ICC context.

The idea that such a coalition of organizations and individuals could contribute to accountability is not new. In fact, the role of “Track II” practices has been studied for several decades. This refers to non-state mechanisms that use civil society, NGOs, and technology to document human rights abuses and ensure accountability when formal legal avenues are constrained or delayed. Starling Lab and Hala Systems embody this approach through their use of decentralized technologies and cryptographic tools to capture, secure, and preserve evidence of war crimes in Ukraine.

Contents

Context
Framework Technology Learnings Archive

Context

The monitoring compiled by the DFRLab included messages from several Telegram “Channels,” which the investigative team systematically searched using English and Ukrainian keywords. In a busy environment with considerable noise in the beginning of the conflict, this methodology permitted the monitors to narrow down their focus on 144 Telegram posts containing instances of artillery fire which could be geolocated with some certainty. Geographical location was established using details included in the media and cross-referenced with publicly available information such as satellite imagery.

From this original dataset, a team of legal experts at Hala Systems picked incidents they thought would be the most relevant and reliable for further investigation. Drawing on the qualifications and experience of Ashley (rostered Justice Rapid Response expert and advisor to the Crimean Prosecutor’s office, both on crimes against children), and inspired by Starling’s roots in higher education, the investigative team decided to focus on allegations of attacks against schools – protected objects under international law. Article 8(2)(b)(ix) of the Rome Statute prohibits intentionally attacking buildings dedicated to religion, education, art, science or charitable purposes, historic monuments, hospitals and places where the sick and wounded are collected, which are not military objectives, as a serious violation of the laws of war in the context of an international armed conflict.

The legal team put forward an online investigation plan going beyond the Telegram messages and looking into the prior and subsequent attacks on the same schools, nearby military installations, and the context of the armed conflict at the time.

The Starling Lab identified a methodology and tools for the forensic capture of the Telegram messages, as well as their verification, along with corroborating online information that required archiving.

To illustrate the complexities and need for corroboration, one incident involved public counter-allegations aiming to justify Russia’s targeting of the school by claiming that it hosted a group of soldiers from Ukraine’s Azov Battalion. Indeed, protected objects can lose their status and become legitimate military objectives if taken over, even partially, by combatants and other active participants in hostilities.

Remnants from an air-delivered bomb which hit the running track around a Kkarkiv school playground.

Over the course of the investigation, the need arose to document consequences of the attacks beyond what could be observed through the Telegram posts and online verifications (e.g., closer shots of a wall pockmarked by shrapnel, and of the damage from what appeared to be an air-delivered bomb). Individuals local to Kharkiv were hired to take photographs and videos of two impacted schools: School 17 and School 35.

Starling Lab had two objectives when assembling tools to support the field verifications. First, photographs and video captured by the team should be demonstrably attributable to them in the future. Particularly, should the photographers not be available to testify in the future, the attribution and metadata of the photos must have a mechanism to independently establish their authenticity and credibility. Without direct photographer testimony, documented provenance and digital verification methods (such as tamper-evident technology or cryptographic signatures) are essential to uphold the legal validity of the evidence. Second, for safety reasons, producing strongly-authenticated data could not hamper or slow down the field work.

To make the most of the field photographs and their ability to better contextualize the physical space surrounding the two schools, the Lab produced an interactive map. Photographs were laid out to show where the camera was aiming, and each included strongly-authenticated metadata pulling directly from image registration records. The interactive map itself was hosted on and served from distributed systems.

The resulting evidence base included original Telegram posts documenting the allegations, online OSINT verifications by investigators, and field photographs confirming specific points of detail. This supported the team drafting of two communications to the ICC – one detailing the original allegations and a follow-up focusing on the production of the field photographs. Both were shared with the ICC Office of the Prosecutor under the terms of Article 15 of the Rome Statute, which stipulates:

“The Prosecutor shall analyze the seriousness of the information received. For this purpose, he or she may seek additional information from States, organs of the United Nations, intergovernmental or nongovernmental organizations, or other reliable sources that he or she deems appropriate, and may receive written or oral testimony at the seat of the Court.”

Following these submissions on June 10, 2022, and Jan 17, 2023, a briefing took place with investigators at the Office of the Prosecutor, as well as subsequent conversations with some of the Court’s IT staff. The object of these meetings was to socialize the concepts, as well as potential benefits and costs should they be considered beyond cases and more broadly into how the Court manages digital information.

Contents

Framework
Technology Learnings Archive

Framework

The Challenge

Starling Lab wanted to maximize the admissibility chances and probative weight of the material: web archives of social messages, supporting online documentation, and photographs and videos from the field.

All data in these investigations are subject to risks of loss, tampering, and damage. Content hosted on the web, including social media and messaging, is particularly vulnerable and must be preserved separately and independent from the platform through web archiving. In all cases, the material must be protected against deliberate attacks (tampering) and accidental loss or damage.

Moreover, we believe the digital evidence collected today will be assessed and examined in a near future when cheap and accessible generative AI can upend our expectations of “trustworthiness.” We aim to collect, verify, and put forward stronger, better contextualized data by preserving digital content alongside provenance and integrity records.

ukraine-building — Caption TK Caption TK Caption TK Caption TK Caption TK Caption TK Caption TK Caption TK Caption TK

A single Telegram message, which can be captured as a web archive using specialized tools. (left). Many such messages can be aggregated and presented to provide context and corroborating evidence for an event (right).

Contents

Framework
Technology Learnings Archive

The Prototype

These investigations were underpinned by the Lab’s research into the creation of authenticated data and the claims that can be made about it, notably as it relates to digital evidence. The methodology, jointly developed by Starling and Hala, is best summarized as three phases in the lifecycle of a piece of evidence: capture to adequate evidentiary standards; preservation for the long-term; and availability for analysis, scrutiny and investigation. This “Capture, Store, Verify” Framework is the subject of a Technical Whitepaper <internal link>

The capture of web pages and social messages
We previously identified notable threats in the current practice of producing web archives and backups of web pages, and summarized them in a November 2022 whitepaper, “Best Practices for Web Archiving.” In short: Content on the web, and social media / messages in particular, is subject to rapid rot and deletion, and it is crucial to preserve it should it be required as evidence. The whitepaper also outlines criteria of strong web archives, as well as preservation techniques for their use as evidence:

For maximal probative value, “the ideal web archive (…) could have been produced by anyone, meaning that the material was publicly accessible (i.e. not privileged or obtained through deception) and the tools to produce it are available to everyone (i.e. not proprietary). This could be either or institutional actor or an individual (see characteristic No. 1 above); and Is of high fidelity, meaning it was carried out by a tool that preserved much if not all of the original material (see Characteristic No. 2 above); and Includes the content itself, its surrounding metadata, the metadata of the web scraping software, its hash value, and the signature of these hashes authenticating it to the author (see characteristic No. 3 above). The aforementioned hashes and signatures must be preserved, that is to say stored securely and made available for the long term, as would the content itself.”

At this point in time, the whitepaper’s criteria for authentication are satisfied by the use of modern web archiving tools such as Browsertrix, the Webrecorder suite, and the cryptographically-signed WACZ standard. Another important benefit of the use of the aforementioned tools is their unique offering in terms of contextualization and renderability (see Berkeley Protocol). These archives not only include the files of the webpages themselves, but also a replay engine, making the web archive effectively browsable. Investigators can interact with a web page as it was at the time of archiving, including interactive elements like clickable links. The inclusion of the replay engine defends against technological change which could render such viewing and engagement impossible, thus making the archive obsolete.

Downstream from this capture of web archives, the Starling Framework further confers forensic qualities to the material, by establishing additional data points corroborating provenance and distributing proofs of integrity. The Framework storage policies and choice also aim to render the material available for the long term.

The legal team included verifications of counter-claims which could have turned out to be exculpatory. Above, an archived Telegram message from a group assessed as pro-Russian, claiming the bombardment of School 17 was legitimate due to the prior presence of armed forces of Ukraine’s Azov Battalion – a claim our submission refutes.

The production of strongly-authenticated field photographs
The field photographs, taken at selected schools in Kharkiv, were captured using guidelines from the Framework. They include an additional layer of context around them, in the shape of metadata. Traditionally this includes verifiable location, date, time, etc. Our legal submissions make the point that this metadata should also provide corroborating information about the photographs, the photographer, as well as their smartphone.

From a custody perspective, this layer of metadata gives the ability to make determinations regarding the person who used the device to take photographs, as well as where and when the photographs were taken. Notably, these determinations rely on the attributability of cryptographic keys. In other words, the photographer could provide access to the smartphone and prove the photographs come from the key contained in a camera app. The method used to cryptographically seal the photographs provides secure timestamping of their capture. According to the guidelines of the Framework, this is attested by a third-party, who also stores proofs of this timestamp, permitting later auditing.

Once again, downstream from this capture of photographs, the Starling Framework further confers forensic qualities to the material, by establishing additional data points corroborating provenance and distributing proofs of integrity. The Framework storage policies and choice also aim to keep the material available for the long term.

The verification displays available use both Content Credentials Verify tools as well as metadata registered on distributed ledgers (see paragraph below).

The inclusion of strong metadata and cryptographic registration receipts
Both submissions consist solely of digital evidence. Both sets of original assets were preserved and stored according to the methodology. As such, they possess unique properties. Overall, the Framework aims to produce a more authenticated, traceable, and auditable digital evidence base:

Authenticated: The authenticity of digital items preserved through the Framework can be established by comparing the hash value of the files with those submitted to the Court or stored with third-party record holders.
Traceable: The provenance of digital items preserved through the Framework can be demonstrated by referring to timestamps proving date of reception (typically through registration with those third parties), and cryptographic receipts proving the preservation of the material on cold storage network systems.
Auditable: The auditing and verification of all digital items preserved through the Framework can be carried out by referring to the third-party record holders.

We argue that these additional integrity characteristics, while not strictly necessary today considering the permissive standards of the International Criminal Court (which rest on a three-part test of relevance, probative value, and prejudice), will be the markers of the strong evidence bases of tomorrow. The Lab’s research covers several industries, all of them dealing with issues related to synthetic content and generative artificial intelligence. We have found that these characteristics are often present in distributed ledgers like blockchains, which also don’t require their users to place their trust in single centralized authorities. As international criminal proceedings tend to span years, if not decades, the evidence bases created today must seize this opportunity to demonstrate their integrity and resilience over time.

One of the photographs, presented in situ and in the context of the school’s geography and the other images, displaying in addition integrity data of the photograph.

Cryptographic signatures, employed throughout the submissions to secure the digital items, their integrity proofs, and the transport to and through the archive, are one such strong marker permitting the verification of attestations to individuals and organizations. These might be supported by witness testimony or an affidavit verifying the individuals were in possession of the keys and carried out such and such action.

The inclusion of registration receipts in the submissions, including hyperlinks to online “block explorer” tools pointing to these receipts, aimed to permit some level of auditing by the staff of the Office of the Prosecutor. While conversations with the Office revealed a lack of familiarity with these techniques, we argue that the presentation of this data in a familiar spreadsheet environment and interactive map of photographs enabled a relatively accessible auditing beyond prima facie consumption of the photographs.

The submissions themselves contained lengthy methodological descriptions of the processes followed by the investigation, as well as of the meanings and claims of the integrity layer.
http://starlinglab.org/wp-content/uploads/2024/11/output.mp4

Contents

Context
Framework
Technology
Learnings Archive

Technology

Capture

Web archives (of webpages and social messages) were created using tools developed by Webrecorder. Each of these archives results in a cryptographically-signed file matching the WACZ specification, and can be browsed independently and offline using the ReplayWeb.Page tool. For more about WACZ in this context, see this introductory dispatch from the Lab.

The cryptographic signatures of the web archives used AuthSign, another Webrecorder tool, as well as Hala Systems’ and Starling’s SSL certificates. In short, the very certificates used by both organizations to secure their online websites are also used in this instance to digitally sign the captured webpages. Notably, this demonstrates support to and oversight of the crawls, as both signing organizations needed to take active steps to enable these signatures. These steps aim to demonstrate that the archiving party “has a secure, private connection” to the signing server on the designated domain, as per the Webrecorder spec.

The field photographs of the schools were taken using Android smartphones equipped with ProofMode, a mobile application from the Guardian Project collecting surrounding metadata for a media capture, as well as notarizing the image hash value and timestamp through third-parties.

Signal Messenger was used to transfer ProofMode bundles from the field to Starling servers. Prior to transfer, we established trust between the Signal phone numbers involved.

From the point where Starling took custody of the field photographs, as well as from the moment the legal team finished assembling web pages they wished to preserve, the bundles of data assembled went through the Starling Integrity Pipeline, a proposed implementation and assembly of tools aiming to standardize the way data is organized and preserved, as well as to trigger cryptographic registrations on the Avalanche and LikeCoin blockchains, which are both publicly inspectable ledgers.

The registration of web archives and field photographs using the Numbers Protocol and International Standard Content Number (ISCN) specification.

Store

Once integrity records were secured on public ledgers, data bundles were prepared for long term preservation. These bundles containing the original assets were encrypted at rest on Starling’s storage pools.

Non-critical metadata about the assets, limited to the hash values of the media and in accordance with the legal team’s investigative plan, were made available on the IPFS public storage network.

The preservation of the data bundles is backed by best-in-class digital storage. The core of the infrastructure resides in a data center belonging to OVH, one of the world’s largest cloud hosting providers, protected by security badge control system, video surveillance and security personnel 24/7 on-site, and further protected by encryption at rest.

Encrypted copies of the files (field photographs, web archives) were also preserved on two decentralized storage providers: Filecoin and Storj. This multi-tiered approach ensures redundancy, data integrity, and compliance with the investigative requirements, while also providing a secure, immutable record of the assets that can be independently verified if needed. While Filecoin offers a robust, blockchain-based verification system for storage proofs, Storj focuses on high performance through its distributed network, providing faster access times and scalability, making each provider uniquely suited to different aspects of secure, decentralized data storage.

Verify

A key element of the demonstration of non-tampering of the files is the aforementioned distribution of integrity data on third-party ledgers, thus permitting verification of hash values and signatures at a later stage. In this instance, such non-critical metadata was registered on the Avalanche blockchain through Numbers Protocol, and on the LikeCoin blockchain following the International Standard Content Number (ISCN) specification. While Avalanche's Numbers blockchain offers fast transaction times and a flexible framework for integrating integrity proofs, LikeCoin's ISCN-native blockchain provides a specialized content registry tailored for media assets, making it ideal for precise tracking and verification of digital works.

ISCN integrity record of a web archive on LikeCoin viewed through its media asset explorer.

The investigative team manually marked relationships between the initial Telegram posts and their subsequent verifications. The resulting “web” of relations can be leveraged and displayed by asset-management software downstream (e.g. Uwazi, a flexible AMS for human rights defenders).

For the purpose of socializing some of these concepts to the Office of the Prosecutor, the submission included spreadsheets mapping all included digital items with their registration and timestamp data, as well as links to the various ledgers on which the data can be confirmed.

Finally, the submission included a bespoke online map of the schools visited by the in-country team. Satellite images of the schools were overlaid with the location of the photographers, as well as the vantage points of the photographs themselves, permitting viewers to explore the damage through different angles. Each display of the photographs also directs viewers to the integrity records of each image, permitting manual verification.

Contents

Context
Framework
Technology
LearningsArchive

Learnings

Articulation of the Framework Benefits in a Legal, Investigative Context

This joint project represented an important milestone for the Lab, insofar as it involved presenting the methodology and the practices the Lab had been working on to an external, legal audience. Abstracting from the seriousness of the allegations contained in the submissions, as well as from the difficult context of a conflict causing suffering for civilians, the team felt an obligation to present its work to the Court as tersely and intelligibly as possible – keenly aware of the novelties it contained.

The Starling Framework had been applied in the past in the shaping of prototypes and projects related to pieces of journalism, e.g. election coverage in Hong Kong or reporting on deforestation in Brazil. This project, however, was the first articulation of the methods, merits and limitations of the Framework in an international law context. It was the legal team’s first contact with some of the specific implementations as well.

We were pleased to be able to rely on the technical literacy brought by the publication of the Berkeley Protocol for Open Source Investigations. Published in December 2020, this seminal collaboration between Berkeley’s Human Rights Center and the UN Human Rights Office set the stage for investigations relying on openly-accessible material collected online – more commonly referred to as OSINT. The Protocol notably formulates technical high-level recommendations related to hashing, signing, and storage. We built on the Protocol’s guidelines to develop specific technical designs that implement and fulfill these standards.

Similarly, the conversation between the two parts of the team was key to identifying how the methodology and the Framework fit in a legal context. The vehicle for this project, a submission under Article 15, was extremely permissive – it is in fact a free-form communication to the Office of the Prosecutor, who then reviews and assesses on merit. This freedom led to questions regarding the merit of the efforts and innovations deployed in this instance. We found it helpful to reframe the methodology from its original claims of solving problems in the practice of international law, and towards presenting opportunities to produce digital evidence that is of a stronger kind, and possibly more fit to stand forensic scrutiny over a long period of time.

Articulation of the Framework Benefits in a Legal, Investigative Context

urther to the section above, project members sometimes had issues finding semantic common ground.

The Lab focuses on three practice areas (Journalism, History, and Law) and builds upon a software engineering background. Consequently, much of the terminology used in our methodology aims to speak to a broad audience. But sometimes, professions see specific words and phrases as terms of art (i.e., holding a narrow meaning within their field).

Take “Authentic”, for example. To a legal audience, “authentic” is a quality of a piece of evidence which was demonstrated as not being a forgery, i.e. genuine. While English dictionaries similarly converge towards this definition (Merriam-Webster: “Conforming to an original so as to reproduce essential features. Not false or imitation.” Collins: “Of undisputed origin or authorship. Reliable or accurate”), it is common vernacular to use “authentic” interchangeably with “genuine”, “truthful”, “real”, etc. Meanwhile, psychologists have tried to define different types of authenticity, including historical, categorical, and values.

What is “Verified”? “Verification” is narrowly-defined in law as a specific process by which a person, under oath or before a notary public, declares that a statement is true. The term is used more broadly in everyday life. One notable example is the choice of this terminology by the Content Authenticity Initiative, a consortium of various companies with stakes in visual media, who call one of their tools Verify. This tool permits the inspection of a digital item in view of surfacing provenance information it might contain.

A Need for Cheap, Accessible, Open Tools

In several instances, the issue of accessibility of the tools involved in the project arose. How difficult is it to reproduce the Framework’s processes? Is preserving and storing each item this way prohibitively expensive? Does it require a dedicated engineering team, or advanced technical knowledge to operate?

The underlying question was the opportunity for all parties to potentially benefit from the tools and the methodology. Any choice of tool or technique which would be too complex or expensive to run, or too bespoke for the project, would face hurdles related to fair trial and equality of arms.

The SITU digital evidence platform, developed for the ICC’s Office of the Prosecutor (OTP), provided a detailed 3D model of Timbuktu for contextualizing evidence. Originally designed for the Al Mahdi case, the platform featured a detailed digital reconstruction of Timbuktu, allowing judges, defense teams, and prosecutors to visually explore and contextualize evidence within a 3D environment. While effective, critics have pointed out that the high complexity and the resources needed for these systems can be a barrier, particularly for smaller defense teams lacking equivalent technical capacity (see blog from Jonathan Hak KC, for example).

Similarly, OTPLink, another ICC platform, aimed to streamline evidence submission but faced significant accessibility barriers for victims and witnesses, such as language issues and internet access limitations. Critics argue that such platforms, despite their utility, may unintentionally create an imbalance by favoring those with greater resources.

One additional concern when preserving digital evidence is the cost of storage, especially when dealing with large-scale or long-term needs. Decentralized storage options like IPFS, Filecoin, and Storj offer a different approach to traditional cloud storage providers, using distributed networks that not only reduce reliance on a single provider but also help keep costs down. For instance, Filecoin’s pay-as-you-go model can be much more budget-friendly for storing big datasets over time, while Storj uses a global network of nodes to cut costs by spreading the load. In addition, because these networks are decentralized, they’re less likely to be affected by sudden price hikes or monopolistic control, making them a fairer option for smaller teams that need to manage tight budgets without sacrificing security or data integrity.

Contents

Context
Framework
Technology
LearningsArchive

Law

We are thrilled to announce our participation in an extensive archiving initiative supporting the Atlantic Council’s Digital Forensic Research Lab (DFRLab). This significant project involves preserving 10,000 web pages as part of the research for the “Narrative Warfare” report, providing a robust resource for understanding the complex landscape of digital misinformation and information warfare.

The “Narrative Warfare” report, published by the DFRLab, delves into the intricacies of how narratives are weaponized in the digital age. The report uncovers how state and non-state actors manipulate narratives to influence public perception and destabilize societies.

The dataset of 10,000 web pages represents pro-Kremlin news publications from the 70 days prior to the ground invasion (Dec 16, 2021 to Feb 24, 2022). The team at the DFRLab identified and tracked five primary narratives pushed in support and ahead of the invasion of Ukraine.

Starling Lab, renowned for its pioneering work in data integrity and digital preservation, played a crucial role in this initiative. By leveraging advanced technologies and methodologies, Starling Lab ensured that the archived web pages were meticulously preserved and accessible for future research and analysis. This collaboration underscores the Lab’s commitment to safeguarding digital content and promoting transparency in the digital realm.

Methodology

Given the risk to the material once the report would be published, we preserved and durably stored the material on the request of the DFRLab. Director Andy Carvin’s team shared a spreadsheet containing, for each URL, original research metadata such as the associated broad narrative, the source pushing this narrative, and the article’s reach on social media.

Each page was crawled individually, meaning that all its content (media, code, style) were downloaded in a package permitting to “replay” it in the future and offline, independently of whether the original content would be deleted. Archives were prepared and packaged up using the cloud tools provided by WebRecorder, and all the content cryptographically signed by Starling Lab’s SSL certificate. For more details on these authenticated web archives, read our Dispatch on WACZ files.

The resulting dataset, totalling 400GB, can be accessed by members of the “Dokaz” initiative on request. While each page can be individually browsed as it was captured at the time, regardless of whether it went away, we are looking for support to produce an explorable, searchable interface to facilitate access for researchers.

Conclusion

In conjunction with the DFRLab’s “Dokaz” initiative, which focuses on preserving and verifying evidence in conflict zones, this archiving effort marks a significant step forward in the fight against digital misinformation. The “Dokaz” initiative, supported by Starling Lab, aims to provide reliable and verifiable digital evidence that can withstand scrutiny and support investigative journalism and research.

This collaborative effort between Starling Lab and DFRLab highlights the power of interdisciplinary partnerships in addressing the challenges posed by digital misinformation. By combining expertise in data integrity, digital forensics, and narrative analysis, this project not only enhances our understanding of narrative warfare but also strengthens our ability to counteract its harmful effects.

For more information, visit the Atlantic Council’s Narrative Warfare Report.