The Proof’s in Your Pocket: Proofmode inside Signal
The Proof’s In Your Pocket
What could authentication data look like in your favorite messaging app? We built an easy-to-deploy secure camera for crowdsourcing documentation of Ukrainian schools by integrating ProofMode with the popular Signal messaging app.
Reading Time: 5min
Contents
Background
In response to the large-scale invasion of Ukraine by Russia, the collective response of the Lab was to start a broad project cutting across all of our three research areas: Journalism, Archiving, and Law. Under the project called Dokaz (“Доказ”, Ukrainian for proof), a loose coalition of organizations have shared material and ideas related to the support of Ukraine against aggression. In the context of this new large-scale conflict, documentation has been a cornerstone concern of our research into the creation of stronger digital material:
- The Lab’s law program led the submission of originally-collected field and remote evidence to the International Criminal Court;
- On the journalism side, we have supported the first demonstration of at-source cryptographic authentication, directly in-camera, for Reuters photojournalists;
- And we have prototyped the integration of metadata-rich, authenticated photography in Signal Messenger, most notably for the use case of crowdsourced citizen documentation.
Quite a few of these projects were aimed at professionals and specific uses of information. However, as the specter of deepfakes gives way to the arrival of mainstream photorealistic generative AI, we are faced with even faster-moving challenges that require us to consider how new camera hardware, software, standards, and user experience (UX) can help establish what is an accurate depiction of the real world. This challenge concerns everyone, extending to our day-to-day consumption of media. Simply put, in the near future, just seeing digital photos may not be a reliable means of believing them.
In order to make digital proof more accessible, we set out to incorporate a powerful authenticity tool called ProofMode with a popular communications tool likely already in your pocket – Signal Messenger.
Our collective thanks go to the project partners:
- The team at Guardian Project (makers of ProofMode), who work tirelessly to bring their ideas to the world from the ground up, driven by the “right” approach and choices, with free and open software.
- Our collaborators and Dokaz <link> members Hala Systems, who provided the operational framework ensuring the support of the photographers, notably by red-teaming the risk assessment.
- Our local photographers in Kharkiv who once again went out to document their city.
- The Forté Group, who provided ad-hoc engineering resources for the integration delivery.
- The Signal team, who kindly heard our pitch for adopting this approach, and makes their code open-source for others to build on.
- And finally, the entire Starling Lab team, involving notably Basile Simon on project direction, engineering management and in-country deployment; Alisha Seam on technical advice to the prototype; and Yurko Jaremko on operating the preservation pipeline.
Contents
Context
The first investigations Starling Lab supported in Ukraine were related to attacks against schools in the city of Kharkiv. With our grounding in higher education, we found the words of residents and witnesses especially disturbing:
“Where will the kids go to learn? Can they actually bomb these places?”
To address potential disinformation by Russia, the work of our legal team was to investigate and confirm that facilities had not been co-opted by Ukrainian military personnel – an after-the-fact task fraught with challenges.
In support of the Safe Schools Declaration, an initiative joined by over 100 countries, we sought to empower local communities to lead or crowdsource their own preventative documentation of schools. This entailed frequently visiting campus surroundings in order to document the absence of armed forces and thus the schools remain protected under international humanitarian law.
As a pilot, we organized two weeks of regular “rounds” at several schools in Kharkiv by designated photographers, who were tasked to document the surroundings and insides of the schools. This approach followed recommendations of the Declaration to “make every effort at a national level to collect reliable relevant data on attacks on educational facilities, on the victims of attacks, and on military use of schools and universities during armed conflict,” as well as its numerous guidelines, which in short recommend that armed forces of a country at war never use schools or educational facilities.
However, broadening the pool of documenters poses questions regarding how easily they can adopt technologies with robust authentication features. Our research questions turned to tool ease of use and accessibility with minimal training – important as we seek to understand the burdens placed on both the viewer of authenticated media and the creator.
The pilot deployment of this prototype was trialed in January 2023 in Kharkiv, where Starling engaged with two local photographers. They were able to go in the field only with their Android smartphones. Our prototype app was side-loaded into the phone through a custom APK. This setup permitted them to both preserve their message history and contacts, as well as to file their photographs with the Starling Lab.
The field work was completed in time to present the Lab’s findings and methodologies in a joint submission with Hala to the United Nations Special Rapporteur on the right to education, as well as in thematic presentations at the World Economic Forum 2023.
Contents
Framework
The Challenge
This project’s aims were two-fold:
- Can we devise a lightweight solution permitting the secure capture and transport of authenticated photos and videos?
- What could a user interface presenting contextual and authentication information look like, inside a social messaging application? What data points, if any, further a person’s trust in what they’re seeing?
These questions demonstrate our commitment to approach information systems with the perspective of Authenticity-by-Design.
Contents
The Prototype
ProofMode is a software and smartphone app for authenticated media capture, developed by the Guardian Project, Okthanks and WITNESS. It was already used in several of our Lab’s projects when they released it as a software library for integration into any app. This spurred us to build a proof-of-concept combining its technology with a fork of the code for Signal Messenger. Signal is a popular global messaging service known for offering end-to-end encryption and open source code, and has also been used in Lab projects for secure communications and media transport.
At its core, ProofMode reverses a common approach to combating misinformation and deepfakes. Rather than identifying and debunking the authenticity of fake content, ProofMode promotes trust in what is genuine by providing a means to strongly authenticate the multimedia it generates. It accomplishes this by backing up numerous records of metadata, which come in as corroborating pieces of information supporting media captured in the field. In short, ProofMode helps users to trust the real rather than questioning the fake.
By incorporating the recently-released libProofMode library, we were able to introduce a streamlined experience natively within Signal to create verifiable, provenance-laden media using the app’s camera function.
The resulting app enabled us to deliver strongly-authenticated photographs with industry-leading privacy for both sender and recipient. As a system, multimedia content could be authenticated at the point of capture on a smartphone, then later verified by a recipient. It utilizes enhanced sensor-driven metadata, hardware fingerprinting, cryptographic signing, and third-party notaries to enable a pseudonymous, decentralized approach to the need for chain-of-custody and “proof” by both activists and everyday people alike.
Contents
Technology
Capture
Strongly-authenticated photographs are captured directly in the custom Signal app. Tapping the in-app camera icon opens the camera, and the resulting captured media is co-located with surrounding metadata through the integration of ProofMode.
Ahead of capture and upon loading the app on the phone, a digital identity was created through the generation of an OpenPGP key pair unique to the app / device combination. This key pair is used to sign the ProofMode data files, and this signature in turn permits later attribution of a ProofMode bundle to a person who would have custody of an OpenPGP key pair.
By default, ProofMode collects the following surrounding metadata at the moment of capture: hash of the capture media, information about the phone / device used, information about the device connectivity including nearest cell tower environment, device IP address, GPS coordinates and accuracy thereof, timestamp of the geolocation, and a Google SafetyCheck signature of the media attesting the integrity of the Android environment which ran the app.
The media hash is cryptographically registered on OpenTimestamps (a process ProofMode also calls “notarization”). The resulting anchoring on the ledger permits the demonstration that “this media existed at this time.”
After capture, the photograph and its associated ProofMode bundle are shared with a contact or group on Signal, following the app’s native UI and workflow. The resulting bundle of data is, at the moment of consisting of the photograph and its surrounding metadata, including the above timestamping receipt,is hashed and cryptographically signed
Each media shared this way is followed by an automated MobileCoin transaction, with a memo field containing the first 16 digits of the hash value of the ProofMode bundle (called “proof hash”). This permits the recipient to confirm that the ZIP shared with them is the one the sender meant to send, by comparing the hash of the data received with the hash registered on MobileCoin by the sender. MobileCoin is a micro-payments system and cryptographic ledger natively available in Signal Messenger.
For the specific purposes of this project and prototype, these features can complete Signal’s feature set: to not only encrypt, but also authenticate self-generated assets using cryptographic hashes and signatures. By embedding these road-tested tools natively in the app, they can protect and notarize photos at their source so they have a better chance of being trusted as they move through chaotic information environments.
Store
This project focused on the Capture and Verify phases, without requiring integration of a long-term preservation strategy for the multimedia assets. Beyond proof of concept and prototype however, routine storage considerations were addressed:
The files shared by the in-country team with Starling were automatically validated upon receiving them by our Signal signald client. Custody of the files was asserted and matched to the photographer’s previously-provisioned JSON Web Token (JWT). After these authenticated bundles were validated, they were preserved and encrypted at rest in Starling’s storage pools.
Non-critical metadata (hash values of the media and bundles) was registered on cryptographic ledgers, acting as immutable third-party record holders and timestamp anchors. This included the Numbers blockchain, Avalanche, and the International Standard Content Number through LikeCoin.
Verify
We designed a bespoke user interface inside Signal’s conversation view to demonstrate what surfacing contextual metadata about photos shared in-app could look like for all users. This data comes from Proofmode’s own surrounding environment metadata snapshots, as well as from several third-party record holders (called notaries by ProofMode).
A key element of the demonstration of non-tampering of the files is the aforementioned distribution of integrity data on third-party distributed (and immutable) ledgers, thus permitting verification of hash values and signatures at a later stage by means of comparing the present file hash with expected values registered with third parties.
While our use case involved communicating with an automated Signal client, we designed a rich verification and inspection UI into Signal Messenger. Both sender and receiver are presented with metadata in their normal conversation thread. Further metadata, including hashes and cryptographic signatures are displayed in a separate “See more” screen.
This visual layer of verification, and the inclusion of the micro-transaction on MobileCoin, provides an accessible, very present tool presenting background information about the shared media.
Contents
Learnings
Ease of use
Leveraging the widely tested and familiar user experience of Signal resulted in a prototype that was intuitive for users. The response was dramatic. Investigators in the field, lawyers, and in particular, leaders at the Department of State, indicated to us that direct Signal integration could be transformative for non-governmental organizations and citizen journalists.
Nathan Freitas, Director of Guardian Project and the ProofMode team, has over twenty years of experience providing digital security tools and training for human rights defenders around the world. He said: “Activists and journalists are already burdened with intense physical and digital threats through their work. Asking them to learn a whole new app often can be too much, or put them at more risk. Integrating provenance and authentication features into Signal means they get more benefit from an audited, vetted app most of them already have and rely upon everyday. Less is more!”
A best practice?
Starling Lab made a submission to officials at the UN’s Human Rights Council which outlined the work enabled by this prototype. The Special Rapporteur on the Right to Education<internal link> noted to the Council that our efforts, along with collaborator Hala Systems, was an emerging good practice for documenting evidence.
At the World Economic Forum in January 2023, Starling presented the project as a means to illustrate the need for the continuation of this documentation effort – itself made possible partly by the easy deployment of a top-tier documentation tool.
Cost of maintenance
There are rolling costs to keeping the fork up-to-date with Signal changes and potentially ProofMode itself. Lagging behind means being shut out of the Signal main network. These are important factors to consider when starting an initiative beyond the prototype stage.
Contents
Archive
Materials related to this case study are under review and kept private for now.