Authenticating Election Coverage in Hong Kong
New Standards for Photojournalism Election Coverage
A deep dive into the challenges and technical solutions, including C2PA manifests and cryptographic signatures, that enabled the South China Morning Post to publish authenticated, trustworthy photos.
Reading Time: 5min
Contents
Background
Starling Lab, anchored at Stanford University and the University of Southern California, is an applied research lab innovating with open-source tools, best practices, and case studies to securely capture, store, and verify digital content.
Given the rampant misinformation and disinformation landscape – and Hong Kong’s drop in press freedom rankings – SCMP wanted to explore a method of documenting and publishing events in a way that would be secured against attempts to manipulate public perception and opinion.
As Starling founding director Jonathan Dotan put it: “Would it be possible for the SCMP to deploy a team and use a system that would allow for there to be authenticated photographs? Where every time a photo was taken, there would be a way of preserving the time, the date, and the pixels so we had a record of the authentic, original photograph that was taken?”
SCMP Managing Editor Brian Rhoads remarked: “While we cannot predict controversy in these elections, having the blockchain technology available to authenticate/verify would help if and when controversy arises or simply be an exercise giving us clear provenance over the images.” He added: “In the long run, it would be a useful exercise to know we had proper, reliable verification tools available for other coverage as well.”
For this project Starling Lab and an interdisciplinary team of SCMP journalists, editors, and technologists worked together to capture, store, and verify the information and photographs collected during the Hong Kong Legislative Council and Hong Kong Chief Executive elections.
Contents
Context
Within less than six months Hong Kong saw two major political elections: the Legislative Council polls to elect lawmakers on December 19, 2021, and the Chief Executive Election to select the city’s leader on May 8, 2022. The polls were the first to be held after Beijing’s overhaul of Hong Kong’s electoral system, which led to more pro-Beijing representation in the legislature – and just one candidate running for chief executive.
Compared to previous years, election events were more muted and distrust was high, making the need for transparency and the ability to verify the accuracy of SCMP’s journalism more important than ever. Put simply, proving that we took news photos where and when we said we took them would help to combat misinformation.
Collaboration
Chief Technology Officer Benedict Lau and engineer Yurko Jaremko led Starling Lab’s technology team, overseeing the development and implementation of photo capture and authentication solutions. Starling Lab’s editorial team, led by Managing Director of Journalism Ann Grimes and Executive Editor Sophia Jones, worked with SCMP staff to implement and integrate new technologies into the newspaper’s workflow over the course of the two elections.
Working with Starling Lab to add cryptographic reliability and authenticity to the journalistic process was a natural next step, as SCMP had already begun experimenting with the potential of blockchain technology with print and online media. SCMP’s spin off company, Artifact Labs, creates and sells non-fungible tokens of the publication’s front pages.
We embarked on a collaboration with SCMP to apply authentication technology to the capture, storage, and verification of the unfolding of a sequence of events during these two Hong Kong Elections where parties of either side of an issue have a vested interest in altering public opinions.
Chow Chung-yan, SCMP’s Executive Editor, echoed that view. Speaking with the Starling Lab he said, “misinformation and disinformation has become increasingly a real problem for the newsroom.” The matter of photos being photoshopped now has given way to the use of artificial intelligence to “basically create an AI character that can actually do an interview with reporters” and be used by political parties to manipulate public opinion.
“Here in Hong Kong, I will say that we have experienced the deliberate use of misinformation,” he said, adding: “When people talk about misinformation or disinformation in Hong Kong some might think that this is the government, particularly when you have an undemocratic government, and you will think that the government is the most obvious culprit. In many cases, that’s the case but that’s a simple view, because the fact is, anyone who has the resources and is interested in manipulating the narrative is basically going to seize on this weapon and then use it to their advantage.”
Contents
Framework
The guiding principle at The Starling Lab is establishing “provenance” as the backbone of authenticity and integrity of digital content. To do so, the Lab follows a three-step framework – Capture, Store, Verify.
The Challenge
Before working with Starling Lab, the existing photo capture framework for SCMP consisted of:
- Capture: SCMP photographers in the field capture images with their Canon cameras
- Store: The photographers then download the pictures using SD card readers to their mobile phones, and caption the images before sending them to SCMP’s shared storage using File Transfer Protocol (FTP)
- Verify: Data is stored on Alibaba cloud, archived, and also published in SCMP articles on the world wide web. Content can be changed without a record of changes and edits.
Capture: When images are captured with SCMP’s usual method, limited information is packaged with the photos, making it difficult for anyone in the future to use the images to establish them as irrefutable records of the truth.
Storage: Without a reliable, secure way to store photo metadata, such as the device used, location, time and date, etc., there is little for courts, archivists, or journalists to point to during the validation process to guarantee that photos are a reliable part of a narrative or historical event.
Verification: For the images and other reporting data that is done for SCMP, there is no way to cryptographically validate the identity of who captured a photo and no method for checking or protecting the files to guarantee against modification, interception, or “man in the middle” attacks between source and storage.
Contents
The Prototype
In December of 2022, Journalists using Starling Lab technology were able to take photos and create a reliable and verifiable photographic record.
- Capture: Starling prototyped workflows with mobile apps and camera firmware to authenticate digital photos, and the photos’ metadata and cryptographic signatures, at the time the photos were taken using Canon EOS R3 and R5 cameras in conjunction with the HTC Exodus 1S smartphone.
- Store: Starling used advanced cryptography and decentralized networks that can securely distribute and store content over time, using IPFS and Filecoin to preserve copies, with OpenTimestamps and Likecoin used to preserve immutable records of photos taken.
- Verify: Starling experimented with C2PA and CAI tools to register photos, and created a custom front end element to embed into the SCMP stories, enabling experts to audit, or verify, the provenance and authenticity of the photos.
Contents
Technology
For this experiment with the Starling Framework, SCMP journalists were deployed to cover the two Hong Kong elections with a set of tools to securely capture, store, and verify the photographic evidence for the stories.
The capture methodology used was different for each of the two elections. For the first election, the field team used cameras that were tethered to phones that helped capture metadata using a WiFi connection to transmit photos to the device. During the second election, SD cards were used to capture and store the data from the pictures along with information from the cameras that could be used later in the verification of those photo’s authenticity.
Because of the differences in technology for these two elections, the process of ingesting and storing this information was done differently for each election.
Capture
The first step in this process is capturing the images in a way that is both trustworthy, and ensures that the authenticity of the photos can be validated later on. The capture phase for this evolved between the first and second election, due to learnings about the challenges encountered with the first election.
The most important aspect in the capture step is ensuring you are capturing a provably accurate image, that is accompanied by metadata that supports the authenticity of the image.
Canon EOS R3 and R5 Cameras were used to capture both the photos, and an array of metadata about the device and conditions in which the picture was taken.
HTC Exodus 1S is a mobile phone with an Android operating system and the Zion vault secure enclave for cryptographic key management. The Zion vault is an isolated subsystem on the phone that stores & protects a cryptographic private key that you can use to sign information, such as captured photos or transactions on a blockchain. This is done without risking exposure of your private key to the operating system which could make it vulnerable to attack.
Each of the elections used different workflows and sets of tools for capturing during the election. After the coverage of the first election, we switched from using WiFi to using SD cards, due to limitations of sending over a WiFi network, and a different workflow was used for each.
Election 1: Hong Kong Legislative Council
Canon Capture API (CCAPI) is a set of rules for communicating (also known as an API) that allows the Canon camera to connect to the HTC Exodus1S and transmit the photos and data over WiFi.
Starling Capture is an application developed by Numbers Protocol installed on the HTC Exodus1S phone that enables the phone to collect both the data for the photographs as well as data about the device that received and transmitted the information such as the GPS location and time, and more when the photo is sent using CCAPI to the phone.
Most importantly, the application is authenticated to Starling API via credentials provided by Starling Lab, and it utilizes the cryptographic features of the Exodus 1S to sign every photo coming through CCAPI.
Zion Vault is a secure enclave, an isolated, highly-secure subsystem, on the HTC Exodus 1S that stores private keys used to sign photos before they are sent to the Starling Integrity Pipeline for storage. This vault was set up before the phones were given to journalists, and the public key used (tied to a private key in the secure enclave) with Starling Capture is recorded ahead of time, so that identities of who sent photos and data can be verified later down the line.
Election 2: Hong Kong Chief Executive Election
ProofMode, an application that can work with media captured or uploaded to a phone to add phone-provided metadata to that photo, was used during the documentation of the second election to gather, bundle, and send photos and data.
Safetynet Attestation API is a protocol and service that ProofMode uses to check information about the Android device such as OS version, and type of device you are using.
Signal is an end-to-end encrypted messaging app that was set up on each phone before they were given to journalists and used to send ProofMode zip files. When apps are sent out over the end-to-end encrypted app, a private key and the phone number set up by Starling Labs signs and sends those files, and the public key can be used to verify these messages.
Signal Chat Bot was created using Signal’s API, and was a part of the Starling Integrity Pipeline during the second election only. It was used to monitor and move the packages of media & information from ProofMode on a journalist’s phone to the Starling Integrity Backend where photos, metadata, and signatures can be verified and stored.
In the second election, hashes of photos and metadata were registered to the Bitcoin blockchain using the OpenTimestamps network. The timely inclusion of these hashes onto the Bitcoin blockchain provides strong proof of the existence of these assets at a particular block height, or location within the blockchain, which can be correlated with a point in time.
In the second election, photos were bundled with metadata as encrypted archives, and their hashes were registered onto the LikeCoin blockchain using the ISCN specification for digital content registration. The registrations can be viewed on Likecoin.
Store
Starling Integrity Pipeline
The pipeline is a process and set of tools used to transport photos and data to the storage server for photographs from both elections. The Integrity pipeline includes several pieces:
Starling Integrity Preprocessor and API are set up and maintained by Starling Lab to process the photos and metadata that has been signed and encrypted. These take the photos and metadata, then validate signatures, and pass the data to both Starling servers and web3.storage.
The Starling Integrity Backend is maintained by Starling Lab where the images are initially stored (before they are also processed into distributed storage), and where the C2PA Manifest is created and signed by Starling Lab, which makes it possible to track incremental edits and changes and tie it back to the original photo with a verified source.
Once original photos are bundled with a C2PA Manifest, these are placed in a shared FTP directory or Dropbox folder from which SCMP editors were able to access, make trackable edits to photos with Photoshop (which uses the C2PA standards to record edits), and upload new versions of those images with updated manifests.
web3.storage
This is a storage tool used by Starling Integrity Pipeline that takes the image from the Starling Integrity Pipeline, packages it appropriately for storage on distributed systems, then adds and ensures the persistence of these files on both the IPFS and Filecoin distributed networks.
A distributed cold storage solution used as a part of web3.storage, Filecoin is a token-incentivized storage in a distributed network of providers who are required to regularly prove the integrity and availability of data. When you store with web3.storage, the platform will make deals with storage providers on the Filecoin network, paying them FIL (the Filecoin cryptocurrency) over the lifetime of a storage deal as the Filecoin blockchain runs computational proofs to ensure that data is being stored.
IPFS is a peer-to-peer distributed storage network that allows anyone who wants to maintain a node to add and provide data to a network that is independent of the client-server model and large corporate storage providers. When content is published on IPFS, a node will create a unique content identifier (CID), which functions as a unique digital fingerprint for and pointer to the content. The IPFS CID for data (such as the bundles of photos and metadata created with ProofMode) points to a tamper- and censor-proof copy of that data. An IPFS node is maintained by web3.storage and CIDs are pinned and maintained on the IPFS network.
As a part of this project, Starling Lab worked with Number Protocol to mint NFTs on the Flow blockchain. With this project, photos and captions about those photos can be minted and sold as a kind of ‘digital news clipping’ with value and verifiable ownership. This is an early experiment to the broader ARTIFACTs initiative project at SCMP.
Verify
CAI Toolkit is a set of tools created by the Content Authenticity Initiative is an organization and community made up of companies, NGOs, and academic organizations. This toolkit makes it possible to create C2PA manifests with signatures of contributors, edits, and other changes to media files, embed information from these manifests in websites and applications, as well as provides the specifications used by Photoshop Content Credential to track changes made to photos.
CAI C2PA Command Line Tool is used to generate and read cryptographically signed manifests compliant to the C2PA open technical standard. At the time of the elections, Starling Lab used a pre-release version of the tool from Adobe both in the Starling Integrity Pipeline, and it is also used as a part of Photoshop Content Credentials to record editing steps.
The Photoshop editing tool by Adobe includes a feature called Content Credentials that, using the C2PA CLI and the Rust SDK reads and adds to the -When you open a C2PA file, it allows you to track the edits made and repackage the edited photos with a manifest, that is then shared back with Starling Labs, that has a record on the manifest you can use other tools to see.
This website can be used to preview and see metadata created with a C2PA manifest. You can visually inspect changes to an image, and see data from the manifest about the who produced or created versions of an image, a signature timestamp, how changes were made, what edits were made with Photoshop, and what assets were used.
In order to preview the images and manifests of the signature of the individual making changes, edits, and other changes, the JavaScript Software Development kit was integrated into the SCMP website and allows the user to see information included on the C2PA Manifest.
With this SDK Starling Lab helped SCMP to develop an `info icon` that not only previews publishing and editing data like Verify, but also additional data that was captured in the manifest, such as data about the location, who produced the photo, which app the image was captured with, the date and time, decentralized storage information, and data about the NFT produced from that image and caption.
Contents
Learnings
Due to the back to back elections, events moved quickly and a lot of ideas for improvement were generated in short order. In both elections, Starling Lab worked closely with the photo and engineering teams at SCMP to effectively deploy these technologies. Here is what we learned:
Metadata Collection
Due to GPS sensors on the phones not always having an up-to-date location (especially indoors where GPS signals are weak) some photos ended up missing location information in the metadata collected. Although lack of precise location is problematic, the GPS timestamp data included with the metadata is a valuable record indicating when location data was last acquired.
Signature Challenges
The HTC Exodus 1S’s hardware-backed signer does not make it possible to sign photos as a background process (i.e. in the background, as photos are being taken) which means photojournalists had to manually type in a PIN to unlock the signer after each photo is taken. As journalists take multiple continuous shots at a time, it’s important that any signage can hash and sign many full-size photos without manual PIN input.
To solve this, Starling Capture created an implementation that generates a software session key that can quickly sign camera photos and their metadata as photos are taken. This session key is itself later signed by the Exodus 1S hardware keys and persists on the device for the duration of capture sessions. An ideal solution would be for a hardware signer to support touch-free fast signage natively.
Several issues also came up with the cryptographic signature implementations in the two experiments. These include proper recording of all cryptographic material, such as:
- Public keys per device
- Any intermediary keys and their attestations (e.g. software session keys and their hardware signatures in the trust chain)
- Recording of algorithms used to generate signatures
- Reproducible messages used to generate the hashes for signage
Missing pieces has led to some of the metadata lacking proper cryptographic attestations, and arising issues were addressed throughout the project. The learnings from here is to implement logic at the beginning of the pipeline that proactively validates the full trust chain of signatures, and to adopt standards such as RFC 8785: JSON Canonicalization Scheme (JCS) to ensure signed content produces reproducible hashes.
Logistical Challenges
In the course of the project, we encountered several issues for the first time:
A general need for the capture process is equipment that simultaneously captures location data the moment that photos are taken. Though there are some cameras that can do this, not all journalists have those models of camera, so cell phones were used in order to enable all journalists to capture data with their photos. Collecting the exact time and location of photograph capture can potentially be refutable when there is any delay between capture and timestamp (such as problems with WiFi connectivity, or a lapse in time when an SD card is transferred), and having journalists having to take extra steps to transfer photos to the phone while on-site was a challenge during fast-moving news coverage.
In the first election, a local WiFi connection between the Canon camera and the HTC phone was required for journalists to connect their cameras over the CCAPI to the phones collecting and signing the data & photos with Starling Capture, and the WiFi connection was unstable, meaning that not only was it difficult to transit photos, but that there was some loss of photo records if too many were captured simultaneously. This is especially problematic when in crowded areas with a lot of WiFi noise, such as indoor spaces where an election is held. This led to the use of SD for the second election, where SD cards can be manually moved from the camera to the phone to transfer photos and metadata, eliminating the reliance on an always-on WiFi connection via CCAPI.
Application Improvements
In this experiment, Starling Lab worked with the CAI JavaScript SDK to develop custom previews or ‘info icons’ that display custom C2PA manifest data such as the Filecoin Piece Content Identifier (CID) and IPFS CID, which are identifiers of storage information about the photos. At the time, the pre-release SDK (which is now publicly released) was being actively developed by Adobe, and the SCMP engineering team worked through several technical hurdles to integrate the frontend onto the production news site. The SDK is now publicly released.
Data Storage Considerations
There were also logistical road bumps uncovered as the team tried to work quickly and publish articles to the SCMP as the stories unfolded. The original intention was to have images with proofs of storage embedded at the time of publication, however because of factors like the amount of time it takes to make a deal to store on the Filecoin network (this can take over a day), images without signed metadata (Complete C2PA Manifests) were first published, then they were replaced with images that included proofs of storage (CIDs) a few days after the original article publication.
In addition, although there was a pipeline set up to automatically capture the original photos from journalists with Starling Capture or ProofMode + Signal using an API, the process of sharing edited photos between SCMP and the Starling Lab team was rather manual, involving editors having to manually upload new versions of edited photos, and the Starling Lab team needing to check a shared FTP or Dropbox directory for updated versions of images, so that Starling Lab can prepare photos and data for Filecoin archival and additional signatures.
Though the data transfer process is inconvenient, it isn’t by any means a major consideration. It was clear from the beginning that various tools lack support for attesting to content changes and the Starling Lab team will have to manually retrieve assets to inject attestations. A future solution to this would be for more tools to natively support signed attestations as they are used to edit photos, and that versions of photos can be synced over shared cloud storage.
Authenticity in the Field
This project demonstrated that journalists could successfully deploy sophisticated cryptographic and blockchain-based authenticity tools on a significant live news story under real-world time and operational pressures. The authenticated photographs of this experiment serve as a verifiable and important historical record preserving the time, date, and pixels of SCMP’s photojournalism of two significant elections.
Contents
Archive
Publications / News Articles
Photo published is of the 40 lawmakers who make up the Election Committee; no location data
Published CAI content credentials here
Photos published include photo of Candidate Regina Ip of the New People’s Party calling for votes in Aberdeen and moderate candidate Jason Poon out in Kornhill seeking support.
Published CAI content credentials here
Published CAI content credentials here
Published photo shows Ng Chau-pei (left) and Edward Leung who defeated Jason Poon in the Hong Kong Island East constituency
Published CAI content credentials here
Published CAI content credentials here
Published photo shows Tik Chi-yuen (second from left) of the middle-of-the-road party Third Side who won the seat for the social welfare functional constituency.
Published CAI content credentials here
Published photo shows two women walking out of a polling center at Shek Wu Tong in Kam Tin.
Published CAI content credentials here
Mapping of Photojournalists at the Two Elections
Spreadsheet of photographs & GPS data
Map: https://www.google.com/maps/d/edit?mid=17-NQuxlFtazY-zjBnvc8xAt66CPcegY&usp=sharing
Participating Photographers & Published Photos
- May Tse – no Starling Framework photographs published in SCMP
- Felix Wong – Article 3160469 (1 image, LegCo)
- Nora Tam – Article 316033 (2 images, LegCo), Article 3176982 (3 images, Executive)
- K.Y. Cheng (Cheng Kok Yin) – Article 3160321 (2 images, LegCo)
Dickson Lee – Article 3160475 (1 image, LegCo), Article 3160466 (1 image, LegCo)
