In late summer of 2014, the Ukrainian Armed Forces battled pro-Russian separatists for control of the town of Ilovaisk, in the border region of Donetsk, eastern Ukraine. By September, the separatists had won a resounding victory, routing the retreating Ukrainian army. Even as the battle was ongoing, however, allegations swirled that regular units of the Russian army had joined the battle on the side of the separatists, tipping the balance decisively in their favour.

Russia denied the charges. But, in what became a watershed moment for open source investigation, communities of researchers, reporters, and citizen journalists gathered substantial and compelling open source evidence for the presence of the Russian military in the region.

The turret of a destroyed T-72B3 tank in eastern Ukraine. In 2014, this model of T72 tank was operated exclusively by the Russian military. (Askai707/Bellingcat)

A convoy of vehicles is visible on a Ukrainian road, approaching the border with Russia, on 3 Sep 2014⁠—immediately following the defeat of the Ukrainian Armed Forces. (Google Earth)

Forensic Architecture was commissioned by the European Human Rights Advocacy Centre (EHRAC) and the Ukrainian Legal Advisory Group (ULAG) to gather together the available open source evidence for the presence of the Russian military in eastern Ukraine, in support of a legal claim against the Russian and Ukrainian states, currently being heard by the European Court of Human Rights (ECtHR).

The case concerns the capture and detention of Ukrainian volunteer combatants during the battle of Ilovaisk, by the Russian military and pro-Russian separatists. In total, we brought together evidence of almost 300 Russian military vehicles in the regions around Ilovaisk and the nearby town of Luhansk.

The open source evidence we gathered together is presented within an interactive cartographic platform. The platform allows users to move forward and backward throughout the period of the battle, exploring over 150 incidents, supported by thousands of sources. This platform is an instance of timemap, an open source codebase we developed to facilitate the creation of interactive cartographic platforms, which we have used across multiple projects to reveal connections in time and space between incidents and events.

The Ilovaisk platform also contains our own verification, geolocation, and short video investigations.

The Battle of Ilovaisk platform gathers together the open source evidence for Russia's military presence in eastern Ukraine in August 2014. (Forensic Architecture)

The platform is not only the first example of the use of interactivity in an evidence submission before the ECtHR. It is also the first example of a submission which is based, in part, on machine learning techniques.

Open source investigations, such as that which uncovered and assembled convincing evidence for the presence of Russian military units in eastern Ukraine, invariably run into a variation of the same problem of labour, time, and resources: open source researchers are required to find, and watch, many hours of video footage, in order to find evidentiary material. Much of the material that researchers end up reviewing is not relevant, or evidentiary.

Forensic Architecture set out to examine whether some of that process could be automated, saving valuable time for researchers and investigators. In gathering open source material for this project, we deployed pre-trained machine learning classifiers through a piece of bespoke software, to search YouTube according to a set of terms and a date range, download, and analyse those videos frame-by-frame.

In this case, we used classifiers trained to recognise military vehicles, and tanks. We also experimented with training machine learning classifiers on ‘synthetic data’, or photorealistic digital images. You can read more about our experiments in synthetic data here, and more about our use of synthetic data in our Triple-Chaser project here.

Forensic Architecture used machine learning to search through videos for military vehicles. Custom software, designed by the agency, presents the results of that search in an interactive, navigable format. (Forensic Architecture)

Our thanks to all the open source investigators, researchers, and reporters whose findings have contributed to this platform.