Pennsylvania’s Facial Recognition System, with 25 Million more Faces than the Commonwealth has Residents

It’s been a while since I posted to the Techlaration, and I’m cranking it back up to document the protracted installation of a massive surveillance infrastructure in Philly between now and the Democratic National Convention next summer.

By Kenneth Lipp

For three years Pennsylvania State Police and other law enforcement agencies in the Commonwealth have been using a facial recognition software that generates 3-D images of suspects and searches them against a database, through the Pennsylvania Justice Network (JNET). Updated in September and again in December of 2012, the JNET Facial Recognition System (JFRS) began in 2006, with a database of about 3 1/2 million images from the Commonwealth Photo Imaging Network (CPIN) and other state and local law enforcement sources that was made available to more than 39,000 municipal, county, state and federal justice professionals.

A 2013 announcement of expansion highlighted an apparently very serious shortfall in available faces:

Pennsylvania has over 3.5 million criminal booking photographs for comparison in JFRS. Unfortunately, this represents only a fraction of the approximately 13 million residents in the state.

In May of 2013 JNET filled its face-gap with 35 million images from the Pennsylvania Department of Transportation’s (PennDOT) own facial recognition database and added its MorphoTrust engine to the JFRS search. The current image universe available to everyone with access to the state justice network – representing about 500 agencies – now has a population of 25 million greater than that of the Commonwealth.

The JFRS combines the three largest proprietary facial recognition engines: MorphoTrust, NEC’s NeoFace, and Cognitec’s FaceVAC. What that means is that three different algorithms or sets of algorithms are used to attempt to match a picture submitted with pictures in the database. There are only 3 basic algorithms in wide industry use, and a lot of the differences between software are in at which stage in the process a given algorithm is applied, and what kind of preparations are done with the raw data. It’s important to keep in mind that computers don’t see color and features, they just decode 1’s and 0’s in blocks and sequences.

The Animetrics ForensicaGPS platform operated by the JFRS attempts to solve one of the persistent problems of automated image recognition – subjects that won’t look straight at the camera. Facial recognition technology is powerful because computers are able to use a strictly defined metric, like the relative pixel distance between facial features or identifying marks like scars, to process a huge collection the same way every time.

But a partial profile and a straight-on shot of the same face look to a computer to have concrete differences that are in fact relative to the viewer. ForensicaGPS uses a preparation algorithm to generate a 3-D view from a 2-D picture to fill in gaps in perspective. Results are mixed, in my testing. You can try a basic version for yourself. Below is “Weedney Houston” before and after 3-D treatment.

And here’s former Attorney General Eric Holder, from a picture taken at a conference in Philly in 2013. Take note that this is not identical to ForensicaGPS, which allows for much more fine-tuning than the online “ID-Ready” app.

I had less luck with this side-view of Mayor Michael Nutter

JNET stresses to users that JFRS is for “investigative purposes” only and does not make positive identification, but that’s not really reassuring. There doesn’t seem to be any threshold for suspicion before one of the almost 40,000 people with access to the system can submit an image for search, allowing for easy fishing expeditions. It has the potential to be used, and to some extent is already used, like Automatic License Plate Reader (ALPR) technology, with the digital signatures from facial scans taken by passing camera vehicles checked against a “hotlist” of persons of interest, as scanned plate numbers are checked against hotlists of stolen vehicles, and scofflaws.

This is not conspiracy theory, anyone who has a Facebook account, for example, willingly submits her own photos and those of others for computational analysis and commodification.

In the ALPR industry it is all about who has the biggest database – read the embedded documents for language which treats collections of tagged image data like business assets. The term “enterprise solution” is s refrain throughout JFRS documentation. That’s especially the case in the first of the two documents embedded below, which is an html copy saved by Google of a Pennsylvania law enforcement working group Powerpoint presentation, that is no longer online.

JNET is the first law enforcement application of Animetrics ForensicaGPS in the US, where its niche has been the commercial market and retail applications.

The video below provides a sort of demonstration of the software, muting is recommended, the music is annoying, then creepy.

[youtube https://www.youtube.com/watch?v=hHp-FDfDQCw]

The Declaration has a Harris Hailstorm of public records requests in the works.