DeepPlate is our new AI-based license plate reading service for investigative use. It currently supports 8 countries and is freely accessible for users with a valid Amped FIVE SMS plan through the Amped Support Portal.
Dear Amped friends, we’re super excited to share some great news! Starting today, we’re launching DeepPlate, an online system for AI-powered reading of severely degraded license plates (for investigative purposes only). Keep reading to find out more!
License Plates: an Evergreen Challenge
It’s no secret that reading a poor-quality license plate is one of the most common challenges for forensic video analysts and investigators. Despite the constant increase in the spatial and time nominal resolution of surveillance cameras, it is still very common to deal with heavily compressed or poor-resolution license plates that are hard to read.
Amped FIVE’s Perspective Super Resolution is often a lifesaver when confronted with a low-resolution license plate. Frame Averaging can also significantly improve the quality of a decently sized but very noisy plate. One thing FIVE won’t do, though, is read the license plate characters for you. DeepPlate will allow users to evaluate and compare their results with an “impartial reading” of the plate.
What Is DeepPlate
For this reason, we’ve developed DeepPlate, a deep-learning-based algorithm for “decyphering” license plates affected by the common issues introduced by surveillance systems: perspective distortion, poor resolution, optical and motion blurring, and compression noise.
How did we do it? For several countries/states, we have trained a dedicated neural network with millions of synthetically generated and distorted license plates of that country/state. Generating synthetic license plates is fairly easy once you know the font, the spacing, and the allowed character structure (customized license plates are not considered at this stage). Thanks to this approach, we didn’t have to scrape license plates from the web: no data provenance nor privacy issues in our model!
If you’re curious about our system technicalities, we’ve published them in a scientific paper, together with the experimental validation on Italian license plates. The model has improved since then, and more countries have been added, but the general idea is still the same.
Currently Supported Countries
The currently supported countries are: France, Germany, Italy, Netherlands, Spain, Sweden, United Kingdom, and United States of America. For countries with multiple license plate formats, we show the supported formats during the license plate upload phase.
Using AI for Forensics and Investigations
Before we continue showing how DeepPlate works and its recommended workflow, let us briefly discuss the use of AI.
We acknowledge that, depending on the local law and jurisdiction, the use of AI-based tools for evidentiary purposes may be admissible. However, if you follow our blog and events, you know Amped is very cautious about using AI in forensic applications.
We maintain that using AI for image enhancement, such as improving the quality of license plates or facial images, is not currently reliable for legal evidence. The reason is that the explainability of the AI’s results is limited. Additionally, there is a risk that the outcomes may be biased by the data used when training the AI model.
Nevertheless, we believe that AI can be helpful for image analysis (e.g., reading a license plate or detecting the presence of an object/subject). This is especially true when it is done for investigative purposes.
For the following reasons:
- DeepPlate will not provide you with an enhanced image of the license plate, only a proposed reading of it;
- We make it very clear, both in the service portal and in the generated report, that results are subject to errors and should not be used for evidentiary purposes;
- We warn the users that seeing the output of DeepPlate may induce bias in them. It is recommended that a user first processes the image using standard methods. After this initial step, if tasked with reading a license plate, they should attempt to do so before proceeding to use DeepPlate for further analysis;
- DeepPlate implements a bias mitigation approach whereby you’ll only see results after clicking on a “Show Results” button. They are always given on the second page of the PDF report.
DeepPlate Access Conditions and Data Storage
DeepPlate is currently offered at no extra cost to Amped FIVE users who have an active subscription license or an SMS plan. To qualify, users must be part of an institution located in one of the supported countries listed above. The service is accessible from the Amped Support Portal.
Each institution meeting the requirements above will be gifted 50 DeepPlate uses/month for every Amped FIVE license (and for each seat, in case of multi-seat licenses). Please be aware that this usage cap is shared among all users associated with that license. We don’t impose any constraints on how usages are split among users.
Data Storage
We will only hold your data on our servers for the short time needed to process them. Everything will be deleted after processing. We will not retain your images nor use them to train our AI model; we’ll just run DeepPlate on them and delete them. Can’t do better than that 😉
How To Use DeepPlate
Accessing the Service
If you work for an agency/entity located in a DeepPlate-supported country, you’ll automatically find an additional tab called “DeepPlate” on your Support page.
When you click on DeepPlate, our system will check that you have an active Amped FIVE SMS plan. If you don’t, you will see a message about this missing requirement. Otherwise, the first time you access DeepPlate, you’ll be asked to agree to the Terms and Conditions of the service. Please read them carefully, as they contain important information on the allowed use of DeepPlate.
You’ll then be brought to the upload page to see the remaining service usage for the current month. If a user is associated with multiple eligible licenses, they can choose which license cap to consume for their attempt.
Please notice that, as indicated on the page, DeepPlate currently supports single instance use only, so don’t use it within multiple browser tabs.
Uploading Your Image
When you select a country, the supported license plate templates list will appear below for all countries except the United States. This is shown to inform you about the supported formats; you don’t have to select one of them. Just be aware that results may be unreliable if the uploaded license plate does not adhere to any of the listed formats.
In the case of the United States, the user has the option to select a state. If you select a state, you’ll also be asked to choose the license plate format you think the uploaded image adheres to. If you’re unsure about the state or the license plate format, leave the state selector to “Unknown”. When you specify a license plate format, the neural network will know the kind of characters (letters vs numbers) expected in each position. If you leave the selection to “Unknown”, no such rule will be applied, possibly reducing the chances of correct reading or the confidence levels associated with the results.
Once you’re done, just click on the Upload button.
Ok, let’s go more practical from now on and use the following image of an Italian license plate as a test. This image is from a real recording and has not been subjected to any artificial degradation. The presence of a double timestamp results from a mishap during footage acquisition, which, unfortunately, is a realistic situation.
Selecting the License Plate
After uploading, you’ll be automatically brought to the license plate selection page. Just follow the instructions on the screen: select the four vertices of the license plate using a right-click, starting from the top-left corner and proceeding clockwise. You can zoom over the image scrolling with the mouse. Make sure to choose pixels according to (one of) the template(s) displayed above the image, excluding any background pixels and including all required plate parts.
Once the selection is done, you can reset it by right-clicking anywhere on the image or using the Clear selection button. There’s also a Select all button for selecting the entire image if you’ve uploaded a picture of a license plate that has already been cropped and rectified. Once you’re happy with the selection, click on Continue and wait for the results page to load.
Interpreting results
The Results page implements a bias mitigation technique by which results are not shown immediately. You’re first reminded with a red box that you should treat DeepPlate output as a second-opinion tool.
Once you click on the Show results button, you’ll be presented with the results, which come in the form of two tables.
The first table shows, for each character position, a list of possible characters sorted by confidence level. The confidence level only tells you how confident the neural network is about its conclusion. A high confidence level does not mean you can be sure the character is correct. The neural network may be 100% confident about a character and still be wrong about it.
The second table is derived from the first one. It shows a list of 60 possible license plates sorted by the aggregated confidence of characters. This is simply computed by multiplying together the individual confidence score of each character on the license plate.
For example, the aggregated confidence for BT716MY (0.2%) is given by 0.158 (the confidence level for “B” being the first license plate’s character), multiplied by 0.93 (the confidence level for “T” being the second license plate’s character), multiplied by 0.338 (the confidence level for “7” being the second license plate’s character), and so on. Therefore, 0.158 x 0.93 x 0.338 x 0.809 x 0.337 x 0.911 x 0.151 = 0.0018, which is rounded to 0.002, which means 0.2%.
As the example above shows, even when the network’s output has a reasonably high confidence level for some characters, the first composed license plate may have a very low aggregated confidence level (0.2% in our case). This outcome is the result of how confidence scores are combined. When you multiply these scores together, even a single low confidence score can significantly reduce the aggregated confidence level for the license plate.
Exporting the PDF Report
By the time you see the results, the data deletion process has already begun on our servers. The results page is stored locally on your browser cache. You can use the Generate PDF button to export results to a PDF file for later use. The PDF will basically show the same information and data as the web page. Results are always presented on the second page as a bias mitigation technique.
Should I Enhance My Image Before Using DeepPlate? (Spoiler: Yes, you should!)
At this point, you may be wondering whether DeepPlate is the solution to all your problems. Will it make your enhancement skills useless? Absolutely not!
Indeed, we do recommend enhancing a license plate using Amped FIVE before submitting it to DeepPlate. In our experiments, even with a partially successful enhancement, this increases the chances of a correct reading.
Still working with the previous example, we’ve used Amped FIVE to enhance the image slightly. This enhancement was applied to that individual frame. However, we could have achieved much more using super-resolution or frame averaging if we were dealing with the complete video:
Analyzing the enhanced image with DeepPlate, yields this result:
It is now time to reveal which was the actual license plate of our test image:
Now, let’s look back at the results of the first and second experiments. Notice that:
- In the first experiment’s result, 3 characters out of 7 were correctly read, and their confidence level was significantly larger than for the 4 misread characters;
- In the second experiment’s result (enhanced picture), 5 characters out of 7 were correctly read;
- Interestingly, in the second experiment, the network misread a “B” in place of a “D” as the most probable character. This mistake looks fairly reasonable as a human also easily confuses the “B” and “D” letters. Also, “D” ranked as the second most likely character after “B”. The same applies to the other misread character, which is an “8” in place of a “0”. Even in this case, “0” was the second most likely character after “8”;
- Although the network returned higher confidence values for characters that were correctly read, it returned a fairly high confidence level also for the two wrong characters (92.2% for the “B” and 92.5% for the “8”). This is another reminder that confidence levels should not be interpreted as the probability that the character is indeed that one. The neural network is far less prudent than a human in reporting confidence levels!
Just a sanity check, let’s run the high-resolution image through DeepPlate:
Thankfully, the network had no issues reading this crystal-clear license plate! Just notice that, even in this case, the second most likely character for each position is a character that has some visual similarity with the one above (“D” and “B”, “T” and “Y”, “2” and “7”, and so on).
Final Remarks
Our research team has worked for years on developing DeepPlate, and we’re excited to share with you the results of our efforts.
If you know Amped Software, then you also know we like to be very clear on what our solutions can do. For this reason, we’re trying to make it very clear that DeepPlate is an AI-based license reading tool that can get things wrong and can never replace a human in taking responsibility for a decision, even if it is solely investigative.
If you upload a test image and the network miserably fails on it, there’s not much we can do about it or explain why (we could only recommend carefully selecting the four points). Lack of explainability is indeed, in our opinion, one of the crucial elements against the use of deep-learning algorithms for evidentiary purposes.
So please use DeepPlate with a pinch of salt. If you have recommendations for improvements, let us know through our contact page!
