Crawler
Intelligent Crawler
We created an intelligent web crawler which we deploy to methodically browse targeted areas of the internet. What makes ours different to others is that it’s loaded with over 566,000 hashes of known child sexual abuse images.
How does using the crawler help our mission?
We use our crawler as one operational tactic in a suite of tools designed to find, remove, and disrupt the availability of child sexual abuse material. Our crawler is deployed in a considered and targeted manner in order to be most effective. In addition, it reduces unnecessary exposure to child sexual abuse imagery for our analysts.
As well as running our crawler to generate proactive reports for our analysts to assess, we’ve been able to provide an additional service to our Members in the domain registry sector.
In 2020 it crawled almost 42 million webpages, and over half a billion images.
By comparing each image our crawler finds to the hashes of known child sexual abuse material, it means we can find duplicate child sexual abuse images hidden across the internet.
“A growing number of domain registry Members are committed to taking preventative steps to stop their services from being abused. We’re helping them by using our crawler to check their domains and notify them if we match any imagery to our hash list of known child sexual abuse images.”
Sarah Smith
Technical Projects Manager, IWF
