Law enforcement and governments are demanding the ability to decrypt communications.
They want backdoors that can get past encryption, and they’re trying to legislate them into existence. For example, in the US, New York and California recently proposed bills that would require either backdoors to bypass phone encryption or fines of $2,500 per unit sold in the respective states.
Encryption is often, as Motherboard’s Joseph Cox succinctly puts it, a polemic issue: an argument that can boil down to a simplistic battle between the advocates of innocent individuals’ privacy rights (and of security that isn’t weakened via backdoors) vs. those who point to encryption that shields criminals.
This debate – a renewal of the decades-old Crypto Wars – needs more nuance, according to two researchers from King’s College London.
Specifically, it needs to be fleshed out with data showing, as accurately as possible, what encryption is actually hiding, be it journalists protecting their sources, political dissidents, pedophiles selling photos of their own children, or murderers for hire.
To get at that nuance, researchers Daniel Moore and Thomas Rid have carried out an in-depth scan of hidden-services websites within the Tor network.
Hidden services, also known as dark web sites or .onion sites, use the sophisticated, multilayered encryption of the Tor network to hide themselves and the source of their traffic.
💡 LEARN MORE: What is… Tor?
The researchers’ conclusion: dark web sites are, in fact, most commonly used for crime.
From their paper, Cryptopolitik and the Darknet, which explores the relationship between privacy and security:
The results suggest that the most common uses for websites on Tor hidden services are criminal, including drugs, illicit finance and pornography involving violence, children and animals.
To conduct their research, the pair scraped Tor hidden services with a website crawler that automatically hopped from site to site via links, the same as a human site visitor would do.
They had manually come up with a taxonomy with which to classify websites into categories that included, among other things, drugs, extremism (including militant how-to guides and support for terrorist violence), finance (including money laundering, counterfeiting, and trade in stolen credit cards or accounts), hacking (including distribution of malware), “illegitimate” porn (i.e., porn that’s nonconsensual or which involves children, violence, or animals).
Over the course of five weeks, Moore and Rid found 5,205 live websites, 2,723 of which they were able to classify by such content categories.
Of those, some 57% – 1,547 – hosted illicit material.
The researchers point out that there’s no way to index every existing hidden site: it’s a shifting landscape, where sites constantly pop up, only to quickly disappear as they switch addresses and server locations without notice.
To make it more difficult still, Tor addresses are just long strings of characters with a .onion domain at the end.
In order to get a proper sample of all the hidden services on the dark web, the pair built a Python script that crawled the dark web, starting with the popular dark web search engines Onion City and Ahmia.
Moore and Rid said that their methodology, though it’s bound to have missed a “small number” of closely held sites, manages to map a number of sites that roughly matches the estimated number of Tor websites at any given moment.
This is not a full survey: The Tor project has estimated that some 30,000 hidden services are active and announce themselves to the Tor network every day.
The crawler scraped the content from each page and uploaded it for analysis. It did the same when it found a link: it hopped to the linked site and scraped it, too.
Next, an algorithm processed the collected content and automatically sorted it into the categories that Moore and Rid had set up. They spot-checked the automatic categorizing and found it to be, overall, very accurate.
What the researchers captured was a representative sample, they said. Moore told Motherboard that it’s similar to what Tor users encounter:
We don’t make any statements about the entire contents of Tor. We just looked at what is the reasonable offering of hidden services to most users.
We went for what a user can actually see and interact with.
When Motherboard asked The Tor Project to comment on Moore and Rid’s methodology, the nonprofit behind the Tor network declined.
But Kate Krauss, a spokesperson for the project, told Motherboard in an email that the pair seem to have overreached their research:
The researchers seem to make conclusory statements about the value of onion services that lie outside the scope of their research results.
Onion services are a tool with unique security properties used for a wide range of purposes: They are self authenticated, end-to-end encrypted, and offer NAT punching and the advantage of a limited surface area.
Moore and Rid conclude their essay by saying that hidden services have already damaged Tor, as well as trust in the internet as a whole, with the political risk they bring to cryptography.
Should Tor do away with hidden services?
It’s not like hidden services haven’t been considered for the chopping block already.
In 2014, an anonymous user asked Roger Dingledine, one of the original developers of Tor, “Why not scrap hidden services?”
We do think about that option periodically.
From the researchers’ conclusion:
Tor’s ugly example should loom large in technology debates. Refusing to confront tough, inevitable political choices is simply irresponsible. The line between utopia and dystopia can be disturbingly thin.
Users, what’s your take: are hidden services worth the political firestorm they generate? Are they worth criminals escaping justice?
Please let us know your thoughts below.
Image of Dark Web courtesy of Shutterstock.com
28 comments on “Dark web is mostly illegal, say researchers”
The political debate isn’t about the TOR network. Who cares, shut TOR down it wouldn’t make a difference anyways. The debate is about our cell phones. the trusty little devices we put our hole life on and carry around with us everywhere that we go. Cell phone’s are what needs encryption. the NSA already has control of the exit nodes on the TOR network and is able to find and shut down .onion sites at will as they have already shown. Saying that the encryption debate is about TOR is misleading.
I don’t disagree that phones are critically important but I don’t think you can write off Tor either.
Control of exit nodes has no bearing on hidden services because you don’t exit the Tor network when you use them – you exit the Tor network when you browse regular sites.
A quick tour of the Dark Web will demonstrate why it’s such a hot issue – it’s the best place to do the worst criminal business and the fact that the sites that are there are there at all is testament to the fact that they cannot be shut down at will, which is of course why they attract the people they do.
Where sites have been shut down it seems to be the result of undercover police work, malware and statistical correlation attacks that are complex, costly and take a long time to bear fruit. It may be possible to take out hidden services but it’s far more difficult than taking out regular websites.
Millions of dollars have been spent on research and tools like Memex to create cracks in Tor’s armour and research papers such the one produced by the Global Commission on Internet Governance (see https://nakedsecurity.sophos.com/2015/02/20/the-dark-web-anarchy-law-freedom-and-anonymity/) demonstrate what a serious issue it is.
I have to agree with critique of the study; their “representative sample” would likely have excluded a much larger proportion of legitimate uses/users than illegal.
Why? Just face the fact the TOR is a giant shithole with the majority of users being the scum of the Earth.
Interesting stats. So to put it another way, 1547 of 5205 websites (29.7%) were illegal.
Numbers are funny like that.
Actually it’s about 5.2% (1,547 of 30,000 websites) if you’re going to stack the numbers like that but I’m sure you’re not suggesting that the research indicates that 94.8% of the Dark Web is legal hidden services.
25,000 sites could not be crawled. Of the 5,000 or so sites that could be crawled 29.7% were found to be illegal, 22.6% were found to be legal and about 48% could not be classified according to the researchers’ algorithm.
So a much, much larger proportion of the total population was tested than in, say, an opinion poll.
Given that there is no validation error rate given for their classifier, I’m not going to suggest we can draw any conclusions at all from this.
Well, I think you could go for this conclusion: “Remember that Silk Road thing on Tor? There’s more…”
Half of me says, “That’s a bit like saying, ‘Sun to rise in morning’,” and leaves me shrugging at the whole story in a sort-of “You don’t say?” attitude.
The other half says, “OK, you actually tried to measure it, which is to be applauded. And it seems likely that there is a surprising amout of non-bad stuff on there, which is worth knowing.”
It’s not 30,000 websites, it’s 30,000 hidden services. A hidden service can host *any* TCP application, not just HTTP. Probably half of all hidden services are just SSH servers to allow remote configuration of the other half. How many hidden services are PGP key servers? Or Bitcoin nodes? Or something else? It’s a bit premature to talk about how many hidden services are illegal websites when we have no idea how many are even websites to begin with.
Yes, you’re right, I should have said ‘hidden services’ other than ‘websites’ but beyond that you’re making my point for me, which was that it wasn’t appropriate (implicitly in the case of the parent) to compare the hidden services for which there was an ‘illegal’ result with everything that didn’t get an ‘illegal’ result.
But I think the idea is that you’re invited to assume that sample of sites that could be found, and of those the ones that could be categorised automatically into legal-or-not, are representative.
Therefore you are invited to treat from 1547 of 2723 as a likely badness ratio rather than 1547 of 5205 or 1547 of 30,000.
You only get 1547/5205 if you assume that everything not categorised was legit, or 1547/30,000 if you assume that everything that could not be found automatically at all was 100% legit.
Interestingly to me, all those numbers (even 1547/2723) are lower than I would have guessed.
The other interesting thing was the orotundity of the Tor spokelady’s reply. I think Tor can be its own worst enemy sometimes, acting surprised when people associate Tor with crookery, and using lots of words when fewer would do. (“The researchers seem to make conclusory statements about the value of onion services that lie outside the scope of their research results.” I think she meant “They may have overstated their case.” 🙂
I speculate that the methodology – spidering – skews the results in favour of the non-criminal. Spidering is the easy way to crawl a hypertext space and therefore the easiest way to be found.
If you’re interested in hiding in a hypertext space then it’s the first thing you’d try to avoid.
It’s the non-spiderability of ‘Deep’ bits of the Dark and regular webs that led the US Gov asking DARPA to create Memex.
I have seen those numbers disputed quite handily. But here is something to consider. Governments around the world use Tor. Activists use Tor. Bloggers use Tor. I have used Tor on a message board just to see if it would work, and it did. Plenty of people use Tor because they’re tired of being followed around all over the internet. And it’s usually not by the government. That being said, I bet there is far more crime on clearnet than there is on Tor. I bet there is far more crime perpetrated through the US mail than on Tor. The crimes committed openly with guns every year make Tor look like a public playground. This idea of Tor being mostly criminal is designed politically to get support for banning it. Nothing more. There are those who want automatic access to everything…..just because.
If you’ve got nothing to hide being “followed” is irrelevant.
We all have something to hide from somebody.
So, just to clarify, you wouldn’t mind being watched by a government agent 24/7, including areas commonly considered private (Such as bathrooms, your own house, etc) without your consent?
Why do people always trot out that ridiculous remark? What you “have to hide” can easily be a lot less about the relative morality of what you’re doing and a lot more about *who’s looking*.
In Steve Gibson’s words: “Does the fact that I have nothing to say make ‘free speech’ irrelevant?”. It is not only about myself; there are people who have to say something and there are people who have to hide something for very good reasons.
Good. Then you won’t mind if some strangers come into your house and look around whenever they want to. Take notes about what you own, what you buy, what shows you watch on TV, and record all of your personal conversations…..and then sell the info and put you on various lists for targeting. So you can feel free to allow these strangers into your home, uninvited and unannounced. I think I’ll choose another path.
Encryption is not Tor.
Encryption is not “The dark web”.
Using those examples as a reason to cripple encryption is like
trying to “ban roads because they have gutters”.
If Tor never existed and your task was to redesign the internet to keep mobile users safe from privacy violations and hacking then I’m fairly certain that the idea of onion routing would be high up on the list. Removing the darkweb element to it would go a long way for improving public trust. There are plenty of legitimate mobile apps that use can use Tor – Chat Secure, Twitter and even Facebook.
I once visited the Dark Web… and was effectively terrified of going to sleep that night. It’s an evil place. So there might be somebody, somewhere who uses it for some good purpose (whatever that may be; I’m guessing someone in a corrupt government). But lets face it. People in the past didn’t need TOR to hide secrets from Hitler or Stalin. The only people who are being dragged to court in our country are pedophiles, frauds, murderers, and terrorists. The rest of us are “clear”. The world–that is, the good world–would carry on just fine, without hidden services. And frankly, is privacy really so important that we are willing to lay down hapless young women and children on the altar? Believe me, someone who’s been to that miserable place, when I say that we are better off without it.
“People in the past didn’t need TOR to hide secrets from Hitler or Stalin.”
No, instead they were tortured and murdered by the tens of millions. How many would have lived if there was something to hide their identity through?
Since this article was posted I have been doing a little research on Tor. Admittedly I know very little about it, but I support it. What I am finding out is that hidden servises are only a small part of Tor. I also found that child exploitation makes up 2% of hidden services. That’s bad, of course. And there are other bad things there as well. But those same bad things are everywhere. I think it is important for people to have the option of being anonymous and to be able to speak freely. I also believe in a right to privacy. Here is an interesting article.
“That is a huge, and important, distinction. The vast majority of Tor’s users run the free anonymity software while visiting conventional websites, using it to route their traffic through encrypted hops around the globe to avoid censorship and surveillance”.
I’ll see your interesting article from Wired and raise you one of our very own 🙂
I would even question the relevance of “the number of hidden services in category X” as a relevant measure for how hidden services are used.
“How many users uses them” is far more relevant imho.
In the normal web, a very tiny amount of websites account for most internet uses.
In the dark web, this might be even more skewed towards “a few big legal hidden services, a lot of small illegal hidden services” since an illegal website needs to avoid attracting to much attention (eg : SilkRoad).
So, even if the numbers are accurate, they are probably not as much relevant as they seem.
The same argument can be made on analysis relying on bandwidth usage/tragic. Of course child porn make for most of traffic. It’s video.
Regular porn also uses up most of the regular web traffic. Should we shut down the regular web too ?
This is not about preventing crime or stopping criminals. The reason why the government hates things like Tor & encryption for the general population is because it allows those of us in this population to do things, make plans, buy things, etc, to combat the government if we decided to, whether as individuals or in groups. The government wants complete & total control. It always has but has also always lagged behind in current trends concerning technology. This pisses them off to no end. It is an agitation. Trust me, if they could make a “law” making it illegal except for cops/other govt entities/agents/politicians to possess access to all of this stuff & this includes guns/ammo/other weapons too, they WOULD. They’re working on doing it, but everything being done is a slow & steady thing. Eventually, they will succeed in passing some unconstitutional laws that make it impossible almost for us to have these tools, including guns/encryption equipment, software, etc.
The government hates Tor so much it invented it (via the US Navy and DARPA), made it open source so it couldn’t possibly control it and to this day continues to fund it in spite of the fact that it’s quite obviously in the hands of citizens.