The Invisible Web: The Internet Is Much Bigger Than You Think

Browsing the world wide web is extremely fun, somewhat addicting, and also very easy to do. We type our queries into Google Search or Bing and hit enter. Bang! Billions of search results instantly pop up on the screen (a good Internet connection helps with this too). Sounds simple enough, right? Well, what if I told you that the part of the world wide web that is accessible through search engines like Google and Bing makes up only 4% of all web content?

I know, that’s exactly how I reacted when this was pointed out to me by a friend. I still find it hard to believe, but after digging into this strange hidden world, it’s hard to deny. So what are we waiting for? Let’s find out what lies beyond the reaches of the Internet as we know it!

The Invisible Web

Also known as the Deep Web, Deep Net or Hidden Web, the Invisible Web is essentially the part of the world wide web that is not indexed by search engines. The name “Deep Web” was coined by Mike Bergman in 2000. The size of this part of the web was estimated, in 2001, to be 7,500 terabytes. Keep in mind that the Deep Web is said to be growing exponentially. Scary, huh?

It mostly consists of database-driven pages that can only be accessed via major database sites, similar to job search sites. They are essentially dynamic pages that are constantly updated with the latest information, but disappear the moment they are closed. It’s almost like magic!

This part of the web may also be inaccessible due to unlinked content, which prevents spiders (web-crawling programs) from accessing the content.

Private intranet content is also present on the Deep Web, where 11+ billion static pages are hidden from the public. These pages are obviously owned by specific companies and are password-protected.

Credits:Brian Senic/Shutterstock

The remainder of the Invisible Web is made up of limited access content (sites technically shut-off by CAPTCHAs), Non-HTML/text content (which isn’t handled by search engines), Software (content accessible only by Tor software) and other related material.

Unexplored World

The Invisible Web is so vast, but it can’t be accessed by the usual search engines. In that case, how exactly DO you take a trip into this unexplored world?

First, use a search engine (Google, Yahoo, Bing etc) and find a primary database page. Then, search within that database for exactly what you’re looking for. This is not easy and can be quite time-consuming, due to the isolated nature of these “invisible” pages.

The Dark Side

As mentioned above, there is a significant portion of the Deep Web that is encrypted and only accessible by special software. This is called the Dark Web. One such special program used for accessing the Dark Web isTor. Unfortunately, it has devolved into a hub for numerous criminal activities. These criminal networks use Tor to their advantage, as it protects the identity of a person online by continually changing the user’s IP address. Obviously, caution is advised when using such software. It’s important to note, however that the majority of Tor users actually require this software to safely access social media, since certain social media sites are restricted in some countries.

Taking advantage of this unique use of the Tor software and its connection to the Dark Web, Facebook announced last year that its users could access the site through the Tor program. Now, even people living in countries with Internet restrictions can access Facebook. See, even the dark side has its advantages!

So in  the end, these secret pages might turn out to contain some very useful and interesting information, while also helping people safely access certain parts of the web. I guess it’s time to turn invisible!