Security+ and the Deep, Dark Web

One of the topics in the CompTIA Security+ exam objectives covers how the Internet is and isn’t indexed. Learners often question why this topic is included in the exam objectives. Too many assume that search engines like Google and Bing provide an index of the entire Internet. When we discuss this topic, others will point out that it’s important to understand it, as this is where criminal elements who use the Internet hide. Both of these views (‘everything is indexed’ and ‘only criminals use the dark web’) are wrong.

The deep web is any part of the internet that is hidden from public search engines. It makes up the vast majority of the web and includes everyday secure sites like private email accounts, online banking portals, subscription services, and corporate intranets that require credentials to access. The web crawlers that build Internet indices can’t access these sites. Alternatively, site owners use a simple text file called robots.txt to give a command: User-agent: * \n Disallow: /. This acts like a “Do Not Enter” sign for web crawlers like Googlebot.

Then there is the dark net. The darknet is a hidden portion of the internet that requires specific, anonymizing software to access. It is entirely unindexed by traditional search engines and is known for providing users with strict privacy, allowing individuals to communicate, browse, and share data securely without being tracked.

But if you want to explore the Internet anonymously, can’t you use a web browser feature like incognito mode? That web browser setting (also called Private Browsing) prevents your browser from saving your browsing history, cookies, and search data locally. When you close the browser window, temporary session files are erased, making this feature ideal for shared computers or for logging into accounts without leaving traces. Despite popular belief, those features do not make you anonymous.

Who can detect your private browsing activity? Network administrators overseeing your device can check their logs for IP addresses you've visited. Similarly, your Internet Service Provider's operators can review their log files. Additionally, most website administrators can see which IP addresses accessed their servers through their own log files.

The dark web is an intentionally hidden layer of the internet that cannot be accessed through standard search engines. Visitors often reach the dark web via the dark net. The dark web requires specialized, anonymity-focused software (like the Tor Browser) to mask a user’s identity and location, making it a hub for both illicit activities and privacy-conscious communication.

Based on the recently released CompTIA Security+ exam objectives, this topic will be included in the new SY0-801 exam.