Dark internet and the invisible web
Search engines have become the main means of accessing information on the Internet. In the early days directories were used but these have now relinquished first place to the search engines. This is due to the enormous number of websites. Their number has grown exponentially and Google tells us that today there are over eight billion of them. There is no way that all these sites could be viewed by a human reviewer and placed in a directory. Our ability to find information on the Internet is now very much dependent on the qualities of the chosen search engine.
This does mean is that, if there are websites which contain information which cannot properly be accessed by the search engine spider, then we will never see it listed in the results. Indeed there are areas of the web which are never found at all by the searching spiders. A current problem is with search engines being unable to index databases and other websites which generate dynamic content. In other words, the content is not set out on the pages of the website but is generated each time in response to a user accessing the site via a browser. Websites with some content not in HTML also cause difficulties. Examples are proprietary text formats or multimedia. PDF files have also been problematic.
If the search engines are not able to find the information then the only hope of finding its existence is by using a Directory. You can obtain more information and links to directories of resources by going to http://websearch.about.com and searching for invisible web.
You can also read more about the Invisible Web here:
http://library.rider.edu/scholarly/rlackie/Invisible/Inv_Web_Main.html
We've now reached the end of this section, which is also the end of Outcome 2. You can go back to any topic by using the menu at the left-hand side of the screen, or return to the Unit 6 home page by clicking on the ICTL logo at the top left of the screen.
