Deep Web Harnessing Tools

Abstract

Abstract
Much of the information is valuable and accessible on the internet, but a vast amount of high-quality information is not searched even by the best search engines. Estimates put the volume of the hidden internet at up to 500 times the size of the surface web. Surfing the visible web failed to satisfy many searchers’ information needs. For this reason, the deep web lying outside the scope of Google and other general search tools has excited many including library and information centers. Much of the hidden information resides in dynamic databases; the contents of these sites will never be indexed by any web crawler because the pages are generated on the fly and then discarded. It also includes other contents such as audio/video clips, flash movies, and documents in non-standard formats, etc. The present study tends to identify and explore such tools including search engines, directories and databases that help the research community to harness the invisible web.

Keywords: Deep web, invisible web, hidden web, search engines, directories, databases, search engines