Crawling The Deep Web
Researchers have been exploring how the deep Web can be crawled in an automatic fashion. In 2001, Sriram Raghavan and Hector Garcia-Molina presented an architectural model for a hidden-Web crawler that used key terms provided by users or collected from the query interfaces to query a Web form and crawl the deep Web resources. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho of UCLA created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms. Several form query languages (e.g., DEQUEL) have been proposed that, besides issuing a query, also allow to extract structured data from result pages. Another effort is DeepPeep, a project of the University of Utah sponsored by the National Science Foundation, which gathered hidden-Web sources (Web forms) in different domains based on novel focused crawler techniques.
Commercial search engines have begun exploring alternative methods to crawl the deep Web. The Sitemap Protocol (first developed by Google) and mod oai are mechanisms that allow search engines and other interested parties to discover deep Web resources on particular Web servers. Both mechanisms allow Web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not directly linked to the surface Web. Google's deep Web surfacing system pre-computes submissions for each HTML form and adds the resulting HTML pages into the Google search engine index. The surfaced results account for a thousand queries per second to deep Web content. In this system, the pre-computation of submissions is done using three algorithms: (1) selecting input values for text search inputs that accept keywords, (2) identifying inputs which accept only values of a specific type (e.g., date), and (3) selecting a small number of input combinations that generate URLs suitable for inclusion into the Web search index.
Read more about this topic: Deep Web
Famous quotes containing the words crawling, deep and/or web:
“We learn through experience and experiencing, and no one teaches anyone anything. This is as true for the infant moving from kicking to crawling to walking as it is for the scientist with his equations. If the environment permits it, anyone can learn whatever he chooses to learn; and if the individual permits it, the environment will teach him everything it has to teach.”
—Viola Spolin (b. 1911)
“We are not very much to blame for our bad marriages. We live amid hallucinations; and this especial trap is laid to trip our feet with, and all are tripped up first and last. But the mighty Mother who had been so sly with us, as if she felt that she owed us some indemnity, insinuates into the Pandora-box of marriage some deep and serious benefits, and some great joys.”
—Ralph Waldo Emerson (1803–1882)
“Any newspaper, from the first line to the last, is nothing but a web of horrors.... I cannot understand how an innocent hand can touch a newspaper without convulsing in disgust.”
—Charles Baudelaire (1821–1867)