Dissertation > Excellent graduate degree dissertation topics show

Research on Crawling Deep Web Information

Author: JiangKai
Tutor: TangShuoFei
School: Harbin Institute of Technology
Course: Computer Science and Technology
Keywords: Deep Web automatic form filling Deep Web information extraction semantic metadata acqusition re-crawling
CLC: TP393.09
Type: Master's thesis
Year: 2010
Downloads: 30
Quote: 0
Read: Download Dissertation

Abstract


The entire web information can be categorized into Surface Web and Deep Web,based on the information’s depth. The Surface Web mainly includes the“static”pages, which are statically stored in server in forms of file. Comparatively, Deep Web means the“dynamic”pages whose content hidden behind search forms and only produces results dynamically in response to a direct request. These dynamic information are also known as Deep Web information or Hidden Web information, which have characteristics of large amount, high quality and highly structured. But traditional search engine crawlers can’t probe the Deep Web information. Techniques of crawling Deep Web information in large scale become a hot spot in research.In this paper, we first present the current ways of searching Deep Web information and analyze their advantages and disadvantages. Then we decide to adopt the method of extending the traditional search engine crawlers with the ability of automatically filling in forms, constructing the query URL, obtaining the result pages. In this way, traditional search engine can index Deep Web information the same as Surface Web.To achieve these goals, we need to address the following main issues:(1) How to correctly find the Deep Web sources and extract their useful information in spite of the complexity of HTML forms.(2) In condition of diversity of form control, how to select the combination of queries, especially how to generate values of text box input.(3) How to extract valueable information from result pages. To solve the above problems, we first analyze and optimize the traditional search engine crawlers, then present solutions and algotithms of Deep Web data sources finding, form interface extraction, automatically filling in forms, Deep Web information extraction, semantic metadata acqusition, re-crawling result pages. Finally, based on the traditional crawler, we deploy a Deep Web information crawling system. Meanwhile, we present experimental evaluation validating the correctness of our algorithms.

Related Dissertations

  1. The key component vertical search engine technology research,TP391.3
  2. Topic-based Web Information Monitoring System Strategy and Implementation,TP393.09
  3. Research of Data Source Selection with Similar Theme in Deep Web Integrated System,TP311.13
  4. Deep Web Data Cleaning Method Research and Application,TP393.09
  5. The Study on Deep Web Interface Integration and Search Strategy,TP393.09
  6. Deep Web Interface Discovery Based on Domain Knowledge,TP393.09
  7. The Research of Data Extraction and Semantic Annotation in Deep Web,TP393.09
  8. Selection of Deep Web Database,TP311.13
  9. Study and Design of Focused Search Engine,TP391.3
  10. The Research of Deep Web Uncertain Schema Matching Based on Domain Ontology,TP393.09
  11. Deep Web Ontology-based pattern matching technology research,TP393.09
  12. Design and Implementation of Domain Specified Deep Web Information Integration System,TP311.52
  13. Research on Web Information Extraction Technology Based on Deep Web,TP393.09
  14. Automatic Data Extraction of the Query Answer from the Deep Web,TP311.13
  15. Research on Web Crawling Strategies,TP393.092
  16. Design and Implementation of the Spider system based on Web image search engine,TP391.3
  17. The Research and Design on Intelligent Vertical Search Engine,TP391.3
  18. Hierarchical Modeling-based Approach to Integrate Query Interfaces on the Deep Web,TP311.13
  19. Service-oriented data integration framework,TP393.09
  20. Research and Implement O F Web in for Mation Integration Technology,TP393.09

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Computer network > General issues > The application of computer network
© 2012 www.DissertationTopic.Net  Mobile