Dissertation > Excellent graduate degree dissertation topics show

The Design and Implementation of Vertical Search Engine Based on Duplicated Web Pages Elimination

Author: ZhaoLiLei
Tutor: PanDongHua
School: Dalian University of Technology
Course: Management Science and Engineering
Keywords: Vertical search engine Topic crawler Full-text retrieval Duplicatedweb pages elimination
CLC: TP391.3
Type: Master's thesis
Year: 2012
Downloads: 291
Quote: 1
Read: Download Dissertation

Abstract


As the rapid development of Internet, web pages have increased constantly in recent years. However, it is impossible for comprehensive search engine to retrieve all the web pages on the Internet, due to issues of storage, computing resources, bandwidth and so on. In order to satisfy users’requirements of a specific field, and to improve the relevance and accuracy of search engine, the study of vertical search engine begins. However, compared to comprehensive search engine, vertical search engine within its own particularity may crawl duplicated or near-duplicated web pages easily. Thus, existing duplicated web pages elimination strategies have some defects when applying to vertical search engines.Around the design and implementation of vertical search engine, first, the paper makes a brief introduction of the vertical search’s engine current research situation, and then analyzes the topic crawler and full-text retrieval in vertical search engine to provide a theoretical basis for the design and implementation of the system in the last section.On account of duplicated web pages elimination in vertical search engine, the reason and type of duplicated web pages, the meaning of removing duplicated pages, as well as the process and common algorithms of duplicated web pages elimination are introduced briefly first. After that, the shortages of these existing algorithms of duplicated web pages elimination in existing vertical search engine are pointed out, i.e. all of them ignore the particularity of vertical search engine, and don’t use vertical search engine’s own characteristics to remove duplicated web pages. Thus, this paper combines the topic crawler algorithm based on content and the duplicated web pages elimination algorithm based on content, and puts forward a duplicated web pages elimination strategy appropriating for vertical search engine, which can filter duplicated or near-duplicated web pages, and alleviate the search engine’s burden of post process and index construction. This paper also attests the strategy’s superiority proposed in the paper by several groups of related experiments.The design and implement of the vertical search engine in the last section applies the theories above. Basing on Solr Server, a vertical search engine with the theme relating to Chinese herbal medicines is designed. During the process of implement, this paper proposes a practically feasible method to acquire seed URLs and topic dictionary relating to Chinese herbal medicines, and improves the topic crawler of search engines by using Java technique, which can guarantee the engine filters duplicated or near-duplicated web pages when crawling web pages.

Related Dissertations

  1. The Design and Implementation of Lucene-Based Network Literature Vertical Search Engine,TP391.3
  2. Union Catalog System Based on OAI-PMH Protocol and Full-text Retrieval Technology,TP391.3
  3. Research and Implementation of Vertical Search Engine Key Technology,TP311.52
  4. The Research and Design of Monitoring System for BBS,G206
  5. Mobile Internet environment Wireless Vertical Search Engine Research and Implementation,TP391.3
  6. Applied Research LUCENE Chinese word in the text of a document retrieval system research,TP311.52
  7. Research and Implementation of Vertical Search Engine Based on Distribution,TP391.3
  8. Research and Application of Full-Text Retrieval Based on Lucene.Net,TP391.3
  9. Massive Data Storage and Full-text Search,TP333
  10. The Design and Implementation of Website Analysis Module Based on Shopping Search Engine,TP393.092
  11. Design and Implementation of Web-Based Resource Digitized Full Text Retrieval System,TP391.3
  12. Research and Realization on Intelligent Question Answering System of Distance Education,TP391.6
  13. Personalized Multi-media Resources to A Vertical Search Engine Technology Research,TP391.3
  14. Retrieval Service Research for Urban Geographic Information,P208
  15. Research and Implementation of full - text retrieval system based on lucene Chinese,G354
  16. The design and development of multi-document text retrieval system,TP391.3
  17. The Research on Focused Crawling Algorithm in Vertical Search Engine,TP391.3
  18. Research and Realization on Full Text Retrieval System Based on Indexing of Single Chinese Character,TP391.3
  19. The Design and Implementation of Second-Full-Text Retrieval System Based on Lucene,TP391.3
  20. Design and Implementation of the Graduation Thesis Full-text Retrieval System with Lucene,TP391.3
  21. The Research of Stock Information Search and Prediction System Based on Mobile Platform,TP393.09

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Retrieval machine
© 2012 www.DissertationTopic.Net  Mobile