Dissertation > Excellent graduate degree dissertation topics show

Topic-Specific Crawling and Search Routing Research Based on Peer-to-Peer Network

Author: LiuJia
Tutor: LiuDongFei
School: Wuhan University of Technology
Course: Applied Computer Technology
Keywords: Topic-specific Search Peer-to-Peer Network Focused Crawler Starting URLs Topic Model
CLC: TP391.3
Type: Master's thesis
Year: 2010
Downloads: 79
Quote: 2
Read: Download Dissertation

Abstract


With the explosive growth of the information on the Internet, search engine has been a necessary access to information effectively. For meeting some users’searching requirements on specialized subject and specific topic, topic-specific search engine is becoming more and more popular in the area of data mining and information retrieval.Based on studies on search engine’s working principle and key technology, this thesis mainly analysis the feasibility and importance to implement a focused crawler in a Peer-to-Peer network environment. At the same time, it designs a general framework for the focused crawler—PPSpider based on Peer-to-Peer network, which has been developed and realized for corresponding experiments about crawling algorithm and optimization. Main contributions are as following:(1) Considering the difficulty in establishing topic model, the thesis creates its term vocabulary for each theme according to universal catalog classification, and then mines local documents on the peer in order to set up its vector space model. Through the relevance calculation between the peer’s VSM and each term vocabulary’s VSM, the topic model can be established.(2) In view of the issue that universal starting URLs always lose some edgy link or hidden web. We propose to choose some local seeds from each peer’s logs which record the peer’s topic trend. Including local seeds, the new staring URLs can not only cover more hot pages but also crawl web information on the edgy page. In another word, it increases PPSpider’s coverage.(3) As to the low efficiency brought by each peer’s frequent join and exit in the distributed network, the thesis put forward a mechanism named sortURL to optimize the crawling algorithm:sequence those URLs in the queue to be crawled through calculating the distance between URL’s hash value and peer’s hash value. The method decreases duplicated URLs and improve the crawling efficiency. AS sortURL mechanism has influence on discovery speed of topic-relevant page, the thesis considers an improved proposal sortURL-Depth. To be concluded, there is a broad application prospect for topic-specific search based on Peer-to-Peer network. As its core part, focused crawler under Peer-to-Peer network has always been a hot spot in this area. For this issue, the thesis proposes an idea of setting up topic model, designing and developing PPSpider to improve crawling algorithm. The experiment that mining peer’s log to choose local seeds proves that the improved starting URLs can discover target page more quickly. At the same time, the results of another experiment show that sortURL mechanism can reduce the duplication rate with no changes of peer’s throughput and overhead.

Related Dissertations

  1. Research on Focused Crawler Based on SVM Classification Algorithm,TP391.3
  2. Method Research of the E-learning Resources Collecting and Automatic Classification Based on Focused Crawling Technique,TP301.6
  3. Research on Topology Optimization of Bittorrent-Like Peer-To-Peer Systems,TP393.02
  4. Research on Network Coding Design in Heterogeneous P2P Streaming,TN919.8
  5. Hierarchical Topic Model Based on Document Frequency,TP391.1
  6. Comparison and Application of Image Classification,TP391.41
  7. The Strategy and Algorithm Designing for P4P Network Stream Optimization,TP393.02
  8. Resource search mechanism based on peer-to-peer network and algorithm research,TP393.02
  9. Application of network traffic behavior analysis,TP393.06
  10. Asymmetric-Prior Author Topic Models,TP391.1
  11. Research and Design of NAT Travsering Scheme Based on Hybrid P2P Network under UDP,TP393.02
  12. The Research and Implementation of Topic Evolution Based on LDA,TP311.13
  13. Based on the experts of the topic model retrieval and mining,TP391.3
  14. Experts search for mining research community,TP311.13
  15. P2P of Data Distribution System of the Realization of the Media,TP393.02
  16. Design and implementation of high - performance P2P streaming media content distribution server,TN919.8
  17. UPRIDE: An Improved PRIDE Reputation Model in P2P Environment,TP393.08
  18. P2P Web search technology research based on the characteristics of small-world optimization,TP393.02
  19. Research on Self-adaptive P2P Network Model Based on Peer’s Serve Capabilities,TP393.02
  20. Trust Model Researching in Peer-to-peer Network,TP393.02
  21. Applied Research of Network Coding on P2P Streaming Media On-demand System,TN919.8

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Retrieval machine
© 2012 www.DissertationTopic.Net  Mobile