Dissertation > Excellent graduate degree dissertation topics show

Research on a Method of Focused Crawler Based on Page Partition

Author: XingMinLing
Tutor: LiXueMing
School: Chongqing University
Course: Computer Software and Theory
Keywords: Focus crawler Tunnel Classifier Page partition Transition probability between classes
CLC: TP391.3
Type: Master's thesis
Year: 2011
Downloads: 74
Quote: 0
Read: Download Dissertation

Abstract


With the increasing of information on internet, the results of general search engine can’t meet the needs of the users, especially those with different area or different background who want to search information in specialized fields. Then vertical search engines which are much more professional and personalized emerge as the times require.A key part of vertical search engine is the focused crawler by which topical portions of the internet can be downloaded whose quality can directly affect the results of vertical search engines. The major factors of the quality of topical portions downloaded are how to predict the relevance between downloaded page and the target topic and how to decide the visit priority of candidate URL in crawling frontier; also the structural characteristics of Web page will result in“tunnel”phenomenon that exists in focus crawling, which will cause big effect on the coverage and accuracy of focus crawler.According to the advantages of page blocks and best-first search strategy and in order to help the crawler cross“tunnel”, this paper proposes a new method based on page partition:①With the thought of the classifier to a focus crawler, a Center for Vector-type classifier is used to determine the topic of a page or a page block and their similarity. The outstanding advantage of this classifier is that it can better describe user’s interest topics; also it runs fast and can improve crawler’s speed;②In consideration of the difference between URLs, the URLs in a page block are classified as special URL and common URL to predict their visit priorities, which can improve the accuracy of prediction, resulting in overcoming the problem that related pages are neglected caused by inaccurately prediction;③Based on page block, using transition probability between classes can help the crawler cross“black tunnel”and using the thought of page partition can help the crawler cross“grey tunnel”.Finally with 3 different target topic from DMOZ, comparison experiments are carried out on Harvest Ratio between three crawler that are separately realized by the method this paper proposed, the text- contents method and the classifier-guided method, and the experimental results show that the method this paper proposed performs better than other two in Harvest Ratio and can better meet the needs of the focus crawler.

Related Dissertations

  1. ISAR Imaging Simulation of Space Targets and Target Recognition Based on ISAR Images,TN957.52
  2. Study on the Reasonable Cover Thickness of Subsea Tunnel,U459.5
  3. The Surrounding Rock Monitoring and Finite Element Simulation of Subway Tunnel,U455.43
  4. The Study of Stability and Deformation for Cao Mao Mountain Tunnel,U451.2
  5. Research and Implementation of Monocular Vision-Based Vehicle Detection Algorithm,TP274
  6. Design and implementation of the multi-channel pressure sensor auto-calibration system,TP212
  7. Design of Classifier of License Plate Location and Relationg Algorithms Research,TP391.41
  8. Multi-feature fusion of visual tracking algorithm,TP391.41
  9. Internetwork accelerate technology research and implementation,TP393.2
  10. Based on wavelet transform and linear subspace face recognition technology,TP391.41
  11. TUNNEL driver behavior and road safety research,U458.1
  12. Training and weighted online dictionary sparse representation of difference,TP391.41
  13. A Study on Deformation Law of Highway Tunnel Mizhi NO.1 Excavated in Loess,U451
  14. The Study of Tunnel Light Environment Artificial Lighting Control and Energy Saving,U453.7
  15. Research on Privacy- Preserving Data Mining Based on K-anonymity Algorithm,TP311.13
  16. The circular flow Tunnel of study of the hydraulic transient problems,TV135
  17. Design and Study on Eye Location and Tracking System,TP391.41
  18. Study on Analytic Solution of Torsional Deflection on Subway Tunnel Structure Inclined Wear Activities Ground Fissure,U451
  19. Simulation of Underground Water Inrush Based on Lattice Boltzmann Method,TD745
  20. The Change of Electromyography Before and after Ulnar Nerve Release,R688
  21. Analysis on the Stability of Surrounding Rocks and Its Safety Guarantee Measures in Qingdao Close Overlapping Underground Engineering,TU457

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Retrieval machine
© 2012 www.DissertationTopic.Net  Mobile