Dissertation > Excellent graduate degree dissertation topics show

Research on Extensible Hash Based Dynamic Load Balancing for Parallel Web Crawling

Author: SunShouXing
Tutor: YeYunMing
School: Harbin Institute of Technology
Course: Computer Science and Technology
Keywords: Parallel reptiles Dynamic Load Balancing Extensible hashing algorithm Hypergraph rezoning
CLC: TP391.3
Type: Master's thesis
Year: 2010
Downloads: 109
Quote: 0
Read: Download Dissertation

Abstract


With the rapid development of the Internet , online information explosive speed constantly enriched and expanded the search engine to obtain information necessary tools , while the Web crawler collection tool as a web page is a core part of the search engine . With the Internet, the amount of information increases , higher search engine website to collect comprehensive and timely performance of the reptiles facing huge challenges . Not only requires the ability to download page as efficiently as possible , and requires the use of computing resources and server resources as fully as possible . Around the core needs to improve the performance of Web crawler , crawler works and related technologies on the basis of in-depth study , research from the two perspectives of the process of load distribution and dynamic load balancing process the reptile dynamic load balancing , from multiple larger upgrade the operating performance of the reptiles . The main results of this paper can be summarized as the following aspects: First , on the basis of in-depth study of reptiles related technologies , proposed two logical node mapping method scalable hash algorithm combining static load distribution strategy for pseudo key distribution problem of abnormal improvement in the practical application of the extensible hashing algorithm . Secondly , based on the static load distribution strategy , the proposed parallel crawler system node to load equivalent model to measure the load distribution , in turn , according to the system load distribution node hypergraph re - partitioning model for dynamic load scheduling modeling , eventually introducing a hierarchical strategy solve the hypergraph re - partitioning problem . Finally , based on the above results of theoretical studies , this paper design and development of a high-performance distributed parallel Web crawler system .

Related Dissertations

  1. Research on Parallel Frequent Graph Pattern Mining,TP311.13
  2. Research and Implementation of Dynamic Distributed Strategy Load Balancing of Grid Service,TP393.01
  3. Distributed load balancing strategy,TP393.03
  4. CORBA environment , dynamic load balancing studies,TP311.52
  5. Based on the exchange of content-oriented Linux cluster system load balancing strategy research,TP393.05
  6. DMS Application Research Based on Distributed Cache Technology,TP311.52
  7. The Research and Realization of the Parallel Spatial Operation in a Simple Feature Model,P208
  8. Research of Dynamic Service Allocation Strategy Based on Java RMI,TP311.52
  9. Partition of Task Type Based on Resources and Real-time Requirements and the Research of Load Balancing on It,TP393.02
  10. Web Service -oriented load balancing strategy research,TP393.09
  11. Research and Implementation of Physical Distribution Management System Based on Three-tier C/S Structure,TP311.52
  12. Research and Implement on Technologies in Cluster Middleware of Spatial Database,TP311.13
  13. Dynamic load balancing technology based training system design and implementation,TP311.52
  14. Research and Implementation of Parallel Logic Simulation System Based on VHDL,TP391.9
  15. Dynamic Load Balancing with Uncertain Factor Based on Cluster,TP338
  16. Object-oriented software system disk controller Research and Implementation,TP311.52
  17. Implementation and Optimization of the Missing Call Notice System in the Mobile Communication Network,TN929.5
  18. The Research of Architecture and Very Important Technologies for Parallel Graphics Rendering System,TP338.6
  19. The Research of Dynamic Load Balancing Strategy with PVM,TP338.6
  20. Distributed system host load prediction and dynamic load balancing of,TP393

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Retrieval machine
© 2012 www.DissertationTopic.Net  Mobile