Dissertation > Excellent graduate degree dissertation topics show

Research and Implementation of Distributed Web Crawl Based on Hadoop Architecture

Author: ChengJinJia
Tutor: PengYong
School: Beijing University of Posts and Telecommunications
Course: Computer Science and Technology
Keywords: Cloud Computing Distributed reptiles Hadoop HDFS MapReduce Search engine
CLC: TP391.3
Type: Master's thesis
Year: 2010
Downloads: 1417
Quote: 7
Read: Download Dissertation

Abstract


Today , cloud computing has become an important trend _ future development . Currently , the world's major IT companies such as Google, IBM, Microsoft , Amazon and others have built their own cloud computing platform . In view of this , the lab has also developed a long-term \This project is part of the long-term plan is developed on the basis of the Hadoop Distributed File System HDFS distributed computing framework MapReduce distributed search engine . It is on this basis that explores the reptilian part of the search engine , which is based on a distributed computing framework distributed reptiles . This paper work including cloud computing , Hadoop distributed platforms , network reptiles principle of three main parts. Part of the cloud computing , the departure from the cloud computing architecture , cloud computing service level and technical level . Then , for the technical basis for Hadoop Distributed platform , this article on its two core technical points . Distributed file system HDFS distributed computing framework MapReduce -depth study . At the same time , the article discusses the basic search engine technology and the basic principles of the web crawler , and distributed the reptiles technology prototype Nutch parsing . On the basis of research work , we propose a specific design for distributed reptiles , determine a distributed crawler system layout , the module is divided and distributed process control and reptiles functions . Based on this , the data stored and data format information distributed reptiles design and complete realization of reptiles sub- module specific class . Finally, on the basis of the large cluster of the laboratory , this reptiles system functional testing, scalability testing and performance testing . On the basis of these test data , a detailed analysis of the advantages and disadvantages of the system , and the work of a summary and outlook .

Related Dissertations

  1. Web search engine related technology research,G354
  2. Research and Application of Map/Reduce Based Distributed Log Analyzer,TP311.52
  3. Design and Implementation of Online Shopping Prototype System Based on Hadoop,TP311.52
  4. Design of the Mobile Learning System Based on Hadoop,G434
  5. Cloud Computing in the Application of Privilege Management,TP309
  6. Cloud computing technology and its application in e-commerce logistics center design and operation of the application,F724.6;F252
  7. HADOOP architecture based on the social security project web log analysis system,TP311.52
  8. The Research of Software Service Platform Based on Cloud Computing,TP311.52
  9. Research on Fast Queryalgorithm of Massive Data,TP311.13
  10. An Intrusion Detection System for High-Speed Networks,TP393.08
  11. Incremental Learning Method Based on Cloud Computing,TP311.13
  12. The Research of Text Classification Based on Hadoop,TP391.1
  13. Research and Implementation on a Distributed Service Registry Based on HADOOP Platform,TP393.09
  14. Resarch of Task-level Data Processing Based on Multicore CPU and Test of Its Performance on Cluster Platform,TP274
  15. Hadoop-based video transcoding system design and implementation,TN919.81
  16. Fault Tolerance for MapReduce in the Cloud Environment,TP302.8
  17. Cloud-based mobile data storage backup system,TP309.3
  18. Cloud storage system for mass data,TP333
  19. Massive Video Conversion Platform Design and Implementation Based on Cloud Computing,TP311.52
  20. Generatioin of Induced Pluripotent Stem Cells from Adult Human Dermal Fibroblasts,R329
  21. IaaS cloud computing - based Web application technology research,TP393.09

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Retrieval machine
© 2012 www.DissertationTopic.Net  Mobile