Dissertation > Excellent graduate degree dissertation topics show

The Performance Optimization and Improvement of MapReduce in Hadoop

Author: HeRongBo
Tutor: LiHui
School: Beijing University of Chemical Technology
Course: Applied Computer Technology
Keywords: Distrubuted computing Hadoop HDFS MapReduce
CLC: TP338.8
Type: Master's thesis
Year: 2011
Downloads: 254
Quote: 1
Read: Download Dissertation

Abstract


Today, the Internet is a data explosion era. People’s work, life and entertainment keep in touch with network tightly. It makes data scale on the internet increase dramatically and enrich the application type. The seemingly chaos of data, in fact, holds enormous business opportunities. As enterprises, future success largely depends on whether it can extract value from the data or not. The coming problem is that data processing ability of single computer can’t meet the current mass data application processing requirements. Distributed computing based on Large-scale computer cluster has been the main route to improve processing performance of future data.Due to the reliable stability, high-efficiency distributed parallel processing ability, easy extension and open source, Hadoop has been the mainstream open source clouds computing platform in just three years. But the development time of Hadoop is relatively short, there is much improvement room. This paper thoroughly analyzes one of the Hadoop’s core technologies, MapReduce computation model. According to the flaws of temporary data management and control which the Map outputs, the optimization and improvement are made. It aims to solve the performance bottleneck generated by the large scale of middle data quantity and imbalance of data distribution when the program is running. Furthermore, it can promote program performance and optimize resource.The main research contents and contribution are as follows:The domestic and overseas cloud computing development situation, application prospect and existing problems are discussed. The distributed systems such as Hadoop Distributed Computing, Grid Computing, Volunteer Computing and so on are distinguished. The paper introduces the background and frame structure of Hadoop platform. The operation mechanism of Hadoop’s two core technologies, HDFS and MapReduce, are researched. After analyzing the read-write process to the data and middle data control of MapReduce, the optimization idea and improvement plan are proposed. Then, they are tested and verified by the specific case. The experimental result suggests the expected objective has been achieved, and the shortcomings of existing framework have been solved.

Related Dissertations

  1. Research and Application of Map/Reduce Based Distributed Log Analyzer,TP311.52
  2. Design and Implementation of Online Shopping Prototype System Based on Hadoop,TP311.52
  3. Design of the Mobile Learning System Based on Hadoop,G434
  4. The Research of Software Service Platform Based on Cloud Computing,TP311.52
  5. An Intrusion Detection System for High-Speed Networks,TP393.08
  6. Incremental Learning Method Based on Cloud Computing,TP311.13
  7. Research and Implementation on a Distributed Service Registry Based on HADOOP Platform,TP393.09
  8. Hadoop-based video transcoding system design and implementation,TN919.81
  9. Hadoop data center deployment and tracking systems research,TP308
  10. Fault Tolerance for MapReduce in the Cloud Environment,TP302.8
  11. Cloud-based mobile data storage backup system,TP309.3
  12. The Research of Distributed Text-based Data Filtering Technology and System Implementation Based on MapReduce,TP391.1
  13. Cloud storage system for mass data,TP333
  14. Massive Video Conversion Platform Design and Implementation Based on Cloud Computing,TP311.52
  15. Generatioin of Induced Pluripotent Stem Cells from Adult Human Dermal Fibroblasts,R329
  16. IaaS cloud computing - based Web application technology research,TP393.09
  17. Parallel Association Rules Algorithm Based on Hadoop,TP311.13
  18. Study on Hadoop-based Inverted Index,TP391.3
  19. Research on Technology of Massive Data Stores Based on Cloud Computing,TP333
  20. Research on the Key Techniques of Massive Image Data Management Based on Hadoop,TP751
  21. Research on Unified Access Plantform for Unstructured Data and Index Technology,TP311.52

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Electronic digital computer (not a continuous role in computer ) > A variety of electronic digital computer > Distributed computer
© 2012 www.DissertationTopic.Net  Mobile