Dissertation > Excellent graduate degree dissertation topics show

MapReduce Performance Research and Optimization Based on Block Aggregation

Author: LiJun
Tutor: AiLiHua
School: Beijing Jiaotong University
Course: Computer Science and Technology
Keywords: Cloud Computing GFS Hadoop HDFS MapReduce BlockAggregation
CLC: TP302.7
Type: Master's thesis
Year: 2014
Downloads: 45
Quote: 0
Read: Download Dissertation

Abstract


ABSTRACT:With the rapid development of computers and Internet technologies, Cloud Computing technology emerges as the times require. Because that big data cannot be stored in a single computer and traditional serial processing needs a lot of time overhead, how to efficiently process big data has been a problem which urgently needs to be solved. As a computational model supporting for distributed and parallel big data processing, MapReduce has been widely adopted in many data-intensive application fields, such as machine learning, data mining and scientific computing. Hadoop is an open source implementation of MapReduce computational model, which has been used in the data sets of search logs and user access logs through the way of data mining by a lot of enterprises like Yahoo, Amazon and Facebook. Although the use value of Hadoop has obtained everyone’s approval, it still has many problems whose performance needs further improvements.The core components of Hadoop include Hadoop Distributed File System (HDFS) and MapReduce computational framework, which are open source implemented versions of Google File System (GFS) and MapReduce. Through deeply researching and practicing in HDFS and MapReduce, this paper discusses the problem that sharing clusters of Hadoop cannot guarantee jobs which have all kinds of data size run efficiently. When a sharing cluster of maintains the parallel performance of jobs which have less data size, jobs which have large data size in this cluster can produce many map tasks. The above situation brings much pressure to the master of cluster and costs more resources when initializing map tasks. Combining the Hadoop Distributed File System (HDFS) with the distributed and parallel computing framework (MapReduce), this paper proposes, the adaptive splitting algorithm based on block aggregation, which calculate the split size accordance to the actual data size, the number of data files and computing resources which jobs can use. This mechanism of cluster task distribution makes the data group a split and stored in the same node. This algorithm ensures the parallelism of jobs with different data size, and appropriately reduces the number of map tasks of jobs which haves a large amount of data. It can reduce the cost of initializing tasks and pressure of masters, and improved running performance of clusters effectively.The current Hadoop implementation assumes that computing power of all the nodes in a cluster is same. Data locality has not been taken into account for launching speculative map tasks, because it assumes that most maps are data-local. Unfortunately? both the homogeneity and data locality assumptions are not satisfied in Heterogeneous Hadoop clusters. The native Hadoop fails to guarantee high performance in such environment. In this paper, we also launche discussions on this problem and propose an optimization scheme to improve the overall performance under the Heterogeneous environment. In this scheme, master of the cluster computes a split size according to the computing capacity of nodes and the real input size, and stores the data in the range of a split to the same nodes, which let each node has a balanced data processing load. Experimental results on the real application show that our optimization scheme on Heterogeneous Hadoop clusters can run computational tasks more efficiently.

Related Dissertations

  1. The Research of Dynamic Trust Model on Cloud Computing Platform,TP309
  2. Research and Application of Map/Reduce Based Distributed Log Analyzer,TP311.52
  3. Design and Implementation of Online Shopping Prototype System Based on Hadoop,TP311.52
  4. Design of the Mobile Learning System Based on Hadoop,G434
  5. HADOOP architecture based on the social security project web log analysis system,TP311.52
  6. 3D Mannequins Generating Engine Based on eMTM with MapReduce,TP391.41
  7. The Research of Software Service Platform Based on Cloud Computing,TP311.52
  8. Research on Fast Queryalgorithm of Massive Data,TP311.13
  9. Web application system design based on Google's cloud computing platform and,TP393.09
  10. An Intrusion Detection System for High-Speed Networks,TP393.08
  11. Research on Cloud Computing Security Based on Combned Public Key Cryptosystem,TP309
  12. Incremental Learning Method Based on Cloud Computing,TP311.13
  13. Digtital Image Processing Platform Baseed on Cloud Computing,TP391.41
  14. Research of Resource Provisioning Technique in Telecom Business Supporting Based on Cloud Computing Platform,TP3
  15. Cloudqueue: An Internet-Scale Messaging Infrastructure Based on Hadoop,TP311.52
  16. The Research of Text Classification Based on Hadoop,TP391.1
  17. Research and Implementation on a Distributed Service Registry Based on HADOOP Platform,TP393.09
  18. The Cloud Computing Strategy of Small and Medium一Sized E-Commerce Enterprice,F276.3
  19. Load Balance Approach to Save Power on Cloud Datacenter,TP308
  20. Resarch of Task-level Data Processing Based on Multicore CPU and Test of Its Performance on Cluster Platform,TP274
  21. Hadoop-based video transcoding system design and implementation,TN919.81

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > General issues > Design and Performance Analysis > Performance analysis, functional analysis
© 2012 www.DissertationTopic.Net  Mobile