Dissertation > Excellent graduate degree dissertation topics show

Join Processing and Optimizing on Large Data Sets Based on Hadoop Framework

Author: SunHui
Tutor: WangXiaoJun
School: Nanjing University of Posts and Telecommunications
Course: Applied Computer Technology
Keywords: Join Cloud Computing MapReduce Hadoop HDFS Bit-map DistributedCache Shuffling Strategy
CLC: TP311.13
Type: Master's thesis
Year: 2013
Downloads: 717
Quote: 0
Read: Download Dissertation

Abstract


Data analysis is an important functionality in cloud computing which allows a huge amount ofdata to be processed over very large clusters. MapReduce is recognized as a popular way to handledata in cloud environment due to its excellent scalability and good fault tolerance. However becauseof its own limitation, the performance of MapReduce is slow when it is adopted to perform complexdata analysis tasks that require the joining of data sets in order to compute certain aggregates.First, through the analysis of the shortage of a general two-way join algorithm--RSJ, aoptimization algorithm is proposed which is based on DistributedCache. The idea of this optimizationalgorithm is preprocessing the data before using RSJ algorithm. Preprocessing refers extractingconnection attribute values from one of join table and compressed into smaller "background" datastorage to a small file by Bit-map, and then transmission this small file to all nodes through theDistributedCache mechanism. At this time, if using RSJ can filter out much data in other table whichnot satisfy the connection condition by "background" data at Map stage. Thus reducing the outputdata from mapper to achieve the effect of optimization.After, because the one-to-one shuffling scheme, MapReduce need divide multiway join tasksinto a sequential subtasks which frequently checkpoints and shuffles intermediate results inintroducing a huge I/O overhead. Here, a new shuffling scheme which is one-to-many shufflingstrategy is used. This new strategy help us only performing one MapReduce task can completemultiway join tasks.Finally, in order to verify the above two kinds of optimization algorithm, we have doneextensive experiments on top of Hadoop platform. From the experimental results,we can see that thetwo optimal methods could impove join performance effciently using MapReduce framework.

Related Dissertations

  1. The Research of Dynamic Trust Model on Cloud Computing Platform,TP309
  2. Higher Engineering Education and Vocational Technical Education Relations,G649.2
  3. Research and Implementation on Model of Educational Knowledge Service System Based on Eucalyptus,TP393.09
  4. The Research and Implementation of Cloud Network Experiment Platform,TP393.09
  5. The Tae Kwon Do Moves Towards the Internationalization to the Chinese Martial Arts the Inspiration,G852
  6. Cloud-based digital library service model,G250.76
  7. Method Study of Team Safety Synergy Management of Zhaoguan Coal Mine,TD791
  8. A Study on Collaborative Education Strategy in Primary School under Network Environment,G629.2
  9. Regional joint prevention and control of air pollution research,X51
  10. Research and Application of Map/Reduce Based Distributed Log Analyzer,TP311.52
  11. The Research on Cloud Computing for the Dynamic Fuzzy Measurement Method,TP274
  12. Cloud-based Library Information Services Research,G252
  13. The Crawler of Education in Web by Cloud Computing,TP391.3
  14. Research and Development of Client Applications Based on Cloud Computing,TP311.52
  15. Design and Implementation of Online Shopping Prototype System Based on Hadoop,TP311.52
  16. Computation on Fuzzy Riemann Integral of Type Ⅱ and Order Structure on Ordered Weighted Geometric Operators,O177
  17. Design of the Mobile Learning System Based on Hadoop,G434
  18. Cloud Computing in the Application of Privilege Management,TP309
  19. Research on Task Scheduling Strategy of Cloud Computing Based on MPSO Algorithm,TP3
  20. Based on the study of resource management in the cloud environment of credibility,TP315
  21. Cloud computing technology and its application in e-commerce logistics center design and operation of the application,F724.6;F252

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer software > Program design,software engineering > Programming > Database theory and systems
© 2012 www.DissertationTopic.Net  Mobile