Dissertation > Excellent graduate degree dissertation topics show

The Research and Practice of Performance Optimization Based on Hive

Author: YeWenZuo
Tutor: ZhengTao
School: Nanjing University
Course: Engineering
Keywords: Large - scale data processing Parallel computing Distributed computing systems HDFS HADOOP Hive Optimization
CLC: TP311.13
Type: Master's thesis
Year: 2011
Downloads: 367
Quote: 2
Read: Download Dissertation

Abstract


With the dramatic growth of the rapid expansion of the market of network information processing and information processing amount of data , the distributed computing processing technology for large-scale and ultra large scale data engineering research topic attracting much attention . , Engineering research and development community is generally concerned about the Hadoop technology can be widely used in the Internet field , domestic universities and research institutes Hadoop data storage , resource management , job scheduling , performance optimization , high availability and security features conducting in-depth analysis . In this paper, the demand for Taobao e-commerce systems , storage processing transaction data and applications , system Hadoop - based data warehouse infrastructure - Hive performance optimization method . This paper first gives the definition of the the Hadoop system calculates characteristics , Hadoop Map / Reduce task allocation and file storage system HDFS . Divided into three levels to achieve the purpose of tuning leads to executable optimization method by introducing the analysis of the underlying parameters and change the value of parameters to achieve the effect . Describe and analyze the similarities and differences of a common database with distributed data processing based on the the the Hadoop system infrastructure Hive SQL-like language HQL . Typical code case , expounded the Hive 's logical characteristics , leads to the corresponding optimization direction and parameter adjustment program , respectively, from the data type of problem to solve data tilt problem with the internal optimizations to reduce the Job , combined with Taobao data demand characteristics such as a few aspects , modify the code script optimization data . The measured data show that the optimization algorithm has achieved good results .

Related Dissertations

  1. Development of the Platform for Compressor Optimization Design and Aerodynamic Optimization Design in the Transonic Compressor,TH45
  2. Research on Feature Extraction and Classification of Tongue Shape and Tooth-Marked Tongue in TCM Tongue Diagnosis,TP391.41
  3. Optimization of Fermentation Conditions, Purification, Cloning and Expression of a Cold-active Lipase from Pseudomonas Sp.RT-1,TQ925
  4. Finite Element Analysis and Optimization Design of a Tractor Cab,S219.02
  5. Research on Subsea Pipeline Repair Coupling,TE973
  6. Study of Ventilation System Optimization and Application on Merged Mine,TD724
  7. Chemical Construction Company Human Resource Management,F426.7
  8. Cao Mine Ventilation Optimization of multi-regional,TD724
  9. Based on Genetic Algorithm Pishihang irrigation canal water allocation marshalling model of,S274
  10. Mining resources based on genetic algorithm optimization model of,O224
  11. Qinghai Oilfield optimal allocation of production costs,F406.72
  12. Nonlinear Structural Dynamics Analysis and Optimization Design on Naval Gun Driver,TJ391
  13. Optimization of Zhenqing two standard K29 450 to K29 900 paragraph excavation slope stability analysis and reinforcement,U418.52
  14. The Structure Design and Toplogical Optimal Analysis for the Key Components of Tunnel Type Washer,TM925.33
  15. Segmentation of cDNA Microarray Image Using Fuzzy C-means Algorithm Optimized by Particle Swarm,TP391.41
  16. Research on the Milling Performance and Parameters Optimization with Large Parts of Heavy Machine,TG54
  17. Research and Application of Algorithms for CT Cerebral Perfusion,R811
  18. Research of Medical Image Registration Method Based on DTCWT and NPSO,TP391.41
  19. Study of Medical Image Registration Methods Based on Multi-features and Artificial Immune Algorithm,TP391.41
  20. Reactive Power Optimization Based on Modified PSO,TP301.6
  21. Analysis of American School-selecting System in Basic Education Stage,G639.712

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer software > Program design,software engineering > Programming > Database theory and systems
© 2012 www.DissertationTopic.Net  Mobile