Dissertation > Excellent graduate degree dissertation topics show

Research and Application of Map/Reduce Based Distributed Log Analyzer

Author: LiuYan
Tutor: PanWei
School: Northeast Normal University
Course: Computer Software and Theory
Keywords: Distributed system Map Reduce Hadoop Performance Optimization I/O Scheduler Filesystem
CLC: TP311.52
Type: Master's thesis
Year: 2011
Downloads: 80
Quote: 0
Read: Download Dissertation

Abstract


In this paper, a MapReduce-Based Framework is implemented to analyze the distributed log generated in cloud computing. The framework is built on top of Hadoop, an open source distributed file system and MapReduce implementation.We first make use of Random Access File to realize an incremental way for aggregating system logs from each node of the monitored cluster, and collect them to the analysis cluster. Then, we integrate the collected logs. After that, we implement a MapReduce-Based algorithm to parser these clustered log files. Furthermore, in order to make the best use of this collected data, a flexible and powerful way is utilized to display monitoring and analysis results.Besides, we quantitatively evaluate and characterize the Hadoop framework through I/O extensive benchmarking, so as to optimize the performance and understand the tradeoffs of system designs for the MapReduce-based data analysis using Hadoop.First, we characterize and evaluate workload performance of I/O intensive benchmarking with different underlying software choices, both on I/O schedulers and native filesystems.Then, we provide some potential enhanced solutions to optimize performance of Hadoop benchmarking, and conclude our experiments in the end.

Related Dissertations

  1. Research of Fault Injection for a Distributed System,TP338.8
  2. The Dynamic Performance Analysis of Oil Pumping Center and Structural Optimization of Improvement,TE933.1
  3. Research and Implementation of Distributed Data Integration Visual Modeling,TP311.52
  4. Design and Implementation of Online Shopping Prototype System Based on Hadoop,TP311.52
  5. Research and Engineering Application of Heat-resistant Sodium Water Glass Adhesive,TQ437
  6. The Research of Software Service Platform Based on Cloud Computing,TP311.52
  7. An Intrusion Detection System for High-Speed Networks,TP393.08
  8. Incremental Learning Method Based on Cloud Computing,TP311.13
  9. A Kernel-Level Intelligent Middleware for Honeypot Filesystem,TP393.08
  10. Hadoop-based video transcoding system design and implementation,TN919.81
  11. Virtual environment multiple network interface card I / O Scheduling System,TP334.7
  12. Fault Tolerance for MapReduce in the Cloud Environment,TP302.8
  13. Cloud-based mobile data storage backup system,TP309.3
  14. Distributed File System Design and Implementation of the client,TP338.8
  15. Cloud storage system for mass data,TP333
  16. Research on Performance Evaluation and Optimization for CPU-GPU Heterogeneous System,TP306.2
  17. Massive Video Conversion Platform Design and Implementation Based on Cloud Computing,TP311.52
  18. Study and Implementation of Performance Optimization SAN Cluster File System,TP316.7
  19. Matching and Optimization of Powertrain for a Hybrid Firefighting Vehicle,U469.68
  20. IaaS cloud computing - based Web application technology research,TP393.09

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer software > Program design,software engineering > Software Engineering > Software Development
© 2012 www.DissertationTopic.Net  Mobile