Dissertation > Excellent graduate degree dissertation topics show

Research on Scientific Data De-duplication Model in Real-Time Data Warehousing Environment

Author: DingHui
Tutor: BaoYuBin
School: Northeastern University
Course: Applied Computer Technology
Keywords: data de-duplication scientific data architecture real-time data warehousing scheduling strategy
CLC: TP311.13
Type: Master's thesis
Year: 2008
Downloads: 21
Quote: 0
Read: Download Dissertation

Abstract


In the information age, the correct decision is becoming to a powerful weapon for competition. To make the strategic business plans, process and the business decisions, companies and organizations are building their data warehouses. However, because data warehouse gets large amounts of data from a variety of data sources and the probability that some data sources include "dirty" data is very high. On the other hand, for data warehouses are used to make decision, data quality is crucial to avoid the wrong decision-making. Duplicate data is an important factor that affects the quality of data. The duplicate data will not only cause redundancy, increasing the amount of data and the burden of data warehouse, but also affect the analysis and decision-making seriously. Therefore, in a data warehouse, data de-duplication is one of the indispensable ways to improve data quality.Real-time data warehouse (RTDWH) is a new research direction of data warehouse technology. It refers to any changes of a data source will be automatically and immediately propaganded into the data warehouse. The development of RTDWH has also brought new challenges to guarantee the data quality. The changes of data sources of the RTDWHs are reflected to the data warehouse immediately, which is done by real-time ETL, which requires guaranteeing the quality of data to support the query and analysis at a real-time level. But the previous researches on data quality are mostly based on traditional data warehouse. Therefore, we need a new scheduling approach to ensure the credibility of the data in data warehouse more accurately and efficiently.This thesis presents a general de-duplication model for scientific data, introduces the characteristics of scientific data, gives a detail description of the de-duplication model, and proves that the traditional method of "sort and merge" isn’t suitable for scientific data and shows the detail of algorithm of the model, and introduces the scheduling process and the architecture of the model. And then, it analyses the difficulties and problems of data quality assurance in the real-time data warehouse, presents de-duplication priority scheduling strategy, real-time scheduling strategy and ETL priority scheduling strategy in the real-time environment, and gives specific analysis for each. In the ETL priority scheduling strategy, evaluate indexes as De-duplication Busy Degree, Cumulative Delay are defined, and scheduling strategy based on time and Scheduling strategy based on events are presented, which apply SD2M in real-time data warehouse. Finally, the results of experiments show the high efficiency and stability of the model, and the scheduling strategy in real-time environment presented by this thesis is reasonable.

Related Dissertations

  1. Sleep Architecture in Acute Stroke and Correlation Factor,R743.33
  2. Independent college teaching system based on career planning,G642.4
  3. A Study on the Curriculum Reform of Higher Vocational Education,G712.3
  4. Research on Traceability and Incremental Consistency of the MDA Model Transformation,TP311.5
  5. Research and Design of Web Reports Based on Service-Oriented Architecture,TP393.09
  6. ECG Detection Analyzer Based on Windows xp Embedded System,R197.39
  7. Research on Human Erector Spinae about the Muscle Architecture and the Muscle Spindles,R322
  8. Reform and opening up three decades PEP high school Chinese teaching materials Review and Reflection,G633.3
  9. Based on cloud model software trustworthiness evaluation model,TP311.53
  10. Southern Jiangxi Hakka Architecture Aesthetics Analysis,TU-80
  11. The Study on Architecture Color in Tibetan Areas,TU115
  12. Materials for Building Regional Factors Influence,TU-0
  13. Research and Development on Wireless Transmission-based Public Transport Vehicle Media Asset Management Systems,TP311.52
  14. Resourse Oriented Architecture Application Research Based on JAX-RS,TP393.09
  15. Research on Construction of iSchool Platform Environment Based IA,TP393.09
  16. Wright Garden Hotel Deyang Management System Design and Implementation,TP311.52
  17. The Reflect of the Traditional Implication in the Modern Architecture Design,TU-0
  18. Research on Video Compression Algorithm Based on Multi-core Computing Platform,TN919.81
  19. Research on Parallel Computation of H.264 Based on CUDA,TN919.81
  20. The Design and Implementation of a Web Based Affairs Management System,TP311.52
  21. Educational Administration and Implementation of the System,TP311.52

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer software > Program design,software engineering > Programming > Database theory and systems
© 2012 www.DissertationTopic.Net  Mobile