Dissertation > Excellent graduate degree dissertation topics show

N-gram Language Model Based on Distributed System

Author: ShenZhenZhong
Tutor: WangKaiDong
School: Xi'an University of Electronic Science and Technology
Course: Computer System Architecture
Keywords: Distribute Language Model Smoothing Methods HDFS MapReduce Hadoop Hbase
CLC: TP391.1
Type: Master's thesis
Year: 2011
Downloads: 23
Quote: 0
Read: Download Dissertation

Abstract


With progress of natural language processing technology and the emergence oflarge-scale corpus, large-scale language model training become a reality. N-gramlaguange model(LM) is a important tool in many research areas of natural languageprocessing,such as information retrieval,machine translation,speech recognition，etc.Using higher order models and more training data can significantly improve theperformance of applications.Therefore,the research on large scale language modelingdraws more attention.However,for limited resources of PC,can not be a large-scalelanguage modeling,so it is necessary to use a distributed system for modeling.This theis presents the work of building a large scale distributed n-gram languagemodel using a MapReduce platform named Hadoop and a distributed database calledHbase.We give a method focusing on the time cost and storage size of themodel,exploring different Hbase table structures and compression approaches. Themethod is applied to build up training and testing processes using Hadoop MapReduceframework and Hbase. The experiments evaluate and compare five different databasetable structures on training40million words for unigram, bigram and trigram models,and the results suggest a table based on half ngram structure is a good choice fordistributed language model. The results of this work can be applied and furtherdeveloped in machine translation and other large scale distributed language processingareas.

Related Dissertations

  1. Research on Structure Transition Technology for SMT,TP391.2
  2. Word Sense Disambiguation Corpus Automatic Acquisition,TP391.1
  3. The Research and Implement of Chinese Input System Based on Dynamic Self-adaptation Language Model,TP391.14
  4. Research and Application of Map/Reduce Based Distributed Log Analyzer,TP311.52
  5. Design and Implementation of Online Shopping Prototype System Based on Hadoop,TP311.52
  6. Design of the Mobile Learning System Based on Hadoop,G434
  7. Sliding Mode Control and Stability for Distributed Parameeter System with Multiple Time-delays,O231
  8. HADOOP architecture based on the social security project web log analysis system,TP311.52
  9. 3D Mannequins Generating Engine Based on eMTM with MapReduce,TP391.41
  10. The Research of Software Service Platform Based on Cloud Computing,TP311.52
  11. Research on Fast Queryalgorithm of Massive Data,TP311.13
  12. An Intrusion Detection System for High-Speed Networks,TP393.08
  13. Incremental Learning Method Based on Cloud Computing,TP311.13
  14. Cloudqueue: An Internet-Scale Messaging Infrastructure Based on Hadoop,TP311.52
  15. The Research of Text Classification Based on Hadoop,TP391.1
  16. Research and Implementation on a Distributed Service Registry Based on HADOOP Platform,TP393.09
  17. Design and Implementation of the WEB Proxy for Mass Storage System,TP393.05
  18. Resarch of Task-level Data Processing Based on Multicore CPU and Test of Its Performance on Cluster Platform,TP274
  19. Hadoop-based video transcoding system design and implementation,TN919.81
  20. Hadoop data center deployment and tracking systems research,TP308
  21. Fault Tolerance for MapReduce in the Cloud Environment,TP302.8

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Text Processing
© 2012 www.DissertationTopic.Net  Mobile