Dissertation > Excellent graduate degree dissertation topics show

Similar pages deduplication algorithm parallelization Research and Implementation

Author: DingYiBin
Tutor: LiRuiXuan
School: Huazhong University of Science and Technology
Course: Applied Computer Technology
Keywords: Web collection Approximate repeat Mapping / Statute Fingerprint Hamming distance
CLC: TP393.092
Type: Master's thesis
Year: 2009
Downloads: 115
Quote: 0
Read: Download Dissertation

Abstract


Because Web mirroring and network reproduced plagiarism, complete duplication and the approximate duplicate pages of data for the current search engine produced a series of problems: it not only increases the amount of storage data indexing web pages to the search engines and search services brought heavy burden , meanwhile , filled with duplicate content search engine search results also give users a poor user experience . So for the search engines , you need to use efficient detection algorithms to find similar pages and remove duplicate pages approximation , reduce web crawlers and search engines burden. Search engine technology has been rapid development in recent years , similar to the weight for Web site data collection is essential. For similar pages to heavy systems, the need to identify the topic of the page content blocks , such as advertising pages at the same time removing noise such content. Web text data according to the dictionary for segmentation algorithm based on Shingle pages feature vectors extracted using Simhash algorithm for computing represents the feature vector web pages feature fingerprints. Possess the characteristics of the fingerprint is that if two web pages with similar content , then it will have a smaller fingerprint Hamming distance . Moreover , the proposed algorithm is based on the traditional Shingle Mapping / parallel protocol model improvements and verified by experiment . Web page gives a practical framework and collection system in which similar site to re- design of the prototype system implementation. Similar pages deduplication prototype system has two operating modes : the first one for the online mode , in this mode , a single page of the same mass fingerprint fingerprint data for historical comparison ; second for parallel processing mode , this mode , a group of web pages of the same mass fingerprint fingerprint data for historical comparison, unlike the former , the massive historical fingerprint data is divided into specific data block is stored in a distributed computing platform, two batches of fingerprint data Hamming distance calculation process will use the mapping / protocol parallel programming model for processing. After experimental verification , the use of parallel processing prototype system effective solution to weight problems similar pages , and to achieve a higher efficiency and accuracy .

Related Dissertations

  1. Study on the Pharmacokinetics of Xiaoaiping and Chlorogenic Acid in Rats and the Quality Standard of Xiaoaiping Preparations,R285
  2. Studies on the Commodity Survey and Quality Evaluation of Radix Paeonia Rubra,R282.71
  3. Quality Standard Revise and Finger Print Study of Pipazhike Granule,R286.0
  4. Drug Susceptibility Testing of Staphylococcus Aureus from Children Impetigo and Typing of Staphylococcus Aureus by Randomly Amplified Polymorphic DNA in Chengdu,R440
  5. Research and Design on Asymmetric Encryption and Decryption Algorithm Based on FFTT,TP309.7
  6. Research on Pressure Sensed Fingerprint Identification System and Algorithem,TP391.41
  7. Research on Methods of Fingerprint Image Segmentation,TP391.41
  8. The Research and Implementation of Iris Recognition Algorithm,TP391.41
  9. Design and Implementation of family planning management system in Yongkang,TP311.52
  10. The Technology of Browser Security Access and Fingerprint Identification,TP393.092
  11. Research on Multi-modalbiometric Fusion Personalidentification Technology Based on Fingerprint and Name Speech,TP391.41
  12. Constructing DNA Fingerprint Database of Hybrid Rice Cultivars by SSR Markers,S511
  13. Drug on the chemical composition of dried ginger - galangal volatile oil and Salvia miltiorrhiza fingerprinting studies,R284
  14. The treating diabetes chemical composition and quality of research,R284
  15. Research of Fingerprint Preprocessing and Recognition Algorithm,TP391.41
  16. Fingerprint recognition algorithm research related,TP391.41
  17. Study on the Detection of the Fumet in Brewed Vinegar,TS264.22
  18. Studies on Genetic Diversity of Germplasm Resources in Yuexuan Series Agrostis Stolonifera L. Based on RAPD and SRAP Molecular Marker,S688.4
  19. SSR Analysis of Sugarcane Parents and Construction of DNA Fingerprinting,S566.1
  20. Studies on Fingerprint and Correlation between Its Chroma-togram and Antibiotic Effect of the Flower of Lonicera Japonica Thunb,R284
  21. Studies on Dynamic Change of Chemical Components of Compound Prescriptions of Chinese Medicines with Different Extract Methods,R286

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Computer network > General issues > The application of computer network > Web browser
© 2012 www.DissertationTopic.Net  Mobile