Dissertation > Excellent graduate degree dissertation topics show

The Research on Key Techniques for Page Segmentation Based Forum Crawler

Author: ZhangDianFang
Tutor: YeYunMing
School: Harbin Institute of Technology
Course: Computer Science and Technology
Keywords: web crawler forum page segmentation crawling strategy document structure model
CLC: TP393.092
Type: Master's thesis
Year: 2009
Downloads: 153
Quote: 0
Read: Download Dissertation

Abstract


Along with Web2.0 technology development, as a typical application of user-created content, Web forums are very popular in the world. The countless topics or issues of all in web pages has been created by Internet users, so the forum has become the bearer data is actually a huge data set of human knowledge. As the network increases the amount of information in the forum, the forum website to collect accuracy and efficiency are faced with enormous challenges. In this situation, we urgently need to develop an effective forum crawler system.This forum crawler around the cutting-edge technology, through in-depth study of the characteristics of Internet forums, web crawler works and related technologies, based on the idea to block the introduction of web pages to the forum, crawlers crawling strategy, as well as in various forums how to crawl on the different forums and servers in this page. The main results can be summarized in the following areas:In this paper, a great deal of research on the internet forums and in-depth analysis, based on summarized the basic characteristics of a forum; and then the problems of the existing crawlers crawling forums were analyzed to identify the root causes that caused these problems; in order to solve these problems, the subject crawlers crawling strategies inspired by the ideas we introduced to the page segmentation in the forum crawler, and some ways to optimize the forum crawler. In this paper, we study some page segmentation algorithms and propose a good page segmentation algorithm -- based on the web page structure of vertical segmentation algorithm (WPS-VSA). The experimental results on several forums show that this algorithm has a better general, but also on the forum pages have a high precision of the page segmentation.This paper studies a number of web crawler crawling strategies, and then proposes a general forum crawling algorithm for the majority of Internet forums -- based on the web page segmentation of forum crawling algorithm (WPS-FCA). We use this algorithm can filter invalid page online links; while taking advantage of the characteristics of the forum solving the page-flipping problem, for those who use the forum web content data mining application has laid a good foundation. The experimental results show that this algorithm not only saves the network bandwidth to download the forum pages and storage space for web pages, but also greatly increased the forum crawl pages accuracy ratio and coverage ratio, but also facilitates a variety of applications using the data content for the forum.Based on the above results of theoretical studies, this paper designs and implements a prototype forum crawler system.

Related Dissertations

  1. A Research on New China’s Cultural Diplomacy in Africa,D822
  2. Network -based mind mapping discussion teaching research,G434
  3. Traditional media agenda and the agenda of comparative research network,G206
  4. Web-based forum for public dissemination of research events,G206
  5. Based WebHarvest the Chinese financial news search engine design and implementation,TP311.52
  6. Forum message text for effective data extraction studies,TP391.1
  7. Forum posts for the text orientation analysis,TP391.1
  8. The Research about the Operation and the Practice of Chinese E-Governance,G206.3
  9. On the academic diplomacy forum -type carrier,D820
  10. Multilateralism and security cooperation in Northeast Asia,D73
  11. Topic-based Web Information Monitoring System Strategy and Implementation,TP393.09
  12. Functional Genre Analysis of Internet Academic Forum,H052
  13. The Difference Research of Government Issue between E-government Public Forum,D035
  14. The network era correct guidance to high school students interested in politics hotspots research,G631
  15. Internet forums discussing the status of public sector research,G206
  16. Research on Data Acquisition and Topic Analysis of Online Public Opinion,TP393.09
  17. Research on Information Extraction Method for Retrieval Result Pages of OA Journals,TP393.092
  18. Vertical Search Engine Technology in Network Control in the Patrol Public Opinion Research and Application,TP391.3
  19. Research on Algorithms of Document Image Processing and Form Image Identification,TP391.41
  20. Research and Implementation on Topic Search Oriented to Enterprise Competitive Intelligence,TP391.3
  21. Gender -based computer-mediated language forum,H315

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Computer network > General issues > The application of computer network > Web browser
© 2012 www.DissertationTopic.Net  Mobile