|
With the development of network and Web resources rich Web text information retrieval system to obtain the required information has become an important part of people's daily lives, and users are increasingly concerned about how to more accurately and efficiently find information. Theory and technology of Web information retrieval system are introduced, and a more in-depth practice of information retrieval in the Web text information retrieval applications. In the second chapter, theoretical papers, such as the type of the search engine, the Chinese word segmentation method, the theory of inverted index used in this thesis Lucene theory and use in the third chapter, according to the the site features two page templates analysis algorithms, the first algorithm is based on the longest common subsequence model using dynamic programming method to find the optimal solution, optimized and extended algorithm of the original literature, in order to achieve find the web template string and the purpose of the string is inserted, the second algorithm uses the statistical theory and principle, the page template abstracted into a mathematical model, to extract the common identity of the web page to start position and end position , different variance different calculated according to the length of the body to determine the position of the body in the into the original, which can be extracted from a web page into the contents of the body, in order to achieve the purpose of saving space and reducing the time of the indexing and search index, and finally compare the advantages and disadvantages of the two algorithms. In the fourth chapter, a web spider Java developers, including the processing of heterogeneous data, such as word, pdf, rtf file content extraction, addition of an HTML file parsing method and multi-threaded use. In the fifth chapter, to achieve a web crawling system, the information specified by the user from the Internet to download pages, including content Next, in order to improve the efficiency of information retrieval using Lucene package, the establishment of full-text search to speed up the search speed and improve the accuracy and timeliness of the information, and save a lot of storage space. From a theoretical and practical point of view, both the design and analysis of algorithms, but also specific program implementation, using Oracle, Tomcat, Jsp, Java, Eclipse, Lucene software and language original HTML parsing method for the Users save time, improve work efficiency.
|