|
The Internet has become the main source to spread and search information with the development of the WWW’s technology. People hope that they can get information faster, accurate and proficient. The young surfing the Internet cannot distinguish black sites, or are easily addicted into games. It is necessary to create a healthy web that can search web pages quickly and filter specified sites avoiding the young getting bad information.With web classification can classify web accurately, search topic easily, and mask specific sites, it becomes hot research in mass information mining.Web page classification processes web crawling, text preprocessing, feature dictionary, and classifier training, which needs feature dictionary can accurately and completely cover page’s feature. Therefore this paper put forward a model for generating page feature dictionary.The author has done following works:1. overviewed the technology of page automatic classification, introduces the whole process of automatic classification, and major classification algorithms;2. researched web crawl and Chinese segmentation technology, focus on commonly crawl methods and crawling strategies, Chinese segmentation methods and the problem to disambiguate words, the management of new words and stop-words, and introduced some usually used open project.3. requirement analysis on the web feature dictionary generated model, divided the model into two modules, six sub-modules:crawling module including URL-management sub-module, crawling sub-module, filtering sub-module, segmentation module including preprocessing sub-module, Chinese segmentation sub-module, and calculating sub-module, and analysis every sub-module.4. design every sub-module, and realization them, focus on the crawl sub-module and Chinese segmentation sub-module, and finally provided some code.5. finally test the whole model, using automotive theme’s feature dictionary, and compared it to different theme page’s feature words, and results obtained that the model is reasonable.The model can used into web page automatic classification, search engine, etc. Therefore it has large research value.
|