Dissertation > Excellent graduate degree dissertation topics show

Study on Feature Selection and Feature Weighting of Chinese Text Classification

Author: WangYong
Tutor: ZhangYuFang
School: Chongqing University
Course: Computer System Architecture
Keywords: Text Classification Imbalanced Data Set Feature Selection FeatureWeighting
CLC: TP391.1
Type: Master's thesis
Year: 2012
Downloads: 126
Quote: 0
Read: Download Dissertation

Abstract


With the rapid development of information technology and Internet, electronicdocument information has increased dramatically, text classification becomes the keytechnique on organizing and processing large scale of texts. But the text data has aobvious natural attribute that the data is often imbalanced, that is a great disparity of thenumber between categories and negative class(big class) may be hundreds of times thanpositive class(small class). This problem is easy to cause the classifier to be moreinclined to the negative class while ignoring the positive class, the document whichbelongs to the positive class may be wrongly assigned to the negative class. Thisproblem leads to lower classification accuracy of the positive class, and affect theperformance of the classifier in the end. At present, the imbalanced data classificationproblem has already become a hot spot in the field of data mining.There are three reasons why the classifier tends to the negative class while ignoringthe positive class on the imbalanced data set. Firstly, the imbalanced class distribution isquite common in many real applications of automatic text classification. Secondly, thedefects of classification algorithm make the classifier not suit to the imbalanced data set.Thirdly, The existing feature selection and feature weighting methods are more inclinedto the features in the negative class. About the first two parts, there many algorithmshave been discussed, but the research on the methods of feature selection and featureweighting is not sufficient. Therefore, to seek some effective feature selection andfeature weighting methods which could be adapt to the relatively balanced orimbalanced data sets is the key problem on text classification.Firstly, aiming at the shortage of Information Gain ignores the term frequencydistribution of feature inside class and document distribution of feature among classes, afactor which measures term frequency and document distribution of features isintroduced for distinguishing features of strong correlation with class, and aiming at theproblem that Information Gain is inclined to the negative features on the imbalanceddata set, a fator is introduced to reduce the contribution of these features. Then,considering the feature distribution characteristics in the positive and negative classes,synthesizes four measures on the category distinguishing ability of features, a newfeature selection method based on the distribution ratio of features is proposed. Finally,aiming at the problem that TF-IDF feature weighting method does not consider the feature distribution characteristics between positive class and negative class, whichleads the classifier to assign the larger weight to rare features and smaller weight to thefeatures with better ability to distinguish class, an improved TF-IDF formula isproposed.In order to verify effectiveness of the feature selection methods are proposed andthe improved formula of TF-IDF, the experiments using relatively balanced data set andimbalanced data set on the chinese text classification experiment platform have beentaken. The results on different data sets show that the feature selection methods areproposed have achieved better effect for reducing dimension, while the improvedformula of TF-IDF achieved better effect than the TF-IDF, the performance of theclassifier is improved.

Related Dissertations

  1. Research on Text Classification Based on Biomimetic Pattern Recongnition,TP391.1
  2. Tourism Comments on the Internet’s Semantic Analysis and Usefulness Research,TP391.1
  3. Chinese folk music feature extraction and classification technology research,J607
  4. Research on Feature Selection and Classification of Epistatic Gene,Q78
  5. Research on Feature Selection for Gene Expression Data,Q78
  6. The Research and Implementation of Web Page Classification in Enterprise Search Engine,TP393.092
  7. Design and Implementation of the Intelligent Mail-head Analysis Mechanisms Based on LDAP,TP393.098
  8. Research of Key Technology of Anomaly Network Intrusion Detection Based on SVM,TP393.08
  9. Misuse Intrusion Detection Based on Weighted Feature Selection,TP393.08
  10. Hot Topic Detection and Hotness Evaluation from Financial BBS,TP393.094
  11. The Research of Intrusion Detection Based on Feature Selection,TP393.08
  12. Chinese Spam Filtering Based on SVM,TP393.098
  13. Web Spam Detection Based on Combination of Multiple Classifiers,TP18
  14. Application Research of Genetic Algorithm in Supporting Marketing of Forestry Enterprise,TP18
  15. SAR Images Segmentation Based on Quantum Evolution Feature Selection Algorithm,TN957.52
  16. A Feature Extraction Method Using Base Phrase and Keyword,TP391.1
  17. Research on Web Text Categorization Technology Oriented to Information Service,TP391.1
  18. Support Vector Classifier Machine Based on Feature Analysis,TP311.13
  19. The Research and Implementation of Text Classification Based on Meta-Information and Optimization,TP391.1
  20. Sentiment Classification by Combining Lexicon-based and Machine Learning Methods,TP391.1
  21. Web Knowledge Service Oriented of Medical Information Classification Approach,TP391.1

CLC: > Industrial Technology > Automation technology,computer technology > Computing technology,computer technology > Computer applications > Information processing (information processing) > Text Processing
© 2012 www.DissertationTopic.Net  Mobile