THUCTC (THU Chinese Text Classification) is a Chinese text classification toolkit launched by the Natural Language Processing Laboratory of Tsinghua University, which can automatically and efficiently implement user-defined text classification corpus training, evaluation, and classification functions. Text classification usually includes three steps: feature selection, feature dimensionality reduction, and classification model learning. How to select appropriate text features and reduce dimensionality is a challenging problem for Chinese text classification. Based on years of research experience in Chinese text classification, my group selected two-character string bigram as the feature unit in THUCTC, the feature reduction method is Chi-square, the weight calculation method is tfidf, and the classification model uses LibSVM or LibLinear. THUCTC has good universality for long texts in the open field, does not depend on the performance of any Chinese word segmentation tools, and has the advantages of high accuracy and fast test speed.

Data Collection

THUCNews is generated by filtering and filtering historical data of Sina News RSS subscription channels from 2005 to 2011. It contains 740,000 news documents (2.19 GB), all in UTF-8 plain text format. On the basis of the original Sina news classification system, we re-integrated and divided 14 candidate classification categories: finance, lottery, real estate, stocks, home furnishing, education, technology, society, fashion, current affairs, sports, horoscope, games, entertainment. Using THUCTC toolkit to evaluate on this data set, the accuracy rate can reach 88.6%.