Massih-Reza Amini
Universit?? Joseph Fourier
Laboratoire d'Informatique de Grenoble
Email : Massih-Reza.Amini '@' imag.fr

Cyril Goutte
National Research Council Canada
Interactive Language Technology group
Email : Cyril.Goutte '@' nrc.ca

Data Set Information:

Uncompressing rcv1rcv2aminigoutte.tar.bz2 will create a directory  that contains 5 subdirectories EN, FR, GR, IT and SP, corresponding to the 5 languages. Each subdirectory in {EN, FR, GR, IT, SP} contains 5 files, each containing indexes of the documents written or translated in that language.  For example, EN contains files:
- Index_EN-EN : Original English documents
- Index_FR-EN : French documents translated to English
- Index_GR-EN : German documents translated to English
- Index_IT-EN : Italian documents translated to English
- Index_SP-EN : Spanish documents translated to English

And similarly for the 4 other languages.

Each file contains one indexed document per line, in a format similar to SVM_light.  Each line is of the form:

Attribute Information:

We focused on six relatively populous categories: C15, CCAT, E21, ECAT, GCAT, M11. For each language and each class, we sampled up to 5000 documents from the RCV1 (for English) or RCV2 (for other languages). documents belonging to more than one of our 6 classes were assigned the label of their smallest class.  This resulted in 12-30K documents per language, and 11-34K documents per class. The distribution of documents over languages and classes are:

             Number of                   Vocabulary
Language      documents     percentage       size
************  **********   ************  ************
English        18,758         16.78        21,531
French         26,648         23.45        24,893
German         29,953         26.80        34,279
Italian        24,039         21.51        15,506
Spanish        12,342         11.46        11,547
-------
Total         111,740

The distribution of classes in the whole collection is
          Number of                
Class      documents     percentage  
*********  **********   ************
C15          18,816         16.84    
CCAT         21,426         19.17    
E21          13,701         12.26        
ECAT         19,198         17.18        
GCAT         19,178         17.16        
M11          19,421         17.39


In experiments that we conducted in cite{AUG09}, we considered each document available in a given language as the observed view for an example and all translated documents were used as the other views for that example, generated using Machine Translation. Results shown in this study were averaged over 10 random samples of 10 labeled examples per view for training, and 20% of the collection for testing.

Relevant Papers:

Massih-Reza Amini, Nicolas Usunier and Cyril Goutte. Learning from Multiple Partially Observed Views - an Application to Multilingual Text Categorization. Advances in Neural Information Processing Systems 22, pp. 28-36, 2009

Massih-Reza Amini and Cyril Goutte. A Co-classification Approach to Learning from Multilingual Corpora. Machine Learning Journal Springer, 79(1-2):105-121, 2010

Abhishek Kumar, Hal Daum?? III. A co-training approach for multi-view spectral clustering. International Conference on Machine Learning, pp. 393-400. 2011

Citation Request:

If you publish results based on this data set, please acknowledge its use, by referring to:

M.-R. Amini, N. Usunier, C. Goutte. Learning from Multiple Partially Observed Views - an Application to Multilingual Text Categorization. Advances in Neural Information Processing Systems 22, p. 28-36, 2009