The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia.
Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies.
In comparison to the Mikolov processed version of the Penn Treebank (PTB), the WikiText datasets are larger. WikiText-2 aims to be of a similar size to the PTB while WikiText-103 contains all articles extracted from Wikipedia. The WikiText datasets also retain numbers (as opposed to replacing them with N), case (as opposed to all text being lowercased), and punctuation (as opposed to stripping them out).

Data Collection

We selected articles only fitting the Good or Featured article criteria specified by editors on Wikipedia. These articles have been reviewed by humans and are considered well written, factually accurate, broad in coverage, neutralin point of view, and stable. This resulted in 23,805 Good articles and 4,790 Featured articles. The text for each article was extracted using the Wikipedia API. Extracting the raw text from Wikipedia mark-up is nontrivial due to the large number of macros in use. These macros are used extensively and include metric conversion, abbreviations, language notation, and date handling.
Once extracted, specific sections which primarily featured lists were removed by default. Other minor bugs, such assort keys and Edit buttons that leaked in from the HTML, were also removed. Mathematical formulae and LaTeX code, were replaced with¡´formula¡µtokens. Normalization and tokenization were performed using the Moses tokenizer, slightly augmented to further split numbers (8,600¡ú8 @,@ 600) and with some additional minor fixes. A vocab-ulary was constructed by discarding all words with a count below 3. Words outside of the vocabulary were mapped to the¡´unk¡µtoken, also a part of the vocabulary.