Wikipedia dumps contain a tremendous amount of markup. WikiMedia Text is a hybrid of markdown and HTML, making it very difficult to use. Wikipedia, however, is an extremely valuable dataset. I wanted a more usable format.

• Blog post about this project

• GitHub repo for this project

Content

This dataset includes ~40MB JSON files, each of which contains a collection of Wikipedia articles. Each article element in the JSON contains only 3 keys: an ID number, the title of the article, and the text of the article. Each article has been "flattened" to occupy a single plain text string. This makes it easier for humans to read, as opposed to the markup version. It also makes it easier for NLP tasks. You will have much less cleanup to do.

Each file looks like this:

[
 {
  "id": "17279752",
  "text": "Hawthorne Road was a cricket and football ground in Bootle in England...",
  "title": "Hawthorne Road"
 }]

Acknowledgements

Absolutely all thanks goes to Wikipedia! Everyone who helped build and fund Wikipedia, and everyone who has contributed their time and expertise to the content of Wikipedia