Context This is a dataset made by modifying the code [from this github repository](https://github.com/eukaryote31/openwebtext) slightly. This is part of the open-webtext movement where people are trying to replicate the data which the GPT-2(a text generator model which is said to be too dangerous to publish) was trained on. This is just roughly the quarter of the entire data but as Venya Cohen stated in the Discussion section, the full dataset is available at https://skylion007.github.io/OpenWebTextCorpus/. Content This dataset contains an id to identify the text up to 2019434 for the large data.db and 743 for the smaller.db. Each of the ids corresponds to a text column which has a bit of text. Acknowledgements Thanks eukaryote31 for github repo the https://github.com/eukaryote31/openwebtext Inspiration The open ai's gpt-2 model is said to have still underfitted the 40GB webtext model so, I'm wondering how much data will be just right?