The wikipedia dump is a giant XML file and contains loads of not-so-useful content. I needed some english text for some unsupervised learning so I spent quite a bit of time extracting and cleaning up the text.

Content

Each line of the txt file is a 'sentence'. I put sentence in quote because the content in these files haven't been read all the way through for errors. Here is what I did:

• Parsed out the opening text on non-disambiguation and non-table-of-contents pages.

• Removed sentences requiring citations, because these were usually poorly formed.

• Parse each block of text into sentences using SpaCy. I then checked for bracket and quote correctness, filtering out sentences that didn't quite match up.

• Removed sentences shorter than 3 letters and longer than 255 characters. This covers 97% of the data.

• Remove duplicate sentences, and, as a byproduct, sorted alphabetically.