Is BERT the right model to fine tune your data on? Or do you need to pretrain from scratch?

Know your model's training data

BERT models have become commonly available and the use of subword tokenization has become widespread. But are these base models suitable for fine tuning against your data? Subword tokenization obscures the vocabulary the base model was trained on. By examining the original training data unigrams and their distributions, you can determine whether or not your data would benefit from training a model from scratch.

Content

This dataset is a best effort reconstruction of the training data used to train the English BERT base uncased model. The dataset comes from the BookCorpus dataset and a processed dump of Wikipedia (August 2019). Following the principles of BERT's tokenization scheme, no punctuation nor stopwords have been removed. The original unicode text was normalized using NFKC, tokenized using SpaCy English model (large), and the total count for each bigram across the corpora was recorded. The bigrams do not cross sentence boundaries, but they do include all tokens, only spaces, newlines and non-printing characters have been omitted. The bigrams are sorted in descending order of frequency. The CSV file column values are tab separated.

This dataset is large because it contains all bigrams; of course, you will likely want to filter out many of the lower counts to reduce some of the noise. There are many different methods one could adopt so I wanted to empower users to apply their own filters. Also, it was important to include complete counts so that the base of this corpus could be extended on, and used with new data to spot language changes and emerging trends.

Acknowledgements

Wikipedia and a public archive site provided the raw data for processing.

Inspiration

Here are some useful ideas

• Use the sister dataset BERT unigrams and:

• experiment with language modeling

• calculate Pointwise Mutual Information to find interesting collocations

• Construct a probability distribution of data in your domain and determine if BERT base is close enough for your task.

• Analyze the training data of a new BERT model (e.g. Bio-BERT, Legal-BERT) and quantify how similar/different they are to BERT base by calculating the Kullback–Leibler divergence for the shared vocabulary.

• Compare with newer language data to spot emerging keywords and trends.