Children¡¯s Book Test (CBT), designed to measure directly how well language models can exploit wider linguistic context. The CBT is built from books that are freely available thanks to Project Gutenberg. Details and baseline results on this dataset can be found in the paper:

Felix Hill, Antoine Bordes, Sumit Chopra and Jason Weston. The Goldilocks Principle: Reading Children¡¯s Books with Explicit Memory Representations, arXiv:1511.02301.

After allocating books to either training, validation or test sets, we formed example ¡®questions¡¯ from chapters in the book by enumerating 21 consecutive sentences. In each question, the first 20 sentences form the context, and a word is removed from the 21st sentence, which becomes the query. Models must identify the answer word among a selection of 10 candidate answers appearing in the context sentences and the query. For finer-grained analyses, we evaluated four classes of question by removing distinct types of word: Named Entities, (Common) Nouns, Verbs and Prepositions

Here is an example of question (context + query) from Alice in Wonderland by Lewis Carroll:

Context:
1 So they had to fall a long way .
2 So they got their tails fast in their mouths .
3 So they could n't get them out again .
4 That 's all .
5 `` Thank you , " said Alice , `` it 's very interesting .
6 I never knew so much about a whiting before . "
7 `` I can tell you more than that , if you like , " said the Gryphon .
8 `` Do you know why it 's called a whiting ? "
9 `` I never thought about it , " said Alice .
10 `` Why ? "
11 `` IT DOES THE BOOTS AND SHOES . '
12 the Gryphon replied very solemnly .
13 Alice was thoroughly puzzled .
14 `` Does the boots and shoes ! "
15 she repeated in a wondering tone .
16 `` Why , what are YOUR shoes done with ? "
17 said the Gryphon .
18 `` I mean , what makes them so shiny ? "
19 Alice looked down at them , and considered a little before she gave her answer .
20 `` They 're done with blacking , I believe . "

Query: `` Boots and shoes under the sea , " the XXXXX went on in a
deep voice , `` are done with a whiting ".

Candidates: Alice|BOOTS|Gryphon|SHOES|answer|fall|mouths|tone|way|whiting

Answer: gryphon