## Background ## ![scroll][1] Recorded historical documents give us a peek into the past. We are able to glimpse the world before our time; and see its culture, norms, and values to reflect on our own. Japan has very unique historical pathway. Historically, Japan and its culture was relatively isolated from the West, until the Meiji restoration in 1868 where Japanese leaders reformed its education system to modernize its culture. This caused drastic changes in the Japanese language, writing and printing systems. Due to the modernization of Japanese language in this era, cursive Kuzushiji (くずし字) script is no longer taught in the official school curriculum. Even though Kuzushiji had been used for over 1000 years, most Japanese natives today cannot read books written or published over 150 years ago. The result is that there are hundreds of thousands of Kuzushiji texts that have been digitised but have never been transcribed, and can only currently be read by a few experts. We've built Kuzushiji-MNIST and sister datasets by taking handwritten characters from these texts and preprocessing them in a format similar to the MNIST dataset, to create easy to use benchmark datasets that are more modern and difficult to classify than the original MNIST dataset. By releasing these datasets, we're also hoping to bring together the fields of Japanese literature and ML ?? ?? [Read the paper](https://arxiv.org/abs/1812.01718) to learn more about Kuzushiji, the datasets and our motivations for making them! ## The datasets ## 1. **Kuzushiji-MNIST** is a drop-in replacement for the MNIST dataset (28x28 grayscale, 70,000 images), provided in the original MNIST format as well as a NumPy format. Since MNIST restricts us to 10 classes, we chose one character to represent each of the 10 rows of Hiragana when creating Kuzushiji-MNIST. - `kmnist-[train/test]-[images/labels].npz`: These files contain the Kuzushiji-MNIST as compressed numpy arrays, and can be read with: `arr = np.load(filename)['arr_0']`. We recommend using these files to load the dataset. - `[train/t10k]-[images/labels]-idx[1/3]-ubyte.gz`: These files contain `Kuzushiji-MNIST` with the same file format and filenames as the original MNIST for drop-in compatibility. - `kmnist-classmap.csv`: A mapping from class IDs to unicode characters for Kuzushiji-MNIST Some examples of Kuzushiji-MNIST with the first column being the modern Hiragana counterpart: ![Kuzushiji-MNIST examples][2] 2. **Kuzushiji-49**, as the name suggests, has 49 classes (28x28 grayscale, 270,912 images), is a much larger, but imbalanced dataset containing 48 Hiragana characters and one Hiragana iteration mark. - `k49-[train/test]-[images/labels].npz`: These files contain the Kuzushiji-49 as compressed numpy arrays - `k49-classmap.csv`: A mapping from class IDs to unicode characters for Kuzushiji-49 3. **Kuzushiji-Kanji** is an imbalanced dataset of total 3832 Kanji characters (64x64 grayscale, 140,426 images), ranging from 1,766 examples to only a single example per class. *We're also planning to release a train-test split for this dataset soon.* - `kkanji2/`: This folder contains subfolders of images, with each subfolder corresponding to one Kanji character class. ![Kuzushiji-Kanji examples][3] ## License ## The dataset is licensed under a permissive CC BY-SA 4.0 license, which requires attribution, and we would suggest to use the following attribution to the KMNIST dataset: "KMNIST Dataset" (created by CODH), adapted from "Kuzushiji Dataset" (created by NIJL and others), doi:10.20676/00000341 [1]: https://i.imgur.com/dhBM3BG.jpg [2]: https://raw.githubusercontent.com/rois-codh/kmnist/master/images/kmnist_examples.png [3]: https://raw.githubusercontent.com/rois-codh/kmnist/master/images/kkanji_examples.png