Context A [significant amount of software][docker file] is available in Kaggle's Python notebook. I had hoped to find a reference somewhere listing which Python packages were available and what each one did. When I didn't find what I was looking for, I decided to build this dataset instead. [docker file]: https://github.com/Kaggle/docker-python/blob/master/Dockerfile Content This dataset was assembled in four steps: 1. [Code inside a Kaggle notebook][query notebook] was used to gather the names of over 600 installed packages. 1. A [package list][Anaconda list] was [scraped from Anaconda][query Anaconda] and cross-referenced against the notebook package list. 1. The roughly 400 packages that remained were carefully [queried from the Python Package Index][query PyPI] using its [JSON API][]. 1. The results were [collated into a manifest][collate]. [query notebook]: https://www.kaggle.com/ottoschnurr/query-notebook-for-installed-packages [Anaconda list]: https://docs.anaconda.com/anaconda/packages/py3.6_linux-64/ [query Anaconda]: https://www.kaggle.com/ottoschnurr/query-anaconda-for-package-metadata [query PyPI]: https://www.kaggle.com/ottoschnurr/gather-pypi-package-metadata [JSON API]: https://warehouse.readthedocs.io/api-reference/json/ [collate]: https://www.kaggle.com/ottoschnurr/assemble-kaggle-notebook-python-package-manifest Reference - [Anaconda's 64-bit Linux Python package list](https://docs.anaconda.com/anaconda/packages/py3.6_linux-64/). - [HTML Scraping - The Hitchhiker's Guide to Python][scraping guide] - The [PyPI JSON API](https://warehouse.readthedocs.io/api-reference/json/). - [Rate Limiting for the PyPI API](https://warehouse.readthedocs.io/api-reference/#rate-limiting) [scraping guide]: https://docs.python-guide.org/scenarios/scrape/ Acknowledgements - Thanks to @nagadomi for the [original script][]. - Thanks to the Kaggle team for creating a [powerful notebook environment][environment]. - Photo by [j zamora][]. [original script]: https://www.kaggle.com/nagadomi/list-of-installed-packages [environment]: https://www.kaggle.com/docs/kernels#the-kernels-environment [j zamora]: https://unsplash.com/@jzamora