Context This corpus of syllabi aims to support the [Nimbus Assistant](https://www.github.com/calpoly-csai/api), an AI similar to Siri/Alexa that answers students’ questions. In the context of syllabi, students may ask questions like: What textbook does MATH 143 need? Do I need to buy a new book after MATH 142? What’s the course website for Anton Kaul’s 143? What’s Dr. Kaul’s grading policy? What’s the bare minimum I need to do to pass Kaul’s 143 class? How do I ace Kaul’s math 143 class? Content Data was scraped using [Thruuu, an awesome and easy to use SERP (search engine result pages) scraper](https://app.samuelschmitt.com/). # Thruuu * `thruuu.xlsx` - the data exported from Thruuu. * `thruuu.pdf` - the preliminary analysis exported from Thruuu. # Notebooks/Process * `step-1-get-documents-from-sheet-urls.ipynb` - a notebook that **inputs** `thruuu.xlsx` and **outputs** `downloads.tar.gz` along with `downloads.csv` * `step-2-extract-document-data-with-OCR.ipynb` - a notebook that **inputs** `downloads.tar.gz` along with `downloads.csv` and **outputs** `extracted.csv` * `step-3-get-simple-logistical-information.ipynb` - a notebook that **inputs** `extracted.csv` and outputs `logistical_info.csv` # Notebook Outputs * `downloads.tar.gz` - 100 PDF files (some files are corrupted). * `downloads.csv` - a table associating search result positions with individual PDF files for a syllabus. * `extracted.csv` - a table associating each PDF file with the extracted OCR text (also the plain text but OCR is preferred). * `logistical_info.csv` - a table associating each PDF file with the logistical info (instructor/office/email/etc) that is found through regular expressions. Acknowledgements Thank you [Samuel Schmitt](samuelschmitt.com) for making Thruuu! Inspiration * What kinds of factoids could you mine from the syllabus text? * What are common phrases used by Cal Poly professors in their syllabi? * What are the rarest phrases found in syllabi? * Can you identify a professor’s writing style from their syllabus?