策展语料库汇集了 40,000 篇专业撰写的新闻文章摘要，并附有文章本身的链接。这个存储库提供了一个抓取工具来访问它们。如果您对商业用途或访问更广泛的 Curation 数据目录感兴趣，包括超过 150,000 个专业编写的摘要和可扩展的按需内容抽象 API（由人类或 AI 驱动），请访问触碰。有关我们希望此版本如何帮助 NLP 社区的想法，请参阅我们介绍数据集的帖子。

Instruction

• Clone this repository (or just copy the code from scraper.py)

• Download the urls, headlines, and summaries from here

• Run web_scraper.py. Give as command line arguments the path to the csv file without article text, the path to a new csv file which will have article text, and a batch size to determine how many urls it will scrape at a time. Larger batch sizes will make it run faster but it may drop more articles due to timeouts. I recommend ~50 on a 2015 Macbook Pro.

git clone https://github.com/CurationCorp/curation-corpus.git
cd curation-corpus
wget https://curation-datasets.s3-eu-west-1.amazonaws.com/curation-corpus-base.csv
python web_scraper.py curation-corpus-base.csv curation-corpus-base-with-articles.csv 50

Some urls will return messy results due to content changing over time, paywalls, etc. We've tried to remove the worst offenders from this release. There is probably still scope though for improving the scraper though.

Citation

@misc{curationcorpusbase:2020,
  title={Curation Corpus base},
  author={Curation},
  year={2020}
}