ynux/arxiv_to_graph

A friends & familiy fun project to create a visual presentation of ArXiv (math preprint server) metadata

arxiv_to_graph's Issues

have fetch_papers_json.py sleep between getting papers, and check (id, version)

The original fetch_papers.py updates a pickle database. It gets papers in batches and sleeps in between to avoid getting throttled by arXiv. And it checks (version, id) and skips duplicates.

We need the data as json, one compact json document per line.

Either use the original pickle db and convert this - didn't work, json_dump either creates pretty json with indents, or json documents without newlines in between.

Or write directly in json, and create a dictionary with the (_rawid, _version) tuples, store it alongside the json file and check against it.