Taposaur -- paper downloader#

This is part of the Cambridge Evidence TAP project to download literature from a variety of sources. Not intended for public use just yet.

taposaur CLI#

The taposaur command maintains the paper corpora and the dataset of derived artefacts built from them. Two corpora are indexed today, each its own namespace, selected with --ns:

Every paper carries a scheme-qualified uid — doi:10.1016/j.x, pmc:PMC10000266 — which is the join key of every manifest. The scheme's own identifier (doi, pmcid) travels beside it so a consumer can filter without parsing uids.

The shard is the unit of manifest files, partial re-runs and incremental ingestion: every command takes shard names as positional arguments and touches only those.

The dataset#

Everything derived lands in one tree, /eeg/datasets/scratch/taposaur (TAPOSAUR_DATA overrides it), which is where each command defaults its input and output:

taposaur/
├── tei/
│   ├── doi/<prefix>/<suffix>.xml.tei.xml     # pub2tei route
│   ├── doi/<prefix>/<suffix>.pdf.tei.xml     # grobid route
│   └── pmc/<block>/PMC<id>.pdf.tei.xml
├── pdf-index/
│   ├── doi/<prefix>.jsonl
│   └── pmc/<block>.jsonl
└── tei-index/
    ├── doi/<prefix>.jsonl
    └── pmc/<block>.jsonl

It is derived and rebuildable in full; the corpora are never written. The legacy /data/papers/tei tree is frozen and no longer read.

Indexing PDFs: taposaur spider#

taposaur spider walks a corpus read-only and writes one <shard>.jsonl manifest, one object per PDF:

{"uid":"doi:10.1006/anbo.1995.1038","doi":"10.1006/anbo.1995.1038",
 "path":"/data/papers/doi/10.1006/anbo.1995.1038.pdf","size":43920,"mtime":1738000000,"has_xml":true}
{"uid":"pmc:PMC10000266","pmcid":"PMC10000266",
 "path":"/eeg/papers-pmc/PMC10000266.pdf","size":3437661,"mtime":1733016566}

has_xml marks a PDF with publisher XML beside it, so the conversion plan is readable from the manifest alone; the PMC dump has none.

taposaur spider                     # /data/papers/doi -> pdf-index/doi
taposaur spider 10.1016 10.1053     # just those prefixes
taposaur spider --ns pmc            # /eeg/papers-pmc -> pdf-index/pmc
taposaur spider --ns pmc 1000       # just that block

A manifest whose content is unchanged is left untouched, so consumers can detect changed shards by mtime. The manifests load straight into ClickHouse for querying, e.g.:

SELECT count(), formatReadableSize(sum(size))
FROM file('pmc/*.jsonl', 'JSONEachRow',
          'uid String, path String, size UInt64, mtime Float64')

Downloading papers: taposaur download#

taposaur download fetches full text by DOI into a directory, named by DOI with the extension the content type says:

taposaur download -o papers 10.1016/j.foreco.2019.117721
taposaur download https://www.tandfonline.com/doi/full/10.1080/03071847.2019.1605022

An argument is a DOI in any presentation form, any URL with a DOI in it, or an article landing page whose own identifier the biblio lookup resolves to one: nature.com/articles/<slug>, a sciencedirect.com or linkinghub.elsevier.com URL with a /pii/ segment, pubmed.ncbi.nlm.nih.gov/<pmid>, or pmc.ncbi.nlm.nih.gov/articles/PMC<id>. Any query string or fragment is ignored.

Each result is one line on stdout — DOI, strategy, status, path — and a DOI nothing answered for is a line on stderr naming the publisher found and what each source said. The command exits non-zero if any argument failed, without stopping the run.

The publisher is looked up for every DOI rather than guessed from its prefix, through the biblio-glutton server at GLUTTON_SERVER_URL (the instance in the ~/biblio compose stack). The name that comes back selects the strategy:

An unmapped publisher, a strategy whose credential is not configured, an answer that is not a 2xx, or a source that outstays --timeout (60 seconds, counted per request rather than per identifier) falls through to the open-access link biblio-glutton knows and then to the first CORE result for the DOI (CORE_API_TOKEN). Each unset variable costs only its own strategy, save GLUTTON_SERVER_URL: it names the server that both identifies the publisher and resolves the open-access copy, so without it every argument fails saying so and nothing is requested.

Each step is reported on stderr as it finishes, with its status, how long it took and what it found, so a source that takes ten seconds looks like a slow source rather than a hang. --quiet turns those lines off, on download and on identify alike, and leaves the outcome lines on stdout as they are:

$ taposaur download -o papers 10.1093/nar/gkae1010
taposaur: 10.1093/nar/gkae1010: biblio lookup: 200 (0.01s) publisher "Oxford University Press (OUP)"
taposaur: 10.1093/nar/gkae1010: no strategy, trying fallbacks
taposaur: 10.1093/nar/gkae1010: oa: 404 (0.00s)
taposaur: 10.1093/nar/gkae1010: core search: 200 (8.64s)
taposaur: 10.1093/nar/gkae1010: core download: 200 (2.31s) 1.9 MiB
10.1093/nar/gkae1010 core 200 papers/10.1093/nar/gkae1010.pdf

Variables are read from the process environment first and from ./.env second, so an exported one overrides the file. That is the same file the original Python tool reads.

taposaur identify runs that resolution and detection and stops there, downloading nothing, which is how to survey a batch before committing to it. Each identifier — which takes the same forms download accepts — gives one line with the resolved DOI, the publisher, the strategy and whether it is configured, and under it the rest of what the biblio record holds:

$ taposaur identify https://pubmed.ncbi.nlm.nih.gov/23903748
10.1038/nature12373 "Springer Science and Business Media LLC" springer ready
  title   Nanometre-scale thermometry in a living cell
  journal Nature
  year    2013
  pmid    23903748
  pmcid   PMC4221854
  oa      https://arxiv.org/pdf/1304.1068

A detail line is there only when the record answered that field, and oa, the open-access PDF link, is the one that costs a second request. Only the first line is unindented, so a batch reads as four fields to the line:

$ taposaur identify 10.1016/j.x 10.1002/anie.202400000 \
    https://pubmed.ncbi.nlm.nih.gov/23903748 \
    10.1371/journal.pone.0306724 10.1093/nar/gkae1010 | grep -v '^ '
10.1016/j.x "Elsevier BV" elsevier ready
10.1002/anie.202400000 "Wiley" wiley needs WILEY_TOKENS
10.1038/nature12373 "Springer Science and Business Media LLC" springer ready
10.1371/journal.pone.0306724 "Public Library of Science (PLoS)" plos ready
10.1093/nar/gkae1010 "Oxford University Press (OUP)" - -

It exits non-zero if any DOI failed to land on a strategy. A needs or - line is not a paper out of reach: download still tries the open-access and CORE fallbacks, and only the open-access one is reported here.

One conversion: taposaur tei#

taposaur tei resolves a PDF (by path or DOI) to the publisher XML stored beside it and converts it to TEI through a Pub2TEI service with every refinement on — sentence segmentation, GROBID header refinement, generated identifiers and full reference consolidation:

A PDF with no publisher XML beside it falls back to full-text extraction through a GROBID service (--grobid, the instance shared with biblio-glutton in the ~/biblio compose stack); --from-pdf forces that route:

taposaur tei -o out/ /data/papers/doi/10.1371/journal.pone.0306724.pdf
taposaur tei doi:10.1038/s41598-022-22562-w > paper.tei.xml

Bulk conversion: taposaur tei-batch#

taposaur tei-batch converts in bulk what tei converts one at a time: it scans a corpus and converts every paper whose TEI is missing from the dataset tree — or present only as a zero-content stub left by a failed run — through the same routes.

In the doi namespace publisher XML goes through Pub2TEI, PDFs without one through GROBID, and the TEI mirrors the archive's layout. The pmc dump has no publisher XML, so every conversion is a GROBID one, and results are bucketed by shard (tei/pmc/1000/PMC10000266.pdf.tei.xml) rather than reproducing a directory of half a million files.

Prerequisites#

• The ~/biblio compose stack up (Pub2TEI on 8060, GROBID on 8070, glutton behind them): docker compose up -d from ~/biblio.
• A writable dataset tree. taposaur creates what it needs under $TAPOSAUR_DATA. (An unwritable directory fails that paper's conversion, not the sweep, so a partial fix still makes progress.)

Running it#

Census first — the plan, one uid per line, with the summary last (scanning all 2.8M DOI sources takes tens of minutes; the flat PMC directory takes seconds):

taposaur tei-batch --dry-run > plan.txt        # whole DOI archive
taposaur tei-batch --dry-run 10.1016           # one prefix
taposaur tei-batch --ns pmc --dry-run 1000     # one PMC block

Then convert, capturing failures for later triage:

taposaur tei-batch -n 20 2> failures.log            # whole DOI archive
taposaur tei-batch -n 20 10.1053 2> failures.log    # per prefix
taposaur tei-batch --ns pmc -n 20 2> failures.log   # all 528k PMC PDFs

--jobs/-n worker fibers keep one connection each in flight (default 20). The bottleneck is the GROBID service, not the client: GROBID sheds load above its configured pool (default 10) with 503, which is retried with backoff, so raising -n much beyond ~2× that pool buys nothing — raise the concurrency in the GROBID service config instead. Observed rates at -n 20: stub reconversions (small publisher XML) run at tens per second; full GROBID PDF extractions at roughly 5–6k/hour, which puts the PMC dump at ~4 days and the PDF-only DOI tail at weeks.

Results are written atomically (temp file, then rename), so killing a sweep leaves no partial TEI and rerunning the same command resumes where it left off: everything already converted is "up to date" on the next scan. Failures don't stop the sweep and are retried on the next run — note that permanently unconvertible PDFs (issue covers and tables of contents fail with [NO_BLOCKS], oversized documents with [TOO_MANY_TOKENS]) retry every sweep.

--upgrade also re-routes DOIs that have publisher XML but only a GROBID TEI through Pub2TEI; --force reconverts everything.

Afterwards#

Reindex what the sweep touched and refresh the site database:

taposaur tei-spider doi/10.1016 pmc/1000
cd ~/biblio && docker compose run --rm loader

taposaur tei-spider walks the dataset TEI tree read-only — both namespaces, one directory per shard — parses each teiHeader and writes one <ns>/<shard>.jsonl manifest, one object per paper:

{"tei_path":"/eeg/datasets/scratch/taposaur/tei/doi/10.1016/j.foreco.2017.08.006.xml.tei.xml",
 "uid":"doi:10.1016/j.foreco.2017.08.006","doi":"10.1016/j.foreco.2017.08.006",
 "title":"Catastrophic wind and salvage harvesting effects on woodland plants",
 "authors":[{"given":"Jonathan S.","family":"Kleinman"}],
 "journal":"Forest Ecology and Management","publisher":"Elsevier B.V.",
 "published":"2017","year":2017,"abstract":"…","source":"pub2tei",
 "mtime":1740146732.5}

Fields absent from a header are omitted. A paper converted through both routes (.xml.tei.xml via Pub2TEI and .pdf.tei.xml via GROBID) yields one record, preferring the publisher-XML route. Manifests whose content is unchanged are not rewritten, so consumers can detect changed shards by mtime, and positional arguments restrict a run to what a conversion batch touched — a whole namespace or a single shard, never a bare shard name, since each namespace numbers its own:

taposaur tei-spider                 # whole tree
taposaur tei-spider pmc             # one namespace
taposaur tei-spider doi/10.1016     # one shard

The manifests join the spider ones on uid in ClickHouse. A PMC paper's DOI is not in its TEI; it comes from the NCBI PMID_PMCID_DOI mapping at load time.

License#

ISC — see LICENSE.md.