文献自动化入库流水线Literature ingestion pipeline
OWNER · 2025.10 – 至今 · 国家生物信息中心OWNER · Oct 2025 – present · China National Center for Bioinformation面向国家级基因组数据库(GSA / GSA Human / OMIX)。文献里提到某个数据集,到底是新提交、是复用、还是只引了一句——把这件事从人工判读变成可审计、可入库的结构化数据。 For the national genomics databases (GSA / GSA Human / OMIX). When a paper mentions a dataset, was it newly deposited, reused, or merely cited? This turns that judgement from manual reading into auditable, ingestible structured data.
1增量怎么发现Finding what's new
原来的做法是拿本地库里数十万条登录号,逐个去外部源反查,跑一次就是一整晚。把方向反过来:每天只拉 PMC 的 500 篇增量,用它们反查本地库。The old way took hundreds of thousands of accession numbers from our own database and looked each one up against external sources — one run took all night. Reverse the direction: pull only the ~500 new PMC papers each day, and look those up against our database.
单次耗时 14 小时以上 → 3 分钟,从月度批量改成每日可跑;审核员单次待审量由数百篇降到每日 10–20 篇。全量比对留作离线兜底,实测补 3% 覆盖,不占线上链路。One run went from 14+ hours to 3 minutes, and from a monthly batch to a daily job; a curator's queue dropped from hundreds of papers per run to 10–20 per day. The full comparison stays as an offline backstop — it adds about 3% coverage and never touches the live path.
2文献怎么进来How papers get in
邮件、PubMed、PMC、Google Scholar、PDF 五个来源统一收口,抽出登录号与它周围的上下文,再补齐 DOI 与期刊信息。Five sources — email, PubMed, PMC, Google Scholar and PDFs — funnel into one intake. It pulls out accession numbers together with the sentences around them, then fills in DOI and journal metadata.
抽的不只是编号,还有编号周围的句子——同一个登录号出现在方法学段落里还是引文列表里,结论完全不同。It isn't only the number that matters. The same accession in a methods paragraph and in a reference list mean entirely different things.
3引用怎么判Classifying the citation
三分类:deposited(自己提交)、reused(复用他人)、reference-only(只是提及)。规则引擎按 YAML 配置的关键词与上下文判一遍;自部署模型用句子相似度动态检索 Few-shot 例子,再判一遍。Three classes: deposited, reused, reference-only. A rule engine judges it once from keywords and context defined in YAML; a self-hosted model judges it again, retrieving few-shot examples by sentence similarity.
在 500 条人工标注的验证集上,三分类准确率约 90%。About 90% three-class accuracy on a 500-item hand-labelled validation set.
4怎么敢自动落库Why auto-ingest is safe
写入的是国家库,误判的代价远高于漏判。所以两路结论一致才自动落库,不一致的一律转人工复核。This writes into a national database, where a false positive costs far more than a miss. So a record is ingested automatically only when both paths agree; anything else goes to human review.
用覆盖率换自动化部分的零误判:自动落库覆盖 74.8%,其余进人工队列。Coverage traded for zero false positives on the automated share: 74.8% auto-ingested, the rest queued for a human.