mining-pubmed-literature

Mining PubMed & PMC literature (NCBI E-utilities)

Search PubMed (citations/abstracts) and PMC (full text) programmatically with NCBI E-utilities — the stable HTTP interface to Entrez. The core pattern is two steps: ESearch returns matching record IDs (PMIDs), then EFetch (or ESummary) downloads the records. The Entrez History server (usehistory=y) lets you chain the two without re-sending thousands of IDs.

E-utilities are public. No key is required, but a free API key raises your limit from 3 to 10 requests/second and is strongly recommended for batch work.

When to use

• OpenMed extracted a diagnosis, drug, or gene and you want supporting literature.
• You need abstracts to summarize or to assemble a corpus for biomedical NER.
• You want MeSH-anchored, reproducible searches (date ranges, article types).

For ClinicalTrials.gov use searching-clinicaltrials; this skill is for the published literature.

Quick start (real E-utilities calls)

Base URL: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/. JSON for ESearch/ ESummary via retmode=json; EFetch returns text or XML (no JSON for PubMed).

import requests, time

BASE = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
API_KEY = None   # set to your free NCBI key to get 10 req/s instead of 3

def _params(**kw):
    if API_KEY:
        kw["api_key"] = API_KEY
    return kw

def esearch(term: str, retmax: int = 50) -> dict:
    """Find PMIDs; usehistory=y stores them on the Entrez History server."""
    r = requests.get(f"{BASE}/esearch.fcgi", params=_params(
        db="pubmed", term=term, retmax=retmax,
        usehistory="y", retmode="json"), timeout=30)
    r.raise_for_status()
    res = r.json()["esearchresult"]
    return {"count": int(res["count"]), "ids": res["idlist"],
            "webenv": res["webenv"], "query_key": res["querykey"]}

def efetch_abstracts(webenv: str, query_key: str, retmax: int = 50) -> str:
    """Pull abstracts by reference to the stored result set (no ID list needed)."""
    r = requests.get(f"{BASE}/efetch.fcgi", params=_params(
        db="pubmed", WebEnv=webenv, query_key=query_key,
        retmax=retmax, rettype="abstract", retmode="text"), timeout=60)
    r.raise_for_status()
    return r.text

hits = esearch('("type 2 diabetes"[MeSH]) AND metformin AND 2023:2025[pdat]')
print(hits["count"], "papers")
abstracts = efetch_abstracts(hits["webenv"], hits["query_key"])

Equivalent cURL (search then fetch one PMID's abstract):

curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=metformin&retmode=json"
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=38000000&rettype=abstract&retmode=text"

ESummary for structured metadata

When you need titles/authors/journal/date as JSON (not the full abstract), use ESummary — it returns one record per ID:

def esummary(ids: list[str]) -> dict:
    r = requests.get(f"{BASE}/esummary.fcgi", params=_params(
        db="pubmed", id=",".join(ids), retmode="json"), timeout=30)
    r.raise_for_status()
    return r.json()["result"]   # keyed by PMID: title, pubdate, source, authors…

For PMC full text, repeat with db=pmc and EFetch rettype=""/retmode=xml (JATS XML). Respect each article's license before redistributing full text.

Workflow

1. Build the query. Combine OpenMed-extracted terms with MeSH tags and field filters: "<disease>"[MeSH] AND <drug>[tiab] AND 2020:2025[pdat]. Use [tiab] (title/abstract), [au] (author), [pdat] (publication date).
2. ESearch with usehistory=y to capture WebEnv + query_key and the count.
3. Batch-fetch with EFetch/ESummary in pages of ≤ ~200 IDs (or by history), sleeping to stay under your rate limit.
4. Parse abstracts/metadata; store PMID, title, journal, date, abstract text.
5. NER the abstracts with openmed.analyze_text to extract diseases, drugs, genes, and oncology entities for downstream synthesis.

Hand-off to / from OpenMed

• OpenMed facts → query. openmed.analyze_text(note) yields Disease, Pharmaceutical, Genomics, and Oncology entities. Turn the top spans into the ESearch term (optionally grounded: ICD-10 label, RxNorm ingredient, gene symbol) to retrieve targeted evidence.
• Abstracts → OpenMed. Feed fetched abstracts straight into openmed.analyze_text(abstract, model_name="disease_detection_superclinical") (or a Genomics/Oncology model) to structure the literature into entities for evidence tables or knowledge-graph edges.
• Queries and abstracts are public literature, not PHI. Still run locally and never embed patient text in a search term.

Edge cases & gotchas

• Rate limits. 3 req/s without a key, 10 with one — exceed it and NCBI returns HTTP 429. Add api_key, throttle, and retry with backoff. NCBI also requests a tool= and email= parameter identifying your application.
• EFetch has no JSON for PubMed. Use retmode=text (human-readable) or retmode=xml (PubMedArticle XML) and parse XML for structured fields.
• History expires. WebEnv/query_key are session-scoped — fetch promptly after searching, or re-run ESearch.
• Large result sets. Page with retstart/retmax (or history) rather than pulling everything at once; cap total fetches.
• MeSH lag. Very recent articles may not yet be MeSH-indexed — include [tiab] term variants so you do not miss them.
• Full-text licensing. PMC full text carries per-article licenses; many are not redistributable. Store PMIDs/abstracts freely; check the license before republishing full text.

Standards & references

• E-utilities In-Depth (parameters & syntax) — https://www.ncbi.nlm.nih.gov/books/NBK25499/
• E-utilities Quick Start — https://www.ncbi.nlm.nih.gov/books/NBK25497/
• General introduction & policies — https://www.ncbi.nlm.nih.gov/books/NBK25501/
• API keys & rate limits — https://support.nlm.nih.gov/kbArticle/?pn=KA-05317
• PubMed search field tags — https://pubmed.ncbi.nlm.nih.gov/help/
• MeSH browser — https://meshb.nlm.nih.gov/