neqsim-pdf-ocr

NeqSim PDF OCR Skill

Extract machine-readable text from PDFs that don't have an embedded text layer — scanned documents, P&IDs, mechanical arrangement drawings, vendor datasheets exported as raster, inspection reports, and old paper-original technical requirements. Complements neqsim-technical-document-reading:

When to Use This Skill

Load this skill when any of these are true:

• The user mentions OCR, scanned PDFs, or P&ID text/tag extraction
• A PDF returns empty or near-empty text from pymupdf / pdfplumber
• A document is a P&ID and you need equipment/instrument tag numbers as strings
• A vendor datasheet is a scan of a paper original (common pre-2010)
• An engineering drawing has labels, line numbers, or revisions that must be searchable

Do not load this skill for born-digital PDFs that already have a text layer — those are faster and more accurate via pdfplumber / pymupdf (covered in neqsim-technical-document-reading).

Detection: Is OCR Actually Needed?

Always try fast text extraction first. Only fall back to OCR when the yield is too low. The utility implements this automatically; the rule of thumb is:

average_chars_per_page < 50  →  treat as scanned, run OCR

import fitz
doc = fitz.open("document.pdf")
yield_chars = sum(len(page.get_text("text")) for page in doc) / len(doc)
if yield_chars < 50:
    # scanned — needs OCR
    ...

Tool Selection Matrix

P&ID-Specific Patterns

P&IDs are the hardest case for OCR. Use these settings:

• DPI ≥ 400 (300 is too low — tag digits get lost)
• Tesseract PSM 11 (Sparse text. Find as much text as possible in no particular order)
• --rotate-pages in OCRmyPDF (drawings often have rotated title-block text)
• Multi-language: eng+nor for Norwegian operator drawings
• Post-filter with regex to recover tags from noisy OCR output:

import re
TAG = re.compile(r"\b[A-Z]{1,4}-\d{3,5}[A-Z]?\b")  # V-100, PT-2301A, FIC-12345
LINE = re.compile(r"\b\d{1,3}\"?-[A-Z]{1,4}-\d{3,6}-[A-Z0-9]+\b")  # 6"-PG-1001-A1A

Code Patterns

Pattern 1: Auto-OCR text extraction (recommended default)

from devtools.pdf_ocr import extract_text

# Tries pymupdf first, falls back to OCR automatically if scanned
pages = extract_text("step1_scope_and_research/references/datasheet.pdf")
for page_num, text in pages.items():
    print(f"--- Page {page_num} ---\n{text}")

Pattern 2: P&ID tag extraction

from devtools.pdf_ocr import extract_tags

# pid_mode=True → 400 DPI, PSM 11 (sparse), force OCR
tags = extract_tags("references/PID-001.pdf", pid_mode=True)
# → ['FIC-2301', 'PT-2305', 'V-100', 'V-101', ...]

Pattern 3: Add a searchable text layer (preserves drawing for archival)

from devtools.pdf_ocr import ocr_pdf

# Produces references/scanned_ocr.pdf with embedded text layer
out = ocr_pdf(
    "references/scanned_datasheet.pdf",
    language="eng+nor",
    dpi=400,
    rotate=True,
    deskew=True,
)
# Now usable by pdfplumber / pymupdf / search tools

Pattern 4: Combine with pdf_to_figures.py for full P&ID analysis

For a P&ID, do both rasterise + OCR. They're complementary:

from devtools.pdf_to_figures import pdf_to_pngs
from devtools.pdf_ocr import extract_tags, extract_text

pdf = "references/PID-001.pdf"

# 1. Visual analysis — render to PNG, then use view_image (multimodal)
pngs = pdf_to_pngs(pdf, outdir="figures/", dpi=400)
# → use view_image(pngs[0]) to read symbols, topology, valve types

# 2. Textual extraction — OCR for tag numbers, line specs, notes
tags = extract_tags(pdf, pid_mode=True)
text_by_page = extract_text(pdf, force_ocr=True, dpi=400, psm=11)

Pattern 5: CLI usage in task workflows

# Extract text to JSON
python devtools/pdf_ocr.py step1_scope_and_research/references/datasheet.pdf `
    --json step1_scope_and_research/datasheet_text.json

# P&ID preset (400 DPI, sparse PSM, force OCR, print tags)
python devtools/pdf_ocr.py references/PID-001.pdf --pid

# Add searchable layer for archival
python devtools/pdf_ocr.py scanned.pdf --add-text-layer --output scanned_ocr.pdf

Common Mistakes

Validation Checklist

When you've extracted text/tags from a scanned PDF:

• Average chars/page > 200 (low yield = OCR likely failed; raise DPI or check language)
• Tag count matches expectation (typical P&ID: 30–150 tags)
• No common confusions in tags: digit 0 vs letter O, 1 vs l, 5 vs S
• If pid_mode used, verify a sample of tags against the visual (view_image on rendered PNG)
• For task workflows, save extracted text as JSON in step1_scope_and_research/
• Source PDF stays in step1_scope_and_research/references/ (per AGENTS.md task-folder rule)

Installation

The OCR stack is an optional extra of devtools/:

pip install ocrmypdf pytesseract pdf2image pymupdf

Plus the system Tesseract binary:

Norwegian language pack (for NCS documents):

The pdf_ocr.py utility checks for these dependencies and prints a clear remediation message if they're missing — never silently produces empty results.

Related Skills

• neqsim-technical-document-reading — text-based PDFs, Word, Excel, image analysis with view_image
• neqsim-stid-retriever — fetch the documents in the first place
• neqsim-process-extraction — convert extracted text into NeqSim JSON