OfflineScribeAI

Local-first clinical AI research prototype

OfflineScribeAI

A source-grounded clinical documentation framework for studying whether useful draft notes can be generated locally, reviewed by a named human, and evaluated under realistic resource constraints.

GitHub repository Read the methods
Author
Zorykto Mykola
Dataset
100 synthetic encounters
Default posture
No cloud inference
Clinical status
Research only

Research question

Can local-first clinical AI generate useful, source-grounded documentation while operating with constrained connectivity and compute?

OfflineScribeAI does not claim to diagnose, treat, replace clinicians, or validate clinical safety. It measures documentation behavior: fact preservation, unsupported statements, negation, temporal context, source evidence, review burden, latency, and local resource use.

System

Every stage is observable

The repository keeps the workstation and the research harness side by side: FastAPI and React for the local review workflow, Python CLI tools for benchmark runs, seeded synthetic data, and repeatable metrics.

Pipeline from synthetic encounter to local ASR, draft note, evidence, safety gate, review, storage, and evaluation.
Local processing path with source evidence and a human review gate before approved documentation.

Local documentation path

Audio or script input moves through faster-whisper, local note drafting, span grounding, and a schema-checked SOAP note.

Human-in-the-loop review

Drafts remain drafts until a reviewer is named. Edits, approvals, and rejections are logged as research signals.

Evaluation harness

Command-line runs compare baseline, retrieval, grounded, and verifier conditions on the same synthetic cases.

Failure analysis

Unsupported claims, negation flips, temporal errors, medication errors, and resource failures are preserved instead of hidden.

Evaluation

Built to measure tradeoffs, not just produce a note

The committed runner uses deterministic stubs so tests and reports can run without GPUs or API keys. Live model comparisons are explicitly marked as planned until they are executed and reviewed.

  • Clinical extraction precision, recall, and F1
  • Grounded and unsupported statement rates
  • Negation and temporality accuracy
  • Human edit, rejection, and approval rates
  • Latency, RAM, CPU, disk, and model size
Research dashboard mockup with evaluation runs, grounding, unsupported claims, latency, and human edit rate.

Failure modes

Failures are findings

The project treats embarrassing cases as part of the evidence. A failed case should show the input, expected behavior, actual output, likely reason, and mitigation.

Hallucination Grounding failure Negation flip Temporal error Medication error Terminology error Formatting error Resource failure

Safety posture

Documentation support only

No autonomous clinical decisions

The system drafts documentation and flags evidence problems. It does not make diagnoses, treatment decisions, or EHR writes.

Synthetic data by default

The benchmark is synthetic and does not represent real clinical populations. Real data would require separate governance.

Prompt injection awareness

Transcript content is treated as data. Instruction-like text inside an encounter is a safety test, not a command source.

Reproduce

Clone, test, evaluate

These commands run the package tests, backend tests, UI tests, and seeded evaluation path used by the repository. 

git clone https://github.com/zoryktom/OfflineScribeAI.git
cd OfflineScribeAI
python -m pip install -e ".[dev]" -r backend/requirements.txt
python -m pytest tests -q
cd backend && python -m pytest -q
cd ../desktop-app && npm ci && npm test -- --run
cd .. && python -m offlinescribe.cli eval run --config experiments/configs/pilot.yaml