Status: the headline scores below were not reproduced
This page used to say every figure came from reproducible runs of the 23-task harness. On 25 Oct 2026 the saved results in ornith-enhanced-pipeline/eval/results/ were checked against it:
- No saved run contains the 9/10, 22/23 or 1.105 figures. The loss appears only in written notes.
- The stock model's run (
week1_stock.json, 30 Aug 2026) scored compile 0/10, spec 0/5, twin 3/3 and protocol 5/5: 8 of 23. - The one full baseline-vs-candidate run (
20260830_201126_report.json) gave both sides an identical 23/23, which points to the harness grading reference answers rather than model output. - The "100% with auto-healing" result measures the repair engine, not the model.
The model itself is real: adarrshDev/ornith-1.5-9b-sft-r16 is a public LoRA adapter on Hugging Face (created 31 Aug 2026), trained on 20 samples for 9 steps. Treat it as a research prototype until a run is saved that produces these numbers.
8 / 23
Stock model, saved run (week1_stock, 30 Aug)
0 / 10
Stock model, ST compile tasks
none
Saved run of the fine-tuned adapter on all 23 tasks
20
Training samples (9 steps)
1. IEC 61131-3 Structured Text Compilation Gate (10 Tasks)
Evaluated strictly using ironplc-plc2x Stage 4 AST parsing and type verification (eval/plc/compile/):
| Task ID | Industrial Control Function Block | IronPLC Stage 4 Gate | AST Auto-Heal Gate | Verification Details & Type Resolution |
|---|---|---|---|---|
q01 |
FB_TON_Pulse | PASS | PASS | Auto-injected IEC standard timer prelude; non-blocking pulse generation. |
q02 |
FC_ScaleAnalog | PASS | PASS | Enforces strict INT_TO_REAL type casting with zero implicit coercion. |
q03 |
FB_Hysteresis | PASS | PASS | Deadband comparator logic verified against high/low trip thresholds. |
q04 |
FB_RampGen | PASS | PASS | Rate-of-change limiter with scan-cycle delta integration. |
q05 |
FB_Blinker | KNOWN BUG | PASS | IronPLC 0.1.1 AST visitor expression member access limitation (passes MATIEC iec2c). |
q06 |
FC_TrapIntRealMix | PASS | PASS | Rejects implicit numeric arithmetic; strictly requires explicit casting. |
q07 |
FC_TrapIntDintMix | PASS | PASS | Double-integer bounds validated; prevents silent integer overflow. |
q08 |
FC_TrapSelGeneric | PASS | PASS | Standard SEL generic selector type conformity verified. |
q09 |
FC_TrapAndOperator | PASS | PASS | Bitwise ANY_BIT operand validated; boolean logic verified. |
q10 |
FC_TrapShlWord | PASS | PASS | Bit-shift boundary checked with explicit WORD sizing. |
2. Full Multi-Gate Evaluation Scoreboard (23 Tasks)
| Benchmark Category | Task Count | Target Verification Gate | Claimed pass rate (not reproduced) | Status |
|---|---|---|---|---|
| IEC 61131-3 ST Compilation | 10 Tasks | ironplc-plc2x Stage 4 AST & Type Resolution |
9 / 10 (90.0%) | Not reproduced |
| Specification to Logic | 5 Tasks | State machine completeness & safety invariant checks | 5 / 5 (100.0%) | Not reproduced |
| Digital Twin Assertions | 3 Tasks | Dynamic tank differential physics & conveyor assertion loops | 3 / 3 (100.0%) | Not reproduced |
| IIoT Protocol Extraction | 5 Tasks | OPC UA (asyncua), Sparkplug B & SenML canonical mapping | 5 / 5 (100.0%) | Not reproduced |
| Total Harness Benchmark | 23 Tasks | Closed-Loop Multi-Gate Verification | 22 / 23 (95.7%) | Not reproduced |
Training Hyperparameters & Hardware
• Base Model: ornith-ai/Ornith-1.5-9B (Qwen3.5 dense multimodal architecture)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (29.1M params, 0.31%)
• LoRA Rank (r): 16 | LoRA Alpha (α): 32 | Dropout: 0.0
• Dataset: data/plant01/sft_plant01.jsonl (20 grounded interlock rows)
• Training Duration: 446.1 seconds | Total Steps: 9 optimization steps (3 epochs)
• Hardware: 1x NVIDIA Tesla T4 (16 GB) on Google Colab
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (29.1M params, 0.31%)
• LoRA Rank (r): 16 | LoRA Alpha (α): 32 | Dropout: 0.0
• Dataset: data/plant01/sft_plant01.jsonl (20 grounded interlock rows)
• Training Duration: 446.1 seconds | Total Steps: 9 optimization steps (3 epochs)
• Hardware: 1x NVIDIA Tesla T4 (16 GB) on Google Colab