Uncanny

A hardware research lab building RL environments for embedded firmware. Every task is graded in an emulator, then on a real board.

Frontier models pass about half of embedded tasks in simulation: 55.6% on EmbedBench with a schematic, 29.4% on ESP-IDF migration. No public benchmark flashes the code onto hardware. The useful training signal is in the tasks that pass in the emulator and fail on the board.

Focus
Embedded firmware
Checks
Renode → real boards
Formats
Harbor · verifiers · OpenEnv
Contact
hello@uncanny.dev
One task, end to end
01 / 06 · The task

A field report, not a spec.

The model gets a small Zephyr project for an nRF52840, a container with the real toolchain, and one sentence:

The I²C temperature sensor gives wrong values after 10 minutes. Find and repair the fault.
02 / 06 · In the emulator

In Renode, everything looks right.

The emulated sensor holds the room at 24 °C. Every UART reading is correct, so the start code passes everything the model can test.

03 / 06 · On the board

On the board, the enclosure warms up.

We flash the same binary to a real board. For ten minutes its readings track a calibrated PT100 next to the sensor.

04 / 06 · The failure

At 32 °C, a 16‑bit sum overflows.

The driver averages eight raw readings in an int16_t. Past 32 °C the sum exceeds 32,767 and the reading jumps to about −32 °C. The emulator never gets warm enough to show it.

05 / 06 · The fix

One line, once you know where to look.

- int16_t sum = 0;+ int32_t sum = 0;

With the wider accumulator, the board tracks the reference across the whole run.

06 / 06 · The reward

Scored in both worlds.

Hidden checks the model never sees: Renode ramps its sensor from 24 to 48 °C, and the board sits on a heated plate at three setpoints. Only a fix that passes both scores 1.

Fig. 1Reported temperature over UARTDraft data
Renode Board PT100 Board, fixed
Renode ramp, 24→48 °C pass Board at 22 / 35 / 48 °C pass Reward 1

What one task contains

Six parts. The model sees only the first three.

01 Instructionvisible

The symptom, written as a field report. It does not name the cause.

02 Start codevisible

A small firmware project with a real fault in it.

03 Containervisible

A Docker image with the pinned toolchain: Zephyr SDK, west, Renode.

04 Emulator checkhidden

Renode tests that push the emulated hardware past what the visible repo exercises.

05 Hardware checkhidden

The binary runs on a real board. Instruments measure the result against a stated tolerance.

06 Reference solutionhidden

Written by us and tested on the board before the task ships.

A checker earns its place three ways

A checker that can be gamed teaches the model to game it. Each test runs in both stages.

Oracle

1

The reference solution must score 1.

No-op

0

Doing nothing must score 0.

Unsolved

0

The unchanged start state must score 0.

The first twenty

Five task types, four tasks each. Each one is checked in Renode and on silicon.

Peripheral drivers
Timing & interrupts
RTOS & concurrency
Debugging
Porting
PublishedIn development

Drivers that read sensors correctly. PWM measured with a logic analyzer. Race conditions in FreeRTOS and Zephyr. Faults found from logs alone. Code ported from Arduino to ESP-IDF and Zephyr.

The model works in several steps: it reads logs, edits, builds and runs, then tries again. Tasks ship in Harbor, Prime Intellect's verifiers and OpenEnv, so they load into the RL stacks labs already run.

Then the rest of the bench

Every hardware domain already has a checker engineers trust.

Now

Firmware

Renode → boards

Next

PCB layout

KiCad DRC

Later

Circuits

SPICE

Later

PLC programs

PLC simulator

If you train models on engineering work, we should talk.

hello@uncanny.dev