Asynchronous inference for a heterogeneous edge fleet

Stop treating inference
as a blocking call.

The Horizon Broker intercepts every request bound for the local Ollama and Qwen endpoints. It routes long-horizon work, batches it for throughput, and guarantees that a dead node does not lose an agent's work.

Read the README ↗Releases ↗Container image ↗
horizon-broker starting up, its health, and the node fleet
The broker comes up, reports its health, and lists the fleet it can dispatch to — each node with its role, batch slot, power headroom and models.

01 / Quick start

Start the broker, then talk to it.

The operator client speaks to a running broker over HTTP. It never touches broker state directly.

# install
pip install horizon-broker

# validate the configuration and exit
horizon-broker check-config --config config/broker.example.yaml

# run the broker and its ingress API
horizon-broker serve --config config/broker.example.yaml

# from another machine: submit a long-horizon job
horizon-broker job submit --model qwen2.5:7b \
  --prompt "Recursive codebase scan of the scheduler package" \
  --tier flex

02 / The sub-systems

Throughput over latency, and recovery as a given.

Scheduler

Sleep on purpose.

Priority queue, per-model windowing and batch planning. A Flex job is held until the batch is worth executing.

Routing

PD-DAG placement.

An arithmetic-intensity model decides which node should prefill and which should decode, instead of sending everything to one box.

Recovery

Hardware death is expected.

State is checkpointed to EdgeRecord continuously. A lost node triggers re-dispatch elsewhere inside a 30-second budget.

Heartbeat

The watchdog only reports.

It produces verdicts. It does not decide what happens next, and it never talks to a node directly.