How a GPU Job Queue Became a Shared Inference Scheduler

TL;DR: Small teams running AI models on shared GPUs need a reliable way to accept work while protecting active models and scarce device memory; this history gives them a sequence for separating job tracking, capacity checks, safe memory release, and routing. While working on the coding project, I ran into this difficulty: a durable queue could remember each request, yet shared GPUs still needed a separate authority for device capacity, model memory, and safe handoff; the Housekeeper grew into that role through admission checks, explicit ownership, verified model placement, and peer routing.

A graphics processing unit (GPU) is a processor built for the many calculations used in AI. Several people or services can send inference requests to one GPU, but they still compete for its memory and execution time. Inference is a trained model applying what it learned to new input and returning a result. A runtime is the server process that loads a model and handles inference. A container is a packaged application with its own files and settings. This project’s history shows a useful design lesson: store each request in a durable queue, then let a device-aware service decide when and where that request can run.

The case study is Pyash’s GPU Housekeeper, a service that manages GPU-backed runtimes such as Ollama. Its history spans a host-local runtime controller, a separate GPU job queue, capacity-aware reclaim, peer routing, and checks that models reached the intended hardware. Each stage answers a different operational question, so the phases are useful to teams planning a similar system.

From ad hoc GPU calls to managed jobs

The first work began on April 15, 2026. An HTTP adapter gave Pyash a consistent way to ask a host-local GPU Housekeeper about runtime health, submit work, inspect status, and release a managed runtime. Container controls gave the service a place to run beside GPU-backed workloads. The initial design made the housekeeper a hardware steward for one host: it could inspect configured runtimes and coordinate their use of that host’s GPU. The early changes are recorded in the first adapter commit and the runtime and container controls commit.

The next step arrived in June. A durable GPU duty queue gave Pyash a place to record accepted work and its final result. A worker claims a duty, sends its job details to a Housekeeper, checks the temporary execution status, and writes the outcome back to the durable handle. On June 14, a GPU duty queue surface appeared, and Ollama model jobs moved through this worker path. On June 22, a follow-up change clarified that the Housekeeper itself stays outside the durable queue. See Ollama jobs routed through the worker and Housekeeper kept off the durable queue.

That split gives each part one clear job. Pyash preserves the request, handle, and final state across worker restarts. The Housekeeper controls a host’s runtime and device admission, then exposes a short-lived status record while the job executes. If an inference call is slow, an operator can inspect the durable handle to learn what Pyash accepted, then inspect the Housekeeper’s queue to see where the work is running. The architecture document describes the same boundary in its current reference.

This is a practical boundary for other inference systems too. A durable queue answers, “What work has the service accepted, and what result belongs to it?” A runtime manager answers, “Which device can execute it, and what state does that runtime have?” Keeping both records linked lets a worker recover its place while preserving one source of truth for completed work.

Admission, ownership, and safe reclamation

By September, the system had to decide whether a device could satisfy a job’s memory request. Video random-access memory (VRAM) is the fast memory attached to a GPU, where an active AI model’s weights and working data reside. Admission means checking a request against available resources before starting it. A job can declare how much VRAM it needs; live device readings then help the Housekeeper compare the request with current capacity.

The September 15 demand-driven reclamation change added a capacity preview and safe ways to release memory when a new request needs it. A runtime registry defines which managed containers the service may inspect. Provider-specific hooks supply a safe action, such as clearing an idle image queue or asking a model server to unload a resident model. This gives operators an explicit list of resources and release actions that the service can manage.

Ownership checks became part of that boundary in the follow-up GPU ownership commit. Before reclaiming capacity, the Housekeeper checks who owns the device and whether a registered runtime has active work. An idle grace period gives a runtime time to absorb short pauses; the recorded default was 300 seconds. When activity is unknown, the runtime stays in place until its status and safe release path can be established. This policy gives the device manager a clear rule: reclaim only managed capacity with evidence that its provider can yield it safely.

A recorded live check illustrates how preview and execution fit together. A request for 22,000 MB exceeded the available capacity at the time, while ComfyUI remained within its idle grace period. The Housekeeper reported the capacity decision and preserved the active runtime; the preview remained an observation step and launched zero model jobs. This kind of dry run gives an operator a way to inspect a resource decision before sending production work through it.

Two public discussions show similar design questions from other projects. In Anemoi issue 206, an author describes two clients sharing one GPU: a voice agent’s requests timed out during a long benchmark session, then model swaps forced the benchmark to reload and reprocess its prompt. That report makes admission policy concrete: should a service queue the request, prioritize it, select a warm fallback, or return a clear capacity result? Mehoy Malloy AI Runtime issue 26 is a design discussion about declared memory accounting and eviction at admission. It asks what signal should authorize a model load when the ledger reaches its limit. These are two individual examples; frequency across GPU services remains unmeasured.

The lessons from the issues and the Housekeeper history meet at the admission boundary. The queue records accepted work, while live device state establishes whether the target has room. A memory estimate combines with ownership and activity checks to establish whether a runtime can yield capacity safely. A useful policy connects the job’s declared need to current device state, the runtime’s ownership and activity, and a verified provider action. The project’s September CPU-fallback guard adds another check: an inference job should meet its intended GPU placement requirement before the worker treats execution as ready.

Routing work across peers

A host-local controller provides a clear first step, but a pool of machines can offer more choices when the local device is busy. In September, federation extended the Housekeeper’s decision to configured peers. Federation means that separate Housekeepers exchange enough capacity information to route a job to a suitable host. The federated routing commit added the first peer-aware route selector; later commits added dependency-aware GPU fanout and capacity-aware Ollama provisioning.

The selector avoids turning a network of peers into an uncontrolled forwarding chain. At its first hop, it considers the local route and probes configured peers that remain unvisited. A route is immediate when its device can begin the requested work now. If one or more routes are immediate, the selector chooses the highest score and favors the local host on an exact tie. When every candidate lacks immediate capacity but the local route remains available, it keeps the job local. When local is also unavailable, it returns a route-unavailable state for the caller to report or handle.

A simplified version of the decision is:

candidates = [local route]
if first hop:
    add results from unvisited peer probes
ready = available candidates that can start immediately
if ready:
    choose the highest-scoring route, favoring local on a tie
else if local remains available:
    keep the local route
else:
    return ROUTE_UNAVAILABLE

This mirrors the control flow in select_route at commit df2c2045. A peer probe failure is recorded as an unavailable candidate, so one unreachable machine can remain out of the candidate set while the selector evaluates other routes. The one-hop rule and visited-host tracking keep forwarding bounded. The route also carries source information, which helps operators connect a remote execution record to the worker that submitted it.

This design has a useful operational effect: a caller asks one entry Housekeeper to route work, while that Housekeeper evaluates peer capacity and keeps the selected device assignment with the job. The worker remains a single-entry dispatcher; the Housekeeper owns peer selection. Dependency-aware fanout helps work that has prerequisites move when those prerequisites finish; capacity-aware provisioning lets the system check a model’s availability and resource needs through the managed path. These mechanisms address separate questions: where a job can run, when dependent work becomes ready, and whether its requested model is available.

Verification, recovery, and warm routes

By late September, the project checked model placement inside the runtime as well as container access to the GPU. A container can see a GPU while its inference runtime places a model in regular system memory. The Housekeeper’s documented placement guard checks the model’s reported residency before inference; its warm threshold is at least 99% of the reported model size in GPU VRAM. A warm model is one already loaded in the memory where inference can use it. This distinction turns “the device is attached” into a checkable execution condition.

October changes filled in recovery around that condition. The stale-claim recovery change handles worker claims that have stopped heartbeating. A heartbeat is a small periodic signal that says the worker still owns a job. Recovery checks the durable handle and active leases before resolving an old claim; it can then record a clear failure state and release stale ownership. This keeps the next job from waiting on a worker that has disappeared, while the recorded terminal state helps an operator investigate an uncertain remote result.

A safe handoff also needs a confirmed release step. The October 25 handoff change adds a wait for runtime handoff before another workload takes the device. In practice, that means checking active work, giving the current owner its idle grace, asking its registered provider to release memory, and checking capacity again before continuing. These steps connect ownership, workload activity, and live device state in one transition.

Warm-route behavior extends the same checks across peers. An already loaded model can save the time required to load it from storage again. The system can favor a verified warm target and serialize work at that target’s device when another job currently holds its slot. The October 26 residency change preserves verified residency for fast Ollama jobs. Later, warm jobs moved ahead of cold capacity waits. The policy weighs model readiness and device availability together, which gives the router a more useful choice than treating every free or busy device as equivalent.

Verification records dated October 31 report 118 passing Python Housekeeper server tests and 35 passing runtime, worker, and adapter tests. The recorded coverage includes placement checks, CPU-fallback recovery, admission, idle grace, safe discharge, per-device slots, stale claims, handoff, peer routing, and forwarded job status. These counts establish automated behavior coverage; deployment-wide latency and utilization measurements remain unreported in this source snapshot.

For an operator or agent diagnosing a slow inference, the history suggests a short sequence: inspect the durable job handle; check the Housekeeper queue for the selected host and device; verify that the requested model is resident on the GPU; then review the runtime’s active-work and capacity signals. Each step asks a different question, which makes a timeout easier to localize than a single generic “GPU busy” message. The project’s architecture gives these records a shared path while keeping their owners distinct.

Current frontier: scheduling across hosts

On October 30, commits recorded preserved route assignments, worker concurrency scaled to discovered GPU devices, and warm-job preference before cold-capacity waits (route assignments, device-based concurrency, and warm-job routing). On October 31, a new multi-host scheduling effort was reported as in progress and under verification. It extends the earlier peer-routing and device-discovery foundation. A useful next proof point is an end-to-end run that records device assignment, waiting, completion, and recovery across hosts.