Scheduling reference

The four modules that decide when a campaign’s jobs are created, where they run and how big they are. This page is the map and the interface; the reasoning lives elsewhere and is not repeated here:

  • Job Queueing — what an operator sees and configures.

  • Developer Guide §*Job admission internals* — why creation is the admission point, and why the queue is global rather than per caller.

The modules

Module

Answers

node_admission

When. The global queue, the in-flight ledger, and the create-on-admit loop.

cluster_capacity

How much room. Reads the live cluster; the only thing that talks to the API for capacity.

node_placement

Where. Node identity labels, the campaign node pool, the taint job pods tolerate.

node_calibration

How big. Per-node sizing measured by a probe run, per campaign.

They are layered, not circular: node_admission knows only the BudgetProvider protocol, and never a Kubernetes object. That is what lets the whole queue be tested without a cluster.

The interface

What a job needs, and what a node has. Both frozen, both plain numbers:

JobSizing(cpu: float, memory: int, gpu: int = 0)      # summed over a pod's containers
Capacity(cpu, memory, gpu)                            # what one node holds when empty
NodeBudget(node_id, free_cpu, free_memory, free_gpu)  # what one node has free NOW
Budget(nodes: tuple[NodeBudget], counted_jobs: frozenset, growable: bool)

counted_jobs is the double-counting fix: the provider reports which Jobs its reading already saw, so a reservation stops being subtracted the instant the real pod starts being subtracted. No timers, and no window where cores are counted twice or not at all.

The seam. One protocol, two methods, and the reason multi-tenancy can be added later without touching the loop:

class BudgetProvider(Protocol):
    def budget(self) -> Budget: ...        # free NOW  — excludes unusable nodes
    def capacities(self) -> list[Capacity]: ...  # could EVER — includes them

The asymmetry is deliberate and was a bug when it was absent. “Free now” must exclude a node that is cordoned, NotReady or carrying an untolerated taint, because counting it promises room that cannot be spent. “Could ever” must include it, because preflight() raises a permanent error and a rebooting node is coming back. Both take the per-node headroom off: a reserve that is never spendable is not part of either answer.

The controller. Everything a campaign does with the queue:

Call

Meaning

submit(owner, items, *, started_at, priority=0, campaign="", sizing_for_node=None, accepts_node=None, pin=None)

Enqueue a whole plan. items are (key, JobSizing, create_fn); create_fn takes the chosen node_id (or None when unpinned). started_at is the campaign’s start, not the batch’s. campaign is whose rank the items take and defaults to owner; a sub-scope owner (<campaign>#probes) must name it, or its work ranks as a stranger to the campaign it belongs to.

drain(*, limit=None) -> int

Create as many globally-highest-ranked items as currently fit. Returns how many. A paused campaign’s items are not candidates at all. 0 means “nothing fits now” — normal, never an error. An item that does not fit is skipped, so a large one cannot idle the cluster — except a pinned one, which holds its node open until it drains (see below).

finished(key)

Release that job’s reservation.

cancel(owner) -> int

Drop an owner’s planned items and release its holds. Runs per batch.

drop_planned(owner) -> list

Drop only what the queue has not created; the owner keeps the holds behind its created items, which are pods that exist. For an owner with nothing left for a planned item to do.

forget_calibration(owner) -> bool

End a campaign’s per-node figures. Runs per campaign.

set_scheduling(campaign, *, priority=None, paused=None)

Set the campaign’s rank, its hold, or both. None leaves that half alone. Reaches the items already queued and the batches not submitted yet.

scheduling(campaign) -> (int, bool)

(priority, paused); (0, False) when nothing was set.

forget_scheduling(campaign)

Drop both, once the campaign is over. Runs per campaign, beside forget_calibration and never merged with it.

preflight(sizing)

Raise AdmissionRefused when no node could ever hold it.

states(owner) -> dict

key -> PLANNED | CREATED, for progress reporting.

refusal(owner) -> str

One line saying what this owner is waiting for — its own item count and the sizing the fit test actually used, not the queue’s total and not the declared figure.

calibration(owner, factory=None)

The campaign’s NodeCalibration, created once and kept for its life.

node_ids() / growable()

Pinnable node identities; whether the cluster can add nodes.

cancel and forget_calibration are separate because their lifetimes are: reservations belong to a batch, calibration to a campaign. Merging them made a search re-probe every node every round.

Workflows

A batch. The campaign thread owns no admit loop — it enqueues, then polls its own items:

submit(campaign, plan, started_at=..., priority=0)
while True:
    reap     -> finished(key) for each vanished CREATED job
    exit     -> if every plan entry is FINISHED: break
    drain()  -> may create OTHER campaigns' jobs; that is the point
    probes   -> nothing PLANNED of its own left: drop_planned(campaign#probes)
    publish  -> waiting_for_capacity from states(), not from pods
    sleep 2
finally:
    cancel(campaign)                # and cancel(campaign#probes)
    delete every probe Job still outstanding

Step 4 is what makes ordering global without a controller thread: whichever campaign happens to be awake advances everybody, in (priority, campaign rank, campaign start) order.

A campaign’s rank and its hold. priority above is the order within a campaign – a probe before the work it gates, postprocessing before both – and it stays the leading key. What an operator sets is the campaign’s own rank, and it orders the runs: that is where a campaign spends all but a moment of its time, and all of what another campaign is waiting for. Every campaign is at 0 unless somebody says otherwise, and there the order is exactly what it was before ranks existed.

The rank deliberately does not reach a campaign’s own preconditions. A probe measures the node its campaign’s work will be sized from, and postprocessing turns finished runs into results; both are bounded and short, and ordinary work placed ahead of them does not make the queue fairer. It makes the campaign behind them fail: a probe is pinned to one node and is the largest pod its campaign asks for, so anything outranking it takes that node every pass, and a node still unmeasured after UNMEASURED_BATCH_LIMIT batches ends the campaign. A rank that reached them would turn “let other campaigns past” into “end this campaign”. Among items of the same kind the rank decides as usual – two campaigns’ probes order by it.

A paused campaign is not a low rank but no rank: its items are not candidates, however idle the cluster is. Neither setting touches work already created – both order what is queued, so a campaign demoted or held keeps the jobs it is running and gives up only the slots they release. That is why both are offered on a live campaign and stopping is not: nothing is preempted and no partial run is produced.

Because a paused campaign never reaches the placement walk, drain records its refusal separately – a campaign waiting for a machine and one waiting for a person need opposite responses, and leaving the last “no node has that free” standing would confuse them.

Scenario runs and postprocessing submit and track their Jobs through one helper, AdmittedJobs: one submit with a create callback per Job, then a poll per round that drains the queue, reads which created Jobs still run in one listing, releases the finished ones, and times the ones that cannot start. It respects two traps:

  • Only CREATED names may be asked about. running_jobs treats a 404 as finished, which is right for a reaped Job and catastrophic for one not yet created — the batch “finishes” on its first poll having produced nothing.

  • ``waiting_for_capacity`` comes from ``states()``, not from pods. A queued campaign has no pods, so a pod-based probe cannot see it and the stall verdict fires on a healthy campaign.

A campaign with per-node sizing. The probe is one ordinary run, pinned, at the declared sizing, before any work is placed on that node:

batch 0:  for each uncalibrated node -> claim_probe(node) -> submit(campaign#probes, pin=node)
          accepts_work(node) is False while its probe is out
          probe finishes -> record(node, measured, completed=<scenario reached a verdict>)
                         -> node accepts work, sized from its own figures
batch 1+: every node already calibrated -> no probes, work placed immediately
campaign end: forget_calibration(campaign)

Probes queue under a second owner (<campaign>#probes) so they stay out of the campaign’s progress counts — which means the batch’s finally must cancel both, or a probe still PLANNED holds its node out of the campaign for good.

A probe outlives its use in two ways, and each costs a node. Once the batch has created every job it defined, nothing of it can be placed on a node still being measured: the probes the queue has not created are dropped there and then, rather than created later at a trial’s cost for a figure no run of this batch can use. One already created is left to finish — it is most of a trial in, and its figure still spares the next batch of a search from re-probing that node — but when the batch ends it is deleted, not merely forgotten. A probe is pinned, so one left running holds its node against every other campaign until its own activeDeadlineSeconds, which is the trial’s outer backstop and many times a trial.

A probe that does not deliver a figure has two very different fates, and the difference is whether it ever ran:

  • It ran and its measurement was refused — no verdict, too few samples, throttled or OOM-killed against its own ceiling. That ends the campaign, at the moment of refusal: the node’s runs would use the starting allocation while every measured node’s used a figure, so the campaign would mix two allocations and nothing in the results would say which run got which. The message names the reason and the remedy for it.

  • It never ran at all — pinned to a node that had no room to spare for it. That is counted, not fatal: the node takes no work for that batch, is re-probed on the next one, and is left out of the campaign only if it goes unmeasured in UNMEASURED_BATCH_LIMIT batches. A probe the batch cancelled because it had nothing left to place is reported but not counted: it lost no race for capacity, so it says nothing about whether the node can be measured. A probe too large for any node’s capacity is a separate case and never reaches here — preflight in _start_probes refuses it before a single job exists.

A pinned item holds its node open. Skipping a non-fitting item is right when it can be served from anywhere later; a pinned one has a single candidate, so “later” only arrives if that node is left room. A probe is also the largest pod a calibrated campaign asks for — the declared sizing, while the calibrated jobs behind it run at a measured fraction of it — so skipping it handed its node to the smaller work it outranked, every pass. Priority ordered the queue and reserved nothing.

So a pinned item that does not fit claims its node for the rest of the pass: nothing further is placed there, the node drains as its work finishes, and the item goes on the pass where it fits. The claim is taken in priority order, so it only shuts out work ranking below the item holding the node, and it is per node — work that can go elsewhere still does.

It is taken only where the wait can end: if the node could not hold the item even empty, the node keeps working. Holding it would drain the machine and keep it drained, which is worse than not reserving at all. That question needs the node’s identity, which is why Capacity now carries an optional node_id — preflight asks only whether some node is large enough, and a probe pinned to the smallest machine of a mixed cluster was being judged against the biggest. An unknowable answer reserves, because a wrong reserve costs one batch and a wrong skip cost the whole campaign.

The count exists for what is left. At the end of one batch the two are indistinguishable, and failing on first sight made a busy minute at campaign start terminal — discarding a batch of finished, correct runs to do it. The probe is the largest pod a calibrated campaign asks for (the declared sizing summed over its containers) and, being pinned, it cannot spread, so n calibrated campaigns starting together need nodes × n of them placed at once. On a cluster of unlike machines that lands on the smallest node first, where one probe can be half the machine — preflight asks only whether the probe fits a node when empty, never whether it fits there alongside anything else. That contention drains; a node still unmeasured a batch later is not waiting on it.

Both cases leave the node on the declared sizing meanwhile, which is what a cluster with calibration switched off does anyway — a worse allocation, never a wrong result.

Because probes queue under their own owner, the batch loop reads the refusal for both keys. Reading only the campaign’s own key left a pinned probe’s wait with no line anywhere: the queue computed why it could not be placed on every drain and printed none of it, and the campaign then ended on a node it could not measure while naming a cause nobody had checked.

Sizing is per role, and this is a validity rule rather than a tuning one:

  • the system under test takes the measured peak as request and limit, so its budget is one it never throttles against;

  • everything else takes the sustained figure as its request and keeps its declared ceiling, because a simulator’s peak-to-mean ratio is ~18 and reserving its peak would cost more than no calibration at all;

  • memory is re-sized too, from the same probe, and for every role alike at the maximum rather than at a percentile — exceeding a memory limit is an OOM kill rather than a slowdown, so there is no tail to discard. Request and limit are set equal for the same reason: a soft memory ceiling only means “allowed to die”;

  • a calibrated figure is clamped to the declared value. Calibration sizes a node’s jobs down to what they need; raising a ceiling the author set is not what it is for, and preflight ran once on the declared sizing and is never re-asked per node.

Failure modes worth knowing

Symptom

Cause

Campaign creates nothing, forever

A sizing that fits allocatable but not allocatable - headroom. preflight now takes the reserve off, so this raises instead of hanging.

Runs discarded in a loop

A dead node read as fully free once its pods were evicted. budget() now excludes unschedulable nodes.

The whole batch created at once

Every item “fits” a growable cluster. Bounded by GROWTH_UNPINNED_LIMIT; a handful of unplaceable pods signals an autoscaler as loudly as a thousand.

Per-node sizing silently off

growable compared an autoscaler maximum for the whole cluster against a pool-filtered reading, so configuring a node pool made it permanently true.

A create retried forever

An RBAC change, a webhook or a quota looks identical to an API blip from here. Bounded, then dropped with its cause.

A finished batch discarded over one unmeasured node

A pinned probe that lost a race for free capacity read as a configuration fault. Counted now, and terminal only if it repeats — UNMEASURED_BATCH_LIMIT.

A probe waits with nothing said

Refusals key on the item’s owner, and probes have their own; only the campaign’s key was read. Both are read now.

A pinned probe never placed at all

Priority gave it first pick but no reservation, so the smaller work it outranked took its node every pass. A pinned item now holds its node open until it drains.

Where the numbers come from

Nothing is remembered between deployments. Capacity is measured each cycle from what the nodes advertise, minus bound pod requests, minus the per-node headroom (ROBOVAST_NODE_HEADROOM_CPU / ROBOVAST_NODE_HEADROOM_MEMORY, set on the service Deployment). Free disk is the smaller of allocatable - requested and what the kubelet measures free on the node’s filesystem less its own evictionHard nodefs.available threshold (read from the node’s configz, the kubelet’s 10% default where it states none): a node’s disk also holds images and logs no pod requested, and a job placed by requests alone could fill it past the threshold and be evicted. A node whose disk cannot be read keeps the request figure. Calibration is measured per campaign and deliberately never inherited by the next one — figures taken under one campaign’s contention describe a load the next never meets.

That is also why there is no cached per-node factor anywhere: measured across two campaigns on the same cluster, a container’s cost per simulated second moved up to 40% and the ranking between nodes inverted, so a factor that transfers does not exist.