Scheduling reference
The four modules that decide when a campaign’s jobs are created, where they run and how big they are. This page is the map and the interface; the reasoning lives elsewhere and is not repeated here:
Job Queueing — what an operator sees and configures.
Developer Guide §*Job admission internals* — why creation is the admission point, and why the queue is global rather than per caller.
The modules
Module |
Answers |
|---|---|
|
When. The global queue, the in-flight ledger, and the create-on-admit loop. |
|
How much room. Reads the live cluster; the only thing that talks to the API for capacity. |
|
Where. Node identity labels, the campaign node pool, the taint job pods tolerate. |
|
How big. Per-node sizing measured by a probe run, per campaign. |
They are layered, not circular: node_admission knows only the
BudgetProvider protocol, and
never a Kubernetes object. That is what lets the whole queue be tested without a cluster.
The interface
What a job needs, and what a node has. Both frozen, both plain numbers:
JobSizing(cpu: float, memory: int, gpu: int = 0) # summed over a pod's containers
Capacity(cpu, memory, gpu) # what one node holds when empty
NodeBudget(node_id, free_cpu, free_memory, free_gpu) # what one node has free NOW
Budget(nodes: tuple[NodeBudget], counted_jobs: frozenset, growable: bool)
counted_jobs is the double-counting fix: the provider reports which Jobs its reading
already saw, so a reservation stops being subtracted the instant the real pod starts being
subtracted. No timers, and no window where cores are counted twice or not at all.
The seam. One protocol, two methods, and the reason multi-tenancy can be added later without touching the loop:
class BudgetProvider(Protocol):
def budget(self) -> Budget: ... # free NOW — excludes unusable nodes
def capacities(self) -> list[Capacity]: ... # could EVER — includes them
The asymmetry is deliberate and was a bug when it was absent. “Free now” must exclude a node
that is cordoned, NotReady or carrying an untolerated taint, because counting it promises
room that cannot be spent. “Could ever” must include it, because
preflight()
raises a permanent error and a rebooting node is coming back. Both take the per-node
headroom off: a reserve that is never spendable is not part of either answer.
The controller. Everything a campaign does with the queue:
Call |
Meaning |
|---|---|
|
Enqueue a whole plan. |
|
Create as many globally-highest-ranked items as currently fit. Returns how many. A
paused campaign’s items are not candidates at all.
|
|
Release that job’s reservation. |
|
Drop an owner’s planned items and release its holds. Runs per batch. |
|
Drop only what the queue has not created; the owner keeps the holds behind its created items, which are pods that exist. For an owner with nothing left for a planned item to do. |
|
End a campaign’s per-node figures. Runs per campaign. |
|
Set the campaign’s rank, its hold, or both. |
|
|
|
Drop both, once the campaign is over. Runs per campaign, beside
|
|
Raise |
|
|
|
One line saying what this owner is waiting for — its own item count and the sizing the fit test actually used, not the queue’s total and not the declared figure. |
|
The campaign’s |
|
Pinnable node identities; whether the cluster can add nodes. |
cancel and forget_calibration are separate because their lifetimes are: reservations
belong to a batch, calibration to a campaign. Merging them made a search re-probe every node
every round.
Workflows
A batch. The campaign thread owns no admit loop — it enqueues, then polls its own items:
submit(campaign, plan, started_at=..., priority=0)
while True:
reap -> finished(key) for each vanished CREATED job
exit -> if every plan entry is FINISHED: break
drain() -> may create OTHER campaigns' jobs; that is the point
probes -> nothing PLANNED of its own left: drop_planned(campaign#probes)
publish -> waiting_for_capacity from states(), not from pods
sleep 2
finally:
cancel(campaign) # and cancel(campaign#probes)
delete every probe Job still outstanding
Step 4 is what makes ordering global without a controller thread: whichever campaign happens
to be awake advances everybody, in (priority, campaign rank, campaign start) order.
A campaign’s rank and its hold. priority above is the order within a campaign –
a probe before the work it gates, postprocessing before both – and it stays the leading key.
What an operator sets is the campaign’s own rank, and it orders the runs: that is where a
campaign spends all but a moment of its time, and all of what another campaign is waiting for.
Every campaign is at 0 unless somebody says otherwise, and there the order is exactly what
it was before ranks existed.
The rank deliberately does not reach a campaign’s own preconditions. A probe measures the
node its campaign’s work will be sized from, and postprocessing turns finished runs into
results; both are bounded and short, and ordinary work placed ahead of them does not make the
queue fairer. It makes the campaign behind them fail: a probe is pinned to one node and is the
largest pod its campaign asks for, so anything outranking it takes that node every pass, and a
node still unmeasured after UNMEASURED_BATCH_LIMIT batches ends the campaign. A rank that
reached them would turn “let other campaigns past” into “end this campaign”. Among items of
the same kind the rank decides as usual – two campaigns’ probes order by it.
A paused campaign is not a low rank but no rank: its items are not candidates, however idle the cluster is. Neither setting touches work already created – both order what is queued, so a campaign demoted or held keeps the jobs it is running and gives up only the slots they release. That is why both are offered on a live campaign and stopping is not: nothing is preempted and no partial run is produced.
Because a paused campaign never reaches the placement walk, drain records its refusal
separately – a campaign waiting for a machine and one waiting for a person need opposite
responses, and leaving the last “no node has that free” standing would confuse them.
Scenario runs and postprocessing submit and track their Jobs through one helper,
AdmittedJobs: one submit with a
create callback per Job, then a poll per round that drains the queue, reads which created
Jobs still run in one listing, releases the finished ones, and times the ones that cannot start.
It respects two traps:
Only CREATED names may be asked about.
running_jobstreats a 404 as finished, which is right for a reaped Job and catastrophic for one not yet created — the batch “finishes” on its first poll having produced nothing.``waiting_for_capacity`` comes from ``states()``, not from pods. A queued campaign has no pods, so a pod-based probe cannot see it and the stall verdict fires on a healthy campaign.
A campaign with per-node sizing. The probe is one ordinary run, pinned, at the declared sizing, before any work is placed on that node:
batch 0: for each uncalibrated node -> claim_probe(node) -> submit(campaign#probes, pin=node)
accepts_work(node) is False while its probe is out
probe finishes -> record(node, measured, completed=<scenario reached a verdict>)
-> node accepts work, sized from its own figures
batch 1+: every node already calibrated -> no probes, work placed immediately
campaign end: forget_calibration(campaign)
Probes queue under a second owner (<campaign>#probes) so they stay out of the
campaign’s progress counts — which means the batch’s finally must cancel both, or a probe
still PLANNED holds its node out of the campaign for good.
A probe outlives its use in two ways, and each costs a node. Once the batch has created
every job it defined, nothing of it can be placed on a node still being measured: the probes
the queue has not created are dropped there and then, rather than created later at a trial’s
cost for a figure no run of this batch can use. One already created is left to finish — it is
most of a trial in, and its figure still spares the next batch of a search from re-probing
that node — but when the batch ends it is deleted, not merely forgotten. A probe is
pinned, so one left running holds its node against every other campaign until its own
activeDeadlineSeconds, which is the trial’s outer backstop and many times a trial.
A probe that does not deliver a figure has two very different fates, and the difference is whether it ever ran:
It ran and its measurement was refused — no verdict, too few samples, throttled or OOM-killed against its own ceiling. That ends the campaign, at the moment of refusal: the node’s runs would use the starting allocation while every measured node’s used a figure, so the campaign would mix two allocations and nothing in the results would say which run got which. The message names the reason and the remedy for it.
It never ran at all — pinned to a node that had no room to spare for it. That is counted, not fatal: the node takes no work for that batch, is re-probed on the next one, and is left out of the campaign only if it goes unmeasured in
UNMEASURED_BATCH_LIMITbatches. A probe the batch cancelled because it had nothing left to place is reported but not counted: it lost no race for capacity, so it says nothing about whether the node can be measured. A probe too large for any node’s capacity is a separate case and never reaches here —preflightin_start_probesrefuses it before a single job exists.
A pinned item holds its node open. Skipping a non-fitting item is right when it can be served from anywhere later; a pinned one has a single candidate, so “later” only arrives if that node is left room. A probe is also the largest pod a calibrated campaign asks for — the declared sizing, while the calibrated jobs behind it run at a measured fraction of it — so skipping it handed its node to the smaller work it outranked, every pass. Priority ordered the queue and reserved nothing.
So a pinned item that does not fit claims its node for the rest of the pass: nothing further is placed there, the node drains as its work finishes, and the item goes on the pass where it fits. The claim is taken in priority order, so it only shuts out work ranking below the item holding the node, and it is per node — work that can go elsewhere still does.
It is taken only where the wait can end: if the node could not hold the item even empty, the
node keeps working. Holding it would drain the machine and keep it drained, which is worse than
not reserving at all. That question needs the node’s identity, which is why Capacity now
carries an optional node_id — preflight asks only whether some node is large enough,
and a probe pinned to the smallest machine of a mixed cluster was being judged against the
biggest. An unknowable answer reserves, because a wrong reserve costs one batch and a wrong skip
cost the whole campaign.
The count exists for what is left. At the end of one batch the two are indistinguishable,
and failing on first sight made a busy minute at campaign start terminal — discarding a batch of
finished, correct runs to do it. The probe is the largest pod a calibrated campaign asks for (the
declared sizing summed over its containers) and, being pinned, it cannot spread, so n
calibrated campaigns starting together need nodes × n of them placed at once. On a cluster of
unlike machines that lands on the smallest node first, where one probe can be half the machine
— preflight asks only whether the probe fits a node when empty, never whether it fits there
alongside anything else. That contention drains; a node still unmeasured a batch later is not
waiting on it.
Both cases leave the node on the declared sizing meanwhile, which is what a cluster with calibration switched off does anyway — a worse allocation, never a wrong result.
Because probes queue under their own owner, the batch loop reads the refusal for both keys. Reading only the campaign’s own key left a pinned probe’s wait with no line anywhere: the queue computed why it could not be placed on every drain and printed none of it, and the campaign then ended on a node it could not measure while naming a cause nobody had checked.
Sizing is per role, and this is a validity rule rather than a tuning one:
the system under test takes the measured peak as request and limit, so its budget is one it never throttles against;
everything else takes the sustained figure as its request and keeps its declared ceiling, because a simulator’s peak-to-mean ratio is ~18 and reserving its peak would cost more than no calibration at all;
memory is re-sized too, from the same probe, and for every role alike at the maximum rather than at a percentile — exceeding a memory limit is an OOM kill rather than a slowdown, so there is no tail to discard. Request and limit are set equal for the same reason: a soft memory ceiling only means “allowed to die”;
a calibrated figure is clamped to the declared value. Calibration sizes a node’s jobs down to what they need; raising a ceiling the author set is not what it is for, and
preflightran once on the declared sizing and is never re-asked per node.
Failure modes worth knowing
Symptom |
Cause |
|---|---|
Campaign creates nothing, forever |
A sizing that fits |
Runs discarded in a loop |
A dead node read as fully free once its pods were evicted. |
The whole batch created at once |
Every item “fits” a growable cluster. Bounded by |
Per-node sizing silently off |
|
A create retried forever |
An RBAC change, a webhook or a quota looks identical to an API blip from here. Bounded, then dropped with its cause. |
A finished batch discarded over one unmeasured node |
A pinned probe that lost a race for free capacity read as a configuration fault.
Counted now, and terminal only if it repeats — |
A probe waits with nothing said |
Refusals key on the item’s owner, and probes have their own; only the campaign’s key was read. Both are read now. |
A pinned probe never placed at all |
Priority gave it first pick but no reservation, so the smaller work it outranked took its node every pass. A pinned item now holds its node open until it drains. |
Where the numbers come from
Nothing is remembered between deployments. Capacity is measured each cycle from what the nodes
advertise, minus bound pod requests, minus the per-node headroom
(ROBOVAST_NODE_HEADROOM_CPU / ROBOVAST_NODE_HEADROOM_MEMORY, set on the service
Deployment). Free disk is the smaller of allocatable - requested and what the kubelet measures
free on the node’s filesystem less its own evictionHard nodefs.available threshold (read
from the node’s configz, the kubelet’s 10% default where it states none): a node’s disk also
holds images and logs no pod requested, and a job placed by requests alone could fill it past the
threshold and be evicted. A node whose disk cannot be read keeps the request figure. Calibration
is measured per campaign and deliberately never inherited by the next
one — figures taken under one campaign’s contention describe a load the next never meets.
That is also why there is no cached per-node factor anywhere: measured across two campaigns on the same cluster, a container’s cost per simulated second moved up to 40% and the ranking between nodes inverted, so a factor that transfers does not exist.