# Wait Desk > Read a queue's four numbers and see what they actually produce. Utilisation is > `arrivals x service / workers`, and the wait is `rho/(1-rho)` times the service time: > 1x at 50%, 4x at 80%, 9x at 90%, 19x at 95%, 99x at 99% - and the service time never > changed. So the same five points of load cost 1.22x at 50% and 6.32x at 94%. A > backlog drains at the SPARE capacity, so an outage takes the SAME `rho/(1-rho)` > times as long to clear as it lasted: a one-minute stop at 90% is nine minutes. > Pooling beats utilisation - at the same 90%, one worker waits 900ms and fifty wait > 7ms. Variability costs as much as the mean. And retries are arrival rate. Free > browser-side Erlang C, Little's Law and Kingman, five paid lanes, nothing inspected. Live at https://wait-desk.skillsafe.ai/ · API at https://wait-desk.skillsafe.ai/api.html Derived from the `trigger-dev` and `upstash-qstash` skills in `sickn33/antigravity-awesome-skills` and the `python-background-jobs` skill in `wshobson/agents`. Not affiliated with or endorsed by their authors. ## The one thing to know **A queue's wait is not its service time, and the last five points of utilisation cost more than the first eighty.** ```text utilisation the wait as a multiple queue depth drain multiple 50% 100ms 1× 0.5 1× 70% 233ms 2.33× 1.63 2.33× 80% 400ms 4× 3.2 4× 90% 900ms 9× 8.1 9× 95% 1.9s 19× 18 19× 99% 9.9s 99× 98 99× ``` One worker, a 100ms service time. **The wait is `rho/(1-rho)` times the service time** and the service time never changes. The depth is `Lq = lambda x Wq` by Little's Law - exact, needing no measurement. And the last column is the same coefficient again: it is how many times the outage a recovery takes. **The same five points of load, at each point on the curve.** ```text from -> to wait before wait after factor 50% -> 55% 100ms 122ms 1.22× 70% -> 75% 233ms 300ms 1.29× 80% -> 85% 400ms 567ms 1.42× 90% -> 95% 900ms 1.9s 2.11× 94% -> 99% 1.57s 9.9s 6.32× ``` From 50% five points cost 1.22×; from 94% they cost 6.32× - 5.17× the effect for identical extra traffic. "Keep it under eighty percent" is a rule that fails exactly where it is needed. **A backlog drains at the SPARE capacity.** ```text utilisation backlog spare time to clear multiple 50% 3,000 50/s 1m 1× 70% 4,200 30/s 2.33m 2.33× 80% 4,800 20/s 4m 4× 90% 5,400 10/s 9m 9× 95% 5,700 5/s 19m 19× 99% 5,940 1/s 1.65h 99× ``` A 1m full stop on 100/s of capacity. The backlog is exactly `arrivals x outage`; it clears at `capacity - arrivals` because **the arrivals keep coming**. At 90% that is 9m of recovery for 1m of failure. **Pooling beats utilisation.** ```text workers arrivals P(wait) the wait vs one worker 1 9/s 90% 900ms 100% 2 18/s 85.3% 426ms 47.4% 4 36/s 78.8% 197ms 21.9% 10 90/s 66.9% 67ms 7.43% 50 450/s 36.4% 7.28ms 0.81% ``` **Every row is at 90% utilisation** with identical load per worker. One waits 900ms and 50 wait 7.28ms - 0.81% of it. Erlang C is convex in the worker count, so "we are at 90%" says nothing without the pool count. **Variability costs as much as the mean.** ```text service time Cs^2 Kingman the wait constant service (M/D/1) 0 0.5 450ms moderately steady 0.5 0.75 675ms exponential (M/M/1) 1 1 900ms a heavy tail 4 2.5 2.25s a very heavy tail 10 5.5 4.95s ``` All at 90% with the same 100ms MEAN. Kingman scales the wait by `(Ca^2 + Cs^2)/2`, so constant service waits 450ms and a `cs2=10` tail waits 4.95s. `Cs^2` is `variance / mean^2` - one query away from being a measurement. **And retries are arrival rate.** ```text failure rate 2 attempts 3 attempts 5 attempts 1% 1.01x 1.0101x 1.0101x 5% 1.05x 1.0525x 1.0526x 10% 1.1x 1.11x 1.1111x 20% 1.2x 1.24x 1.2496x 30% 1.3x 1.39x 1.4251x 50% 1.5x 1.75x 1.9375x ``` Every attempt is a real job, so the multiplier on the ARRIVAL rate is `1 + p + p^2 + ...` - **not the attempt count**. A queue at 90% has only 10% of headroom, so a 10% multiplier consumes all of it - and the retries raise the wait, which is what caused the timeouts that produced the retries. ## The curve | Utilisation | Arrivals | The wait | As a multiple | Queue depth | Response time | | --- | --- | --- | --- | --- | --- | | 50% | 5/s | **100ms** | 1× | 0.5 | 200ms | | 70% | 7/s | **233ms** | 2.33× | 1.63 | 333ms | | 80% | 8/s | **400ms** | 4× | 3.2 | 500ms | | 90% | 9/s | **900ms** | 9× | 8.1 | 1s | | 95% | 9.5/s | **1.9s** | 19× | 18 | 2s | | 99% | 9.9/s | **9.9s** | 99× | 98 | 10s | One worker, a 100ms service time, exponential arrivals and service. **The wait is `ρ/(1−ρ)` times the service time** and the service time never changes: at 50% a job waits one service time, at 90% nine, at 99% ninety-nine. The depth is `λ × Wq` by Little's Law - the figure a dashboard shows and almost nobody predicts. ## What the last five points cost | From | To | Wait before | Wait after | Factor | Added | | --- | --- | --- | --- | --- | --- | | 50% | 55% | 100ms | **122ms** | 1.22× | 22ms | | 70% | 75% | 233ms | **300ms** | 1.29× | 67ms | | 80% | 85% | 400ms | **567ms** | 1.42× | 167ms | | 90% | 95% | 900ms | **1.9s** | 2.11× | 1s | | 94% | 99% | 1.57s | **9.9s** | 6.32× | 8.33s | **The same five points of load, at each point on the curve.** From 50% it costs 1.22×; from 94% it costs 6.32× - 5.17× the effect for identical extra traffic. This is why "keep it under eighty percent" is a rule that fails exactly where it is needed, and why an average utilisation past the knee is not a plan: every excursion above the average costs more than the excursion below it saves. ## The drain | Utilisation | Arrivals | Backlog after the stop | Spare capacity | Time to clear | As a multiple | | --- | --- | --- | --- | --- | --- | | 50% | 50/s | 3,000 | 50/s | **1m** | 1× | | 70% | 70/s | 4,200 | 30/s | **2.33m** | 2.33× | | 80% | 80/s | 4,800 | 20/s | **4m** | 4× | | 90% | 90/s | 5,400 | 10/s | **9m** | 9× | | 95% | 95/s | 5,700 | 5/s | **19m** | 19× | | 99% | 99/s | 5,940 | 1/s | **1.65h** | 99× | A 1m full stop on a queue with 100/s of capacity. The backlog is exactly `arrivals × outage` and needs no queueing theory at all. **It then clears at the SPARE capacity - `capacity − arrivals` - not at the capacity**, because the arrivals keep coming: the multiple is `ρ/(1−ρ)`, which is the same coefficient as the wait. At 90% a one-minute stop is nine minutes of recovery. "We were only down a minute" is a statement about the outage, not the incident. ## Pooling at the same utilisation | Workers | Arrivals | P(wait) | The wait | As a multiple | Versus one worker | | --- | --- | --- | --- | --- | --- | | 1 | 9/s | 90% | **900ms** | 9× | 100% | | 2 | 18/s | 85.3% | **426ms** | 4.26× | 47.4% | | 4 | 36/s | 78.8% | **197ms** | 1.97× | 21.9% | | 10 | 90/s | 66.9% | **67ms** | 0.67× | 7.43% | | 50 | 450/s | 36.4% | **7.28ms** | 0.07× | 0.81% | **Every row is at 90% utilisation** and a 100ms service time - the load per worker is identical. One worker waits 900ms and 50 wait 7.28ms, which is 0.81% of it: Erlang C is convex in the worker count, so N pools of one always wait more than one pool of N at the same load. **So "we are at 90%" says nothing without the pool count**, and splitting a pool to isolate tenants is a latency decision as well as an isolation one. ## What variability does | Service time | Ca² | Cs² | Kingman factor | The wait | Versus exponential | | --- | --- | --- | --- | --- | --- | | constant service (M/D/1) | 1 | 0 | 0.5 | **450ms** | 0.5× | | moderately steady | 1 | 0.5 | 0.75 | **675ms** | 0.75× | | exponential (M/M/1) | 1 | 1 | 1 | **900ms** | 1× | | a heavy tail | 1 | 4 | 2.5 | **2.25s** | 2.5× | | a very heavy tail | 1 | 10 | 5.5 | **4.95s** | 5.5× | All at 90% utilisation with a 100ms MEAN service time, which never changes. Kingman's approximation scales the wait by `(Ca² + Cs²)/2`, so **constant service halves it and a heavy tail multiplies it** - the mean is identical in every row. Halving the variance is worth as much as halving the mean, and it is usually the cheaper project: `Cs²` is `variance ÷ mean²`, so it is one query away from being a measurement rather than an assumption. ## What retries add to the arrivals | Failure rate | 2 attempts | 3 attempts | 5 attempts | | --- | --- | --- | --- | | 1% | 1.01× | 1.0101× | 1.0101× | | 5% | 1.05× | 1.0525× | 1.0526× | | 10% | 1.1× | 1.11× | 1.1111× | | 20% | **1.2×** | **1.24×** | **1.2496×** | | 30% | **1.3×** | **1.39×** | **1.4251×** | | 50% | **1.5×** | **1.75×** | **1.9375×** | The multiplier on the ARRIVAL RATE, because every attempt is a real job on the queue. It is the finite geometric sum `1 + p + p² + …` - **not the attempt count**: at 50% over 5 attempts it is 1.938×, not 5×. Small as those numbers look, a queue at 90% has only 10% of headroom, so a 10% multiplier is exactly enough to consume all of it - and the retries raise the wait, which is what caused the timeouts that produced the retries. ## Constants and thresholds | Constant | Value | What it decides | | --- | --- | --- | | `CLIFF_RHO` | 90% | utilisation past this is called out as being on the steep part of the curve | | `SLOW_DRAIN_MULTIPLE` | 2× | a recovery this many times the outage is a warning. The drain exceeds the outage for any utilisation above 50%, so firing at 1× would have been furniture | | `HIGH_VARIABILITY` | 1.5 | a `cs2=` past this is worth naming, because Kingman scales the wait by it directly | | `RETRY_LOAD_SHARE` | 5% | retries adding this much load are worth naming | | `POOLING_GAIN` | 25% | pooling saving this much of the wait is worth naming | | `POOLING_MATTERS_SHARE` | 10% | …and only if the wait it saves is at least this share of the target. Pooling always wins mathematically, so the relative test alone fired on nearly every multi-class sheet | | `DEFAULT_TARGET_MS` | 1s | the assumed wait target | | `DEFAULT_OUTAGE_S` | 1m | the assumed outage, for the drain | | `DEFAULT_WORKERS` | 1 | the assumed worker count - one, which is the worst case for any utilisation and therefore the pessimistic direction | | `DEFAULT_CA2` / `CS2` | 1 / 1 | the assumed variability: exponential, which is what M/M/c means and makes Kingman's factor exactly 1 | None of these changes a utilisation, a wait or a drain time - those are arithmetic on the four numbers the sheet declares. The thresholds decide only which of the exact facts gets called a warning, and the defaults decide only what is assumed when the sheet is silent - which is always said out loud rather than folded in. ## Sheet grammar A sheet is a header of `KEY: value` lines, then a `QUEUES:` block with **one job class per line**. Order does not matter here - a set of queues is a set - so the reader's only job is to lose nothing. ```text JOB: the thumbnail pipeline, as it runs today TARGET: 500ms the wait a class is allowed OUTAGE: 60s a full stop, for the drain question POOLED: no do the classes share one worker pool? WORKERS: 10 a default for classes that do not say QUEUES: thumbnails arrivals=90/s service=100ms workers=10 fail=10% attempts=3 emails arrivals=20/s service=40ms workers=1 reports arrivals=2/s service=4s workers=10 cs2=4 ``` **Header keys.** `TARGET:` and `OUTAGE:` take `500ms`, `2s`, `10m` or `1h`; assumed 1s and 1m. `POOLED:` takes a yes or a no. `WORKERS:` is a default for classes that do not declare their own. `JOB:`, `SYSTEM:` and `NOTE:` are free prose and are read as headers rather than as broken lines. Anything else is reported by name rather than ignored. **Class properties.** `service=` is required and takes `100ms`, `1.5s` or `2m` - a bare number is MILLISECONDS. `arrivals=` takes `90/s`, `5400/min` or `100000/day`; a bare number is PER SECOND, and the unit decides the answer completely. `workers=` is a count. `cs2=` and `ca2=` are the squared coefficients of variation of the service time and the arrivals (`variance ÷ mean²`); assumed 1, which is exponential. `fail=` takes `10%` or `0.1` and `attempts=` a count - together they multiply the ARRIVAL rate by `1 + p + p² + …`. `target=` overrides the header for one class. **What a class with no `service=` does.** It is dropped, and named. Capacity is `workers ÷ service time`, so without the divisor there is no capacity, no utilisation and no place on the curve - and it also vanishes from the pooling comparison, which is about the SUM of the arrivals against the SUM of the workers. That is why it is an error rather than a note. ## Lanes | Lane | What it produces | | --- | --- | | `plan` | Size a queue that meets its target | | `read` (primary) | What this queue actually waits, and how deep it gets | | `wait` | The wait: what utilisation costs and what a worker buys | | `drain` | The backlog: how deep it goes and how long it takes to clear | | `decide` | Decide what changes: a worker, the variance, or the arrival rate | ## A worked sheet ```text JOB: the thumbnail pipeline, as it runs today TARGET: 500ms OUTAGE: 60s QUEUES: thumbnails arrivals=90/s service=100ms workers=10 fail=10% attempts=3 emails arrivals=20/s service=40ms workers=1 reports arrivals=2/s service=4s workers=10 cs2=4 ``` | Class | Arrivals | Service | Workers | Utilisation | The wait | Drain after the outage | | --- | --- | --- | --- | --- | --- | --- | | `thumbnails` | 90/s (99.9/s with retries) | 100ms | 10 | **0.999** | 9.96s (99.6×) | 16.65h (999×) | | `emails` | 20/s | 40ms | 1 | **0.8** | 160ms (4×) | 4m (4×) | | `reports` | 2/s | 4s, cs²=4 | 10 | **0.8** | 2.05s (0.51×) | 4m (4×) | The sheet in the worked example. 3 classes, 121.9/s against 127.5/s on 21 workers, worst 0.999 waiting 9.96s (99.6× its service time), 1m out drains in 16.65h (999×), 2 past target, pooling saves 99.5%. **Read the last two columns together**: they are the same coefficient, `ρ/(1−ρ)`, applied to the service time and to the outage - which is why a queue that feels comfortable in steady state recovers slowly from anything. ## Findings | Finding | Severity | Scope | What it says | | --- | --- | --- | --- | | `NO-QUEUE-TO-READ` | error | sheet | The sheet declares no job classes | | `A-CLASS-HAS-NO-SERVICE-TIME` | error | class | A class declares no service time | | `THE-SHEET-HAS-LINES-THIS-PAGE-COULD-NOT-READ` | warning | sheet | Some lines were not understood | | `THE-TARGET-WAS-NOT-DECLARED` | note | sheet | The latency target was assumed | | `THE-WORKER-COUNT-WAS-NOT-DECLARED` | note | sheet | The worker count was assumed | | `THE-VARIABILITY-WAS-NOT-DECLARED` | note | sheet | The variability was assumed to be exponential | | `WHAT-EACH-CLASS-COSTS` | note | class | Every class, its capacity and its utilisation | | `A-CLASS-IS-PAST-ITS-TARGET` | warning | class | A class waits longer than its target allows | | `A-CLASS-RUNS-ON-ONE-WORKER` | note | class | A class has a single worker, which is the worst case for its load | | `THE-WAIT-IS-A-MULTIPLE-OF-THE-SERVICE-TIME` | note | wait | What each class waits, as a multiple of its own service time | | `THE-QUEUE-DEPTH-BY-LITTLES-LAW` | note | wait | How deep the queue sits, from the wait rather than from a dashboard | | `THE-UTILISATION-IS-IN-THE-CLIFF` | warning | wait | A class is in the region where the wait grows faster than the load | | `VARIABILITY-COSTS-AS-MUCH-AS-THE-MEAN` | warning | wait | A class's variability is doing as much to the wait as its mean | | `A-CLASS-IS-UNSTABLE` | error | load | A class arrives faster than it can be served | | `RETRIES-ARE-ARRIVAL-RATE` | warning | load | The retries are a measurable share of the arrivals | | `RETRIES-MAKE-IT-UNSTABLE` | error | load | A class is stable until its retries are counted | | `POOLING-WOULD-BEAT-THIS` | warning | load | One shared pool would wait less than these separate queues | | `WHAT-ONE-MORE-WORKER-BUYS` | note | load | What adding a single worker does to each class's wait | | `THE-BACKLOG-AFTER-AN-OUTAGE` | note | drain | How many jobs are waiting when the outage ends | | `HOW-LONG-AN-OUTAGE-TAKES-TO-DRAIN` | note | drain | How long the backlog takes to clear, at the SPARE capacity | | `THE-DRAIN-IS-LONGER-THAN-THE-OUTAGE` | warning | drain | The recovery takes longer than the failure did | | `THE-SAME-COEFFICIENT-TWICE` | note | drain | The wait multiple and the drain multiple are the same number | 22 findings: 4 errors, 7 warnings and 11 notes. **Nothing that fires on every usable sheet is a warning.** Every queue has a utilisation, a wait, a depth and a drain time - so those are notes. What gets a warning is a class past its target, a utilisation on the steep part of the curve, a variability doing more than the mean, retries that are a measurable share of the arrivals, a split that costs more than it isolates, and a recovery more than 2× the outage. The drain warning in particular fires at 2× rather than at 1×, because the drain exceeds the outage for any utilisation above 50% and most real queues run above that. ## What this page cannot do **These are means, not percentiles.** `Wq` is the average wait in the queue. The distribution behind it is roughly exponential, so the p95 is about three times the mean and the p99 about 4.61 times it - which means a queue whose MEAN wait meets your target still misses it for one request in twenty. This page does not compute percentiles and says so rather than implying them. **M/M/c is a model, and Kingman is an approximation.** Poisson arrivals and exponential service are the textbook case; Kingman's `(Ca² + Cs²)/2` correction extends it to any variability and is accurate in the heavy-traffic region and optimistic in the light one. Both assume ONE queue, FIFO, no priorities, no abandonment and no feedback from the wait to the arrival rate - and every one of those assumptions is usually false in a way that makes the real wait worse. **A mean arrival rate is not a traffic model.** Real arrivals are bursty and correlated, so a queue whose average utilisation is comfortable can spend minutes at a utilisation that is not. `Ca²` is where that goes, and setting it above 1 is the honest way to say "our traffic is spikier than Poisson". **The numbers are yours.** This page measures nothing: the arrival rate, the service time and its variance are all figures you supply, so a service time 20% out makes every wait figure here wrong by much more than 20% - the curve is steep exactly where it matters. Measure `cs2` as `variance ÷ mean²` and the whole page becomes exact rather than indicative. **It has not seen your queue, your workers or your dashboard.** No claim about whether your workers really pull from one queue, whether your retries have backoff, whether a job that fails is re-enqueued at the back or the front, or what your autoscaler does. A queue with a wait-dependent arrival rate - users retrying because it is slow - is outside every formula here, and it is the case that turns a slow queue into an outage. **And there is no priority, no fairness and no deadline in this model.** A sheet with several classes is several independent queues unless it says `POOLED: yes`, and even then the pooled figure assumes FIFO across the mixture. Priorities help the class you favour by exactly as much as they hurt the others, and this page will not pretend to know which you meant. ## API `POST https://api.skillsafe.ai/v1/app-api/run` with an app session token. The body IS the input object — never wrapped in an `input` key. Fields: `task` (one of `plan`, `read`, `wait`, `drain`, `decide`), `rules`, `prescan`, plus the lane's own fields. `POST .../estimate` with the same body returns `hold_credits` and costs nothing. ## Provenance Every table and every figure above is generated from `queue.js` by `build-skill.js`. Nothing is typed twice, so nothing can drift.