A glossary of tiny lectures, part 6

Week 6: statistics and ranking

By Tim Menzies, published 2026-10-05, updated 2026-10-05

Summary: Last week ended with twenty holdouts and a spread. This week asks the only question that matters about a spread: is that gap real, or is it noise? Three cheap tests (means, ranks, shapes), all three of which must agree; a rank table where ties share a rank; and the row-versus-row comparison (domination) that refuses to speak when the goals disagree. Demos in ezr-eg6.lua; machinery verbatim from ezr.lua and ezr-apps.lua.

Is that gap real?

New acronyms: KS (Kolmogorov-Smirnov).

cohen: The mean gap, measured in pooled standard deviations. If two bags differ by less than about a third of their own spread, nobody will ever notice the difference in practice — so the default is 0.35. This is an EFFECT SIZE, not a p-value: it asks "how big" rather than "how surprising", which is the right question when your sample size is whatever you could afford.
```
function cohen(xs,ys,    x,y,n,m,sd)
  x, y = adds(xs), adds(ys)
  n, m = x.n, y.n
  sd = sqrt(((n-1)*x:div()^2 + (m-1)*y:div()^2)/(n+m-2))
  return abs(x.mu - y.mu) / (sd + TINY) end
```
cliffs delta: Forget the means; count the comparisons. Of all pairs (x,y), how many have x above y, and how many below? The imbalance, scaled to 0..1, is Cliff's delta, and the default threshold (0.195) is the standard "small effect" line. Because it reads only ranks, no outlier can drag it around — the complaint that kills cohen on skewed data.
```
function cliffs(xs,ys,    gt,lt,j,k)
  gt, lt, j, k = 0, 0, 0, 0
  for _, x in ipairs(xs) do
    while j < #ys and ys[j+1] <  x do j = j + 1; k = j end
    while k < #ys and ys[k+1] == x do k = k + 1 end
    gt = gt + j; lt = lt + #ys - k end
  return abs(gt - lt) / (#xs * #ys) end
```
ks (Kolmogorov-Smirnov): Compare the SHAPES. Walk both cdfs together, one distinct value at a time, and record the widest vertical gap between them. Divide by the critical value √((n_x+n_y)/(n_xn_y)) and the answer reads as "how many critical units apart" — above 1.36 means different at the usual 5% line. Two bags can share a mean and a median and still fail this test, because one is a lump and the other is two humps.
```
function ks(xs,ys,    nx,ny,d,p,q,v)
  nx, ny  = #xs, #ys
  d, p, q = 0, 0, 0
  while p < nx and q < ny do
    v = min(xs[p+1], ys[q+1])
    while p < nx and xs[p+1] == v do p = p + 1 end
    while q < ny and ys[q+1] == v do q = q + 1 end
    d = max(d, abs(p / nx - q / ny)) end
  return d / ((nx + ny) / (nx * ny)) ^ 0.5 end
```
At 5%, this test cries wolf on about one comparison in twenty even when both bags come from the same generator. That is not a bug in the code; it is the price of the threshold, and it is why one test alone should never decide anything.
same: Three tests, three different questions — means, ranks, shapes. Two samples count as the same only when all three agree, so a method that fools one test still has to fool the others. Note the ordering: and is lazy, so the cheapest test runs first and usually decides.
```
function same(xsort,ysort,Cohen,Ks,Cliffs)
  return cohen( xsort,ysort) <= (Cohen  or .35)
     and cliffs(xsort,ysort) <= (Cliffs or .195)
     and ks(    xsort,ysort) <= (Ks     or 1.36) end
```

ranks: The table a paper actually prints. Sort the treatments by median, walk down the sorted list, and give a treatment a NEW rank only when it is not same as the one above it. Ties therefore share a rank, and rank 0 is a SET of winners, not a name. Reporting a single winner when four methods tie is the most common statistical lie in this field.

function ranks(d,big,    mid,dd,sign,out,win,rank,best)
  mid = function(t) return t[floor(#t / 2) + 1] end
  dd  = {}; for k,v in pairs(d) do dd[k] = sorted(v) end
  sign = big and -1 or 1
  out, win, rank, best = {}, {}, -1, nil
  for _, k in ipairs(keysort(keys(dd),
                function(k) return sign * mid(dd[k]) end)) do
    if best == nil or not same(dd[best], dd[k]) then
      rank, best = rank + 1, k end
    if rank == 0 then win[1+#win] = k end
    out[k] = rank end
  return {winners=win, ranks=out} end

dominates: Comparing rows, not samples — and with no weights anywhere in it. Row a dominates b when a is no worse on every goal and better on at least one. That is the honest multi-goal comparison, and its honesty is also its weakness: most pairs are incomparable (better here, worse there), so domination often refuses to speak. Hence disty, which buys a total order with one assumption (all goals weigh the same).
```
function TBL.dominates(i,r1,r2,    d1,d2,better,worse)
  if i.model then i:disty(r1); i:disty(r2) end
  better, worse = false, false
  for _,y in ipairs(i.cols.y) do
    d1 = abs(y:norm(r1[y.at]) - y.heaven)
    d2 = abs(y:norm(r2[y.at]) - y.heaven)
    if d1 < d2 then better = true end
    if d1 > d2 then worse  = true end end
  return better and not worse end
```
front: The rows nothing dominates: the Pareto frontier of whatever sample you hold. Two facts about real fronts, both visible in ezr-eg6.lua --fronts: they are small (a handful of 64 rows), and their mean disty is far better than everything else — which is why the one-number shortcut usually lands in about the right place. Remember the Pareto zoom effect: the good region is rare and clumped, so zooming beats wandering.
wins: A score a stranger can read. Label the whole pool (cheating, but this is scoring, not searching), then rescale: 100 = the best row in the pool, 0 = the median row, negative = worse than doing nothing. Reporting raw disty invites "is 0.31 good?"; reporting wins answers it.
```
function TBL.wins(i,rows,    ys,lo,b4)
  ys = sorted(map(rows or i.rows, i:Y()))
  lo, b4 = ys[1], ys[floor(#ys/2)+1]
  return function(r)
    return max(-100, min(100,
      100*(1 - (i:disty(r)-lo) / (b4-lo+TINY)))) end end
```
Cohen: Statistical power analysis for the behavioral sciences. Jacob Cohen. 2nd edition, Lawrence Erlbaum, 1988.

Cliff: Dominance statistics: Ordinal analyses to answer ordinal questions. Norman Cliff. Psychological Bulletin 114(3), 1993, 494-509.

Massey: The Kolmogorov-Smirnov test for goodness of fit. Frank J. Massey Jr. JASA 46(253), 1951, 68-78.

Arcuri & Briand: A practical guide for using statistical tests to assess randomized algorithms in software engineering. Andrea Arcuri, Lionel Briand. ICSE 2011.