A glossary of tiny lectures, part 2

The machinery under the weekly demos

By Tim Menzies, published 2026-08-31, updated 2026-08-31

Summary: Glossary 1 says what weeks one and two compute: Num, Sym, Tbl, distx, disty. This page is the machinery those demos ride on. Every term here was found the same way homework 1 and 2 find code: start at the demos in ezr-eg1.lua and ezr-eg2.lua, trace only the functions they actually call, down the require chain into ezr.lua and ezr-lib.lua. Same deal as before: each entry is a tiny lecture — a hook, the idea, the math if any, then the code, verbatim from the source.

Lua under the hood

Four language tricks that every file in this code leans on. New acronyms: none.

closure: A function that remembers where it was born. csv opens a file, then returns a function that takes no arguments — yet each call yields the next row. How? The returned function captures the variable f (the line iterator) from its birthplace, and f stays alive between calls. Code plus captured variables: that is a closure — an object with one method, no class required.
```
function csv(file,    f)
  f = io.lines(pathname(file))
  return function(    t,l)
    for line in f do
      l = line:gsub("\239\187\191","")   -- strip any BOM
              :gsub("%%.*",""):match"^%s*(.-)%s*$"
      if l ~= "" then
        t={}                        -- (.-), keeps empty cells
        for s in (l..","):gmatch"(.-)," do t[#t+1]=thing(s) end
        return t end end end end
```
Closures are everywhere here. TBL.Y wraps a whole table into a one-argument key function (used by --disty to sort rows):
```
function TBL.Y(i)
  return function(r) return i:disty(r) end end
```
Python gets the same effect from yield (a generator is a closure the language wrote for you) — see the streaming csv in your port.
iterator: What does for row in csv(the.file) do actually do? Lua's generic for just calls the function once per lap and stops when it returns nil. Anything that returns a function can be looped. iter makes plain lists loop the same way, so adds and Tbl accept a list or a live stream, no flag needed:
```
function iter(src,    at)
  if type(src) == "function" then return src end
  at = 0; return function() at = at + 1; return src[at] end end
```
Python's for loop is programmable in exactly the same way: it loops over anything answering __iter__/__next__, and the quickest way to write one is a generator — yield turns an ordinary function into a resumable stream, so the Lua csv above is four lines of Python:
```
def csv(file):
  for line in open(file):
    yield [thing(s) for s in line.strip().split(",")]

for row in csv("auto93.csv"): ...   # same shape as the Lua
```
Either way the moral is the same: the for loop is not magic syntax over lists — it is an open protocol, and your own code can join it.
metatable (colon calls, __index): n:mid() is sugar for n.mid(n): the receiver rides in as the first argument, which is why every method's first parameter is i. But n is a plain table holding only {n=5, mu=3, m2=10} — there is no mid inside it. When a lookup misses, Lua tries the table's metatable, field __index. The whole trick, in the simplest possible example — methods live in one table, data in another, and __index joins them:
```
NUM = {}
function NUM.mid(i) return i.mu end
function NUM.div(i) return (i.m2/(i.n-1))^0.5 end

n = setmetatable({n=5, mu=3, m2=10}, {__index=NUM})
print(n:mid())  -- 3: no mid in n, so Lua asks NUM
print(n:div())  -- 1.58: same story
```
new wires that up for every object in this system, and it is the entire class system:
```
function new(kl,t)
  kl.__index=kl;kl.__tostring=show; return setmetatable(t,kl) end
```
metamethod (operator overloading): The --without demo subtracts one summary from another with a minus sign: w = adds(...) - b. Minus, on tables? Metatable keys starting __ are metamethods: __index answers a lookup miss (the previous entry), __sub answers the - operator, __tostring answers print. Define NUM.__sub and summaries become subtractable algebra. Seen in action, straight from the --without demo: pour {10,20,30} on top of a summary of {1,2,3,4,5}, then subtract a summary of {10,20,30} with a minus sign, and the first summary comes back:
```
a, b = adds{1,2,3,4,5}, adds{10,20,30}
w = adds({10,20,30}, adds{1,2,3,4,5}) - b
print(show{mu=w.mu, sd=w:div()})   -- {:mu 3 :sd 1.58}
```
As a picture, that minus sign is one gaussian leaving another. When the curves overlap, subtraction moves BOTH moments: take a narrow curve out of a wide one and what remains has an in-between spread — and a different middle. Here a = {4,8,12,16} and b = {18,20,22}:
```
 a+b: mu=14.3 sd=6.6       b: mu=20 sd=2      a: mu=10 sd=5.2
        ****                     *
     **********        -        ***      =        ****
 ******************            *****          **********
---------+---------          ----+----       ------+-------
        14.3                     20                10
```
Check it with the pool equations (next section): pulling b's 60 out of the total 100 drags mu from 14.3 back to 10, and removing a tight clump parked off-center shrinks the spread from 6.6 to only 5.2 — not to 2's neighborhood, because much of the combined width WAS the gap between the two middles.

How the subtraction actually works — the counts and the algebra — is the pool entry, below. Python spells the same idea with dunders: __sub__, __repr__. Overload only when the algebra is real — here, summaries genuinely subtract, so a - b reads as the mathematics it is.

The little verbs

ezr-lib is not batteries-included; it is batteries-handmade. Four verbs carry most of the weight. New acronyms: DSU.

fun (map, kap): Glossary 1's minkowski entry showed the style: pass little functions into bigger ones. fun generalizes it — a function stays a function, but a string becomes "call that method" and a number becomes "grab that field", so callers write map(cols, "mid") or map(xy, 2) instead of a closure:
```
function fun(f)
  if type(f)=="string" then
    return function(v,...) return v[f](v,...) end end
  if type(f)=="number" then return function(v) return v[f]end end
  return f end

function map(t,f,    u)
  f = fun(f)
  u = {}; for _,v in ipairs(t) do u[1+#u]=f(v) end; return u end

function kap(t,f,    u)
  u = {}
  for k,v in pairs(t) do u[1+#u] = f(k,v) end; return u end
```
map walks values in array order; kap walks key-value pairs in any order (Python's d.items()), and since a nil result simply vanishes, kap also filters. This is the open-closed principle by hand: map never changes, yet every caller re-aims it.
keysort (DSU: decorate, sort, undecorate): --disty must rank 398 rows by their gap to heaven. Sorting with a comparator would call disty O(n log n) times; the classic fix (Perl folk call it the Schwartzian transform) is decorate each row as a {key, row} pair, sort the pairs, then undecorate back to bare rows. Python's sorted(key=f) is the same contract. Fun fact: set up your functions right and the third step vanishes. keysort decorates into SIDE tables (px: row to key; ix: row to input position), then sorts the original rows in place, with a comparator that just reads the cache — nothing was wrapped, so there is nothing to unwrap:
```
function keysort(t,f,    px,ix)
  px, ix = {}, {}
  for at, v in ipairs(t) do px[v], ix[v] = f(v), at end
  return sorted(t, function(u,v)
           if px[u] == px[v] then return ix[u] < ix[v] end
           return px[u] < px[v] end) end
```
Two details that matter. Once-per-row is not just style: in later weeks the key function costs money (disty reads labels), so calling it once is budget discipline. And the ix table breaks ties by input order — Lua's table.sort is unstable, and an unstable sort prints different lines on different Lua versions, rotting every frozen transcript (see show, below).
thing (coercion): Every cell in every csv file enters the system through one gate:
```
function thing(s)
  s = s:match"^%s*(.-)%s*$"
  return tonumber(s) or s=="True" or (s~="False" and s) end
```
Trim, then: a number if it parses, true/false for "True"/"False", else the string. Note what survives untouched: "?" stays a string, which is why every column method opens with if v == "?". One gate means one place to change — your Python port calls it atom.
show (the transcript is the test): The whole system prints through one function:
```
function show(t,    u)
  if type(t) ~= "table" then
    return tostring(type(t) == "number" and round(t) or t) end
  u = #t > 0 and map(t, show) or
      sorted(kap(t, function(k,v)
        if tostring(k):sub(1,1) ~= "_" then
          return ":"..k.." "..show(v) end end))
  return "{"..table.concat(u, " ").."}" end
```
Lists print in order; dictionaries print sorted by key; keys starting "_" stay hidden; numbers pass through round, which drops float noise and re-floors whole results so 15.0 prints as 15 on every Lua version. Why the fuss? Deterministic printing turns a transcript into a test: same seed plus same show means the output diffs clean — across runs, across Lua versions, and across your Lua-to-Python port, which is graded exactly that way.

The laws the demos swear by

Weeks one and two end in three asserts that are really three pieces of mathematics. New acronyms: PBT.

pool (the algebra of summaries): The --without demo pours {10,20,30} into a summary of {1,2,3,4,5}, subtracts a summary of {10,20,30}, and the mean and sd of {1..5} come back to nine decimals — with no stored data anywhere. That works because moments pool. For two groups a and b, with d = μ_b - μ_a:
- n_ab = n_a + n_b
- μ_ab = (n_aμ_a + n_bμ_b) / n_ab
- m2_ab = m2_a + m2_b + d² n_an_b / n_ab
The Sym side is the easy half — histograms subtract one count at a time:
```
function SYM.__sub(i,j,    out,n)
  out = Sym(i.name, i.at)
  for k,v in pairs(i.has) do
    n = v - (j.has[k] or 0)
    if n > 0 then out.has[k] = n; out.n = out.n + n end end
  return out end
```
The Num side is the three pooling lines above, solved backwards — given the whole and one part, recover the other part, O(1):
```
function NUM.__sub(i,j,    n,d)
  n = i.n - j.n
  if n < 1 then return Num(i.name, i.at) end
  d = j.mu - i.mu
  return new(NUM, {name=i.name, at=i.at, heaven=i.heaven,
                   n=n, mu=(i.n*i.mu - j.n*j.mu) / n,
                   m2=max(0, i.m2 - j.m2
                             - d*d*i.n*j.n/n)}) end
```
Note the guards: fewer than one row left returns a fresh Num, and max(0, ...) exists because float error can push m2 a hair below zero. Welford (glossary 1) forgets one value at a time; __sub forgets a whole group at once. Together they are why later code can score every split of a column in one linear pass.

Chan, Golub & LeVeque: Algorithms for computing the sample variance: analysis and recommendations. T.F. Chan, G.H. Golub, R.J. LeVeque. The American Statistician 37, 3 (1983), 242-247.
metric (the distance laws): A distance you cannot trust poisons everything built on it — clusters, poles, trees. Mathematicians call a trustworthy one a metric: it obeys

law says

identity d(a,a) = 0

symmetry d(a,b) = d(b,a)

non-negative d ≥ 0 (and here, d ≤ 1)

triangle d(a,c) ≤ d(a,b) + d(b,c)

The --laws demo probes the first three on 100 random pairs. The one at risk is identity: unknowns assume the worst, so SYM.dist("?","?") is 1 — a row holding a "?" would sit at nonzero distance from itself, making distx strictly a pseudometric. So why does self-is-zero never fail on auto93? Its only "?" cells live in HpX, and a trailing "X" means ignored: that column never reaches distx (homework 2, question 6). The design trade is deliberate, and it has a name: the Aha heuristic, from the instance-based learning literature (see the distx entry in glossary 1) — maximum ignorance maps to maximum distance, buying robustness to missing data at the price of textbook metric-hood. Note also what tames the ruler itself: gaps are measured between norm'd values (the cdf, glossary 1), so one wild outlier cannot stretch the scale for everyone else.
PBT (property-based testing): --col asserts an example: the mid of {1,2,3,4,5} is 3. --laws asserts a property: for ALL rows a and b, distx(a,b) equals distx(b,a) — then goes hunting for a counterexample with 100 random probes. Example tests catch the regressions you foresaw; property tests catch the ones you did not. QuickCheck made the recipe famous: state the law, generate random inputs, shrink any failure. This code keeps the cheap half of that recipe, and adds the RNG discipline of glossary 1: since run reseeds before every demo, a failing probe replays exactly, every time. --without and --sub are the same idea wearing algebra: (a+b)-b == a, asserted to 1e-9. When code claims mathematics — distance, probability, algebra — test the law, not the instance.

Claessen & Hughes: QuickCheck: a lightweight tool for random testing of Haskell programs. Koen Claessen, John Hughes. ICFP 2000, 268-279.

law	says
identity	d(a,a) = 0
symmetry	d(a,b) = d(b,a)
non-negative	d ≥ 0 (and here, d ≤ 1)
triangle	d(a,c) ≤ d(a,b) + d(b,c)