A glossary of tiny lectures, part 5

Week 5: active learning

By Tim Menzies, published 2026-09-28, updated 2026-09-28

Summary: Weeks 1-4 got the goals for free. Now suppose a single y value costs a test drive, a trial, a week of CPU — and you get fifty, ever. This week is how to spend them: label only the poles, cull gently, restart when the pool dries, and count every purchase. Then the honesty check: a holdout that picks from rows the search never studied. Demos in ezr-eg5.lua; machinery verbatim from ezr.lua. Each entry is a tiny lecture — hook, idea, math if any, then the code.

Labels cost money

New acronyms: none.

active learning: Until now every y value was free: the csv arrived with its goals filled in. Drop that. Suppose one label costs a test drive, a clinical trial, a week of CPU — and you get fifty, ever. Active learning is the discipline of choosing WHICH rows to label, using only the free x columns to choose, and counting every label you spend. The counting is the science: a result whose price is unknown says nothing.
label: Where the money is spent. A row born with "?" goals stays blank until someone asks; label calls the model, writes the answers into the row, and folds each answer into its goal column — so the summaries used for normalization sharpen as spending grows. Note the seam: everything upstream just asks for disty, and never knows whether the number was looked up or bought.
```
function TBL.label(i,row,    f)
  f = i.model(map(i.cols.x,
        function(c) return row[c.at] end), #i.cols.y)
  for j, y in ipairs(i.cols.y) do
    row[y.at] = y:add(f[j]) end
  return row end
```
budget: Four numbers run the economy: budget=50 (labels, total), check=5 (labels held back to confirm the final pick), more=4 (labels per round) and keepf=0.66 (fraction of the pool kept per cull). Read them as a sentence: spend four labels, throw away a third of what is left, repeat; keep five in your pocket for the final exam. Any experiment that quietly exceeds these is not measuring what it claims.

Where did those numbers come from? From a sweep: run the thing at every (budget, check) pair and colour the result. Test score (100 = the pool's best row) is the colour; the contours mark 75, 85 and 95 (figure from Menzies, Srinivasan & Ganguly, Can AI be Easy? Lessons Learned from the EZR.py Toolkit, SPE 2026):

Read the contours, not the colours: they run almost VERTICALLY. That means the score depends mostly on check — the labels spent confirming the top of the final ranking — and only weakly on budget, the labels spent searching. Going from a budget of 30 to 150 (five times the money) barely moves the colour; moving check from 1 to 5 crosses two contour lines, and the yellow lives out past check 7. The cheap corner is bottom right: a SMALL budget with a few more checks beats a large budget with one.

Why? The search only has to get the right rows NEAR the top of its ranking; the checks are what pick the winner out of that shortlist. Pay for the sort, not for the search. And note the shape of this lesson — it is weak indicators once more: the ranking is a cheap guess, so spend a little confirming it rather than a lot perfecting it.

acquire: One sweep of sway3, in code. Label up to the.more rows in the current pool, sort the pool by the projection onto poles drawn from those labelled rows, keep the best the.keepf of it, repeat until the pool is small or the budget is gone. Nothing here builds a model; the geometry does all the work, and the goals are only ever read for the few rows we paid for.

function TBL.acquire(i,rows,cap,lab,lo,hi,
                     seen,more,new)
  seen = {}
  for _,r in ipairs(lab) do seen[r] = true end
  while #rows >= 2*the.leaf do
    more, new = min(the.more, cap - #lab), {}
    for _,r in ipairs(rows) do -- new = labels in this pool
      if seen[r] then push(new, r)
      elseif more > 0 then
        more, seen[r] = more - 1, true
        push(new, push(lab, r)) end end
    if #lab >= cap then return lab end -- budget spent
    rows = slice(keysort(rows, (i:poles(new, lo, hi))),
               1, max(1, floor(the.keepf * #rows))) end
  return lab end

restart: A log-n chop reaches a tiny pool long before fifty labels are spent. So when the pool dries with budget left, reshuffle and descend again — but anchored at lo, hi, the best and worst rows labelled SO FAR. Every restart inherits everything paid for to date, which is why the second descent is sharper than the first. The loop stops when the budget is gone, or when a whole sweep buys nothing new.
```
function TBL.acquirer(i,cap,    lab,lo,hi,t,b4)
  lab = {}
  while true do
    b4  = #lab
    lab = i:acquire(shuffle(i.rows), cap, lab, lo, hi)
    if #lab >= cap or #lab >= #i.rows or
       #lab == b4 then break end -- full, or no progress
    t = keysort(lab, i:Y())
    lo, hi = t[1], t[#t] end -- best+worst seen
  return keysort(lab, i:Y()) end
```
holdout: The honesty check. Finding a good row among rows you have studied proves nothing; the question is whether what you learned TRANSFERS. So: split the rows in half, spend the budget on the train half, grow a tree from those labels, use that tree to rank the unseen test half, then spend the last the.check labels confirming the top of that ranking. The assert on line five is the whole ethic of the course — the spend is counted before anything is scored.
```
function TBL.holdout(i,how,    rows,n,train,test,lab,t,top)
  how  = how or function(t2,cap) return t2:acquirer(cap) end
  rows = shuffle(i.rows)
  n    = floor(#rows/2)
  train= slice(rows, 1, n)
  test = slice(rows, n+1)
  lab  = how(i:clone(train), the.budget - the.check)
  assert(#lab + the.check <= the.budget) -- spend, counted
  t    = Tree(i, lab)
  top  = slice(keysort(test, function(r) return t:leaf(i, r) end),
           1, the.check)
  return keysort(top, i:Y())[1] end
```
One holdout is an anecdote; twenty of them are a distribution, and the distribution is the result. Which raises next week's question: when is one distribution really better than another?

Menzies, Srinivasan & Ganguly: Can AI be easy? Lessons learned from the EZR.py toolkit. Tim Menzies, S. Srinivasan, Kishan Kumar Ganguly. Software: Practice and Experience, 2026. arXiv:2606.03640. (Source of the budget/check sweep above.)

Settles: Active learning literature survey. Burr Settles. Univ. Wisconsin-Madison, Computer Sciences TR 1648, 2009.

Chen, Nair, Krishna & Menzies: Sampling as a baseline optimizer for search-based software engineering. IEEE TSE 45(6), 2019, 597-614.