A glossary of tiny lectures, part 9

Week 9: models and the seam

By Tim Menzies, published 2026-11-09, updated 2026-11-09

Summary: The last piece: replace the csv with a MODEL. Rows are born with "?" goals and disty buys them, one row at a time, so the label count IS the experiment. DTLZ1-7 give us problems whose true answers are known, so a search can be graded exactly. Then the week's punchline: the classics spend the whole pool; acquire spends 45 and lands in the same place. Demos in ezr-eg8.lua; machinery verbatim from ezr-dtlz.lua.

When the goals cost money

New acronyms: DTLZ (Deb-Thiele-Laumanns-Zitzler test suite).

model seam: Everything so far read goals from a csv. The seam swaps that for a FUNCTION: set t.model and rows may be born with "?" goals, which label fills in on demand. Nothing upstream changes — trees, acquire, holdout, the optimizers all keep asking disty — which is the point of a seam. This is where outsiders plug in their own (maybe very expensive) simulator.
```
function Dtlz(    u,r)
  u = {names()}
  for _ = 1, the.pool do
    r = {}
    for _ = 1, the.Nx do push(r, rand()) end
    for _ = 1, the.M  do push(r, "?") end
    push(u, r) end
  u = Tbl(u)
  u.model = _ENV[the.model]
  return u end
```
dtlz: The standard multi-objective test problems (Deb, Thiele, Laumanns & Zitzler, 2002): x lives in [0,1]^Nx, the objectives have a KNOWN true front, and the difficulty is dialable. The last Nx-M+1 variables set DISTANCE from the front (make them zero and you are on it); the first M-1 set POSITION along it. dtlz1's front is the plane ∑f = 0.5, approached across a landscape full of local traps; dtlz2's is the unit sphere ∑f² = 1; dtlz7's comes in disconnected pieces. Because the answer is known, a search can be graded exactly — which is what makes these problems the field's measuring stick.
```
function dtlz1(x,M,    g,f,v)
  g, f = g1(slice(x, M)), {}
  for i = 0, M-1 do
    v = 0.5 * (1 + g)
    for j = 1, M-1-i do v = v * x[j] end
    if i > 0 then v = v * (1 - x[M-i]) end
    push(f, v) end
  return f end
```
lazy labels (the sharpening ruler): A subtle consequence of buying goals one at a time: disty normalizes each goal against the column summary, and that summary only knows the labels bought so far. So the SAME row can score differently early and late — the ruler sharpens as spending grows. This is a feature, not a bug: with five labels you do not know the range of the world, and pretending otherwise is how leakage sneaks in. It also means any reported score must say how many labels were in hand when it was taken.

baseline: What does doing nothing get you? baseline calls the model on every row and reports each goal's mean — deliberately NOT folding those calls into the column summaries, so the baseline never sharpens the ruler it is measured against. Every claim in this course is "better than X"; baseline is the cheapest honest X.

function TBL.baseline(i,    nums,f)
  nums = map(i.cols.y, function() return Num() end)
  for _,r in ipairs(i.rows) do
    f = i.model(map(i.cols.x,
          function(c) return r[c.at] end), #i.cols.y)
    for j,v in ipairs(f) do nums[j]:add(v) end end
  return map(nums, "mid") end

pure vs generalize: Two ways to report the same search. PURE (--pure): spend the budget over the whole pool and report the best row found — the right number when you own every row and just want the best one (config tuning, say). GENERALIZE (--generalize): spend on half, then pick from the half never seen — the right number when tomorrow's rows are not today's. Pure scores flatter. Say which one you ran; papers that do not are hiding the difference.
the price of a label: The week's punchline, from ezr-eg8.lua --race on a 200-row DTLZ2 pool: the four classic optimizers between them buy around 200 labels — effectively the whole pool — while acquire buys 45 and lands at the same place or better. Neither result says the classics are bad; they say the classics were designed for cheap models. The question to carry into any project is not "which optimizer is best" but "what does one evaluation cost here, and how many can I afford?"

Deb, Thiele, Laumanns & Zitzler: Scalable multi-objective optimization test problems. Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, Eckart Zitzler. CEC 2002, 825-830.

Chen, Nair, Krishna & Menzies: Sampling as a baseline optimizer for search-based software engineering. IEEE TSE 45(6), 2019, 597-614.