csc 591-024, (8290)
csc 791-024, (8291)
fall 2024, special topics in computer science
Tim Menzies, timm@ieee.org, com sci, nc state

---

home :: timetable :: syllabus :: groups :: moodle :: license

Do we know enough to make it simpler?

I want you… to be an (AI) brain surgeon

• I want to make you the AI software engineers:
  • that know how to reach inside smart algorithms and make changes
• Surely, by now, we can do that:
  • If you have been doing something for a while then can you or I:
    • Do it simpler, faster. using fewer resources?
    • Know how to combine things, such that you can more with less?
    • Teach seemingly complex things to newbies?

Can we engineering an AI system? Simply? Quickly?

Here we explore dozens of SE problems ¹ using explainable AI for semi-supervised multi-objective optimization.

• Internally, this is coded via sequential model optimization and membership query synthesis.

Sounds complex, right?

• But it ain’t.
• In fact, as I hope to show, all the above is just a hundred lines of code (caveat: if you are using the right underlying object model).

Which raises the question….

What else is similarly simple?

• How many of our complex problems … aren’t?

My challenge to you is this:

• Please go and find out.
• Take a working system, see what you can throw away (while the remaining system is still useful and fast).
• Let me know happens so I can add your fantastic new, and simple, idea to this code

Why study simplicity?

• Cause we are getting really really good at reasoning with very little data
• Cause its good science
  • If you really understand “it”, can you do “it” again, very very simply
• Cause the world is changing
  • Next generation of satellite internet providers
  • Connecting millions of new programmers willing to work for \(\frac{1}{20}\)th of the salary you want
  • In that world, you do not want to be the programmer
  • You want to be the optimizer who controls and improves the work of others.
• Cause almost no one else is studying reasoning with very little data

Why study simplicity? (2)

• Cause everyone has gone mad on complexity.
  • A small number of very large companies have built empires based on “big data”
  • Five years ago, no on one wanted to head about simplicity
  • But after three years of constant tech lay offs, and increasing challenges for getting jobs at these large organizations …
  • … my students now know they need to graduate with knowledge about “big data” AND alternate approaches.
• Cause big data is running out of data (see next slide).

Surfing the long tail

• LLMs? For everything?
  • LLMs know a lot, about things we do a lot (e.g. “if” statements in code)
  • And they know less about things we do less often
• Model collapse:
  • We are about to run out of training data ² for LLMs.
  • Can’t reply of synthetic data generation (no new information from data already seen)
• So, we make do with less data?
  • Are their domain, were models need less, not more, data?

---

1. github.com/timm/ezr/tree/main/data↩︎

2. Udandarao, V., Prabhu, A., Ghosh, A., Sharma, Y., Torr, P.H., Bibi, A., Albanie, S. and Bethge, M., 2024. No” zero-shot” without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint arXiv:2404.04125.↩︎