What AI skills are durable, and affordable?

Are you an LLM dinosaur, or a rat?

By Tim Menzies, published 2026-08-16, updated 2026-08-16

Summary: Students keep asking me two questions. What can I learn about AI that will not be outdated next year? And what can I afford to practice without renting a data center? This page is an answer. Big AI is a bubble, and the bubble is leaking: data centers age fast, revenues trail spending by a trillion dollars, and the regulators are closing in. When it pops, expect a third AI winter. So do not bet your career on renting a dinosaur. Learn instead to be a rat: use small models, small data, and small tools that one person can build, test, and understand. The durable skills are four rat tricks: reuse, then refactor; keep it simple and stochastic; know when near enough is good enough; and mix old symbolic AI with new neural AI, joined at the seam that pays. The evidence below shows a few hundred lines of code keeping pace with tools that burn thousands of LLM tokens.

Huge LLM companies dominate today's AI scene. They hold more resources than we can ever know or access. They sell a world view that contains only them. They change their technology so fast that no one can stay current. They are like T-rex dinosaurs: they will monster you and your career. Renting big AI is renting a telescope: powerful, but pointed by someone else. You cannot inspect it; it changes without notice, so last year's result may not run this year; and it is wrong in confident ways, so every output needs a human check.

These AI dinosaurs look unstoppable, unquestionable, inevitable. It seems all we can do is bow down, or run away. But a meteorite is coming to wipe out their ecosystem:

1. Data centers die young – the accountants book GPUs as lasting five or six years, but the real economic life is closer to two or three. Michael Burry warns that this slow write-off understates depreciation by some $176 billion, overstating hyperscaler profits by 20% or more. And we keep running out of the power and water needed to drive them.
2. The benefits fall short of the promises – e.g. look at the AI slop generated by automatic coding tools; measured carefully, AI assistants can even make experienced developers slower. Most corporate LLM pilot projects never show a measurable profit.
3. The revenues fall short too – in 2024, Sequoia Capital asked AI's $600B question: where is the revenue to cover all that spending? No one answered. Instead, the gap has since doubled, to some $1.2 trillion.
4. The earnings are circular – large AI companies inflate (fraudulently?) their reported earnings with "I pay you to buy from me" deals.
5. Local models keep getting better – when the model running on your laptop is good enough, the economic case for the expensive cloud disappears.
6. Open models keep getting better too – as open source Chinese models such as DeepSeek and Qwen come on line, companies like OpenAI and Anthropic have less and less that only they can sell.
7. The regulators are waking up – governments respond slowly, but they do respond; see the EU AI Act.

So we are heading for a third AI winter: a steep decline in public interest, venture capital, and corporate funding for all things AI. How will you survive it? Will you move to smart hardware/software combinations? Will you jump to outer space?

Here is another option. Look down. Scurrying around the feet of the LLMs are little rodents, eating the dinosaurs' eggs. The rats are small, cheap, fast, and curious. They need no data center. They run on found food: small models, small data, small tools that one person can build, test, and understand. When the meteorite lands, the dinosaurs starve while the rats inherit the earth¹.

This subject teaches you how to be a rat. We break up AI into numerous small parts and discuss shortcuts that make each part very fast. Kautz's neurosymbolic taxonomy (discussed below) makes the same move at field scale: mix and match the best of the old AI with the best of the new.

So be a discerning user of AI technology. Assess the merits of each approach, and know when to use what. Some of the discussion is theoretical—challenge problems where you can prove me wrong. But as we shall see, other parts are strongly supported by extensive experimental results.

How to be a rat

Here are five tricks that rats can use to sneak around, and tame, hard Ai problems.

1. Reuse, then refactor: Rats poke around the ground, feeding on whatever tidbits they can find. Good software engineers do the same: we look inside the skills we ask our AI to deploy, and before writing new code for a new skill, we reuse old code. This subject demonstrates that with some domain engineering. Suppose we were trying to buy a car. Here are ten skills we might ask of our AI:

   task	skill	%LOC	new LOC
   From 400 cars, find unusually great buys.	remember (representation)	45%	150
   Guess the fuel use before we see the sticker	guess (prediction)	17%	57
   Say when "better" is real, and when it is noise	certify (certification)	12%	40
   Decide on the spot as cars arrive one at a time	flow (streaming)	8%	27
   Say why the bad cars are slow, or thirsty	blame (diagnosis)	6%	20
   Justify any verdict to a skeptical buyer	justify (explanation)	4%	13
   Find the best car, test-driving very few	choose (optimization)	3%	10
   Find the cheapest change that fixes this car	fix (repair)	3%	10
   Spot a strange car: a typo, or a scam	spot (anomaly detection)	2%	6
   Race our skills against the field's best	race (baselining)	0%	0
   Totals		100%	333

   As seen here, the first skill costs 45% of the code: it pays for the data structures everything shares. After that, each new skill mostly rewires old parts: prediction costs 17%, optimization 3%, anomaly detection 2%. Skill i+1 reuses the machinery already built for skills 1..i.
   
   Reuse alone is not enough: rat engineers also refactor. If any fool can do it for a dollar, an engineer asks how to do it for a penny. What code can we throw away, with everything still working? The table above shows the payoff of all that redoing: a few hundred lines of code now support world-class optimization and explanation algorithms. Ganguly's FSE work shows these data-light methods matching far heavier learners. Rayegan's JSS work shows the same small toolkit generating explanations that are both shorter and clearer than those of its rivals. Reuse, then refactor, and everything gets simple, understandable, verifiable.
   
   (Aside: this is a first example of an important idea: neurosymbolic systems. Here I am proposing that humans and LLMs work together to specify the skill; then, under the hood, we implement that skill with highly optimized symbolic algorithms. Neural helps humans say what they want; symbolic actually does it. This mix-and-match is the subject of Kautz's 2020 AAAI address on neurosymbolic AI; more on that below.)
   
   
   Kautz: The third AI summer: AAAI Robert S. Engelmore Memorial Lecture, AAAI 2020. Henry Kautz. AI Magazine 43, 1 (2022), 105–125. doi:10.1002/aaai.12036.
   Menzies: Ten lectures on data-lite AI, at the REPL. Tim Menzies. NCSU course notes, 2026.
   Ganguly & Menzies: How low can you go? The data-light SE challenge. Kishan Kumar Ganguly, Tim Menzies. FSE 2026. arXiv:2512.13524.
   Rayegan & Menzies: Minimal data, maximum clarity: A heuristic for explaining optimization. Amirali Rayegan, Tim Menzies. J. Systems and Software, 2026. arXiv:2509.08667.
2. Keep it simple, keep it stochastic: One reason EZR does so much with so little code is that it makes extensive use of random search. Such searh, while simple to implement, can be spectacularly successful. The classic example is MaxWalkSat. To solve a hard logic problem, start with a random assignment. Find a broken clause. Flip one of its variables: usually the flip that fixes most, but sometimes any variable, chosen at random. That is the whole algorithm. Selman, Kautz, and Cohen showed that this pinch of noise let a simple local search crack constraint problems that defeated the careful, systematic solvers of the day. Nature agrees. Particle swarm optimization copies a flock of birds (see the figure at right). Each particle starts with a random guess. Then it moves: partly toward its own best find, partly toward the swarm's best find, and partly at random. From those three simple pulls, the swarm circles in on the prize. The same lesson holds in machine learning: Bergstra and Bengio showed that random trials tune learners better than careful grid search.
   
   
   Selman et al.: Noise strategies for improving local search. Bart Selman, Henry Kautz, Bram Cohen. AAAI 1994, 337–343.
   Kennedy & Eberhart: Particle swarm optimization. James Kennedy, Russell Eberhart. ICNN 1995, 1942–1948. doi:10.1109/ICNN.1995.488968.
   Bergstra & Bengio: Random search for hyper-parameter optimization. James Bergstra, Yoshua Bengio. J. Machine Learning Research 13 (2012), 281–305.
3. Pick your battles: good enough is near enough optimization (NEO): A little maths explains the success of random search. Random search assumes that near enough is good enough. That assumption makes search cheap.
   
   First, see what search can cost. Some tasks are inherently complex. Reward-free reinforcement learning for autonomous cars must learn enough to satisfy any future, as-yet-unwritten safety law. Jin et al. show that even simple versions of this task need 10¹⁵ samples.
   
   Costs fall if you know how solutions will be judged. Search then becomes the Best Arm problem. Hoeffding says: to distinguish A alternatives with accuracy ε and confidence C,
   • samples(best-arm) >= (2/ε²) · log( 2A / (1−C) )
   So a dozen binary choices (A=2¹²), at C=95% and ε=5%, needs 9,605 samples. Much better than 10¹⁵, but still impractical in many domains.
   
   (Aside: labeled samples cost money, all over SE, is is hard to find good labeled data. Human experts are slow at labeling and grow error-prone when rushed; Easterby-Smith reports hours of work for just a few cases. Historical logs are big but unreliable: Herbold et al. report that labeling tools debugged for decades still make many errors, and when Yu et al. audited real-world SE data sets, 90% of the technical-debt comments labeled "false positive" were themselves mislabeled. Even a real oracle can be ruinous: exhaustively exploring just the 11 configuration parameters of the x264 video encoder — compile each variant, then run a large test suite — took over 1,000 hours of CPU. And do not expect LLMs to label those samples for you: for SE problems, Ahmed et al. warn that LLMs assist labeling but cannot automate it.)
   
   Costs fall again if we can assume that near enough is good enough. Real solutions can vary by a small amount, so a rat may stop at any solution indistinguishable from the best. Cohen calls two solutions "small to medium" different when they are 0.2σ to 0.5σ apart, and a normal curve spans ±3σ. So "indistinguishable" means p = ((0.2+0.5)/2)/6 ≈ 6%. Hamlet gives the confidence C of finding an event of probability p in n independent trials:
   • C = 1 - (1-p)ⁿ,
   • so samples(NEO) = log(1 - C) / log(1 - p).
   At C=0.95 and p=0.06, rats can stop after only n=48 samples.
   
   Costs fall once more with prior knowledge. Suppose those 48 samples built a model that can guess which of two solutions is worse. A binary chop then finds good solutions in:
   • samples(clever) = log2(samples(NEO)) = log2(48) ≈ 6 samples.
   This maths rests on wildly optimistic assumptions: solutions are evenly distributed, along one dimension, and that dimension is Gaussian. Yet in practice, the numbers hold.
   
   A tiny tutorial: the csv format. Checking that maths needs real data, so meet the data. Every problem in this subject is one csv file. The first row names the columns, and those names are the entire schema. Uppercase first letter means a numeric column (Volume); lowercase means symbolic (origin). A trailing "-" or "+" marks a goal, to be minimized or maximized; a trailing "X" says ignore this column; a name with none of those marks is an input. A "?" cell is a missing value. So in the cars below, a good car is light (Lbs-), quick (Acc+), and economical (Mpg+):

   Clndrs	Volume	HpX	Model	origin	Lbs-	Acc+	Mpg+
   4	90	48	80	2	2085	21.7	40
   4	140	92	76	1	2572	14.9	30
   4	91	?	82	3	1965	15.7	30
   8	429	198	70	1	4341	10.0	15

   Note what is cheap and what is dear here. The input columns come free with every row. The goal columns are the labels — someone must weigh, drive, and measure each car — so in practice most goal cells start as "?", and the game is choosing the few rows worth labeling.
   
   Figure 2 of arXiv:2606.03640 built models from 10 to 200 samples, then checked the top 1 to 10 sorted items in a holdout set. This ran 100,000 times, picking data sets and hyperparameters at random each time, over 127 data sets from the MOOT repository. The vertical axis is the build budget y; the horizontal axis is the top x items checked. Above a budget of about 50 and a check of about 5, everything is flat at 95. The paper's own summary: at `(budget, check) = (50, 6)` the learner "usually achieves above an 85% win", and "checking beyond 7 items seems not particularly useful".
   
   In a result that should please every rat, those numbers (50, 6) sit very close to the (48, 6) computed above.
   
   
   Jin et al.: Reward-free exploration for reinforcement learning. Chi Jin, Akshay Krishnamurthy, Max Simchowitz, Tiancheng Yu. ICML 2020.
   Hoeffding: Probability inequalities for sums of bounded random variables. Wassily Hoeffding. J. Amer. Statist. Assoc. 58, 301 (1963), 13–30. doi:10.1080/01621459.1963.10500830.
   Ahmed et al.: Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. Toufique Ahmed, Premkumar Devanbu, Christoph Treude, Michael Pradel. MSR 2025. arXiv:2408.05534.
   Herbold et al.: Problems with SZZ and features: An empirical study of the state of practice of defect prediction data collection. Steffen Herbold, Alexander Trautsch, Fabian Trautsch, Benjamin Ledel. Empirical Softw. Engg. 27, 2 (Mar 2022). doi:10.1007/s10664-021-10092-4.
   Yu et al.: Identifying self-admitted technical debts with Jitterbug: A two-step approach. Zhe Yu, Fahmid M. Fahid, Huy Tu, Tim Menzies. IEEE Trans. Software Engineering 48, 5 (2022), 1676–1691. doi:10.1109/TSE.2020.3031401.
   Easterby-Smith: The design, analysis and interpretation of repertory grids80032-0). Mark Easterby-Smith. Int. J. Man-Machine Studies 13, 1 (1980), 3–24. doi:10.1016/S0020-7373(80)80032-0.
   Menzies et al.: MOOT: a repository of many multi-objective optimization tasks. Tim Menzies, Tao Chen, Yiqi Ye, Kishan K. Ganguly, Amirali Rayegan, Suvodeep Srinivasan, Andre Lustosa. MSR 2026, 584–589.
4. Look for the cracks between the parts: AI is not one thing. It is parts, joined at seams. My challenge to you is this: we drop our little nimble rats into those seams, can we build a better faster AI?
   
   The following table extends Henry Kautz's 2020 AAAI address, and his 2024 update of it, which together map eight ways to join old symbolic AI (logic, search, solvers) to new neural AI (deep learners, LLMs). We add two more: symbolic alone, and neural alone, for ten cells in all. Most folks who just use ChatGPT, and nothing else, do not realize they are stuck in one tiny corner of this space of possibilities. The other nine cells are where the rats can zip in and speed things up.
   
   Kautz's notation is a little arcane, so here is a simpler one. Write S for a symbolic part, N for a neural part. Then read the joins off the table, each with an SE example:

   pattern	meaning	SE example
   S	symbols alone	EZR's domain engineering (trick 1, above): optimize and explain in 333 lines
   N	a net alone	LLMs label SE optimization data; Senthilkumar's EMSE work makes those raw labels robust
   S; N; S	symbols in, net thinks, symbols out	prompt an LLM with code, read back a review; the ChatGPT cell
   S[N]	symbolic boss calls a neural helper	an issue-triage pipeline (the boss) calls a sentiment net to score each incoming report (Lin et al.); AlphaGo's tree search calling its neural scorer is Kautz's classic case
   N; S	net proposes, symbols check	LLMs translate requirements to temporal logic, then model checkers test them (nl2spec); LLM warm-starts seed a classical active learner (Senthilkumar's TOSEM work). See also, the above ppint (1) on domain engineering the car yard.
   S; N	symbols build the net's diet	SE knowledge builds the training pairs; rules compile away into weights (Lample & Charton; Hu et al.)
   S as N	rules are templates for the net	logic clauses shape the network's structure (Logic Tensor Networks)
   S in loss	rules graded into training	coding standards as semantic loss: breaking a rule costs the net at training time (Xu et al.)
   N[S]	neural boss calls a symbolic helper	an LLM that calls calculators, search, and solvers mid-answer (Toolformer); Kautz's own favorite, and his demo teaches GPT to call a SAT solver
   S then N	cheap symbols first, dear net second	classical search finds good seeds; the LLM polishes them (Srinivasan)

   The N[S] row hides a joke at the dinosaurs' expense. When Claude's own source code recently bumped into public view, readers found much S inside that N: prompts, rules, regular expressions, and tool calls, all wrapped around the net. Even the flagship dinosaurs are secretly neurosymbolic.
   
   The last row is the newest trick, and the figures show why it works. Srinivasan shows that order matters: run the cheap classical optimizer first, then hand its best results to the LLM. Across 103 MOOT data sets (above), that hybrid, SNAP2, matches or beats SNAP (the LLM-heavy tool) and every other method tried. And note how well the little symbolic EZR tracks the winners.
   
   Then look at the bill (above). Per repeat, SNAP spends 8 LLM calls, 26,000 tokens, and 158 seconds. The SNAP2 hybrid spends 5 calls and 111 seconds. EZR alone spends zero calls, zero tokens, zero dollars, and 0.04 seconds. So if near enough is good enough (trick 3), the little symbolic engine does the job in a blink, with tiny code and no tokens at all.
   
   The lesson of the whole table is the same. Do not buy one giant AI. Be a discerning shopper. Ask what each part does cheaply and well, then join the parts at the seam that pays.
   
   
   Kautz: The third AI summer: AAAI Robert S. Engelmore Memorial Lecture. Henry Kautz. AI Magazine 43, 1 (2022), 105–125. doi:10.1002/aaai.12036.
   Senthilkumar & Menzies: From brittle to robust: Improving LLM annotations for SE optimization. Lohith Senthilkumar, Tim Menzies. Empirical Software Engineering, 2026. arXiv:2603.22474.
   Lin et al.: Sentiment analysis for software engineering: How far can we go?. Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, Rocco Oliveto. ICSE 2018, 94–104. doi:10.1145/3180155.3180195.
   Cosler et al.: nl2spec: Interactively translating unstructured natural language to temporal logics with LLMs. Matthias Cosler, Christopher Hahn, Daniel Mendoza, Frederik Schmitt, Caroline Trippel. CAV 2023. arXiv:2303.04864.
   Senthilkumar & Menzies: Can large language models improve SE active learning via warm-starts?. Lohith Senthilkumar, Tim Menzies. ACM Trans. Software Engineering and Methodology, 2026. arXiv:2501.00125.
   Hu et al.: Harnessing deep neural networks with logic rules. Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, Eric Xing. ACL 2016. arXiv:1603.06318.
   Lample & Charton: Deep learning for symbolic mathematics. Guillaume Lample, François Charton. ICLR 2020. arXiv:1912.01412.
   Xu et al.: A semantic loss function for deep learning with symbolic knowledge. Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, Guy Van den Broeck. ICML 2018. arXiv:1711.11157.
   Schick et al.: Toolformer: Language models can teach themselves to use tools. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom. NeurIPS 2023. arXiv:2302.04761.
   Badreddine et al.: Logic tensor networks. Samy Badreddine, Artur d'Avila Garcez, Luciano Serafini, Michael Spranger. Artificial Intelligence 303 (2022). arXiv:2012.13635.
   Srinivasan & Menzies: Better together, in the right order: Classical-then-LLM optimization for SE. Srinath Srinivasan, Tim Menzies. arXiv:2607.02583, 2026.