How do we trust AI-using machines —
when they fly, drive, and make real-world decisions?
Ben Wooding
Postdoctoral Researcher, Vanderbilt University
Institute for Convergent Software Integrated Systems (ICONS)
Formal methods for safe AI in cyber‑physical systems
🌍Originally from England — ten years in Uganda as a missionary kid
💍Married to Lucy
⛪Immanuel Church Nashville
How do we know what’s real? And what’s just noise?
“Love the Lord your God with all your heart… and love your neighbour as yourself.” Matthew 22 : 37–39
“The Lord God took the man and put him in the Garden of Eden to work it and take care of it.” Genesis 2 : 15
Developing new technologies and machines with behavioral guarantees is loving to the neighbor who will fly in them, be diagnosed by them, and live beside them.
We want new technologies to be fruitful not harmful. Machines cannot take responsibility — but the humans who design and build them can.
People are forming attachments to chatbots in the middle of an epidemic of loneliness.
Convincing fake voices, images and video — used for scams, humiliation and deception.
AI is proliferating social media, with concerns already for the young and the vulnerable.
Genuine concerns about the future of work, jobs, and livelihoods.
The electricity and water required for this compute — and its climate cost.
Anthropic and OpenAI looking to go public soon, many believe an AI market bubble.
Genuine concerns, but outside my field of expertise.
Tonight: AI in the physical world — and how we try to guarantee safety.
A radiation‑therapy machine. Six massive overdoses in two years, across the US and Canada. Three patients died of the overdose.
Earlier models had hardware interlocks — physical switches that made dangerous behaviors impossible. Software checks were substituted for many traditional hardware interlocks.
A race condition: an operator who edited the prescription too quickly could trigger the raw beam to fire without shielding.
The Therac‑20 contained the same software error but its hardware caught what the software missed.
Three years of investigation: “system testing alone is not adequate.”
Intel's answer ever since:
mathematically prove the arithmetic,
don't just test it.
A 64‑bit decimal was forced into a 16‑bit integer slot and overflowed, in code that serves no purpose after lift‑off, kept from Ariane 4 “for commonality”.
Two inertial reference computers ran identical software — so both halted, 0.05 s apart. The autopilot then steered on their error message as if it were flight data.
The code had flown for years — on Ariane 4, a slower rocket. “Correct” is always relative to a specification — change the rocket and the guarantee changes too.
The ground software reported thruster impulses in pound‑force seconds; the interface specification said newton seconds.
Nine months of navigation quietly absorbed the 4.45× error. The $125 M orbiter’s first pass came in at an estimated 57 km — below the ≈80 km survival line.
The right unit was written down. A specification only protects you if something checks it.
“When you invent the ship, you also invent the shipwreck.”
autopilots, collision avoidance
braking, lane‑keeping, cruise control
pacemakers, insulin pumps
balancing supply & demand
So why are we letting them in at all?
ILO estimate, 2023 — 330,000 in accidents, 2.6 million from occupational disease.
WHO, 2021 data — the leading cause of death at ages 5–29.
The strongest argument for automation is not simply convenience.
A person decided in advance what to measure — edges at particular angles, corners, patches of texture — and wrote it down as code. The computer only scored features a human had chosen.
Nobody writes the features down. The network starts from random numbers, sees 1.2 M labelled photographs, and every mistake nudges each number a little. Useful features appear on their own.
Where no first‑principles model exists, fitting to data beats writing rules. Retinal scans, skin lesions, speech, driving: the same curve. Caveat: 3.6% is an average over one fixed test set.
Beat three champion pilots — two of them world champions — 15 races to 10.
One reinforcement‑learned policy replaced a bank of separate PID loops.
Trained in simulation; crossed mud and rubble where the model‑based controller fell.
A 36.7 M‑parameter graph neural network beat ECMWF’s operational physics model.
Classical control is still vital but data can help us close gaps to real behavior.
ImageNet: 1.2 M photographs, each labelled by a person.
Objective: predict the next token — the data labels itself, so the training set can be the whole internet.
An architecture that introduces context — and more data and compute keep helping, predictably.
“General methods that leverage computation are ultimately the most effective, and by a large margin.”
Trained on 960 h of clean audiobooks. Superb there — brittle everywhere else.
Trained on 680,000 hours of everyday audio from the web — 700× more, most of it noisy.
On speech neither had seen — and it still ties the specialist on the specialist’s own benchmark.
The breakthrough: breadth also appears to bring robustness.
A web‑trained vision‑language model that outputs robot actions directly.
One network, straight from camera pixels to steering and braking.
A vision‑language model directing a humanoid that loads car parts.
A fine‑tuned speech model transcribing live air‑traffic radio.
They work brilliantly on average —
but what about edge cases people meet in reality?
A photo is a vector of pixel values — a point in a million‑dimensional input space.
All imperceptible perturbations of the photo form an ε‑ball of radius ε around that point.
An adversarial perturbation slides the point in the ε‑ball — across the network’s decision boundary.
The ε‑ball contains infinitely many images.
No amount of testing can try them all.
“Program testing can be used to show the presence of bugs, but never to show their absence!”
Audio adversarial examples: a clip that sounds like ordinary talk transcribes as “okay google, browse to evil.com” — any audio, any target phrase.
Physical‑world adversarial example: a few stickers, and a stop sign is classified as “Speed limit 45.”
LLM jailbreaks are adversarial examples too: one optimised suffix of gibberish tokens, and an aligned chatbot writes the malicious code it just refused to write.
The ACM A.M. Turing Award — “the Nobel Prize in Computing” — has 81 laureates. Twelve of them laid the foundations of formal methods: the logics, automata and algorithms for proving that programs do what they should.
“composed correctly, not just debugged into correctness”
inductive assertions (1967) — the root of program verification
Hoare logic (1969): {P} S {Q}
nondeterministic finite automata (1959, with Scott)
the same paper; later denotational semantics
LCF: machine‑assisted proof; ML; CCS
NP‑completeness (1971) — the wall we hit five slides from now
temporal logic for programs (1977): “always”, “eventually”
model checking (1981, with Emerson): search every state
model checking; CTL, a branching‑time temporal logic
model checking, independently (1982, with Queille)
safety and liveness (1977); logical clocks; TLA+
Recently AI too: 2018 — Bengio, Hinton, LeCun; 2024 — Barto & Sutton.
Portraits via Wikimedia Commons — Dijkstra: Hamilton Richards, CC BY‑SA 3.0 · Rabin, Cook: Andrej Bauer, CC BY‑SA 2.5 · Scott: George M. Bergman, CC BY‑SA 4.0 · Hoare, Sifakis: Rama, CC BY‑SA 2.0 FR · Pnueli: David Monniaux, CC BY‑SA 1.0 · Clarke: Dennis Hamilton, CC BY 2.0 · Emerson: E. Allen Emerson, CC BY‑SA 4.0 · Lamport: lamport.org, free use
Every image in the ε‑ball around this photo gets the same label — noise, dirt, glare and all.
Two applicants who differ only in a protected attribute must get the same decision — provably.
Where can the closed‑loop system — network and physics — be, one second from now?
The ε‑ball of “nearby images” becomes a set of people, scripts of code , or of aircraft states. The maths doesn’t mind.
Wx + b — like 3×flour + 2×sugar. Well‑behaved, fully predictable.
The only non‑linearity. All the intelligence — and all the trouble — lives here.
So instead of testing images one at a time, we propagate the entire infinite ε‑ball through the layer in one step.
Approach 1 — exact: split the set and each half meets an affine map again — handled exactly, no approximation anywhere. The price: one set becomes two.
Inside a real network, every ReLU the set straddles forces the split — one set becomes two.
Each set meets more ReLUs in the next layer.
Two becomes four. Four becomes eight…
Just 50 straddled ReLUs → 2⁵⁰ ≈ 1,000,000,000,000,000 activation patterns. More than we can store.
In 2017, researchers proved ReLU network verification is NP‑complete — it joins the class of problems computer science believes admit no polynomial‑time algorithm.
Whether that class truly has no polynomial‑time algorithm is the P vs NP problem — one of the seven Millennium Prize Problems. Solve it: $1,000,000.
Only one of the seven has ever fallen — though this month an AI‑generated proof of Navier–Stokes is under review. For P vs NP, ~9 in 10 experts bet P ≠ NP. So we don’t wait for a polynomial‑time algorithm — we over‑approximate.
In practice, a hybrid: over‑approximate first; case‑split only where “unsafe” appears.
Formal verification caught a data‑loss bug that needed a 35‑step chain of events to appear. No test plan on Earth finds that.
Formal verification proves the flight code is free of runtime errors — for every input, on every flight.
Ships with a machine‑checked proof that the code matches its formal specification, line by line.
Bug‑hunters found hundreds of bugs in mainstream rivals — and none in its formally verified core.
Industry’s sentence is changing — from “we tested it extensively”
to “we proved it cannot do the bad thing.”
the bad thing never happens
reach the goal set, avoiding the unsafe set
the good thing eventually happens
after a disturbance, recover within k steps
Written in temporal logic — mathematics for sentences about time:
□ ¬crash · ◇ goal · dip → ◇≤k recover
A barrier certificate is a function B(x), non‑increasing along every trajectory — so its zero level set is never crossed.
Learned from data — even when the system’s dynamics are unknown.
Conditions hold? Keep it — the certificate. Violated? The counterexample goes back into training. Repeat.
The network is allowed to be creative, because it never gets the last word. The checker does.
Even a sound and complete verifier can only report the truth — and sometimes the truth is: this model really is that sensitive.
Local robustness proofs start from the image we tested.
A flying drone meets unseen inputs every frame — it can’t pause mid‑air to verify each one.
Can a guarantee generalise from the photos we tested to the next photo?
“for foundational discoveries and inventions that enable machine learning with artificial neural networks”. Hopfield network (1982): 30 nodes, 435 connections. Boltzmann machine (1985). Hinton also holds the 2018 Turing Award.
Half “for computational protein design”; half “for protein structure prediction”. AlphaFold2: structures for nearly all ~200 million known proteins; at CASP14 (2020), median backbone error 0.96 Å against 2.8 Å.
“for developing the conceptual and algorithmic foundations of reinforcement learning”. Temporal‑difference learning (Sutton, 1988); the textbook (1998). The paradigm behind the tokamak and ANYmal controllers of slide 16.
“…when we create digital beings that are more intelligent than ourselves. We have no idea whether we can stay in control.”
“Speedbird 245, descend flight level 80” — heard by a 1.5‑billion‑parameter speech model over a noisy VHF channel.
Independent draws of real VHF‑band noise. Count how often the meaning survives — which aircraft was addressed, which altitude was given.
“For the next, unheard burst of noise: under 5% probability the meaning changes — Clopper–Pearson, confidence 99.9999%.”
No exact verifier scales this far; sampling needs only forward passes.
each sign: 1,000 noise samples → its own local certificate
The tested images become random samples themselves — a second scenario‑approach bound covers unseen inputs: images nobody has seen yet.
A verified controller is flying — but its proof was made against a model of the world. On board, every split‑second, the monitor checks model against reality.
The moment they diverge — the world doing something the model never promised — a second controller steps in and corrects the course.
We flew it — a union of neural control barrier certificates, in our cluttered two‑room lab, through a 90 cm doorway. With the monitor: 0 of 4,810 states left the certified set. Without it: 177.
Fig. 3, work under review
25 of 30 Olympiad geometry problems — the previous best method solved 10.
Silver‑medal standard — and every proof machine‑checked in Lean.
Gold‑medal standard, in plain language, graded by humans. Faster — no certificate.
the network proposes the solution, the symbolic checker certifies it
“We are not trying to meet some abstract production quota of definitions, theorems and proofs. The measure of our success is whether what we do enables people to understand and think more clearly and effectively about mathematics.”
Do we lose something by taking this approach? Was math only ever about finding an answer?
A router wakes 2 of 8 experts per token — most of the network stays dark.
top‑k routing is a hard case split — the ReLU split of slide 28, over whole sub‑networks
The learned pieces live on the edges, as splines — interpretable, ≈10× slower to train.
small symbolic pieces can be handed to an SMT solver: KAN barrier certificates (IJCAI 2025)
Each node listens to its neighbours — molecules, power grids, the weather model of slide 16.
unbounded graphs: reachability undecidable; bound the node degree and it becomes decidable (Sälzer & Lange, ICLR 2023)
Every new architecture re‑asks the question of slide 28: where do the case splits hide?
Energy spent only on events: TrueNorth classified video at 63 mW; Hala Point packs 1.15 B neurons — research prototypes.
Intel’s own survey: ordinary feedforward networks see “modest if any benefit” — the gains need spikes used as spikes.
Data centres: ≈415 TWh in 2024, heading for ≈945 TWh by 2030, AI the main driver (IEA 2025).
robustness must cover spike timing too — today’s SNN verifiers are “barely scalable” (Seong et al., 2024)
the fine print under the fourth line: “Use RL only when planning doesn’t yield the predicted outcome, to adjust the world model or the critic.”
“IF YOU ARE INTERESTED IN HUMAN‑LEVEL AI, DON’T WORK ON LLMs”
the last line of the slide — verbatim, and red in the original
Happy to share my slides — and more resources.
Ben Wooding · woodingben.com