Jonathan Mugan

Missy Cummings Keynote video from SafeComp 2025: "What California’s Self Driving Car Data Tells Us About Academia’s Shortcomings" Missy talks about the progression of types of reasoning with uncertainty, robotaxi incident reporting statistics, why the industry is getting statistics wrong, robotaxi crash types, sensor fusion, phantom braking, why Tesla fans are stalking Waymo robotaxis in Austin, the Cruise pedestrian dragging mishap, and a determinism/criticality heat map. "We have never needed safety research more than we need it right now." https://lnkd.in/efjCFBZM #SafeComp2025

77

6 Comments

We will present two exciting papers at #AAAI2025 later this month, advancing AI safety, learning-enabled control and automated reasoning. If you will also be in Philly and would like to connect, do reach out! 1. Neural Control and Certificate Repair via Runtime Monitoring (joint work with Emily Yu and Tom Henzinger) Learning-based methods and the use of neural networks as controllers provide a promising approach to solving highly non-linear control tasks. To ensure satisfaction of safety constraints, neural control methods jointly learn a control policy together with a neural certificate function (such as a Lyapunov or barrier function). In this work, we propose a novel framework for certifying and repairing neural control policies and certificates. We utilize runtime monitoring to detect safety constraint violations, extract new training data and ultimately repair the control policy and the certificate. We demonstrate the effectiveness of our approach empirically by using it to repair and to boost the safety rate of neural policies learned by a state-of-the-art neural control method. 2. Quantified Linear and Polynomial Arithmetic Satisfiability via Template-based Skolemization (joint work with Krishnendu Chatterjee, Ehsan K., Mehrdad Karrabi, Harshit J Motwani and Maximilian Seeliger) Satisfiability checking of linear real arithmetic (LRA) and non-linear real arithmetic (NRA) formulas has broad applications, in particular, they are at the heart of logic-related applications to AI and program verification. However, the problem of checking satisfiability of quantified LRA and NRA formulas remains a significant challenge. We propose a novel method for efficient quantifier elimination in quantified LRA and NRA formulas, using a template-based Skolemization approach. Our method offers a range of appealing theoretical properties combined with a strong practical performance, achieving competitive performance compared to state-of-the-art SMT solvers, both on LRA and on NRA benchmarks. Paper 1: https://lnkd.in/d8xUvJ8M Paper 2: https://lnkd.in/dtf2wy-A #AAAI2025 #AI #formalmethods #AISafety

81

I've started a research blog. One of the first posts is to announce our new sandboxing framework for Browser AI agents (think OpenAI Operator, Anthropic Computer Use, Google Project Mariner, Browser-use, etc.) Our framework, ceLLMate, uses systems security ideas and constructs the sandbox around the AI rather than relying on probabilistic model hardening techniques. It is available as an extension for Chrome! Read the blog, try it out, send us feedback! https://lnkd.in/gEMJStzC

89

4 Comments

I wrote a post recently (linked in comments below) on why agentic systems require structures for semantic-associative memory, and why LLMs lack the architecture to do anything but procedural memory. Therefore: LLMs aren't sufficient to build agentic systems. Someone replied with a very thoughtful question about Retrieval-Augmented Generation (RAG), a method that enhances language models by retrieving relevant external documents at query time and using them as additional context for generating responses. The question was whether RAG didn't actually provide the solutions to the inadequacies I outlined in my original post. The answer is: no. RAG is great for fact recall (well...let's say it's solidily good). But it falls short when it comes to forming and using *semantic categories* — the kind of abstract, relational knowledge humans build up over time. Here’s why: 🚩 No integration. RAG fetches documents, but they don’t get internalized. Humans integrate knowledge into structured representations that guide future learning and generalization. 🚩 No schema formation. Humans *compress* experience into categories (e.g., "tools", "emotions", "rules"). RAG retrieves specific items but doesn’t abstract across them to form stable, general concepts. 🚩 No consolidation. There’s no memory strengthening. Humans reinforce category boundaries over time — through repetition, use, and relevance. RAG pipelines don’t naturally evolve or stabilize concepts with repeated exposure. 🚩 Flat associations. Retrieval is based on surface similarity (e.g., vector distance). Human categories are rich with *associative structure* — linking things by function, causality, affordance, etc. 🚩 No cross-episode learning. RAG doesn’t accumulate concepts across interactions. Each retrieval is a fresh lookup. Humans merge partial exposures to build coherent, resilient categories. In short: RAG helps with lookup, not learning. It’s a patch for recall, not a path to structured, practical decision-making. Until we give agents mechanisms for abstraction, consolidation, and associative generalization — they won’t have anything close to human-like semantic memory. In my opinion, it's more realistic to create those mechanism separately and let them interact with LLMs than it is to think that we can somehow train or augment LLMs to the point that they'll be able to do something they were not designed to do.

67

15 Comments

An excellent piece in Newsweek by former Bell Labs President Marcus Weldon on "the 8 principles for the future of AI." In a remarkably clear and concise manner, the piece distills some basic predictions from a series of interviews with roboticist Rodney Brooks, neuroscientist David Eagleman, and myself. https://lnkd.in/dsJSSvGM

864

49 Comments

I very much agree with this and the rest of the piece: "Looking at LLM usage and promotion as a cultural phenomenon, it has all of the markings of a status game. The material gains from the LLM (which are usually quite marginal) really aren't why people are doing it: they're doing it because in many spaces, using ChatGPT and being very optimistic about AI being the "future" raises their social status. It's important not only to be using it, but to be seen using it and be seen supporting it and telling people who don't use it that they're stupid luddites who'll inevitably be left behind by technology. [...] [T]he only reason that LLMs took root in the first place was because our societies in the anglosphere have already developed cultures solely devoted to gaining status and keeping up the appearance of doing things rather than actually doing them." https://lnkd.in/dut2bq_z

130

16 Comments

https://lnkd.in/eNhfS-AM An important paper was released introducing vec2vec (Thanks to my friend Kyle at Booz who messaged me about it). "An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference."

37

3 Comments