Jonathan Ho

New Short Course: Getting Structured LLM Output! Learn how to get structured outputs from your LLM applications in this course, built in partnership with DotTxt, and taught by Will Kurt, a Founding Engineer, and Cameron Pfiffer, Developer Relations Engineer. It's challenging for software to automatically parse through an LLM's freeform text outputs. Structured outputs—like JSON—solve this by converting natural language into consistent, clear, data that a machine can read and process. This course teaches you how to generate structured outputs while building several use cases, including a social media analysis agent. You’ll learn about structured outputs and efficient ways to generate outputs in your defined schema or format. You’ll begin by using structured output APIs, then use re-prompting libraries like “instructor” to generate structured output. Finally, you’ll learn how constrained decoding works; this is a very clever technique in which constraints are applied on each subsequent token generated, blocking any tokens that don’t fit your defined schema. In detail, you’ll: - Learn why structured outputs are important, how they allow for scalable software development, and the different approaches to generate them, including vendor-provided APIs, re-prompting libraries, and structured generation. - Build a simple social media agent using OpenAI’s structured output API, learn how to define a model's desired structured output using Pydantic, and perform basic programming with your outputs, such as importing structured data into a data frame using pandas. - Learn how to use the open-source library "instructor," which checks the structured output of the model and re-prompts the model until it validates the desired output, and explore the limitations of this approach. - Understand how structured generation by the “outlines” library works by modifying LLM logits, on a per-generated-token basis based on the desired format, to give a particular output structure. - Learn how regular expressions, which outlines works with, are represented as finite-state machines, and how they can be used to develop a range of structured outputs beyond JSON. By the end of this course, you’ll have broadened your knowledge of the approaches you can use to get structured outputs from your LLM applications. Please sign up here: https://lnkd.in/gpkNZvwQ

5,134

141 Comments

🚀𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 We’re excited to share our latest work on improving inference-time efficiency for LLMs through KV cache quantization—a critical step toward making long-context reasoning more scalable and memory-efficient. 🧠𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹𝘀 & 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲-𝘁𝗶𝗺𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 Modern reasoning models often require long responses to “think” through problems before arriving at a final answer. Inference-time scaling methods make this even more compute-intensive. While these approaches improve model performance, they incur higher latency and demand more GPU memory. 💡𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 The KV cache stores the intermediate representations from previous tokens to accelerate autoregressive decoding. Think of it as the model’s short-term memory—just as humans recall previous parts of a conversation to respond quickly, the KV cache helps models maintain and build on prior context. For long sequences, the KV cache can consume more GPU memory than the model weights. During inference, LLM decoding becomes memory-bound, with most of the time spent on data transfer rather than computation. This has led to active research on KV cache quantization but quantization errors can accumulate as more tokens are generated, causing later tokens to deviate from expected outputs. ✨𝗪𝗵𝗮𝘁’𝘀 𝗻𝗲𝘄 𝗶𝗻 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? We introduce 𝚂̲𝚀̲𝚞̲𝚊̲𝚝̲ (Subspace-orthogonal KV cache quantization)—a new method that significantly reduces memory overhead and latency while maintaining model accuracy. SQuat constructs a subspace that captures critical task-relevant information, then enforces quantization errors to lie orthogonal to this subspace, minimizing their effect on the output of the attention mechanism. 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: ✅Training-free: No fine-tuning or calibration data needed ✅On-the-fly: Runs during inference without modifying the model ✅Theory-grounded: Built on a theoretical foundation we developed ⚡𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁: • Reduces GPU peak memory by 2.17× to 2.82× • Improves throughput by 2.45× to 3.60× • outperform existing KV cache quantization methods on benchmark tasks 📄paper: https://lnkd.in/emKhAVZu 💻 code: https://lnkd.in/e8TJ7N3R 👏 Joint work with my amazing co-authors Ligong Han, Kai Xu, Akash Srivastava, at the Red Hat AI innovation team (https://lnkd.in/exd6QDbk) Chris Wright, Ruchir Puri, Steven Huels, Joe Fernandes, Tushar Katarki, Ritika Gunnar, Jason McGee, Máirín Duffy, Luke Inglis, Mark Kurtz, Nick Hill, Tyler Michael Smith, vLLM 

206

1 Comment

Our team ByteBrain, in collaboration with Tsinghua University, proposed ChatTS, a time-series multimodal LLM, accepted to VLDB 2025. Here is a brief introduction: Understanding time series is crucial for its application in real-world scenarios. Recently, large language models (LLMs) have been increasingly applied to time series tasks, leveraging their strong language capabilities to enhance various applications. However, research on multimodal LLMs (MLLMs) for time series understanding and reasoning remains limited, primarily due to the scarcity of high-quality datasets that align time series with textual information. This paper introduces ChatTS, a novel MLLM designed for time series analysis. ChatTS treats time series as a modality, similar to how vision MLLMs process images, enabling it to perform both understanding and reasoning with time series. To address the scarcity of training data, we propose an attribute-based method for generating synthetic time series with detailed attribute descriptions. We further introduce Time Series Evol-Instruct, a novel approach that generates diverse time series Q&As, enhancing the model’s reasoning capabilities. To the best of our knowledge, ChatTS is the first MLLM that takes multivariate time series as input for understanding and reasoning, which is fine-tuned exclusively on synthetic datasets. We evaluate its performance using benchmark datasets with real-world data, including six alignment tasks and four reasoning tasks. Our results show that ChatTS significantly outperforms existing vision-based MLLMs (e.g., GPT-4o) and text/agent-based LLMs, achieving a 46.0% improvement in alignment tasks and a 25.8% improvement in reasoning tasks. We have open-sourced the source code, model checkpoint and datasets at https://lnkd.in/gYDJ9S4j Paper: https://lnkd.in/gqXW9tPY About ByteBrain Team: ByteBrain is a platform of AI for Infra (AI for System) at ByteDance. The goal is to leverage AI to automatically optimize system performance and reduce labor and resource costs. ByteBrain integrates machine learning, LLM and OR techniques to support both internal infrastructure systems at ByteDance as well as the Volcano Engine Cloud.

151

3 Comments

I first got started with Reinforcement Learning in 2016-2017, when it still lived mostly in research circles after the Atari results from DeepMind using RL. Fast forward to today, RL is having a moment again, largely because it has become a prominent way to make LLMs learn better. I’m glad to see the surge in interest, so I put together a quick handout on the basics. I’ll follow up with more posts to break down key ideas and how they’re used in practice. Here’s a crisp primer to get you started (in the handout below) If you’re curious to learn, grab the handout and stay tuned. I’ll share follow-ups on PPO, reward modeling, DPO vs RLHF, and practical pitfalls to avoid. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

261

37 Comments

Recent research by UIUC and Intel Labs has introduced a new jailbreak technique for Large Language Models (LLMs) known as InfoFlood. This method takes advantage of a vulnerability termed "Information Overload," where excessive linguistic complexity can circumvent safety mechanisms without the need for traditional adversarial prefixes or suffixes.   InfoFlood operates through a three-stage process: Linguistic Saturation, Rejection Analysis, and Saturation Refinement. Initially, it reformulates potentially harmful queries into more complex structures. If the first attempt does not succeed, the system analyzes the response to iteratively refine the query until a successful jailbreak is achieved. Empirical validation across four notable LLMs—GPT-4o, GPT-3.5-turbo, Gemini 2.0, and LLaMA 3.1—indicates that InfoFlood significantly surpasses existing methods, achieving success rates up to three times higher on various benchmarks.   The study underscores significant vulnerabilities in current AI safety measures, as widely used defenses, such as OpenAI’s Moderation API, proved ineffective against InfoFlood attacks. This situation raises important concerns regarding the robustness of AI alignment systems and highlights the necessity for more resilient safety interventions. As LLMs become increasingly integrated into diverse applications, addressing these vulnerabilities is crucial for ensuring the responsible deployment of AI technologies and enhancing their safety against emerging adversarial techniques.  Arxiv: https://lnkd.in/eBty6G7z

53

4 Comments

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training This recent paper provides compelling evidence for RL's utility in Vision Language Models (VLMs). The authors demonstrate that RL generalizes effectively in both rule-based textual and visual environments, while SFT tends to memorize training data and struggles with out-of-distribution generalization. Their attempts at RL training on the base model directly failed due to poor instruction-following capabilities, unlike DeepSeek-R1-Zero (though it had readability and language mixing issues). The poor instruction following prevented the retrieval of task-related information and rewards needed for RL training. The key findings: While RL proves superior for out-of-domain generalization and works well with VLMs, SFT remains crucial for effective RL training by stabilizing the model's output format, enabling RL to achieve its performance gains. Paper: https://lnkd.in/efi-savV

121

2 Comments

Last few months, I have invested some of my personal time to guide AI startups and graduate students across several universities on projects related to GenerativeAI and Speech Processing. ✨ Excited to share promising findings from one of such collaborations led by awesome graduate students Sagnik and Abhiram from Stanford University. ⁉️ Challenge: Generative models for discrete data face two critical limitations: 1️⃣ They rely on continuous embeddings for inherently discrete distributions 2️⃣ They optimize variational bounds instead of exact likelihood, leading to inaccurate probability estimates 🎯 Our Solution: Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM) - the first approach to tackle both issues simultaneously 📊 Results: Our theoretical proofs and exhaustive empirical validation demonstrate that ItDPDM achieves better likelihood estimates compared to existing generative methods for discrete data Super proud of Sagnik, Abhiram for extensive experiments to validate superiority of information theory driven discrete state modeling of discrete data. Please refer pre-print of the manuscript in the comment. Looking forward to how community use this work, feel free to DM to discuss the work. #generativeAI #diffusionmodel #AI #deeplearning #informationtheory

94

2 Comments