Andrew Liang

Our team ByteBrain, in collaboration with Tsinghua University, proposed ChatTS, a time-series multimodal LLM, accepted to VLDB 2025. Here is a brief introduction: Understanding time series is crucial for its application in real-world scenarios. Recently, large language models (LLMs) have been increasingly applied to time series tasks, leveraging their strong language capabilities to enhance various applications. However, research on multimodal LLMs (MLLMs) for time series understanding and reasoning remains limited, primarily due to the scarcity of high-quality datasets that align time series with textual information. This paper introduces ChatTS, a novel MLLM designed for time series analysis. ChatTS treats time series as a modality, similar to how vision MLLMs process images, enabling it to perform both understanding and reasoning with time series. To address the scarcity of training data, we propose an attribute-based method for generating synthetic time series with detailed attribute descriptions. We further introduce Time Series Evol-Instruct, a novel approach that generates diverse time series Q&As, enhancing the model’s reasoning capabilities. To the best of our knowledge, ChatTS is the first MLLM that takes multivariate time series as input for understanding and reasoning, which is fine-tuned exclusively on synthetic datasets. We evaluate its performance using benchmark datasets with real-world data, including six alignment tasks and four reasoning tasks. Our results show that ChatTS significantly outperforms existing vision-based MLLMs (e.g., GPT-4o) and text/agent-based LLMs, achieving a 46.0% improvement in alignment tasks and a 25.8% improvement in reasoning tasks. We have open-sourced the source code, model checkpoint and datasets at https://lnkd.in/gYDJ9S4j Paper: https://lnkd.in/gqXW9tPY About ByteBrain Team: ByteBrain is a platform of AI for Infra (AI for System) at ByteDance. The goal is to leverage AI to automatically optimize system performance and reduce labor and resource costs. ByteBrain integrates machine learning, LLM and OR techniques to support both internal infrastructure systems at ByteDance as well as the Volcano Engine Cloud.

151

3 Comments

Huge news for open-source. Qwen just released a 32B model that rivals DeepSeek-R1 while being 95% smaller. Why this matters: 🔹 RL scaling breakthrough – Reinforcement learning boosts reasoning, math, and coding, making mid-sized models competitive with giants. 🔹 Efficient & open – Fully open-weight (Apache 2.0), available on Hugging Face & ModelScope. 🔹 Long context & tool use – 131K tokens + agent-based reasoning for complex problem-solving. Blog: https://lnkd.in/ggxwZRua HF: https://lnkd.in/gYiQPdiB Demo: https://lnkd.in/gwK766xd Qwen Chat: https://chat.qwen.ai ↓ Are you an AI developer? Check out https://AlphaSignal.ai to get a daily summary of breakthrough models, repos and papers in AI. Read by 200,000+ devs.

484

28 Comments

Vision Large Language Models (vLLMs) extend text-based LLMs to understand visual modalities like images and video. Here’s exactly how they work... Top-level view. vLLM architectures have two primary components: 1. LLM backbone: standard decoder-only transformer. 2. Vision encoder: usually a CLIP / ViT model (with optional Perceiver Resampler for handling videos). Visual tokens. The vision encoder takes an image (or video) as input and returns a fixed-size set of visual “token” vectors to represent this image as an output. Usually, we create visual tokens by sampling vectors from multiple layers of the CLIP model and concatenating them together, which ensures little perceptual information is lost. Unified embedding. Now, we have a set of text and image (or video) token vectors as input. The first common vLLM architecture simply: - Concatenates the two modalities of vectors together. - Passes these concatenated vectors as input to a decoder-only transformer. The size of the visual token vectors may not match that of the text token vectors, so we linearly project the visual token vectors into the correct dimension. The unified embedding architecture is conceptually simple, but it increases the length of input passed to the LLM, which increases computational costs during training / inference. These visual tokens are passed through every layer of our powerful LLM backbone! Cross-modality attention. Instead of concatenating text and vision token vectors, we just pass text token vectors as input to the LLM. To incorporate vision info, we can add cross-attention modules that perform cross-attention between the text and vision token vector into select layers of the LLM—usually every second or fourth layer. This architectural variant, which looks similar to the transformer decoder, is called the cross-modality attention architecture. This architecture merges visual info into the LLM using cross-attention (more efficient) instead of increasing input length. Additionally, it adds new layers into the model for fusing visual / text info, rather than relying on existing LLM layers. So, we can leave the LLM backbone fixed during training and only train the added layers, ensuring the LLM’s performance on text-only tasks stays the same. Training vLLMs. vLLMs are trained similarly to any other LLM–using next token prediction. There are two strategies for training a vLLM like this: 1. Natively multi-modal: train the model from scratch using multi-modal data from the beginning and throughout the entire training process. 2. Compositional: first pretrain the vision encoder / LLM backbone separately, then perform a separate training stage to fuse them together. Many examples of natively mutli-modal LLMs now exist (e.g., Gemini or GPT-4o). However, natively multi-modal training can be complex, and compositional training has several benefits (e.g., model development can be parallelized, we can leverage existing text-based LLMs, etc.).

419

25 Comments

Last few months, I have invested some of my personal time to guide AI startups and graduate students across several universities on projects related to GenerativeAI and Speech Processing. ✨ Excited to share promising findings from one of such collaborations led by awesome graduate students Sagnik and Abhiram from Stanford University. ⁉️ Challenge: Generative models for discrete data face two critical limitations: 1️⃣ They rely on continuous embeddings for inherently discrete distributions 2️⃣ They optimize variational bounds instead of exact likelihood, leading to inaccurate probability estimates 🎯 Our Solution: Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM) - the first approach to tackle both issues simultaneously 📊 Results: Our theoretical proofs and exhaustive empirical validation demonstrate that ItDPDM achieves better likelihood estimates compared to existing generative methods for discrete data Super proud of Sagnik, Abhiram for extensive experiments to validate superiority of information theory driven discrete state modeling of discrete data. Please refer pre-print of the manuscript in the comment. Looking forward to how community use this work, feel free to DM to discuss the work. #generativeAI #diffusionmodel #AI #deeplearning #informationtheory

94

2 Comments

NVIDIA Research just published an intersting paper stating that SLMs aren't just "budget alternatives" to LLMs, but they're often superior for real-world applications. Their analysis of popular agents like MetaGPT, Open Operator, and Cradle shows that 40-70% of current LLM calls could be effectively replaced by specialized SLMs. Meanwhile, Andrej Karpathy envisions an even more radical future: a small "cognitive core" model that runs locally by default, with always-on capabilities, multimodal input/output, and the ability to dial up reasoning power when needed while delegating complex tasks to cloud-based oracles. The convergence is clear: we're moving from monolithic, cloud-dependent LLMs toward modular, efficient architectures. SLMs offer 10-30x better cost efficiency, lower latency, and can be fine-tuned faster and at a reasonable cost. For agentic systems performing repetitive and specialized task are better handled with careful context engineering and specialized SLMs. The future isn't about choosing between big and small models, but it's about intelligent orchestration of both. Paper: https://lnkd.in/gZAgH4Y6 Andrej Karpathy's X post: https://lnkd.in/g23777Mc

114

2 Comments

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

70

2 Comments

Recovering high-quality 3D scenes from a single RGB image is a challenging task in computer graphics. Analyzing inter-object spatial relationships to make coherent physics interactions is even more a struggle! ShanghaiTech University, Deemos and Huazhong University of Science and Technology introduces "CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image" CAST starts by extracting object-level 2D segmentation and relative depth information from the input image, followed by using a GPT-based model to analyze inter-object spatial relationships. This enables the understanding of how objects relate to each other within the scene, ensuring more coherent reconstruction. CAST then employs an occlusion-aware large-scale 3D generation model to independently generate each object's full geometry, using MAE and point cloud conditioning to mitigate the effects of occlusions and partial object information, ensuring accurate alignment with the source image's geometry and texture. To align each object with the scene, the alignment generation model computes the necessary transformations, allowing the generated meshes to be accurately placed and integrated into the scene's point cloud. Finally, CAST incorporates a physics-aware correction step that leverages a fine-grained relation graph to generate a constraint graph. This graph guides the optimization of object poses, ensuring physical consistency and spatial coherence. By utilizing Signed Distance Fields (SDF), the model effectively addresses issues such as occlusions, object penetration, and floating objects, ensuring that the generated scene accurately reflects real-world physical interactions. Experimental results demonstrate that CAST significantly improves the quality of single-image 3D scene reconstruction, offering enhanced realism and accuracy in scene recovery tasks. CAST has practical applications in virtual content creation, such as immersive game environments and film production, where real-world setups can be seamlessly integrated into virtual landscapes. Additionally, CAST can be leveraged in robotics, enabling efficient real-to-simulation workflows and providing realistic, scalable simulation environments for robotic systems. Checkout the comments for more #sceneunderstanding #scenereconstruction #assetreconstruction #generativeai #deeplearning #machinelearning

137

4 Comments

Recent research by UIUC and Intel Labs has introduced a new jailbreak technique for Large Language Models (LLMs) known as InfoFlood. This method takes advantage of a vulnerability termed "Information Overload," where excessive linguistic complexity can circumvent safety mechanisms without the need for traditional adversarial prefixes or suffixes.   InfoFlood operates through a three-stage process: Linguistic Saturation, Rejection Analysis, and Saturation Refinement. Initially, it reformulates potentially harmful queries into more complex structures. If the first attempt does not succeed, the system analyzes the response to iteratively refine the query until a successful jailbreak is achieved. Empirical validation across four notable LLMs—GPT-4o, GPT-3.5-turbo, Gemini 2.0, and LLaMA 3.1—indicates that InfoFlood significantly surpasses existing methods, achieving success rates up to three times higher on various benchmarks.   The study underscores significant vulnerabilities in current AI safety measures, as widely used defenses, such as OpenAI’s Moderation API, proved ineffective against InfoFlood attacks. This situation raises important concerns regarding the robustness of AI alignment systems and highlights the necessity for more resilient safety interventions. As LLMs become increasingly integrated into diverse applications, addressing these vulnerabilities is crucial for ensuring the responsible deployment of AI technologies and enhancing their safety against emerging adversarial techniques.  Arxiv: https://lnkd.in/eBty6G7z

53

4 Comments

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training This recent paper provides compelling evidence for RL's utility in Vision Language Models (VLMs). The authors demonstrate that RL generalizes effectively in both rule-based textual and visual environments, while SFT tends to memorize training data and struggles with out-of-distribution generalization. Their attempts at RL training on the base model directly failed due to poor instruction-following capabilities, unlike DeepSeek-R1-Zero (though it had readability and language mixing issues). The poor instruction following prevented the retrieval of task-related information and rewards needed for RL training. The key findings: While RL proves superior for out-of-domain generalization and works well with VLMs, SFT remains crucial for effective RL training by stabilizing the model's output format, enabling RL to achieve its performance gains. Paper: https://lnkd.in/efi-savV

121

2 Comments