Misha Laskin

🎉 Happy to share that our paper is now published as the Version of Record in eLife The spatial frequency representation predicts category coding in the inferior temporal cortex 👉 https://lnkd.in/eNtsPN4H 🔬 We show that the inferior temporal (IT) cortex explicitly encodes spatial frequency (SF) at both single-neuron and population levels. The coding unfolds coarse-to-fine (low SF decoded first, high SF later), and a neuron’s SF profile can even predict category coding at the population level, especially for faces. Interestingly, SF and category rely on distinct, uncorrelated coding mechanisms, with SF coded more sparsely by individual neurons. 👀 In simple words: The brain’s object-recognition hub first takes in the blurry big picture and only later fills in the sharp details. Neurons tuned to fine detail are particularly important for recognizing faces. And the brain seems to handle “detail level” and “object type” using separate systems. 🙏 Huge thanks to my brilliant co-authors for this collaboration, and to eLife Sciences Publications, Ltd. for their innovative publish-then-review model, which makes science and peer review open and transparent. If you’re curious about vision, the IT cortex, or bio-inspired AI, I’d love to hear your thoughts. #Paper #Research #Neuroscience #Brain #CognitiveNeuroscience #OpenScience #eLife

124

7 Comments

Happy to announce AlignVLM 📏 – a novel approach to bridging vision and language latent spaces for multimodal understanding in Vision-Language Models (VLMs) 🌍📄🖼 🔗 Read the paper: https://lnkd.in/dw9v4i7P 🧐 What’s the challenge? Aligning visual features with language embeddings remains a major bottleneck in VLMs. Existing connectors such as Multi-layer perceptron (MLPs) often introduce noise that degrades performance. ❌ 🎯 Our Solution: ALIGN Connector We propose AlignVLM, a method that maps vision features into a weighted average of LLM text embeddings, ensuring they remain in a space that the LLM can effectively interpret. ✅ 🔬 How does it perform? We compared ALIGN against common connectors like MLPs, Perceiver Resampler, and Ovis trained under similar configurations. The results? ALIGN outperforms them all 🏆 on diverse document understanding tasks 📄. 📊 Meet the AlignVLM Model Family! We trained Llama 3.1 (1B, 3B, 8B) using our connector and benchmarked them against various models. The results: ✅ AlignVLM surpasses all Base VLMs trained under similar configurations. ✅ Our models also perform competitively against Instruct VLMs such as Qwen2-VL and InternVL-2.5 🚀. 🤔 What about robustness to noise? We injected Gaussian noise (μ=0, σ=3) into the vision encoder’s outputs before feeding them to the connector: ✅ ALIGN Connector: Minimal drop (↓1.67%) – proving its high robustness! ❌ MLP Connector: Severe degradation (↓25.54%) – struggling with noisy inputs. Code & model weights coming soon! Stay tuned! 🔥 Huge thanks to our amazing collaborators! 🙌 ServiceNow Research York University Mila - Quebec Artificial Intelligence Institute #AlignVLM #VisionLanguageModels #AI #MultimodalAI #MachineLearning #LLMs #DeepLearning #Research #DocumentUnderstanding

183

5 Comments

Kimi k1.5: A Next Generation Multi-Modal LLM Trained with Reinforcement Learning on Advancing AI with Scalable Multimodal Reasoning and Benchmark Excellence Researchers from the Kimi Team have introduced Kimi k1.5, a next-generation multimodal LLM designed to overcome these challenges by integrating RL with extended context capabilities. This model employs innovative techniques such as long-context scaling, which expands the context window to 128,000 tokens, enabling it to process larger problem contexts effectively. Unlike prior approaches, the Kimi k1.5 avoids relying on complex methods like Monte Carlo tree search or value functions, opting for a streamlined RL framework. The research team implemented advanced RL prompt set curation to enhance the model’s adaptability, including diverse prompts spanning STEM, coding, and general reasoning tasks. Kimi k1.5 demonstrated significant improvements in token efficiency through its long-to-short context training methodology, enabling the transfer of reasoning priors from long-context models to shorter models while maintaining high performance and reducing token consumption. The model achieved exceptional results across multiple benchmarks, including a 96.2% exact match accuracy on MATH500, a 94th percentile on Codeforces, and a pass rate of 77.5% on AIME, surpassing state-of-the-art models like GPT-4o and Claude Sonnet 3.5 by substantial margins. Its short-CoT performance outperformed GPT-4o and Claude Sonnet 3.5 on benchmarks like AIME and LiveCodeBench by up to 550%, while its long-CoT performance matched o1 across multiple modalities, including MathVista and Codeforces. Key features include long-context scaling with RL using context windows of up to 128k tokens, efficient training through partial rollouts, improved policy optimization via online mirror descent, advanced sampling strategies, and length penalties. Also, Kimi k1.5 excels in joint reasoning over text and vision, highlighting its multi-modal capabilities...... Read the full article here: https://lnkd.in/g47WGYXB Paper: https://lnkd.in/gY-2MiD2 GitHub Page: https://lnkd.in/gx3Y7PmW

37

1 Comment

🚀𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 We’re excited to share our latest work on improving inference-time efficiency for LLMs through KV cache quantization—a critical step toward making long-context reasoning more scalable and memory-efficient. 🧠𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹𝘀 & 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲-𝘁𝗶𝗺𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 Modern reasoning models often require long responses to “think” through problems before arriving at a final answer. Inference-time scaling methods make this even more compute-intensive. While these approaches improve model performance, they incur higher latency and demand more GPU memory. 💡𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 The KV cache stores the intermediate representations from previous tokens to accelerate autoregressive decoding. Think of it as the model’s short-term memory—just as humans recall previous parts of a conversation to respond quickly, the KV cache helps models maintain and build on prior context. For long sequences, the KV cache can consume more GPU memory than the model weights. During inference, LLM decoding becomes memory-bound, with most of the time spent on data transfer rather than computation. This has led to active research on KV cache quantization but quantization errors can accumulate as more tokens are generated, causing later tokens to deviate from expected outputs. ✨𝗪𝗵𝗮𝘁’𝘀 𝗻𝗲𝘄 𝗶𝗻 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? We introduce 𝚂̲𝚀̲𝚞̲𝚊̲𝚝̲ (Subspace-orthogonal KV cache quantization)—a new method that significantly reduces memory overhead and latency while maintaining model accuracy. SQuat constructs a subspace that captures critical task-relevant information, then enforces quantization errors to lie orthogonal to this subspace, minimizing their effect on the output of the attention mechanism. 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: ✅Training-free: No fine-tuning or calibration data needed ✅On-the-fly: Runs during inference without modifying the model ✅Theory-grounded: Built on a theoretical foundation we developed ⚡𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁: • Reduces GPU peak memory by 2.17× to 2.82× • Improves throughput by 2.45× to 3.60× • outperform existing KV cache quantization methods on benchmark tasks 📄paper: https://lnkd.in/emKhAVZu 💻 code: https://lnkd.in/e8TJ7N3R 👏 Joint work with my amazing co-authors Ligong Han, Kai Xu, Akash Srivastava, at the Red Hat AI innovation team (https://lnkd.in/exd6QDbk) Chris Wright, Ruchir Puri, Steven Huels, Joe Fernandes, Tushar Katarki, Ritika Gunnar, Jason McGee, Máirín Duffy, Luke Inglis, Mark Kurtz, Nick Hill, Tyler Michael Smith, vLLM 

206

1 Comment

🚀 New Preprint on arXiv! We’re thrilled to share our latest research on scalable multi-user communication powered by deep learning: "Learning to Interfere in Non-Orthogonal Multiple-Access Joint Source-Channel Coding" 📄 Read it here → https://lnkd.in/dinUR-s4 🔍 What’s it about? Traditional wireless systems avoid interference by orthogonalizing users — but at the cost of spectral efficiency. We challenge this paradigm by learning to embrace interference using a novel deep learning framework for joint source-channel coding (JSCC) over NOMA. 📌 Highlights • A multi-view autoencoder with user-specific projections — enabling a shared encoder/decoder across users • A progressive fine-tuning algorithm — scaling from 1 to 16+ users with <1% increase in parameters • Outperforms TDMA-based DeepJSCC, separation-based baselines (e.g., BPG+LDPC), and prior NOMA systems • Strong performance across diverse datasets, bandwidth settings, and channel models (AWGN, Rayleigh) 👥 Authors: Selim Firat Yilmaz, Can Karamanlı and Deniz Gunduz 💬 Feedback and discussion are very welcome! #arXiv #DeepLearning #WirelessAI #SemanticCommunication #NOMA #JSCC #DeepJSCC #6G #MachineLearning #Autoencoders #MultiUser

51

Mixed curvature models have had a lot of success in graph neural network training, where local embeddings account for local curvature. This paper applies mixed curvature techniques to knowledge graph embeddings and Tucker decompositions (a favorite tool from psychometrics) to fill in knowledge gaps in knowledge graphs. https://lnkd.in/euM9z839

65

5 Comments

No pretraining. No datasets. Just pure inference-time gradient descent on the target ARC-AGI puzzle itself, solving 20% of the evaluation set. Introducing *ARC‑AGI Without Pretraining* Check out MLD PhD student Isaac Liao's technical blog post here: https://lnkd.in/eY_kvbQj

17

A new rendering model just appeared for full global illumination. It's ... a transformer! The start of Mesh-to-image, end-to-end physics based rendering? Let's have a look 👇👇👇 Zhejiang University, Microsoft Research Asia and College of William & Mary present "Transformer-based Neural Rendering of Triangle Meshes with Global Illumination". This paper will be presented at ACM SIGGRAPH 2025. RenderFormer is a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects and that does not require per-scene training or fine-tuning. Instead of taking a physics-centric approach to rendering, they formulate rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted to a sequence of output tokens representing small patches of pixels. RenderFormer follows a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the the view-independent stage. Both stages are based on the transformer architecture and are learned with minimal prior constraints. No rasterization, no ray tracing Amazing stuff, more info in the comments #globalillumination #neuralrendering #transformers #machinelearning #endtoend #deeplearning

335

11 Comments