Jon A.

I just published Vol. 119 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Theoretical Limits of Single-Vector Embedding Models in Information Retrieval, from Google DeepMind 📚 Investigating Why Randomly Truncating Text Embeddings Barely Hurts Performance, from Takeshita et al. 📚 Vector Quantization Attention for Ultra-Long User Behavior Modeling in Recommender Systems, from Kuaishou 📚 Conditional Two-Tower Models for Bootstrapping User-to-Item Retrieval Systems, from Pinterest 📚 Computational Scaling Laws for Zero-Shot Information Retrieval with Decoder Models, from Databricks 📚 A Comprehensive Analysis of LLM-based Reranking Methods, from the University of Innsbruck 📚 Lazy Decoder-Only Architecture for Industrial-Scale Generative Recommendation, from Kuaishou 📚 Dynamic Multi-Task Learning for Scalable Recommendation Systems, from Kuaishou 📚 Enabling Compact Language Models for Agentic RAG Through Distillation-Guided Reinforcement Learning, from Kotoge et al. 📚 Combining ID and Content Embeddings Without Architectural Complexity, from Albatross AI #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

48

1 Comment

You're in a ML Inference Engineer interview at NVIDIA, and the interviewer asks: "Our team wants to switch from API to a self-hosted. Which serving framework and why?" Here's how you answer: Don't say: "vLLM is fastest" or "Ollama is easiest." Wrong framing. The real answer isn't about features - it's about matching serving philosophy to your constraints. Local prototype vs. production scale vs. complex workflows = completely different frameworks. NOTE - if you want to read this kinda content daily, consider subscribing my free newsletter - https://lnkd.in/g8ZJGsWj Most teams pick based on benchmarks, not bottlenecks Your serving choice matters less than your use case. PagedAttention isn't magic - it's just better memory management "Fastest" framework can be slowest for YOUR workload The infrastructure reality everyone misses: Single user prototype = Ollama/LLaMA.cpp wins Multi-user production = vLLM dominates AI agent workflows = SGLang is your only option Edge deployment = LLaMA.cpp or bust Architecture drives the decision, not features. "But what about performance?" Interviewer: "How do you handle 1000 concurrent users?" Performance without context is meaningless. > vLLM's PagedAttention gives you 24x throughput, but only if memory fragmentation was your bottleneck. > SGLang's RadixAttention only helps with cache reuse scenarios. By the way, just reading about inference frameworks won't help you land a job. I personally learnt a lot by doing stuff. If I were to do all over, I would rent a gpu from Hyperbolic for dirt cheap, and play with some open source model with different frameworks. The business framework that matters: > Rapid prototyping + Local dev = Ollama > Known high load + GPU budget = vLLM > Complex multi-step LLM workflows = SGLang > Constrained environments + Portability = LLaMA.cpp Match the tool to the business constraint. The evolution path most teams miss: > Start: Ollama (prototype fast) > Scale: vLLM (handle users) > Optimize: SGLang (complex workflows) > Edge: LLaMA.cpp (deploy anywhere) It's not replacement - it's graduation based on needs. The answer that gets you hired: "Framework choice isn't about better vs worse. It's about constraints vs capabilities. Ollama gives you speed-to-market. vLLM gives you user scale. SGLang gives you workflow complexity. Pick based on your biggest constraint, not your biggest benchmark." That's it for today. See ya tomorrow! And if you want to read more about these frameworks, checkout this blog: https://lnkd.in/eiH2sphA #llm #machinelearning #gpu #chatgpt #ai #agents

398

17 Comments

CAG is RAG’s quiet but powerful cousin -- and it fixes RAG’s biggest weakness. Over the past year, most of the industry has been focusing on RAG. But Cache Augmented Generation (CAG) is showing up as a powerful alternative. Why: - RAG is great when you need fresh data and external retrieval. - CAG shines when you work with data that changes rarely but is accessed very frequently. This avoids repeated retrieval and reduces latency and cost. How CAG works: - You preload “cold” data (rarely changing, high-usage datasets) directly into the model’s KV cache. - The cache is then reused across queries without recomputation. - For example, long compliance rules, internal guidelines, or product manuals can be stored once and applied instantly for every request. - This removes the overhead of retrieval from databases or embeddings for information you always need. It also allows the LLM to respond faster and with higher consistency. Why it’s better than RAG (in some cases): - RAG adds context at runtime -- higher latency, more complexity. - CAG brings the context “pre-baked” into the session -- faster, cheaper, and more reliable for static data. Of course, the real power comes when you combine RAG and CAG -- where you're caching what rarely changes, and retrieving what must stay fresh. RAG retrieves. CAG remembers. And in doing so, it eliminates RAG’s biggest problem: latency :) ♻️ Share it with anyone who is trying to get RAG to work :) I regularly share AI Agents and projects on my newsletter: https://lnkd.in/gaJTcZBR #AI #GenAI #AIAgents

249

8 Comments

Pretty exciting new work from Kuaishou in the Generative Recommender domain. They replaced the entire video recommendation funnel (retrieval - ranking - value model) with a single encoder/decoder architecture trained on user action sequences with next-item prediction task, similar to how LLMs are pre-trained to predict the next token. The key to make this all work is their tokenizer, which is trained to map video content and metadata to semantic item IDs — so that similar videos share similar IDs, making generation feasible even at billion-scale vocabularies. On top of that, they apply reinforcement learning with user feedback to break through the ceiling of the traditional recommender stack. Based on their A/B tests this resulted in +0.5% App Stay time in the Kuaishou app. This finding shows just how much headroom there is left on top of traditional recommender systems! More about this in my blog: https://lnkd.in/gCr6k-MD

148

2 Comments