Harrison Chase

dbt Labs announcement of #dbt (con)Fusion has a lot of things that we as a data community need to discuss. dbt Fusion is a complete rewrite of dbt Core in Rust. Unlike dbt Core, which is completely free and open source under Apache 2.0, dbt Fusion is not open source as it is under the more restrictive Elastic 2.0 license. Although Fusion is free to use, it restricts its usage in a hosted or managed offering to 3rd parties. You may think this is fine, but there are far-reaching implications. Open source is amazing because it incentivizes individuals and companies to invest in it without risk. A company can go all in on open source because no matter what the direction of the core maintainers is, it can be forked and used for whatever reason you want. A non-permissive license like Elastic will disincentivize companies from investing. Don't get me wrong, there's nothing ethically wrong about dbt Labs' decision. It may even be in their best financial interest to do so. However, I want to analyze what led to this situation and what it might mean for the future of dbt Core. What I believe this means is that dbt's strategy is to put dbt Core in "maintenance mode" to focus on Fusion and their other proprietary offerings. The wording of the announcement was very carefully selected to be vague. In particular, when referring to dbt Core support, it was only highlighted that bug fixes, security patches, and compatibility would be ongoing. According to their dbt Core roadmap, they've separated out the dbt language from the runtime. There's a specific callout that Fusion and Core will inevitably diverge because Fusion has additional capabilities that cannot be added to Core. To me, it makes sense that this is the chance for dbt Labs to invest in more restrictive and profitable software while slowly deprecating what not only made them great but is also their biggest challenge to financial growth. Ultimately, resources are finite, and companies must prioritize what makes sense for the business. Given dbt Core's foundational importance to modern data infrastructure, Analytics Engineers deserve a free, open, and continually evolving transformation platform. Otherwise, your career will be dangerously dependent on the decisions of a single company. To safeguard the continuing innovation and development of the transformation space, it may be time to start a discussion about an open standard for defining data transformations.

795

49 Comments

Making RAG systems super fast From an operational perspective, we need a low time to first token(TTFT) and time per output token(TPOT). These two metrics define latency, which can be calculated as TFTF + TPOT * tokens. RAG has two steps. The first one is to find chunks of text which will be added to the prompt. This operation is primarily CPU-bound, which involves querying a vector store and selecting top-k chunks. The second step is to send an extended prompt to the GPU for LLM inference. An obvious solution is to use a cache, saving CPU and GPU cycles. In the LLM world, when we talk about cache, we mean cache for the KV matrix in Transformer architecture. This KV matrix calculates an attention score, which is the core of Transformers. The matrix size is big, so it might be problematic to load it into GPU every time we need it. From the other perspective, we send several chunks of text to LLM: C1, C2, and C3. This sequence is not the same as C3, C1 and C2. Chunk ordering might affect attention score. Together with an order bias of LLM, the final answer might be affected. So when we cache the KV matrix, we have to sync about the order of chunks, too. RAGCache, HCache, TurboRAG, and Don't Do RAG papers review different caching solutions for RAG. RAGCache builds a cache tree - a knowledge tree. This tree keeps a structure that follows the order of chunks to cache. Also, the paper shows a multi-level cache structure, where the most used cache items are stored in GPU and less used in host memory. The replacement policy which is used is called PGDSF. All of this helped reduce TTFT by 4x and improved throughput by 2.1x. In RAGCache, we cache the whole KV matrix, but what if we can restore it with just a bit of calculation? This will save memory in exchange for computation time. HCache caches a hidden state instead of a KV matrix. It's 2x smaller and needs just a bit of computation. As a result, faster transfer time and 'up to 5.73x faster than recomputation'. The best improvements happen when we review our fundamental beliefs. What if the order of chunks does not matter much? TurboRAG explores this idea. It precomputes the KV matrix for each chunk in advance and delivers it to the GPU when needed. The paper introduced a new attention mechanism - Independent Attention, which concatenates caches at inference time in LLM. TTFT is improved by up to 9.4x In the beginning, I said that RAG is essential and used everywhere. But maybe we don't need RAG; we need the ability to add additional information to prompts. We use RAG primarily because of a limit on prompt size and a need for real-time information. Don't Do RAG paper reviews LLMs with long context. The approach is called - Cache-Augmented Generation. The idea is to load all the documents in LLM and save the resulting KV matrix to a cache. It only works for an LLM with a big enough context window. The absolute winner will be an approach that dynamically mixes the four solutions described above.

81

6 Comments

Wild. You can now hit 100 tokens/sec on open-source models like DeepSeek R1. Baseten just launched the fastest Model API for the new R1 5280 weights, currently leading on OpenRouter. It’s powered by their new Inference Stack, an impressive engineering feat built for high-throughput, low-latency inference. They shared a deep dive on what’s behind the speed: ▸ Runtime optimizations ▸ Custom kernels ▸ Speculation engine ▸ Smart routing + autoscaling ▸ Fully multicloud Read the deep dive: https://lnkd.in/gRRa7HVS Try the API: https://lnkd.in/gzwW7UNQ

159

10 Comments

Most Helm Charts are garbage. I’ve used Helm for 6+ years. Written charts. Debugged broken ones. Audited 150+ for clients. But most Helm charts are bloated, fragile, and impossible to scale. They weren’t built for your infra. They were built for any infra. 𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐡𝐚𝐩𝐩𝐞𝐧𝐬 𝐢𝐧 𝐫𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐭𝐞𝐚𝐦𝐬: ☠️ 1 toggle breaks 3 other things ☠️ You want to disable a feature, but now the whole values.yaml explodes ☠️ PRs go stale because nobody wants to touch that spaghetti of conditionals ☠️ 20 templates in 1 chart… and no one knows what’s actually deployed 𝐒𝐨 𝐡𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭 𝐰𝐞 𝐮𝐬𝐞 𝐢𝐧𝐬𝐭𝐞𝐚𝐝 𝐨𝐟 𝐇𝐞𝐥𝐦 𝐂𝐡𝐚𝐫𝐭𝐬: ✅ 1. Kustomize + Jsonnet for opinionated builds Forget general-purpose. We build narrow, composable layers. Each env has exactly what it needs. No more, no less. ✅ 2. Internal Helm forks with pre-baked policies For unavoidable 3rd-party Helm charts, we fork and harden them. We strip out what we don’t use. We codify security defaults. No more trusting upstream chaos. ✅ 3. GitOps-native templating with reusable overlays We write infra the way devs write code. Envs are just overlays, not duplicate manifests. Auditable. Portable. Reviewable. ✅ 4. YAML lint + Policy-as-Code in CI Before anything hits a cluster, we validate against org-wide standards. This catches the “hidden footguns” Helm often lets slip. 𝐀𝐧𝐝 𝐡𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭 𝐧𝐨𝐛𝐨𝐝𝐲 𝐭𝐞𝐥𝐥𝐬 𝐲𝐨𝐮: Helm is great for demos. Disasters in production. - Most values.yaml files turn into unmaintainable state machines - Real infra needs opinionated defaults, not endless if-else toggles - GitOps doesn’t need a templating engine. It needs consistency and clarity If your team is fighting Helm more than using it, you’re not alone. We’ve rebuilt entire CI/CD stacks just to get rid of Helm-induced debt. Want a faster way to ship secure, scalable infra? You don’t need 300-line values files. You need reproducible, testable infra with clear ownership.

147

31 Comments

Meta just announced the release of Llama 4 models. 10M CONTEXT WINDOW??? Here’s what you need to know: Llama 4 Scout - 210GB - Superior text and visual intelligence.   ↪ Class-leading 10M context window.   ↪ 17B active parameters x 16 experts, 109B total parameters.  Llama 4 Maverick - 788GB - The most powerful open-source multimodal model.   ↪ State of the art intelligence and fast responses at a low cost.   ↪ 17B active parameters x 128 experts, 400B total parameters.  Llama 4 Behemoth - still under training.   ↪ 288B active parameters x 16 experts,  massive 2T total parameters.  These models are not reasoning models.  Llama 4 use the "Mixture of Experts" (MoE) architecture.   ↪ Breaks tasks into smaller models.   ↪ Smaller models work together for results like a massive model.  What new possibilities would a 10M context window open ? What would you like to try these models for? 

29

2 Comments

How can a game studio double its portfolio + triple its launch pace, all with the same risk team? One of infinite use cases for AI: ↳ quicker & more in depth outcomes ↳ no technical user needed (just plain English) ↳ taking away repetitive tasks (the stuff humans hate) ↳ keep headcounts flat while enriching the human team's roles ↳ integrated like a human co-worker via Slack, email, Teams or WhatsApp A combination of business intelligence, analytics, and automation. (all secure incl. private networking etc.) Well done Philipp, Aivaras and teams. More customer stories to follow.

57

12 Comments

Is Next Token Prediction the only way Transformers work? Well, No. The main problem: >> Zero-shot text classification typically relies on prompt engineering, but the inherent prompt brittleness of large language models undermines its reliability. >> Minor changes in prompt can cause significant discrepancies in model performance. This attribute this prompt brittleness largely to the NARROW FOCUS on next token probabilities in existing methods. What is the solution, which this paper discusses? To address this, we propose Place-holding Parallel Prediction (P3 ), a novel approach that predicts token probabilities across multiple positions and simulates comprehensive sampling of generation paths in a single run of a language model. Read More in the paper

29

2 Comments

Tensormesh just came out of stealth and didn't tiptoe onto the scene, they kicked the door in with a $4.5M seed round led by Laude Ventures, with Pete Sonsini behind the deal and an angel check from #database pioneer Michael J. Franklin Founded in 2025 and already reshaping the AI infrastructure game, this San Francisco-based team isn't chasing headlines, they're chasing efficiency. CEO Junchen Jiang, Associate Professor of Computer Science at the University of Chicago and co-creator of LMCache Lab, knows how to make complex systems move fast without breaking a sweat. Pair that with CTO Yihua Cheng, fresh out of his PhD at the University of Chicago and loaded with experience from Conviva, Microsoft Research Asia, Alibaba Cloud & MEGVII, and you've got a founding duo that blends deep academia with hands-on engineering swagger. Tensormesh lives at the intersection of AI and #infrastructureoptimization. Their mission is simple: kill redundant computation during #inference. Every time an enterprise model runs the same sequence of logic twice, that's wasted compute, wasted power, wasted money. Tensormesh's caching-accelerated inference tech stops that nonsense cold, cutting latency & GPU costs by up to 10x while keeping enterprises in full control of their data & stack. The backbone of this magic? LMCache. What started as an #opensource project now sits at 5K+ GitHub stars with 100+ contributors and integrations across heavyweights like Bloomberg, Red Hat, Redis, Tencent, GMI Cloud, WEKA, Google & NVIDIA. Tensormesh is the commercial remix, turning distributed caching into a revenue-driving optimization engine. They've already integrated with vLLM NVIDIA Dynamo & Redis, and the results speak louder than any pitch deck. Laude Ventures saw the writing on the wall: #inferenceoptimization is the next trillion-dollar bottleneck. As models scale, the infrastructure costs explode. Tensormesh is quietly flipping that equation. With Ion Stoica, Co-Founder & Executive Chairman of Databricks, advising, this crew isn't guessing, they're engineering the future of scalable AI. This isn't hype, it's inevitability. LMCache proved caching can change the performance game. Tensormesh is proving it can change the economics. When your platform can make LLMs run 10x faster for a fraction of the GPU bill, that's not incremental innovation, that's structural advantage. Junchen Jiang & Yihua Cheng aren't building a tool. They're building the infrastructure layer the next generation of AI will depend on. The AI arms race just found its optimizer. #Startups #StartupFunding #EarlyStage #VentureCapital #SeedRound #AI #AIInfra #Infrastructure #GPU #InferenceTech #Technology #Innovation #TechEcosystem #StartupEcosystem If software engineering peace of mind is what you crave, Vention is your zen.

44

4 Comments