Arnaud Flament

dbt Labs announcement of #dbt (con)Fusion has a lot of things that we as a data community need to discuss. dbt Fusion is a complete rewrite of dbt Core in Rust. Unlike dbt Core, which is completely free and open source under Apache 2.0, dbt Fusion is not open source as it is under the more restrictive Elastic 2.0 license. Although Fusion is free to use, it restricts its usage in a hosted or managed offering to 3rd parties. You may think this is fine, but there are far-reaching implications. Open source is amazing because it incentivizes individuals and companies to invest in it without risk. A company can go all in on open source because no matter what the direction of the core maintainers is, it can be forked and used for whatever reason you want. A non-permissive license like Elastic will disincentivize companies from investing. Don't get me wrong, there's nothing ethically wrong about dbt Labs' decision. It may even be in their best financial interest to do so. However, I want to analyze what led to this situation and what it might mean for the future of dbt Core. What I believe this means is that dbt's strategy is to put dbt Core in "maintenance mode" to focus on Fusion and their other proprietary offerings. The wording of the announcement was very carefully selected to be vague. In particular, when referring to dbt Core support, it was only highlighted that bug fixes, security patches, and compatibility would be ongoing. According to their dbt Core roadmap, they've separated out the dbt language from the runtime. There's a specific callout that Fusion and Core will inevitably diverge because Fusion has additional capabilities that cannot be added to Core. To me, it makes sense that this is the chance for dbt Labs to invest in more restrictive and profitable software while slowly deprecating what not only made them great but is also their biggest challenge to financial growth. Ultimately, resources are finite, and companies must prioritize what makes sense for the business. Given dbt Core's foundational importance to modern data infrastructure, Analytics Engineers deserve a free, open, and continually evolving transformation platform. Otherwise, your career will be dangerously dependent on the decisions of a single company. To safeguard the continuing innovation and development of the transformation space, it may be time to start a discussion about an open standard for defining data transformations.

795

49 Comments

Not all large language models are created equal. The architecture behind an LLM determines how fast it runs, how well it scales, and where it shines. Here are 6 architectures every AI builder should know: ➤ 1. Encoder-Only → Focus: Deep understanding, embeddings → Trained on: Masked tokens → Ideal for: Search, classification Examples: BERT, RoBERTa ➤ 2. Decoder-Only → Focus: Next-token prediction → Trained on: Left-to-right context → Ideal for: Chat, code, agents Examples: GPT-4, Claude, LLaMA ➤ 3. Encoder-Decoder (Seq2Seq) → Focus: Input → Output mapping → Trained on: Paired data → Ideal for: Translation, summarization Examples: T5, BART ➤ 4. Mixture of Experts (MoE) → Focus: Sparse routing for scale → Trained on: Expert selection → Ideal for: High performance, lower cost Examples: GLaM, DeepSeek-V2 ➤ 5. State Space Models (SSMs) → Focus: Long context, fast compute → Trained on: State transitions → Ideal for: Large documents, low latency Example: Mamba ➤ 6. Hybrid Models → Focus: Best of all worlds → Trained on: Mixed paradigms → Ideal for: Balanced performance Examples: Jamba, Grok 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources)  channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

220

24 Comments

Making RAG systems super fast From an operational perspective, we need a low time to first token(TTFT) and time per output token(TPOT). These two metrics define latency, which can be calculated as TFTF + TPOT * tokens. RAG has two steps. The first one is to find chunks of text which will be added to the prompt. This operation is primarily CPU-bound, which involves querying a vector store and selecting top-k chunks. The second step is to send an extended prompt to the GPU for LLM inference. An obvious solution is to use a cache, saving CPU and GPU cycles. In the LLM world, when we talk about cache, we mean cache for the KV matrix in Transformer architecture. This KV matrix calculates an attention score, which is the core of Transformers. The matrix size is big, so it might be problematic to load it into GPU every time we need it. From the other perspective, we send several chunks of text to LLM: C1, C2, and C3. This sequence is not the same as C3, C1 and C2. Chunk ordering might affect attention score. Together with an order bias of LLM, the final answer might be affected. So when we cache the KV matrix, we have to sync about the order of chunks, too. RAGCache, HCache, TurboRAG, and Don't Do RAG papers review different caching solutions for RAG. RAGCache builds a cache tree - a knowledge tree. This tree keeps a structure that follows the order of chunks to cache. Also, the paper shows a multi-level cache structure, where the most used cache items are stored in GPU and less used in host memory. The replacement policy which is used is called PGDSF. All of this helped reduce TTFT by 4x and improved throughput by 2.1x. In RAGCache, we cache the whole KV matrix, but what if we can restore it with just a bit of calculation? This will save memory in exchange for computation time. HCache caches a hidden state instead of a KV matrix. It's 2x smaller and needs just a bit of computation. As a result, faster transfer time and 'up to 5.73x faster than recomputation'. The best improvements happen when we review our fundamental beliefs. What if the order of chunks does not matter much? TurboRAG explores this idea. It precomputes the KV matrix for each chunk in advance and delivers it to the GPU when needed. The paper introduced a new attention mechanism - Independent Attention, which concatenates caches at inference time in LLM. TTFT is improved by up to 9.4x In the beginning, I said that RAG is essential and used everywhere. But maybe we don't need RAG; we need the ability to add additional information to prompts. We use RAG primarily because of a limit on prompt size and a need for real-time information. Don't Do RAG paper reviews LLMs with long context. The approach is called - Cache-Augmented Generation. The idea is to load all the documents in LLM and save the resulting KV matrix to a cache. It only works for an LLM with a big enough context window. The absolute winner will be an approach that dynamically mixes the four solutions described above.

81

6 Comments

Just discovered TOON (Token-Oriented Object Notation), which provides an alternate token size optimized notation for use in the context of LLMs. It offers "typically 30–60% fewer tokens than JSON." Worth looking into! 🔗 TOON repo: https://lnkd.in/gUkFFyDc Explore this and other AI Tools, Tutorials, and News for software developers in Altered Craft's weekly AI review at alteredcraft.com.

2,274

179 Comments

𝐖𝐡𝐢𝐜𝐡 𝐦𝐞𝐭𝐡𝐨𝐝 𝐚𝐮𝐠𝐦𝐞𝐧𝐭𝐬 𝐋𝐋𝐌𝐬 𝐛𝐞𝐭𝐭𝐞𝐫: 𝐌𝐂𝐏 𝐨𝐫 𝐑𝐀𝐆? When context window have crossed 10M (Llama 4’s) tokens, is RAG as relevant as before? 𝐑𝐀𝐆 is about integrating a retrieval system that helps to bring relevant context to LLMs to reduce hallucination and introduce domain or recent knowledge to LLM. 𝐌𝐂𝐏 is a protocol that enables LLM applications (like Claude, Cursor, and Windsurf) to connect to tools in a standardized way. The standardization is helpful because there is an explosion of tools, and it's useful to have a common way to connect tools as opposed to building out M different ways to connect to N different applications. So in summary, for agentic systems, we need both: RAG (via a retrieval engine) and MCP (via tool connections) help to pull in important information to the LLM to improve the accuracy of outputs and to make better decisions, and take actions. People saying 'RAG is dead' do not understand the important of retrieval (which is R in the RAG). Building a great retrieval system is hard and requires tons of iteration and optimization. #rag #mcp #agenticai

40

7 Comments

𝗪𝗵𝗮𝘁 𝗶𝘀 𝗠𝗖𝗣? MCP is similar to a “USB-C for AI”, acting like a universal connector. It’s a standard that allows AI models (like Claude, GPT, Gemini, etc.) to easily connect to any tool, app, or data, regardless of the creator or location. 𝗪𝗵𝘆 𝗪𝗮𝘀 𝗠𝗖𝗣 𝗖𝗿𝗲𝗮𝘁𝗲𝗱? Before MCP, if you wanted an AI to: Read your files, Look up info in a database, Send a message on Slack, Or use any other tool, ...you needed to code specifically for every tool and AI model.It was messy, slow, and didn’t scale.MCP solves this by creating one standard way for AIs to “talk” to any tool or data source.It’s super simple to add new tools to your AI now. 𝗛𝗼𝘄 𝗗𝗼𝗲𝘀 𝗠𝗖𝗣 𝗪𝗼𝗿𝗸? Think of MCP as a translator and connector between AIs and the digital world. The Main Parts: Host:The main AI app or environment (like Claude Desktop, Cursor IDE, or a chatbot) where you interact. MCP Client:The “middleman” inside the host. It knows how to speak MCP and connects to MCP servers. MCP Server:A small program that “wraps” a tool, app, or data source (like Google Drive, GitHub, or your local files) and exposes its abilities in a standard way. Protocol:The agreed-upon “language” (using JSON messages) that clients and servers use to talk. 𝗛𝗼𝘄 𝗮 𝗧𝘆𝗽𝗶𝗰𝗮𝗹 𝗠𝗖𝗣 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗟𝗼𝗼𝗸𝘀: You ask the AI:“Summarize this YouTube video and save the summary to a file.” The AI (host) realizes it needs help:It can’t do this alone—it needs to fetch the transcript and save a file. The MCP client checks what’s available:It discovers there’s a YouTube MCP server (for transcripts) and a File MCP server (for saving files). The AI sends requests via MCP: It asks the YouTube server for the transcript. It asks the File server to save the summary. The servers do the work:Each server talks to its tool (YouTube, file system, etc.), gets the result, and sends it back. The AI combines everything:It uses the transcript, writes the summary, and saves it—all through MCP, with no custom code for each step . 𝗪𝗵𝗮𝘁 𝗖𝗮𝗻 𝗠𝗖𝗣 𝗖𝗼𝗻𝗻𝗲𝗰𝘁 𝗧𝗼? Files and folders, APIs, Databases, Apps ( github, slack), Browsers, and custome tools There are already thousands of MCP servers for all sorts of tools, and anyone can build more. 𝗪𝗵𝗮𝘁 𝗠𝗮𝗸𝗲𝘀 𝗠𝗖𝗣 𝗦𝗽𝗲𝗰𝗶𝗮𝗹? Open and universal:Any AI, any tool, any developer can use it—no lock-in . Plug-and-play:Add a new tool by running its MCP server. The AI instantly knows how to use it—no extra coding. Chaining and orchestration:The AI can use multiple tools together, passing results from one to another, to do complex tasks. Security and control:You (the user) decide what the AI can access. The protocol supports permissions and privacy. Reusable prompts and workflows:MCP servers can offer not just tools, but also reusable prompt templates and workflows for the AI to use #MCP #AI

198

42 Comments

Embedding RL in Agent Design Most AI agents today are over-engineered workflows pretending to be intelligent static decision trees, brittle toolchains, and fine-tuned prompts trying to hold together under dynamic, real-world conditions. The second the environment shifts, they break. But intelligence doesn’t come from preprogrammed behavior. It comes from feedback tight, continuous loops of action, evaluation, and adjustment. That’s where Reinforcement Learning steps in. RL doesn’t just help agents act; it helps them learn how to act better over time. It introduces context-awareness, delayed gratification, and runtime adaptability into agent design. Instead of assuming the right answer, agents can discover it-through interaction, iteration, and measured exploration. If we want AI agents to operate reliably in production-where conditions are noisy, users are unpredictable, and value is tied to long-term outcomes we don’t need more hardcoded logic. We need agents that evolve. That’s the alignment RL brings. Not just smarter decisions, but systems that get smarter the more they’re used. https://lnkd.in/eTgRnxbC

36

3 Comments

🧠 𝗔𝗜 𝗿𝗮𝗻𝗱𝗼𝗺𝗻𝗲𝘀𝘀? 𝗠𝗮𝘆𝗯𝗲 𝗻𝗼𝘁 𝘀𝗼 𝗿𝗮𝗻𝗱𝗼𝗺 𝗮𝗳𝘁𝗲𝗿 𝗮𝗹𝗹... Kyrtin Atreides noticed something odd: Five major LLMs — ChatGPT, Claude, Gemini, Perplexity, Meta — were all asked the same simple prompt: “Guess a number between 1 and 50.” Every single one guessed… 27. Coincidence? Not likely. This isn’t a failure of intelligence — it’s a clear example of pattern recognition at work. These models aren’t reasoning like humans — they’re predicting the most statistically probable output based on their training data. And apparently, “27” shows up a lot in that context. 🔐 𝗪𝗵𝘆 𝗱𝗼𝗲𝘀 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿? Because what looks like a harmless glitch could have serious implications — especially in cybersecurity. Predictable outputs can be exploited for social engineering, prompt injection, or more subtle vulnerabilities. 💡 It’s more than a fun AI quirk — it’s a real reminder: How much randomness can we actually expect from LLMs? 📢 𝗧𝗼 𝘁𝗵𝗲 𝘁𝗲𝗰𝗵 𝗮𝗻𝗱 𝗔𝗜 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆: How should we address this? Do we need better prompt design, external randomness injection — or simply more awareness around predictable patterns?

127

44 Comments

🚀 PRESS RELEASE: OpenBOM Announces Private Beta of AI BOM Agent with MCP Support I'm excited to share the news about availability of OpenBOM private beta of AI BOM Agent with MCP (Model Context Protocol) support. OpenBOM is building a new generation of intelligent, invisible engineering workflows. Instead of forcing engineers to change how they work, OpenBOM introduces AI agents that understand existing tools, files, and data formats. These agents connect to multiple data sources, structure information, automate routine tasks, and enable conversational access to product data. This vision is anchored in the concept of product memory—a persistent, connected knowledge model that spans design, engineering, manufacturing, and procurement. With MCP support, OpenBOM agents can ingest, structure, and link data into the digital thread, enabling reasoning, plain-language queries, and smarter decision-making. Looking ahead, OpenBOM sees AI agents playing a vital role in BOM quality and validation—helping companies ensure accuracy, consistency, and readiness of data before it flows into procurement and production. By transforming CAD data and unstructured Excel spreadsheets into structured, connected product knowledge inside OpenBOM’s graph-based model, we’re: ✅ Linking CAD data and BOMs into the digital thread ✅ Preserving product memory across teams and projects ✅ Laying the foundation for AI-driven workflows in engineering and manufacturing This is the first step in our roadmap to bring intelligence, automation, and conversational access to engineering data—moving from data ingestion and natural language queries to agent-driven actions and connected product memory. As I said in the release: “Excel remains the most widely used tool in manufacturing, but it is also the biggest bottleneck. The OpenBOM Agent with MCP support turns spreadsheets into structured product knowledge—enabling collaboration, traceability, and AI-powered decisions.” 👉 Learn more about the private beta and how to join here: [https://lnkd.in/eD9gmA_X] Over the next few days, I’ll be sharing examples of how OpenBOM AI agents can be used and how product data can be embedded into agentic workflows. #OpenBOM #PLM #AI #DigitalThread #ManufacturingInnovation #MCP

49

5 Comments

Why Some Labs don’t expose Log Probabilities OpenAI shows them in GPT-4 & 4o but not in GPT-5. Google provide that via VertexAI for Gemini. Anthropic’s Claude doesn’t provide & recently even cut off Claude API access to OpenAI staff. Why the secrecy? 𝐌𝐨𝐝𝐞𝐥 𝐭𝐡𝐞𝐟𝐭: Logprobs reveal a fingerprint that can accelerate model cloning. 𝐁𝐢𝐚𝐬 𝐞𝐱𝐩𝐨𝐬𝐮𝐫𝐞: Hidden skews on sensitive topics become visible. 𝐒𝐚𝐟𝐞𝐭𝐲 & 𝐈𝐏: Adversaries can probe weak spots, while probability distributions leak a lab’s secret recipe. So while logprobs are gold for research and interpretability, many labs unfortunately keep them locked down to protect safety and competitive edge. 𝐈 𝐰𝐫𝐨𝐭𝐞 𝐚 𝐛𝐥𝐨𝐠 𝐨𝐧 𝐭𝐡𝐢𝐬: https://lnkd.in/ekF7sUaF #genai #artificialintelligence #largelanguagemodels #probability

22

2 Comments

As frontier model context windows hit 1M tokens, it’s tempting to dump everything—tools, docs, instructions—into a prompt and let the model sort it out. "Lost in the middle" is real — research shows models perform best when relevant info is at the start or end of the input, and degrade when it’s buried in the middle, even for long-context models. That’s where context engineering comes in: writing, selecting, compressing, and isolating context for accuracy and efficiency. Three evolving approaches for optimizing context: 1️⃣ Scratchpad – Store info outside the active context window (e.g., Anthropic’s Think tool) so agents can pause mid-response, reflect, and refine. 2️⃣ Context isolation – Give each sub-agent only what it needs for its task. Anthropic’s multi-agent system does this, with a lead agent merging results for accuracy. 3️⃣ Prompt caching – Cache shared instructions or background once (e.g., Anthropic’s Prompt Caching) and send only new input each time to cut cost and latency.

24

This section on how dbt Labs transitioned from a purely PLG motion to layering on enterprise is a fascinating one. Two instructive passages that I have bolded (below from First Round Review's path to PMF series featuring dbt Labs) TLDR: GTM comprises your ICP, channel, and message. When you transition from PLG / bottom up to Enterprise / top down motion, naturally your ICP and channel also changes, but your messaging / proposition needs to change too, e.g., the enterprise may be multiple personas and are buying assurance as much as solution. --- "Handy found product-market fit organically for dbt as an open-source tool mostly used by data practitioners and developers. But a few years into running a commercial business, he realized he had to build a growth curve all over again with C-suite data leaders. “Even though we had an unbelievable amount of market pull, as we initially commercialized, it wasn’t easy for us to transform this open-source command line tool into a product that enterprises would pay a million dollars for,” says Handy. “When you have enough product-market fit, sometimes it allows you to get away with not being super tight on product marketing or sales motions. So around 2022, we went from this gigantic acceleration curve and overnight we realized, we have to sit at the adults’ table and figure things out real fast,” says Handy. After the PLG flame started to fizzle, Handy turned his attention to layering on a sales-led motion for the cloud platform. “We had to focus our efforts on telling cohesive stories to senior data leaders. We had to have a very clear, explainable answer to the question, ‘Why should I use the commercial product and not the open-source product? And it had to be digestible by someone with a C in their title,” he says. Handy’s answer: dbt Cloud can handle complex data for companies of every size. “The longer people used dbt, the more complex their code became,” he says. “It was a problem for the most sophisticated dbt users, who were often at the largest companies. So there was a real opportunity for us to step in and solve that for them with dbt Cloud.” To tell that story to enterprise customers, Handy relied on data, naturally. “At a user conference we presented a chart that showed the number of dbt projects that had a certain number of models in them — over 100, over 1,000, et cetera,” he says. “We watched that number climb and we knew as users ourselves, ‘Oh my God, trying to work in a dbt project with 5,000 models in it is challenging.’ So we started with that quantitative data point and asked folks in our community about their experiences with these very large, complex dbt projects, and validated that this was a pain in the ass without a cloud platform.”

23

4 Comments