Ignacio G L.

🚀 Self Attention vs. Causal Attention (Masked Attention): An Intuitive Explanation When i talk to many people they dont understand the foundational difference between different attention mechanisms which is the driving power of all generativeAI models and the below is my simple contribution to the community. In the world of Transformers and Generative AI models, two attention mechanisms dominate—Self Attention and Causal Attention. Let's simplify these with a clear example! 🔍 🤔 Self Attention: Imagine you're reading a sentence, "Learning Generative AI is extremely interesting." While understanding the word "extremely", you simultaneously glance at every other word (including the ones after) to grasp context fully. In self-attention, every word "attends" to every other word in the entire sentence, providing rich context from both sides. 🌐✨ ⏳ Causal Attention (Masked Attention): Now, imagine you're writing or predicting the next word of the same sentence, but at the word "extremely", you haven't seen the word "interesting" yet. Here, you can only rely on previously seen words—no future peeking! Causal attention restricts each word from seeing any future tokens, which aligns closely with how language models predict words sequentially. 🔒➡️ In Short: Self Attention: All words see each other—full context. 🌍 Causal Attention: Words only see previous words—sequential context. 🔗 Hope this clears things up! 💡😊 Note: numbers in the diagram are just to understand the concept. #AI #GenAI #LLM #AIAgents #RAG #AgenticWorkflows

45

2 Comments

lya Sutskever has confirmed that LLMs are not scaling with massive gains anymore Enough if Compute continues to scale, but data is not keeping up. New or synthetic data doesnt significantly improved performance. In next wave models can actually get smaller and do one job right. 1)Agentic AI system is the future 2)LLMs can be replaced with SLMs to create optimal systems 3)LLM/SLM not being absolutely deterministic in outputs wont matter as long as system design is robust.

46

3 Comments

Apple just published some great research on the limited reasoning capabilities of LLMs. Some key takeaways: - The paper reveals a sharp drop in performance when problems exceed a certain complexity threshold — showing that even so-called reasoning models fail on tougher tasks - What looks like reasoning is often just pattern-matching learned from training data—not true logical problem-solving. Some researchers like Gary Marcus and Subbarao Kambhampati have been claiming this for a long time. - So LLMs can appear smart, until you give them problems they truly haven't seen before. Then they collapse. Link below to the Apple research. My take: This research feels pretty spot-on. I think we are pretty far from AGI and AI that invents stuff like crazy. But for AI to be quite useful, or even very useful, we don't need AGI. We are at "useful" today and the continuously improving models and falling price of intelligence will enable lots of new applications. https://lnkd.in/gmPzUtfW

31

4 Comments