Douglas Eck

🚀 I’m thrilled to share that our paper “STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data” has been accepted at NeurIPS 2025, San Diego, 2-7 Dec 2025! 🎉 Our contribution: https://lnkd.in/eHcQb6a6 This work is the product of a collaboration between University of Glasgow School of Computing Science (Knowledge & Data Engineering Systems #KDES) and the BMW Group. Huge thanks to my co-authors & collaborators: Maximilian Forstenhäusler, Daniel Külzer, Shameem A Puthiya Parambath, and Dr. Natascha Weber (Harth). 🔍 What problem we tackled Real-world sequential (e.g. spatiotemporal / time series) data often poses serious challenges: --Non-stationarity: signal properties changing over time. --Irregular sampling (missing data / varying intervals). --Need for latent representations that are task-informed, i.e. tuned to downstream classification / anomaly detection tasks, not just generic embeddings. 🎯 STaRFormer Contribution We propose STaRFormer, a framework that combines two main ideas: [*] Dynamic Attention-Based Regional Masking by helping the model learn robustness to irregularities (missing/unreliable/malformed parts) and focus on informative regions. [*] Semi-Supervised Contrastive Learning, where STaRFormer instead of treating labeled and unlabeled data separately, it blends them through a task-aware contrastive learning objective. 💡 Why this matters --Better representations: STaRFormer yields embeddings that are more useful for downstream tasks. --General framework: STaRFormer handles regular, non-stationary, irregular-sampling, sequence-level and element-wise tasks—so less need for highly tailored models per scenario. NeurIPS 2025 Record: https://lnkd.in/eB6YzG26 #NeurIPS2025 #MachineLearning #RepresentationLearning #AIResearch #DeepLearning #TimeSeries #KDES

111

15 Comments

Excellent piece by HuggingFace CEO Clément Delangue on the necessity of a robust open source / open weight / open research approach to AI to keep the pace of progress. Closed/proprietary strategies slow down overall progress. I have made that point many times over the last dozen years, and became more vocal about it over the last 3 years, as prominent American AI companies started clamming up. https://lnkd.in/euTH8qwk

2,727

135 Comments

Yesterday, at #GoogleIO, we announced Gemini 2.5 Pro #DeepThink, with state-of-the-art performances across math, coding, image understand and much more! Very proud of our research team for the relentless effort over the past 6 months towards advanced deep reasoning with parallel thinking! It's been a wild run over the last 2 weeks to land Deep Thinking just in time for I/O! Heroic efforts across many teams at Google DeepMind from research to production for a mission that felt almost impossible last month! The attached video showed a cool example of how DeepThink can nail a complex solution for an algebra problem at USAMO 2025, something we haven’t seen before in previous models! DeepThink achieved a new state-of-the-art performance of 49.4% (more than double the previous performance) on USAMO 2025, an extremely challenging Math benchmark for AI that requires detailed proof writing. See more details at https://lnkd.in/gEJCNHtJ.

123

1 Comment

I wrote a post recently (linked in comments below) on why agentic systems require structures for semantic-associative memory, and why LLMs lack the architecture to do anything but procedural memory. Therefore: LLMs aren't sufficient to build agentic systems. Someone replied with a very thoughtful question about Retrieval-Augmented Generation (RAG), a method that enhances language models by retrieving relevant external documents at query time and using them as additional context for generating responses. The question was whether RAG didn't actually provide the solutions to the inadequacies I outlined in my original post. The answer is: no. RAG is great for fact recall (well...let's say it's solidily good). But it falls short when it comes to forming and using *semantic categories* — the kind of abstract, relational knowledge humans build up over time. Here’s why: 🚩 No integration. RAG fetches documents, but they don’t get internalized. Humans integrate knowledge into structured representations that guide future learning and generalization. 🚩 No schema formation. Humans *compress* experience into categories (e.g., "tools", "emotions", "rules"). RAG retrieves specific items but doesn’t abstract across them to form stable, general concepts. 🚩 No consolidation. There’s no memory strengthening. Humans reinforce category boundaries over time — through repetition, use, and relevance. RAG pipelines don’t naturally evolve or stabilize concepts with repeated exposure. 🚩 Flat associations. Retrieval is based on surface similarity (e.g., vector distance). Human categories are rich with *associative structure* — linking things by function, causality, affordance, etc. 🚩 No cross-episode learning. RAG doesn’t accumulate concepts across interactions. Each retrieval is a fresh lookup. Humans merge partial exposures to build coherent, resilient categories. In short: RAG helps with lookup, not learning. It’s a patch for recall, not a path to structured, practical decision-making. Until we give agents mechanisms for abstraction, consolidation, and associative generalization — they won’t have anything close to human-like semantic memory. In my opinion, it's more realistic to create those mechanism separately and let them interact with LLMs than it is to think that we can somehow train or augment LLMs to the point that they'll be able to do something they were not designed to do.

67

15 Comments

New paper! Available benchmarks for long-context reasoning in Large Language Models do not distinguish the effect of task complexity, presence of irrelevant information, and task length. @daniel Kaiser invented CogniLoad, a synthetic benchmark grounded in Cognitive Load Theory. CogniLoad is used to test 22 reasoning LLMs, and reveals distinct performance sensitivities, identifying task length as a dominant factor, varied tolerances to intrinsic complexity and a never seen before U-shaped response to presence of distracting information. Take a look! https://lnkd.in/dVwAST9B Ali Ramezani-Kebrya, Benjamin Ricaud, Integreat - Norwegian Centre for Knowledge-driven Machine Learning, University of Oslo, UiT- The Arctic University of Norway

59

1 Comment