Francisco Vicente Carrasco

In his latest post from #TheBatch, Andrew Ng offers his argument on why #California's proposed law SB-1047 is likely to stifle #AI builders and impose unnecessary guardrails on #AI innovation. The law established a “hazardous capability” designation that makes *builders* of AI models liable if someone uses their models to do harm. Andrew argues that laws should not regulate the technology (the tool itself) but its application (how it is implemented).   It's a thoughtful read --> DeepLearning.AI #AIgovernance #AIregulation https://lnkd.in/eBD-9rja

3

Research Alert: Pyramidal Flow Matching for Efficient Video Generative Modeling Latest work titled "Pyramidal Flow Matching for Efficient Video Generative Modeling," by Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Key Challenge:  Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. Traditional approaches reduce complexity using cascaded architectures, but they sacrifice flexibility by optimizing each stage separately. Our Solution:  Authors introduced a unified pyramidal flow matching algorithm that reinterprets the denoising process into pyramid stages, where only the final stage operates at full resolution. This design allows more efficient video generative modeling while maintaining high quality. What makes it special? - Unified optimization with a single Diffusion Transformer (DiT) - Efficient autoregressive video generation using a temporal pyramid to compress full-resolution history - Capable of generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS The entire framework is optimized end-to-end, achieving impressive results within 20.7k A100 GPU training hours.

13

#KDD2024 is just around the corner! We are excited to share details about our upcoming tutorial on **Graph Neural Networks in Epidemic Modeling** (joint work with B. Aditya Prakash, Max Lau, and my talented students): 📅 Date: Sunday, August 25 🕙 Time: 10:00 AM - 1:00 PM (Barcelona Time) 📍 Location: Room 119 Epidemiological tasks often involve complex graph data, where GNNs have demonstrated remarkable predictive power and played a key role in monitoring and preventing disease spread. In this tutorial, we’ll provide a comprehensive overview of epidemiological graphs, ML methodologies in this domain, and discuss future directions. We’ll also be showcasing EpiLearn (https://lnkd.in/eFyqPrN7), our Python Library for Machine Learning in Epidemic Modeling, which has been accepted to the KDD #epidamik workshop on August 26. 🔗 For more details, visit our website: kdd2024epidemic.github.io Don’t miss this opportunity to learn how to harness machine learning and GNN techniques in epidemiology! #GNN #machinelearning

101

Excited to share our latest work: 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗜𝗺𝗮𝗴𝗲 𝗮𝘀 𝗔𝗰𝘁𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀, in collaboration with Mohit Shridhar* and Stephen James 🚀 Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. 𝗖𝗮𝗻 𝘄𝗲 𝘀𝗶𝗺𝗶𝗹𝗮𝗿𝗹𝘆 𝘂𝗻𝗹𝗼𝗰𝗸 𝗶𝗺𝗮𝗴𝗲-𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗺𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝘃𝗶𝘀𝘂𝗼𝗺𝗼𝘁𝗼𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝗹? We present 𝗚𝗲𝗻𝗶𝗺𝗮, a behavior-cloning agent that fine-tunes Stable Diffusion to “draw joint-actions” as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state- of-the-art visuomotor approaches, 𝗲𝘀𝗽𝗲𝗰𝗶𝗮𝗹𝗹𝘆 𝗶𝗻 𝗿𝗼𝗯𝘂𝘀𝘁𝗻𝗲𝘀𝘀 𝘁𝗼 𝘀𝗰𝗲𝗻𝗲 𝗽𝗲𝗿𝘁𝘂𝗿𝗯𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗻𝗱 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘇𝗶𝗻𝗴 𝘁𝗼 𝗻𝗼𝘃𝗲𝗹 𝗼𝗯𝗷𝗲𝗰𝘁𝘀. 🌐 Website: https://lnkd.in/eUTFW3dB 📄 Paper: https://lnkd.in/eQ95x_6k 💻 Code: https://lnkd.in/eh-KuR8B #ai #artificalintelligence #robotics #robotlearning #stablediffusion #diffusionmodel #imitationlearning #behaviorcloning

34

Pretty cool paper by Tamer Abuelsaad, Deepak Akkil Ravi Kokku and team on their implementation of Agent-E [1,2] - a multiagent workflow (built with #autogen) for accomplishing web based tasks (driving web interfaces). Paper: https://lnkd.in/gNk-nUAY The paper includes lots of nuggets that help us improve on our collective understanding of how to build interface agents[3], including: - Hierarchical agent architecture with separation of roles e.g, planning, web navigation, etc and the benefits it provides (task verification, error recovery etc) - DOM distillation to reduce noise in web pages. - Results (based on evaluation on the WebVoyaGer benchmark) indicate that a text only representation the use (DOM) does outperform existing multimodal approaches. Great example of where a multi-agent but less powerful setup can result in overall better performance. References [1] Paper on arxiv https://lnkd.in/gNk-nUAY [2] Paper github repo: https://lnkd.in/gaXyuTje [3] Interface Agents - Building Multi-Agent Applications that Act via Controlling Interfaces (Browsers, Apps) https://lnkd.in/eVdFqc_J #autogen #multiagent #multiagentsystems

59

1 Comment

NVIDIA unveils MM-Embed for finetuning multimodal retrieval models using multimodal LLMs. The idea of finetuning MLLMs with multimodal retrieval data is not new. In fact, many previous works have used CLIP to align multiple modalities. In this paper, the authors introduce dynamic hard-negative mining to improve the performance of the retrieval. #MachineLearning #GenAI #LLM #DataScience #RAG

68

2 Comments

[Paper Update] I am pleased to announce that our paper, "LoAS: Fully Temporal-Parallel Dataflow for Dual-Sparse Spiking Neural Networks," has been accepted at #MICRO2024! In this work, we delve into accelerating sparse matrix-sparse matrix (spMspM) operations for dual-sparse SNNs, where both the spike and weight matrices are sparse. Deploying dual-sparse SNNs on spMspM accelerators using existing dataflow and architecture designs is suboptimal due to repetitive access to expensive memory and sparsity-handling units across timesteps. To address this, we spatially unroll and parallelize all timesteps, creating a fully temporal parallel (FTP) dataflow. To enhance the efficiency of the FTP dataflow, we developed an FTP-friendly spike compression mechanism. Neurons that fire no output spikes across timesteps, termed silent neurons, are not processed. Additionally, we proposed an innovative SNN-tailored inner join circuit that is more power and area-efficient without sacrificing throughput. These innovations are integrated into LoAS, a low-latency inference accelerator for dual-sparse SNNs. 📰 Preprint (early version): https://lnkd.in/e3ivB54C 💻 Sample source codes: https://lnkd.in/e_UtXZAY Authors: Ruokai Yin Youngeun Kim Collaborator: Di Wu #SpikingNeuralNetworks #SparseMatrices #TemporalParallelDataflow #Neuromorphic #MICRO2024

155

3 Comments

How to fully leverage the zero-shot models for Monocular Visual Odometry is still interesting to explore 🤔 Here, we challenge this problem by borrowing the principle of self-supervised depth and ego-motion learning. Please check the arXiv for details. Link: https://lnkd.in/ge99rJ2N This is the work collaborated with Igor Vasiljevic,Vitor Guizilini, and @Kazuhiro Shintani! #deeplearning #computervision #robotics #3dvision

112

3 Comments

So I decided to take NVIDIA's powerful NIM and the meta/llama3-70b-instruct model for a spin, building a legal Q&A app. It's still under development, and that's where you come in! Let me know if you're interested in testing, or have any training documents (legal documents, contracts or articles) to help this app become robust. You can check the deployed app here: https://lnkd.in/gv5cawzM #legaltech #AI #LLMs #openinnovation

68

4 Comments

Mamba S5 - Selective Structured State Space Models! This blog took a while to complete! Foundation models often use the Transformer architecture, which faces inefficiencies with long sequences. Mamba AI improves this by enhancing state space models (SSMs) to dynamically respond to inputs, allowing selective information processing. This results in better efficiency and performance without traditional convolutions, significantly outperforming Transformers in speed and scalability across multiple modalities. #ArtificialIntelligence #MachineLearning #Math

1

Just share two experiments with ChatGPT4o. First experiment, I uploaded a paper TOC as a GIT image with 20 KB with a schematic on the left and SFG curves on the right. GPT4o recognized the right part as SFG spectroscopic data and pointed out the key points from those curves correctly. It also recognized the schematic on the left and all organic molecules and presented the right information. In the second experiment, I uploaded a CSV file with nonfouling and fouling peptides. Then, I requested GTP4o summarize what it learned from these peptides and predict another group of peptides' nonfouling properties. Interestingly, GTP4o got 82.9% correct. This is a very nice value, considering we have not played any prompt or RAG yet.

12

After collecting data approximately five times using ALPHA-α and training with "Bi-ACT," the system can currently operate autonomously as shown (though further verification is needed). Data collection is relatively simple, as each session only takes a few tens of seconds. For reference, the ALOHA paper mentions requiring about 50 iterations. During autonomous operation, input updates are optimized to run at about 100 Hz, while current control operates at 1000 Hz. "ALPHA-α and Bi-ACT Are All You Need: Importance of Position and Force Information/Control for Imitation Learning of Unimanual and Bimanual Robotic Manipulation with Low-Cost System" 🤖 project page: https://lnkd.in/g7cTpuHB 💡 X(more details): https://lnkd.in/g6cqtDu7

33

Our review on “#biofabrication of living actuators” is out now in Annual Reviews of Biomedical Engineering! We discuss tissue-scale muscle-powered robots that walk, swim, pump, grip, and compute! Learn more about what’s now and what’s next in #biohybrid #robotics 💪🏽🦾 https://lnkd.in/edcucEQM Disclaimer: since this review was submitted a year ago, it’s missing some of the latest developments in this rapidly emerging field, including some of our own recent work (https://lnkd.in/ed8GjPKz) 😅 Please consider this the beginning of a story and not the complete story!

518

16 Comments

Nvidia Star Attention: Efficient LLM Inference Over Long Sequences up to 11x times faster inference times with 95%/100% accuracy ... Efficiently processing long sequences in large language models (LLMs) presents significant challenges due to the inherent complexities of traditional self-attention mechanisms. In the research paper titled "Star Attention: Efficient LLM Inference Over Long Sequences," a novel algorithm is proposed to address these shortcomings, making LLMs faster and more effective without compromising accuracy. 👉 Overview of Star Attention Star Attention introduces a two-phase approach to long-context inference, fundamentally transforming how LLMs manage extensive data inputs: 1. "Context Encoding": The context is divided into blocks and processed across multiple hosts. Each host operates independently, reducing the demand for inter-host communication and allowing for linear scaling with respect to context length. 2. "Query Encoding & Token Generation": A global attention mechanism is employed to efficiently process query tokens against cached tokens from the context. This phase ensures that relevant information is quickly accessible, enhancing responsiveness. 👉 Efficiency Gains The results speak for themselves: Star Attention can achieve up to 11 times faster inference times, while still maintaining an impressive 95-100% accuracy compared to traditional attention methods. This efficiency allows organizations to deploy LLMs more effectively, especially in applications that require processing extensive datasets, such as: - Multi-document summarization - Large-scale code analysis - Repository-level information retrieval 👉 Key Innovations - "Blockwise Local Attention": By utilizing blockwise-local attention, Star Attention minimizes the computational load, enabling significant speed enhancements while retaining the ability to focus on relevant context areas. - "Seamless Integration": This algorithm can be easily integrated with existing Transformer architectures without the need for extensive model retraining, making it a practical solution for many organizations. 👉 Real-World Applications With its ability to handle contexts up to "1 million tokens", Star Attention opens new avenues for LLM applications across various industries. For instance, businesses can leverage this technology for advanced natural language processing tasks, enhancing their capabilities in customer service, data analysis, and content generation. 👉 Conclusion As we continue to push the boundaries of AI technology, advancements like Star Attention signify a vital step in optimizing the performance of large language models. I encourage colleagues and industry professionals to delve into this research to explore how such innovations can enhance our work in AI and machine learning.

55

1 Comment

Excited to share that our extended abstract titled "Incorporating Foundation Model Priors in Modeling Novel Objects for Robot Instruction Following in Unstructured Environments" has been accepted for presentation at the Workshop on 3D Visual Representations for Robot Manipulation at ICRA 2024! Visit our website for more details: https://lnkd.in/gcmfMJgu Workshop: https://lnkd.in/gA38HrpH 📝 Authors: Moksh Malhotra, Aman Tambi, Sandeep Zachariah, P.V.M. Rao, Rohan Paul 🙏 Special thanks to our project advisors Prof. Rohan Paul and Prof. P.V.M. Rao for their invaluable guidance and support throughout this research journey! In our paper, we tackle the challenge of acquiring object models when a robotic manipulator is sent in unstructured and unknown environments. We proposes a novel approach to utilizes foundation models with RGB-Depth images to acquire high-fidelity semantic 3D model of objects, required for robotic manipulation. We demonstrate our pipeline in simulation as well as real-world scenarios. Looking forward to discussing our research and exchanging ideas with fellow researchers and professionals at the conference. Stay tuned for more updates! #ICRA2024 #Robotics #Research #AI #3DVisualization #RobotManipulation

11

2 Comments

A question to the community: Do we need to include diffusion inside the model's architecture as done by NVIDIA's EDM paper for UNET, or just tinkering with the input data should be okay ? What I mean is adding noise in time steps and teaching the model to map from noisy data to the target in steps is okay ? I want to add the diffusion as in NVIDIA's EDM work to all models from SRCNN to Swin Transformer to IPG to SRGAN ... For example in the EDM networks.py code at https://lnkd.in/gUJ2azPk for DhariwalUNET:   def forward(self, x, noise_labels, class_labels, augment_labels=None):     # Mapping.     emb = self.map_noise(noise_labels)     if self.map_augment is not None and augment_labels is not None:       emb = emb + self.map_augment(augment_labels)     emb = silu(self.map_layer0(emb))     emb = self.map_layer1(emb)     if self.map_label is not None:       tmp = class_labels       if self.training and self.label_dropout:         tmp = tmp * (torch.rand([x.shape[0], 1], device=x.device) >= self.label_dropout).to(tmp.dtype)       emb = emb + self.map_label(tmp)     emb = silu(emb)     # Encoder.     skips = []     for block in self.enc.values():       x = block(x, emb) if isinstance(block, UNetBlock) else block(x)       skips.append(x)     # Decoder.     for block in self.dec.values():       if x.shape[1] != block.in_channels:         x = torch.cat([x, skips.pop()], dim=1)       x = block(x, emb)     x = self.out_conv(silu(self.out_norm(x)))     return x the map_noise is used even in the forward function. I wanted to use diffusion for models like Swin Transformer and IPG and changing within the model architecture seems challenging. Background: Recently I have been dabbling a lot with the diffusion models, and now am interested to add diffusion to all image to image models from SRCNN to Swin Transformers. The inspiration has spiked from Peter Manshausen's work on diffusion based data assimilation. Also, there is a very nice paper from NVIDIA titled "Elucidating the Design Space of Diffusion-Based Generative Models" by Tero Karras and others. It provides clear design choices for diffusion models.

52

I wrote two blog posts about SN, Leon Bottou and Yann LeCun's 1988 𝘚𝘪𝘮𝘶𝘭𝘢𝘵𝘦𝘶𝘳 𝘥𝘦 𝘕𝘦𝘶𝘳𝘰𝘯𝘦𝘴. One is an English translation of the original paper, for which I've reproduced the figures. The other is a tutorial on how to run their code on Apple silicon. https://lnkd.in/dMFxMZeT

77

2 Comments

Again, this is the dumbest AI will ever be. It only gets smarter from here. What are you doing today to partner with AI? How will you use these new superpowers?

7

In this article titled "Information Channels of Deep Neural Networks" https://lnkd.in/gJEmWi3y we analyze Deep Neural Networks (DNN)  which consist of convolutional layers. Each convolutional kernel along with the corresponding activation function is viewed as an Information Channel (IC). We use the spatiotemporal properties of the images to compute the Mutual Information (MI) between the channel input and output and demonstrate that the DNNs generalize and learn by reducing the Shannon capacity of these ICs while maximizing the accuracy of prediction. The Gray level Co-occurrence Matrix (GLCM) technique used in our analysis encodes the spatial and temporal correlation between pixels of the image in the probability matrices that are used to compute the mutual information. This approach improves the accuracy of the mutual information estimate between images. 

9