Fanjing(Frank) Liu

Stanford Ph.D. researcher Cheng Chi discusses the development of the UMI gripper and the use of diffusion AI models for robotics. The post Stanford researcher discusses UMI gripper and diffusion AI models appeared first on The Robot Report. https://lnkd.in/gbTiRwh4

1

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

✨ Excited to share my first open-source contribution to the Hugging Face Transformers library! I recently added a feature that enables the interpolation of positional and padding embeddings for two models: VIT_MAE (Vision Transformer with Masked Encoder) and TVP (Time-Varying Parameter model). This enhancement allows fine-tuning these models without resizing high-resolution images. This experience was incredibly rewarding. It provided me with deep insights into model architecture and the intricacies of building production-level models. The support from the Hugging Face team was invaluable, and their guidance helped me immensely. Open source is crucial—without it, I wouldn't have had the opportunity to work hands-on with these cutting-edge models. Looking forward to making many more contributions! 🚀 VIT_MAE :- https://lnkd.in/gzfja-9f TVP :- https://lnkd.in/gn6ucr5w #OpenSource #MachineLearning #ArtificialIntelligence #AI #Transformers #DeepLearning #HuggingFace

41

11 Comments

The paper "Attention Is All You Need" presents the Transformer model, a novel architecture for sequence transduction based entirely on attention mechanisms. https://lnkd.in/gbe9HSzE Key IDEAS: - Traditional sequence models rely heavily on recurrent networks, limiting parallelization. - The Transformer architecture replaces recurrence with self-attention, improving training times. - Self-attention allows the model to draw connections across the entire input sequence. - The Transformer achieves a BLEU score of 28.4 on the English-to-German translation task. - Processing can occur in parallel, significantly speeding up computation. - Multi-head attention enables the model to focus on different parts of the input simultaneously. - This model can generalize to other tasks beyond translation, such as constituency parsing. - The design simplifies model architecture by removing the need for complex recurrent structures. - Positional encodings are introduced to maintain the order of sequences. - Attention layers improve the model's ability to learn long-range dependencies. - The paper emphasizes the efficiency gains over state-of-the-art models. - Transformers outperform existing methods with considerably less training cost. - The model can be fine-tuned for various language tasks easily. - Experiments showed better performance with increased attention heads up to a point. - Utilization of dropout and label smoothing during training improves performance. - The model architecture allows for easier interpretation of attention distributions. - Attention heads often learn specific tasks, such as syntactic and semantic relations. - The architecture allows adapting to larger input and output sequences efficiently. - Adequate regularization contributes to superior generalization in learning from data. - Ease of implementation on hardware accelerates research and application. Key INSIGHTS: - Replacing recurrence with attention simplifies architecture and allows for efficient model training. - Self-attention is crucial for capturing dependencies across various positions in input sequences. - The Transformer demonstrates that parallel processing can drastically reduce training time and increase efficiency. - The design showcases how innovations in architecture can redefine state-of-the-art results in machine translation. - By employing a consistent embedding across input and output, the model minimizes errors in translation tasks. - Regularization methods like dropout and label smoothing enhance the robustness of predictions. - Understanding various attention heads can provide insights into the inner workings and decision-making of the model.

7

Facebook just released a very different kind of LLMs. These are focused on code optimization and compiler optimization. This is super important because it can fundamentally change the way code is executed on any machine. Here's how it works right now: Programs written in many languages like C/C++ have intermediate steps before they are run on the computer. For example they are converted into Byte Code and then converted into machine code via LLVM (Low level virtual machine). But Facebook's new model can take in the byte code as input and convert it to into a more efficient representation so that when this new code is converted into machine code, it will be faster. "In theory", it could mean that all existing code might be made faster without any change by developers. Our games can have higher frame rates on the same GPUs. Photoshop might run faster on the same laptop. Our software might complete tasks faster without any changes at all. But that will probably take quite a long time. To be honest I don't expect these models to actually be used in production since unless they were 100% as good as compilers they will probably break the code. And are they 100% good? No, not even close. But the speed at which improvements are happening, they will probably reach that level eventually. And that's when we will see models like these everywhere. 

11

More on the power of combining DSLs (domain specific languages) and LLMs. Last time I made some general statements about DSLs, and I gave a trivial example. Now I'm going to describe how I used a DSL to accomplish something that otherwise would not have been possible with LLMs, (at the time). When ChatGPT3.5 was first released there was no JSON mode, no tools, and certainly no output schemas. The advice at the time was to prompt engineer GPT3.5 to output JSON, and then run a bunch of validators on the JSON to see if it was correct, try to fix up common issues that may pop up, and if all else fails, send the prompt through GPT again and hope it gave better output next time! Not sound engineering practices, and also way too much effort for an after work side project. I was working at HBO Max with Jiunwei at the time and I was inspired to make a Game of Thrones style drama web toy, where you can enter the name of your friends and family and everyone fights to the bloody death for control of a fantasy kingdom. After entering all the names, ChatGPT3.5 would create a 7 or 8 page long story. While the text was nice, I wanted some way to show who was fighting, and have animations and sound effects and the like. Typically in JSON land you'd do something like { participant1: name; participant2: name, action: string} but again, no JSON. I instead came up with a DSL that very nicely solved my problem, and it was also a lot more powerful than what I could have easily expressed in JSON. Examples are easier than explaining, so here we go! (taken from https://lnkd.in/gJ7bxKpf) ---- John and Sarah want to ensure Sarah's ascension to the throne. (John, Sarah 🤔 Sarah 👑) Tim gathered his army and attempted to conquer Arcadia. (Tim ⚔️ Arcadia) John knows that an marriage between Sarah and Tim will ensure Sarah's ascension to the throne. (John 🤔 Sarah 💍 Tim) ---- Did you catch on how it works? I made an Emoji based DSL! I needed a limited number of "verbs" to trigger animations and sound effects, but early ChatGPT3.5 releases didn't follow guidance very well. So instead of trying to prompt engineer a limited number of verbs I just told GPT to use any emoji it wanted to, creating a natural limit on what I had to parse. What made me fall in love with LLMs is when ChatGPT3.5 naturally extended the grammar itself! From multiple people acting together, to changing how groupings were done. Sometimes I'd suggest some particular grammatical form and ChatGPT would instead return results in some other, sensible but different, format. Rather than trying to correct the output I leaned into the LLM's creativity and reinforced its suggestions by adding the format it wanted to use as examples in the initial prompt. Working with the LLM actually resulted in better stories! And of course, feel free to check out old AI project, now a bit long in the tooth: https://lnkd.in/gWNfg9Tk

4

🚀 𝗥𝗲𝗰𝗲𝗻𝘁 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀 𝗶𝗻 𝗔𝗜 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 🚀 𝗚𝗿𝗼𝗸𝗙𝗼𝗿𝗺𝗲𝗿: Graph Fourier Kolmogorov-Arnold Transformers Introducing a novel Graph Transformer network that surpasses conventional self-attention mechanisms. 1. Tackles limitations of existing GTs in modeling complex node label patterns. 2. Incorporates learnable activation functions in the graph spectrum. 3. Demonstrates superior performance on multiple node and graph classification datasets. Paper: https://lnkd.in/gDKKk-2x 𝗦𝗧𝗔𝗥 𝗔𝗧𝗧𝗘𝗡𝗧𝗜𝗢𝗡: Efficient LLM Inference Over Long Sequences Recent advancements in Large Language Models (LLMs) now handle contexts extending to millions of tokens. 1. New attention mechanism designed to address computational challenges during inference while preserving accuracy. 2. Opens up new applications, including repository-level code analysis, multi-document summarization, and large corpus retrieval. Paper: https://lnkd.in/gtj29ghz 𝗚𝗹𝗶𝘁𝗰𝗵 𝗧𝗼𝗸𝗲𝗻𝘀 𝗶𝗻 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀: 1. Categorization Taxonomy and Effective Detection. 2. Addressing the issue of glitch tokens that lead to hallucinations and erratic behavior in LLMs. 3.Introduction of GlitchHunter, a novel clustering-based detection method. 4.Constructs a Token Embedding Graph (TEG) and uses clustering for efficient detection. 5.High precision and recall across multiple LLMs. Paper: https://lnkd.in/gTVr8jgK #MachineLearning #AIResearch #GraphTransformers #LLM #DeepLearning

4

𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝗡𝗲𝘄𝘀 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠𝘀) 𝗦𝗽𝗮𝗰𝗲! PyTorch has officially released TorchTune, a new tool designed to fine-tune Large Language Models (LLMs) with ease and efficiency. This groundbreaking tool promises to streamline the fine-tuning process, making it more accessible to researchers and practitioners alike. https://lnkd.in/eMf6dNZJ 𝗧𝗼𝗿𝗰𝗵𝘁𝘂𝗻𝗲 𝗢𝗳𝗳𝗲𝗿𝘀: ▪ 𝗣𝘆𝗧𝗼𝗿𝗰𝗵-𝗻𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀: Build on familiar ground with composable building blocks for popular LLMs. ️ ▪ 𝗠𝗲𝗺𝗼𝗿𝘆-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗿𝗲𝗰𝗶𝗽𝗲𝘀: Train on consumer-grade GPUs with 24GB VRAM – no beefy machines needed! ▪ 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗮𝗻𝗱 𝗲𝗮𝘀𝗲 𝗼𝗳 𝘂𝘀𝗲:  YAML configs and support for common dataset formats streamline your workflow. 𝗙𝗼𝗿 𝘁𝗵𝗼𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗱𝗶𝘃𝗶𝗻𝗴 𝗱𝗲𝗲𝗽𝗲𝗿, 𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Code: https://lnkd.in/gEgETe5t Blog: https://lnkd.in/gP3mctA6 Tutorials: https://lnkd.in/gvh2eEsU This is a fantastic opportunity to explore and innovate, I'd be very interested in hearing your thoughts. ▪ 𝗪𝗵𝗮𝘁’𝘀 𝘆𝗼𝘂𝗿 𝘁𝗮𝗸𝗲 𝗼𝗻 𝘁𝗵𝗶𝘀?  ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝗶𝗺𝗽𝗮𝗰𝘁𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗼𝗿 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵? ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝘄𝗶𝗹𝗹 𝗶𝗺𝗽𝗮𝗰𝘁 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗜 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁? #AI #PyTorch #TorchTune #LLMs #MachineLearning

24

1 Comment

Our paper on "Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models" was also accepted to Neurips! One of the coolest parts of the paper was finding an "en passant" feature in our SAE, which activates whenever an en passant capture is available. I consider this strong evidence that LLMs and SAEs are learning something real in their training process. ChessGPT is trained solely to predict the next character in PGN strings, and an SAE is trained solely to reconstruct ChessGPT activations. Despite the minimal amount of supervision in this process, we find very meaningful features in our SAE. More information here: https://lnkd.in/g6avcMTw

33

2 Comments

This week we are presenting our paper "LiNR: Model Based Neural Retrieval on GPUs at LinkedIn" accepted at CIKM 2024 (https://lnkd.in/gUrWqRcD). Please stop by and say hi to Aman Gupta, who will be there in person :) We discuss our experiences and challenges in creating scalable, differentiable search indexes using TensorFlow and PyTorch at production scale. In LiNR, both items and model weights are integrated into the model binary. Viewing index construction as a form of model training, we describe scaling our system for large indexes, incorporating full scans and efficient filtering. We believe LiNR represents one of the industry's first Live-updated model-based retrieval indexes at production scale. Talented co-authors include Fedor Borisyuk, Qingquan Song, Mingzhou Zhou, Ganesh Parameswaran, Madhulekha Arun, Siva P., Tugrul Bingol, Zhoutao Pei, Stanley(Kuang) Lee, Lu Z., Hugh Shao, Syed Ali Naqvi, Sen Zhou, Aman Gupta

171

4 Comments

Good teachers can convey the information in a simple way! I am working on my third Machine Learning project in CS598 PSL offered by UIUC. The model that I am trying to build have to learn whether the review of IMDB is positive or negative. I have to use BERT emdeddings, TFIDF, ... which are new terms for me. I jumped into Youtube and search for a video to have an idea about them. I found this Channel which I have always benefited from it, and I highly recommend it. This guy is really a good teacher. He can make you understand the awful math formulas, horrible terms in a fun and simple way :) https://lnkd.in/d2ZXg46j Enjoy! #ML #UIUC #Machine_learning #Deep_Learning #AI

1

Is Prompt Engineering Becoming Less Crucial with Larger Language Models? Recent research indicates that LLMs show reduced sensitivity to prompt variations, suggesting that prompt engineering may be less critical for straightforward queries. However, some practical experiences highlight the ongoing importance of tailored prompts noting that specific context enhances task comprehension and efficiency. Also incorporating roles and personas is beneficial when developing agents for specialized functions. These insights suggest that while larger LLMs may handle general prompts effectively, prompt engineering remains vital for complex or domain-specific applications. How have you approached prompt engineering in your work? #AI #MachineLearning #PromptEngineering #LLM Paper: https://lnkd.in/eutnf_uE

7

We are happy to announce the addition of knowledge distillation to torchtune, a PyTorch library for easily authoring, fine-tuning and experimenting with LLMs. Knowledge distillation is a technique for imparting knowledge from a larger teacher model to a smaller student model. This was a key step in Llama 3.2 pretraining. Check out how we can leverage knowledge distillation in post-training to distill Llama 3.1 8B to Llama 3.2 1B using torchtune: https://hubs.la/Q02YyqPP0

816

1 Comment

In this episode, we discuss NVLM: Open Frontier-Class Multimodal LLMs by Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping. The paper introduces NVLM 1.0, a set of advanced multimodal large language models that achieve state-of-the-art performance on vision-language tasks and improve upon their text-only capabilities. It outlines the benefits of a novel architecture that enhances training efficiency and reasoning abilities using a 1-D tile-tagging design, emphasizing the importance of dataset quality and task diversity over scale. NVLM 1.0's models excel in multimodal and text-only tasks through the integration of high-quality data, and the model weights are released with plans to open-source the training code.

3

1 Comment

Easily generate one billion embeddings in under a day... and with limited budget! How? With Ray Data from Anyscale and Pinecone! Cheng Su and Roy Miara present this can't-miss session during the #DataAISummit tomorrow. Catch it in-person or online.

43

2 Comments

Check out this guided series on introductory RL concepts using the environments from the #OpenAI Gymnasium Python package. By Jesse Xia

16

Many will say that if you write Python code correctly, you would never end up using nested loops, so in theory you would never end up hitting the speed wall highlighted in this video. The key word here is "never". 1. Not every single problem out there is vectorizable. You can try to vectorize as much as possible in your code, sometimes using forced and costly (readability, safety and efficiency costs) artifacts, but in the end of the day, even after flexing all the brain muscle and all the forced vectorization, you are going to raise a while flag, do the unthinkable and write a "while" loop in python, knowing that it will cripple the speed of your code, because there is no other way of doing it. 2. You are a purist an never, ever write nested loops in python. You have then to admit that python is not as versatile as C, or rust. In conclusion, python has to compromise at either speed, or versatility. The speed compromise is highlighted in the attached video. The versatility comprimise is more tricky, but if you ever did solve real (not homework) problems in python, you have encountered it.

22