It's stunning to see the progress of open source foundation models in the past year - you've probably heard a lot about that. But I would like to point out that the progress in open source AI software infrastructure also made incredible progress. Just to name a (very) few: vLLM for LLM serving, MLC-LLM (off-shoot of Apache TVM) for portable deployment across a wide range of cloud and edge hardware, LoRAX and Punica for multiplexed fined-tuned model inference, Deepspeed for training, etc. There is a great symbiosis happening in open source model and infrastructure innovation, and I suspect we will see even more in 2024. Please name some of your favorite open source AI infra projects.
Couldn’t agree more! Thanks for the LoRAX shoutout, and congrats on MLC-LLM and Punica.
QLoRA!
SkyPilot
Co-founder/CEO at TitanML | Secure Enterprise GenAI | Forbes 30 Under 30
10moWe love the Triton language - we use it extensively in Titan Takeoff Inference Server. https://github.com/openai/triton It allows us to be super flexible when it comes to adding new models & support non-nvidia hardware