Hi everyone! I'm Iva Vrtaric, AI Researcher/Developer from Belgrade, Serbia. I write courses for TowardsAI. As a side quest - I was invited to give a lecture about LLMs/MLOps in Data Science Club (Belgrade) next month. While I was researching about what would be interesting subject to cover, I stumbled upon the Flyte orchestration concept and loved the idea. I've read a few blogs and checked out the documentation - it's very well written. Anyways, I was thinking if you have any ideas, like fine-tuning LLM (with QLoRA) maybe? I'd like to use Flyte for the production phase or something similar -haven't decided yet. So, I'd like to find the best way to incorporate Flyte in the presentation, it should be 45min lecture for intermediate level. Thanks!

Hi @adamant-waiter-10076, it's great to hear that you're interested in presenting Flyte, and we're glad you liked our documentation and blog posts. We've an active repository for fine-tuning LLMs with Flyte: https://github.com/unionai-oss/llm-fine-tuning. Here are a couple of articles we've recently published: • https://www.union.ai/blog-post/fine-tune-llama-2-with-limited-resources • https://www.union.ai/blog-post/fine-tuning-insights-lessons-from-experimenting-with-redpajama-large-language-model-on-flyte-slack-data If you're specifically looking for QLoRa, here it is: https://github.com/unionai-oss/llm-fine-tuning/blob/main/fine_tuning/llm_fine_tuning_qlora.py. Let us know if you require any more resources for your presentation; happy to help!

Thank you, kindly for the resources, they seem really informative. I see you have done a bunch of experiments, immediately I followed you on Medium. I'll take some time to read and go through all this, and I'll let you know what I've come up with. I see that you've been using Union - as a cloud solution, I'm already paying for Google Cloud, so I'll reconfigure that environment. I suppose there aren't any differences in the script itself (regarding the Flyte part of the code). Best thing it would be to try to replicate some of your experiments, maybe with a different dataset. I'd also have to check if I have sufficient resources for running this, as I've seen you have been using single T4. Once again, thanks for sharing these valuable insights, I think this will work for now!

I see that you've been using Union - as a cloud solution, I'm already paying for Google Cloud, so I'll reconfigure that environment. I suppose there aren't any differences in the script itself (regarding the Flyte part of the code).

The script has to work on Flyte (you needn't make any changes.)

@broad-monitor-993 cc also @adamant-waiter-10076 you can use union and it gets deployed to your GCP. Ofcourse you can use Flyte too. My name is Ketan and I started the Flyte project am7 years ago. Thank you for the kind words

Nice to meet you Ketan! I'll try to make this work over weekend, hopefully:)