Coding Transformers, the framework that ChatGPT is built on, from Scratch
Thursday, 16. May 2024 | 09:00 - 17:00
Description
This workshop explored how to code, train, and use transformers from scratch in Python with PyTorch and Lightning. Participants then leveraged pre-made transformer classes to build, train, and use a Large Language Model (LLM) Transformer capable of answering basic questions.
Topics
Step-by-step overviews of how transformers work
An introduction to Lightning Studios for easy access cloud computing
Coding a transformer from scratch, including word embedding, positional encoding, self-attention, masked self-attention, and encoder-decoder attention.
Training our transformer with a simple dataset to translate English to Spanish.
Using our transformer to translate user input.
Downloading and fine-tuning an open source large language model.
Using the large language model to answer basic questions.
Requirements
Basic coding skills in Python.
Target audience
Anyone who wants to learn how artificial intelligence (AI) like ChatGPT works. No previous experience with AI is necessary.