Step-by-Step Implementation of Generative Pre-Trained Transformers (GPT)

In this post, we explore the implementation of a small GPT model using Keras and Tensorflow.

I will not delve into too many details, as I am already planning to publish a post that explains how GPT was created and sheds some light on the theoretical frameworks behind it.

If you’d like to skip straight to the implementation, here’s a link to the python notebook https://github.com/mouadk/GPT/blob/main/GPT.ipynb.

Attention Mechanism