{"id":5152,"date":"2024-04-12T05:18:36","date_gmt":"2024-04-12T05:18:36","guid":{"rendered":"https:\/\/towardsdatascience.com\/deep-dive-into-transformers-by-hand-%ef%b8%8e-68b8be4bd813\/"},"modified":"2025-01-08T19:13:51","modified_gmt":"2025-01-08T19:13:51","slug":"deep-dive-into-transformers-by-hand-%ef%b8%8e-68b8be4bd813","status":"publish","type":"post","link":"https:\/\/towardsdatascience.com\/deep-dive-into-transformers-by-hand-%ef%b8%8e-68b8be4bd813\/","title":{"rendered":"Deep Dive into Transformers by Hand \u270d\ufe0e"},"content":{"rendered":"

There has been a new development in our neighborhood.<\/p>\n

A ‘Robo-Truck,’ as my son likes to call it, has made its new home on our street.<\/p>\n

It is a Tesla Cyber Truck and I have tried to explain that name to my son many times but he insists on calling it Robo-Truck. Now every time I look at Robo-Truck and hear that name, it reminds me of the movie Transformers where robots could transform to and from cars.<\/p>\n

And isn’t it strange that Transformers as we know them today could very well be on their way to powering these Robo-Trucks? It’s almost a full circle moment. But where am I going with all these?<\/p>\n

Well, I am heading to the destination – Transformers. Not the robot car ones but the neural network ones. And you are invited!<\/p>\n

\"Image
Image by author (Our Transformer – ‘Robtimus Prime’. Colors as mandated by my artist son.)<\/figcaption><\/figure>\n

What are Transformers?<\/h3>\n

Transformers are essentially neural networks. Neural networks that specialize in learning context from the data.<\/p>\n

But what makes them special is the presence of mechanisms that eliminate the need for labeled datasets<\/strong> and convolution or recurrence<\/strong> in the network.<\/p>\n

What are these special mechanisms?<\/h3>\n

There are many. But the two mechanisms that are truly the force behind the transformers are attention weighting and feed-forward networks (FFN).<\/p>\n

What is attention-weighting?<\/h3>\n

Attention-weighting is a technique by which the model learns which part of the incoming sequence needs to be focused on. Think of it as the ‘Eye of Sauron’ scanning everything at all times and throwing light on the parts that are relevant.<\/p>\n

Fun-fact: Apparently, the researchers had almost named the Transformer model ‘Attention-Net’, given Attention is such a crucial part of it.<\/p><\/blockquote>\n

What is FFN?<\/strong><\/h3>\n

In the context of transformers, FFN is essentially a regular multilayer perceptron acting on a batch of independent data vectors. Combined with attention, it produces the correct ‘position-dimension’ combination.<\/p>\n

How do Attention and FFN work?<\/strong><\/h2>\n

So, without further ado, let’s dive into how attention-weighting<\/strong> and FFN<\/strong> make transformers so powerful.<\/p>\n

This discussion is based on Prof. Tom Yeh’s wonderful AI by Hand Series on Transformers<\/a> . (All the images below, unless otherwise noted, are by Prof. Tom Yeh from the above-mentioned LinkedIn posts, which I have edited with his permission.)<\/p>\n

So here we go:<\/p>\n

The key ideas here : attention weighting and feed-forward network (FFN)<\/strong>.<\/p>\n

Keeping those in mind, suppose we are given:<\/p>\n