Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix vocab_size in PositionalEmbedding in Transformer notebook #1266

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Fix vocab_size in PositionalEmbedding in Transformer notebook #1266

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8f6ddd6
Fix vocab_size in PositionalEmbedding in Transformer notebook
8bitmp3 Apr 9, 2024
fca8889
Update transformer.ipynb
8bitmp3 Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions docs/tutorials/transformer.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
"source": [
"This tutorial demonstrates how to create and train a [sequence-to-sequence](https://developers.google.com/machine-learning/glossary#sequence-to-sequence-task) [Transformer](https://developers.google.com/machine-learning/glossary#Transformer) model to translate [Portuguese into English](https://www.tensorflow.org/datasets/catalog/ted_hrlr_translate#ted_hrlr_translatept_to_en). The Transformer was originally proposed in [\"Attention is all you need\"](https://arxiv.org/abs/1706.03762) by Vaswani et al. (2017).\n",
"\n",
"Transformers are deep neural networks that replace CNNs and RNNs with [self-attention](https://developers.google.com/machine-learning/glossary#self-attention). Self attention allows Transformers to easily transmit information across the input sequences.\n",
"Transformers are deep neural networks that replace CNNs and RNNs with [self-attention](https://developers.google.com/machine-learning/glossary#self-attention). Self-attention allows Transformers to easily transmit information across the input sequences.\n",
"\n",
"As explained in the [Google AI Blog post](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html):\n",
"\n",
Expand Down Expand Up @@ -138,7 +138,7 @@
"To get the most out of this tutorial, it helps if you know about [the basics of text generation](./text_generation.ipynb) and attention mechanisms. \n",
"\n",
"A Transformer is a sequence-to-sequence encoder-decoder model similar to the model in the [NMT with attention tutorial](https://www.tensorflow.org/text/tutorials/nmt_with_attention).\n",
"A single-layer Transformer takes a little more code to write, but is almost identical to that encoder-decoder RNN model. The only difference is that the RNN layers are replaced with self attention layers.\n",
"A single-layer Transformer takes a little more code to write, but is almost identical to that encoder-decoder RNN model. The only difference is that the RNN layers are replaced with self-attention layers.\n",
"This tutorial builds a 4-layer Transformer which is larger and more powerful, but not fundamentally more complex."
]
},
Expand Down Expand Up @@ -186,8 +186,8 @@
"## Why Transformers are significant\n",
"\n",
"- Transformers excel at modeling sequential data, such as natural language.\n",
"- Unlike the [recurrent neural networks (RNNs)](./text_generation.ipynb), Transformers are parallelizable. This makes them efficient on hardware like GPUs and TPUs. The main reasons is that Transformers replaced recurrence with attention, and computations can happen simultaneously. Layer outputs can be computed in parallel, instead of a series like an RNN.\n",
"- Unlike [RNNs](https://www.tensorflow.org/guide/keras/rnn) (like [seq2seq, 2014](https://arxiv.org/abs/1409.3215)) or [convolutional neural networks (CNNs)](https://www.tensorflow.org/tutorials/images/cnn) (for example, [ByteNet](https://arxiv.org/abs/1610.10099)), Transformers are able to capture distant or long-range contexts and dependencies in the data between distant positions in the input or output sequences. Thus, longer connections can be learned. Attention allows each location to have access to the entire input at each layer, while in RNNs and CNNs, the information needs to pass through many processing steps to move a long distance, which makes it harder to learn.\n",
"- Unlike [recurrent neural networks (RNNs)](./text_generation.ipynb), Transformers are parallelizable. This makes them efficient on hardware like GPUs and TPUs. The main reasons is that Transformers replaced recurrence with attention, and computations can happen simultaneously. Layer outputs can be computed in parallel, instead of a series like an RNN.\n",
"- Unlike [RNNs](https://www.tensorflow.org/guide/keras/rnn) (such as [seq2seq, 2014](https://arxiv.org/abs/1409.3215)) or [convolutional neural networks (CNNs)](https://www.tensorflow.org/tutorials/images/cnn) (for example, [ByteNet](https://arxiv.org/abs/1610.10099)), Transformers are able to capture distant or long-range contexts and dependencies in the data between distant positions in the input or output sequences. Thus, longer connections can be learned. Attention allows each location to have access to the entire input at each layer, while in RNNs and CNNs, the information needs to pass through many processing steps to move a long distance, which makes it harder to learn.\n",
"- Transformers make no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects (for example, [StarCraft units](https://www.deepmind.com/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii)).\n",
"\n",
"\u003cimg src=\"https://www.tensorflow.org/images/tutorials/transformer/encoder_self_attention_distribution.png\" width=\"800\" alt=\"Encoder self-attention distribution for the word it from the 5th to the 6th layer of a Transformer trained on English-to-French translation\"\u003e\n",
Expand Down Expand Up @@ -1007,8 +1007,8 @@
},
"outputs": [],
"source": [
"embed_pt = PositionalEmbedding(vocab_size=tokenizers.pt.get_vocab_size(), d_model=512)\n",
"embed_en = PositionalEmbedding(vocab_size=tokenizers.en.get_vocab_size(), d_model=512)\n",
"embed_pt = PositionalEmbedding(vocab_size=tokenizers.pt.get_vocab_size().numpy(), d_model=512)\n",
"embed_en = PositionalEmbedding(vocab_size=tokenizers.en.get_vocab_size().numpy(), d_model=512)\n",
"\n",
"pt_emb = embed_pt(pt)\n",
"en_emb = embed_en(en)"
Expand Down Expand Up @@ -1340,7 +1340,7 @@
"id": "J6qrQxSpv34R"
},
"source": [
"### The global self attention layer"
"### The global self-attention layer"
]
},
{
Expand All @@ -1360,7 +1360,7 @@
"source": [
"\u003ctable\u003e\n",
"\u003ctr\u003e\n",
" \u003cth colspan=1\u003eThe global self attention layer\u003c/th\u003e\n",
" \u003cth colspan=1\u003eThe global self-attention layer\u003c/th\u003e\n",
"\u003ctr\u003e\n",
"\u003ctr\u003e\n",
" \u003ctd\u003e\n",
Expand All @@ -1378,7 +1378,7 @@
"source": [
"Since the context sequence is fixed while the translation is being generated, information is allowed to flow in both directions. \n",
"\n",
"Before Transformers and self attention, models commonly used RNNs or CNNs to do this task:"
"Before Transformers and self-attention, models commonly used RNNs or CNNs to do this task:"
]
},
{
Expand Down Expand Up @@ -1415,7 +1415,7 @@
"- The RNN allows information to flow all the way across the sequence, but it passes through many processing steps to get there (limiting gradient flow). These RNN steps have to be run sequentially and so the RNN is less able to take advantage of modern parallel devices.\n",
"- In the CNN each location can be processed in parallel, but it only provides a limited receptive field. The receptive field only grows linearly with the number of CNN layers, You need to stack a number of Convolution layers to transmit information across the sequence ([Wavenet](https://arxiv.org/abs/1609.03499) reduces this problem by using dilated convolutions).\n",
"\n",
"The global self attention layer on the other hand lets every sequence element directly access every other sequence element, with only a few operations, and all the outputs can be computed in parallel. \n",
"The global self-attention layer on the other hand lets every sequence element directly access every other sequence element, with only a few operations, and all the outputs can be computed in parallel. \n",
"\n",
"To implement this layer you just need to pass the target sequence, `x`, as both the `query`, and `value` arguments to the `mha` layer: "
]
Expand Down Expand Up @@ -1470,7 +1470,7 @@
"source": [
"\u003ctable\u003e\n",
"\u003ctr\u003e\n",
" \u003cth colspan=1\u003eThe global self attention layer\u003c/th\u003e\n",
" \u003cth colspan=1\u003eThe global self-attention layer\u003c/th\u003e\n",
"\u003ctr\u003e\n",
"\u003ctr\u003e\n",
" \u003ctd\u003e\n",
Expand Down Expand Up @@ -1499,7 +1499,7 @@
"source": [
"\u003ctable\u003e\n",
"\u003ctr\u003e\n",
" \u003cth colspan=1\u003eThe global self attention layer\u003c/th\u003e\n",
" \u003cth colspan=1\u003eThe global self-attention layer\u003c/th\u003e\n",
"\u003ctr\u003e\n",
"\u003ctr\u003e\n",
" \u003ctd\u003e\n",
Expand All @@ -1515,7 +1515,7 @@
"id": "Yq4NtLymD99-"
},
"source": [
"### The causal self attention layer"
"### The causal self-attention layer"
]
},
{
Expand All @@ -1524,7 +1524,7 @@
"id": "VufkgF7caLze"
},
"source": [
"This layer does a similar job as the global self attention layer, for the output sequence:"
"This layer does a similar job as the global self-attention layer, for the output sequence:"
]
},
{
Expand All @@ -1535,7 +1535,7 @@
"source": [
"\u003ctable\u003e\n",
"\u003ctr\u003e\n",
" \u003cth colspan=1\u003eThe causal self attention layer\u003c/th\u003e\n",
" \u003cth colspan=1\u003eThe causal self-attention layer\u003c/th\u003e\n",
"\u003ctr\u003e\n",
"\u003ctr\u003e\n",
" \u003ctd\u003e\n",
Expand All @@ -1551,7 +1551,7 @@
"id": "0AtF1HYFEOYf"
},
"source": [
"This needs to be handled differently from the encoder's global self attention layer. \n",
"This needs to be handled differently from the encoder's global self-attention layer. \n",
"\n",
"Like the [text generation tutorial](https://www.tensorflow.org/text/tutorials/text_generation), and the [NMT with attention](https://www.tensorflow.org/text/tutorials/nmt_with_attention) tutorial, Transformers are an \"autoregressive\" model: They generate the text one token at a time and feed that output back to the input. To make this _efficient_, these models ensure that the output for each sequence element only depends on the previous sequence elements; the models are \"causal\"."
]
Expand Down Expand Up @@ -1608,7 +1608,7 @@
"id": "WLYfIa8eiYgk"
},
"source": [
"To build a causal self attention layer, you need to use an appropriate mask when computing the attention scores and summing the attention `value`s.\n",
"To build a causal self-attention layer, you need to use an appropriate mask when computing the attention scores and summing the attention `value`s.\n",
"\n",
"This is taken care of automatically if you pass `use_causal_mask = True` to the `MultiHeadAttention` layer when you call it:"
]
Expand Down Expand Up @@ -1650,7 +1650,7 @@
"source": [
"\u003ctable\u003e\n",
"\u003ctr\u003e\n",
" \u003cth colspan=1\u003eThe causal self attention layer\u003c/th\u003e\n",
" \u003cth colspan=1\u003eThe causal self-attention layer\u003c/th\u003e\n",
"\u003ctr\u003e\n",
"\u003ctr\u003e\n",
" \u003ctd\u003e\n",
Expand Down Expand Up @@ -1679,7 +1679,7 @@
"source": [
"\u003ctable\u003e\n",
"\u003c/tr\u003e\n",
" \u003cth colspan=1\u003eThe causal self attention layer\u003c/th\u003e\n",
" \u003cth colspan=1\u003eThe causal self-attention layer\u003c/th\u003e\n",
"\u003ctr\u003e\n",
"\u003ctr\u003e\n",
" \u003ctd\u003e\n",
Expand Down