Comments (1)
Hi @labdmitriy ! Thanks again for finding these! I just double-checked the first one, and it seems like it was something I previously already fixed:
![Screenshot 2024-08-04 at 1 46 35 PM](https://private-user-images.githubusercontent.com/5618407/354923223-db3daea5-4b9d-4446-9541-a73cbbae685b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxNDk4MTMsIm5iZiI6MTcyNDE0OTUxMywicGF0aCI6Ii81NjE4NDA3LzM1NDkyMzIyMy1kYjNkYWVhNS00YjlkLTQ0NDYtOTU0MS1hNzNjYmJhZTY4NWIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjBUMTAyNTEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTAzMjg0ZjRhZTJmYmFhYjA0OWM2OGVmMzdjY2I2MmIyZTljYzM5YjUyOGY1YTg3M2QyOWE2ZGNjODFkNzE5MiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.3fSepRSTFJWQpKGC17_KVqDHGNGrkoFGgauy2_Q7LFQ)
However the other ones are indeed still incorrect in the latest drafts. Regarding 2), this should definitely be "six".
And regarding 3)
As we have seen in the previous section, based on the DummyLayerNorm
placeholder, in GPT-2 and modern transformer architectures, layer normalization is typically
applied before and after the multi-head attention module and before the final output layer.
I also agree. A better wording could be:
In GPT-2 and modern transformer architectures, layer normalization is typically applied before and after the multi-head attention module, and, as we have seen with the DummyLayerNorm placeholder, before the final output layer.
I just emailed the layout team to get these fixed!
Thanks!
from llms-from-scratch.
Related Issues (20)
- Ch-07 instruction tuning HOT 3
- The GPT 2 Model url link needs to be corrected HOT 3
- Duplicated line in the Listing 2.6 (2.6 Data sampling with a sliding window) HOT 1
- Detaching a Tensor from the graph doesn't have effect (Embedding Layers and Linear Layers) HOT 3
- (Enhancement) Applying mask to attention in one operation (3.5 Hiding future words with causal attention) HOT 2
- Inconsistency in cases for applying dropout in the book and in the notebook (3.5.2 Masking additional attention weights with dropout) HOT 3
- Question of training on conversational datasets HOT 1
- generate_model_scores function bug? HOT 3
- lower validation&train loss with poorer performance
- Edge case: Gradient accumulation HOT 3
- Typo of figure labeling? (5.1.2 Calculating the text generation loss) HOT 1
- An unusual link in the pdf version (5.1.2 Calculating the text generation loss) HOT 1
- Different figures in the book and jupyter notebook for Figure 5.9 (5.1.3 Calculating the training and validation set losses) HOT 1
- Inconsistencies in the book and Jupyter notebook (5.2 Training an LLM) HOT 4
- Missing word in a sentence (5.3.1 Temperature scaling) HOT 4
- Output and code cells are in the wrong order (5.3.1 Temperature scaling) HOT 1
- Incorrect formatting of the text as code (5.3.1 Temperature scaling) HOT 3
- Different function names for the same function (5.3.3 Modifying the text generation function) HOT 1
- Duplicated code cell (Exercise 5.3: Deterministic behavior in the decoding functions) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llms-from-scratch.