Comments (1)
Thanks for reaching out! If I remember correctly, TAG assigns different layers in a language model with different weights, where the layers closer to the output get assigned larger weights.
Now consider the case of GPT-2. As you may know, in the model the first layer is the word embedding layer, which translates a token from a vocabulary of size 50257 into an embedding of length 768. In fact, the last layer of GPT-2 is the fully connected layer that reuses the weights of the first layer to translate an internal state of size 768 into an output token from a 50257-token vocabulary.
Since they share the weight parameters
In some cases, it is the first array of the gradients; while in other cases, it is the last array of the gradients. This depends on the PyTorch or transformer library we use. I did observe both cases in my experiments in different platforms, but just can't remember exactly the precise condition.
However, this matters because in one case,
Thus, we need to make sure that the gradient of
Line 28 in 8b4cda7
Back to your problem, if the condition is not met, so be it. It means that the order of your computed gradient is the expected one, and thus the following code can be just skipped.
from grad_attacks.
Related Issues (1)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grad_attacks.