Comments (19)
Most of modern hardware architectures uses FMA instructions for operations with tensors.
FMA computes a*x+b as one operation. Roughly GMACs = 0.5 * GFLOPs
from flops-counter.pytorch.
I think GFLOPs = 2 * GMACs as general each MAC contains one multiplication and one addition.
from flops-counter.pytorch.
MAC = Multiply–accumulate operation
from flops-counter.pytorch.
@sovrasov, in this case would you consider changing the variable flops
to mac
to avoid confusion?
flops-counter.pytorch/sample.py
Line 36 in 1ad0ed1
from flops-counter.pytorch.
In the original resnet paper authors mixed up macs and flops. As far as I remember, they provided a definition of flops that considers one flop as multiply & add operation. Please check up the paper, correct me if I'm wrong.
from flops-counter.pytorch.
GMACs = 2 * GFLOPs, because MACs includes addition and multiplication operation, GFLOPs only has add operation.
from flops-counter.pytorch.
@cmj18 @jerryli1981
No, it should be GFLOPs = 2 * GMACs
.
MACs
stands for multiply–accumulate operation that performs a <- a + (b x c)
(they are counted as one operation)
FLOPs
is abbreviation of floating operations which includes mul / add / div ... etc.
(each is separately counted as a single floating operation)
from flops-counter.pytorch.
@cassie101 makes sense, I'll change it
from flops-counter.pytorch.
Thank you, sir! Whether the output value is directly FLOPs? Do we need to divide it by 2 to get FLOPs?
from flops-counter.pytorch.
what does MAC stands for? Multi-Add Calculation?
from flops-counter.pytorch.
Roughly GMACs = 2 * GFLOPs
@sovrasov is there a typo here? I did a little reading and it seems that @snownus has it right. In general a multiply-accumulate is one multiplication and one addition, which can each be floating point operations. So 1 GMAC counts as 2 GFLOPs, meaning GMACs = .5 * GFLOPs (I'm not sure if this is what was already meant).
As for fused multiply-add (FMA) it seems that (if it is supported on a given chip/system) the two FLOPs are indeed computed "in a single step" (see here) or "at once" (see here). But this confuses our conversion. Perhaps in the case of FMA it is more accurate to say 1 GMACs = 1 GFLOPs? Hopefully someone with more expertise than me can clarify!
from flops-counter.pytorch.
@chnadell yes, you're right! @snownus also figured it out. I'll edit the first post to avoid any future confusions.
from flops-counter.pytorch.
Thank you, sir! Whether the output value is directly FLOPs? Do we need to divide it by 2 to get FLOPs?
I am also confused. Shouldn't we multiply it by 2 to get FLOPs?
from flops-counter.pytorch.
@code-by-jin yes, exactly, we should multiply GMACS by 2 to get FLOPS
from flops-counter.pytorch.
@code-by-jin yes, exactly, we should multiply GMACS by 2 to get FLOPS
Thanks for your response. I checked ResNet-50 using your tool. It has around 4 GMACS, which is close to the number of FLOPS claimed in the resnet paper. Now I am confused, do I really need to multiply your output GMACS by two?
from flops-counter.pytorch.
Most of modern hardware architectures uses FMA instructions for operations with tensors. FMA computes a*x+b as one operation. Roughly GMACs = 0.5 * GFLOPs
hi, I've never seen GMACs like this before, it means 10^9 about macs? As far as I know the capital letter before the word is related to FLOPS, not FLOPs and MACs, which is easy to confuse me. Looking forward to your reply, thx
from flops-counter.pytorch.
It isn't always true that GMACs = 2 * GFLOPs. For example, two models with the same the GMACS, may have very difference GFLOPS. It depends how you implement model efficiently
from flops-counter.pytorch.
I want to know is there any relation between GOPS(Giga operations per second ) and GFLOPS ,like if i know GFLOPS then ,can i determine GOPS , or are they independent ??
from flops-counter.pytorch.
GOPS is a characteristic of hardware, it can only be determined by measurements. ptflops just shows an approximation to theoretical amount of operations required for one forward pass. Time is not considered by ptflops.
from flops-counter.pytorch.
Related Issues (20)
- There was a bug with computing MultiheadAttention flops HOT 9
- Is the input size of function "get_model_complexity_info()" must be fixed to 3 demensions? HOT 2
- How do I calculate the FLOPs of a model with some frozen layers during training? HOT 2
- Does this code also calculates MACs for back propagation? HOT 1
- how to calculate the flops if one module have 'einsum' option? HOT 2
- flops are counted multiple times if a module is shared by other modules HOT 4
- Support LayerNorm? HOT 1
- support for torch.compile? HOT 1
- The Conv1d with the same architecture yields different Macs in different models HOT 4
- integer overflow, when calculate the MACs of the ViT on Windows HOT 1
- There was a bug with computing FLOPs in OpenPCdet HOT 1
- Do this work with the 'deformable convolution' as well? HOT 5
- Is there some bug in the 'input_constructor' function? HOT 2
- Can't work with `F.interpolate` HOT 2
- FLOPs for a linear layer with 3D input HOT 2
- How to work with two input or more HOT 6
- Support ViT from timm huggingface HOT 1
- Fail to install the newest version HOT 1
- failed to install HOT 2
- Question about add operation count in different case
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flops-counter.pytorch.