Comments (3)
I have temporarily solved this issue by changing num_workers=4 and prefetch=250 to num_workers=2 and prefetch=125. However, I'm not sure why setting a higher number of num_workers would lead to this error. It seems that different numbers of GPU cards need to match the appropriate number of num_workers.
from wenet.
may some oom occurs in training,
num workers * gpus <= cpus cores
from wenet.
may some oom occurs in training,
num workers * gpus <= cpus cores
I am using a total of 4 machines, each equipped with 4 V100 GPUs with 16GB of memory and 100 cores of dedicated CPU.
from wenet.
Related Issues (20)
- 日语训练的asr模型
- Not able to open prebuilt Android apk and x86 demo
- pip install 遗漏了一些包 HOT 7
- [SSL] Config for BESTRQ HOT 2
- recognize.py --dict option HOT 3
- 定制热词
- streaming onnx export HOT 1
- torch.jit.Error: The following operation failed in the TorchScript interpreter. HOT 1
- wenet v3.1.0训练u2++_conformer结束内存buff无法清除,发生警告ResourceWarning。 HOT 1
- 请问大佬,如果想拿wenet encoder的结果去做下游任务,怎么做呢? HOT 3
- finetune whisper largeV3,and run in grpc server,got inference error.
- wenet2.2.1 recognize.py uses attention decoding method, the result is abnormal HOT 2
- 项目建议 HOT 6
- 【提问】需要一个char+bpe联合建模时tokenizer的yaml配置 HOT 4
- 在github中使用action 编译runtime报错 HOT 5
- 【Help needed】训练迭代一定step后开始频繁出现loss = nan的情况 HOT 4
- 在不能联网环境下编译onnxruntime,如何离线下载WeTextProcessing HOT 1
- Error Building Android Demo
- torch_ddp retraining from checkpoint , optimizer state does not resume from checkpoint. HOT 1
- Null Pointer Exception in Android Demo Project
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wenet.