Segmentation fault

[wuhuikai]

Segmentation fault

[Anikily]

I have switched the pytorch version to 1.0.0 by pip install torch==1.0,and still segmentation fault.

[Anikily]

好像是我外部编译的时候用的c++，pytorch编译用的g++，我咋用g++编译外部应用呢

[Anikily]

$ c++ -v
Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,fortran,lto --enable-plugin --enable-initfini-array --without-isl --without-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.2 20131212 (Red Hat 4.8.2-8) (GCC)

[wuhuikai]

I do not think it's caused by g++.
Can you install PyTorch with conda?

[wuhuikai]

conda install pytorch=1.0.0 torchvision cudatoolkit=9.0 -c pytorch

[Anikily]

@wuhuikai Sorry to bother you again,but after install pytorch by conda,i stil have this problem,and i cant figure out the why this happen.Actually,i have run this project successful in another machine and got better scores than paper without any modify,FastFCN is a good project,i want to use it to make more good work.

[Anikily]

/home/yulu/anaconda3/envs/fastfcn/lib/python3.6/site-packages/torch/nn/_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
warnings.warn(warning.format(ret))
Using poly LR Scheduler!
Starting Epoch: 0
Total Epoches: 80
0%| | 0/312 [00:00<?, ?it/s]
=>Epoches 0, learning rate = 0.0010, previous best = 0.0000
Segmentation fault

[wuhuikai]

Can you compare the environment (versions of gcc, python, pytorch, cuda, cudnn, GPU, GPU Driver) between the two machines?
In my three machines, all work well except that sometimes I need to run it twice to get rid of segmentation fault.

[Anikily]

I have passed the different compiler problem by setting export CXX=g++, but segmentation fault still.

[wuhuikai]

One possible reason is the wrong gcc version. In my machines, gcc==5.5.x
You have mentioned that you successfully ran the program on another machine, is it Ubuntu or RedHat?

[Anikily]

Yes,it is ubuntu with gcc=5.5,mybe the cause of segmentation fault is the version of gcc

[wuhuikai]

@Anikily The code in branch latest is now ready, which uses official SyncBatchNorm and can run on all OS with PyTorch>=1.1.0.