Mac OSX install

[outlace]

Mac OSX install

[hughperkins]

Hi, sorry for the delay in replying; I like to think I normally answer faster than that. So... building on Mac is a bit hit and miss unfortunately, it depends on a few factors, works for some people, less well for others. I think you should try using the linux method for building, and see what happens, ie:

git clone --recursive https://github.com/hughperkins/DeepCL.git
cd DeepCL
mkdir build
cd build
ccmake ..
# in ccmake:
# - press 'c'/configure
# - choose the options you want
# - press 'c' /configure again
# - press 'g' / generate, then `q` / quit
make -j 4 install
./deepcl_unittests

More details on pre-requisites and stuff at https://github.com/hughperkins/DeepCL/blob/master/doc/Build.md

Edit: by the way, please let me know how this goes, success/failure, and any logs and so on, if any issues.

[outlace]

Sorry for late reply, was out of town.

Tried installing. Seemed to install okay, but I tried ./deepcl_unittests and get this error

$ ./deepcl_unittests
dyld: Library not loaded: libEasyCL.dylib
Referenced from: /Users/brandon/DeepCL/build/./deepcl_unittests
Reason: image not found
Trace/BPT trap: 5

Although I've installed/built EasyCL

[hughperkins]

Hi outlace, Ok, I think you will need to set some environment variables, such as DYLD_LIBRARY_PATH, something like:

export DYLD_LIBRARY_PATH=../dist/lib
export LD_LIBRARY_PATH=../dist/lib
./deepcl_unittests

[outlace]

I think it worked? Does that one failed test matter?

[ OK ] testGpuOp.addscalarinplace (8 ms)
[----------] 4 tests from testGpuOp (35 ms total)

[----------] 1 test from testjpeghelper
[ RUN ] testjpeghelper.writeread
[ OK ] testjpeghelper.writeread (13 ms)
[----------] 1 test from testjpeghelper (13 ms total)

[----------] Global test environment tear-down
[==========] 159 tests from 29 test cases ran. (67203 ms total)
[ PASSED ] 158 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testNorbLoader.load1000

1 FAILED TEST
YOU HAVE 2 DISABLED TESTS

[hughperkins]

That one test doesnt matter. It means that the norb dataset is not present, which is fine. Looks like it works, you have succeeded :-)

[outlace]

Sweet! Thank you very much!!

[outlace]

Actually.. Now I'm trying to install the Python wrapper and I get this error:

---

Failed building wheel for DeepCL
Failed to build DeepCL
Installing collected packages: DeepCL
Running setup.py install for DeepCL
Complete output from command /Users/brandon/anaconda/bin/python3 -c "import setuptools, tokenize;file='/private/var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-build-k35n57y6/DeepCL/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-sgk6frfv-record/install-record.txt --single-version-externally-managed --compile:
version: 8.0.0
running install
running build
running build_ext
building 'PyDeepCL' extension
gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/brandon/anaconda/include -arch x86_64 -Imysrc -Imysrc/lua -I/Users/brandon/anaconda/include/python3.4m -c PyDeepCL.cxx -o build/temp.macosx-10.5-x86_64-3.4/PyDeepCL.o
In file included from PyDeepCL.cxx:251:
In file included from /Users/brandon/DeepCL/dist/include/deepcl/DeepCL.h:6:
In file included from /Users/brandon/DeepCL/dist/include/easycl/EasyCL.h:31:
In file included from /Users/brandon/DeepCL/dist/include/easycl/deviceinfo_helper.h:22:
/Users/brandon/DeepCL/dist/include/easycl/mystdint.h:11:14: fatal error: 'cstdint' file not found
#include
^
1 error generated.
error: command 'gcc' failed with exit status 1

----------------------------------------

Command "/Users/brandon/anaconda/bin/python3 -c "import setuptools, tokenize;file='/private/var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-build-k35n57y6/DeepCL/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-sgk6frfv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-build-k35n57y6/DeepCL

[GOFAI]

Looking at the setup.py file, I think that the problem is that under Darwin clang isn't told to set the '-std=c++0x' option.

[hughperkins]

@GOFAI Ah, ok. Any thoughts on how to update the setup.py to include this option?

[hughperkins]

Ah, in this bit:

if osfamily == 'Windows':
    compile_options.append('/EHsc')
elif osfamily == 'Linux':
    compile_options.append('-std=c++0x')
    compile_options.append('-g')
else:
    pass

outface, dont suppose.. can you open a python prompt, and do:

import platform
print platform.uname()[0]

[outlace]

import platform
print(platform.uname()[0])
Darwin

[GOFAI]

Don't have time to test at the moment, but I think it's probably as simple as:

if osfamily == 'Windows':
    compile_options.append('/EHsc')
elif osfamily == 'Linux' or osfamily == 'Darwin':
    compile_options.append('-std=c++0x')
    compile_options.append('-g')
else:
    pass

[hughperkins]

Ok. Fixed in cdb0438 outface, can you pull down the latest version, and retry?

[m-smith]

The above didn't work for me, but the following did!

if osfamily == 'Windows':
    compile_options.append('/EHsc')
elif osfamily in ['Linux']:
    compile_options.append('-std=c++0x')
    compile_options.append('-g')
elif osfamily == 'Darwin':
    compile_options.append('-mmacosx-version-min=10.7')
    compile_options.append('-stdlib=libc++')
    compile_options.append('-g')

[hughperkins]

Thanks! Merged.

[outlace]

I'm getting a different error now
$ python setup.py install
...
...
running install_lib
running build_ext
building 'PyDeepCL' extension
creating build
creating build/temp.macosx-10.5-x86_64-3.5
gcc -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/brandonbrown/anaconda/include -arch x86_64 -Imysrc -Imysrc/lua -I/Users/brandonbrown/anaconda/include/python3.5m -c PyDeepCL.cxx -o build/temp.macosx-10.5-x86_64-3.5/PyDeepCL.o -mmacosx-version-min=10.7 -stdlib=libc++ -g
PyDeepCL.cxx:250:10: fatal error: 'CppRuntimeBoundary.h' file not found
#include "CppRuntimeBoundary.h"
^
1 error generated.
error: command 'gcc' failed with exit status 1

[m-smith]

is it possible that you forgot to run source dist/bin/activate.sh ?

[outlace]

Whoops, indeed I did. Thanks

On Wednesday, November 4, 2015, Matthew Smith [email protected]
wrote:

is it possible that you forgot to run source dist/bin/activate.sh ?

—
Reply to this email directly or view it on GitHub
#32 (comment).

[outlace]

Now I have an error importing. Sorry guys...Thanks so much for your patience and help

Brandons-Air:DeepCL brandon$ python
Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 20 2015, 14:39:26)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import PyDeepCL
Traceback (most recent call last):
File "", line 1, in
ImportError: dlopen(/Users/brandonbrown/anaconda/lib/python3.5/site-packages/DeepCL-0.0.0-py3.5-macosx-10.5-x86_64.egg/PyDeepCL.cpython-35m-darwin.so, 2): Library not loaded: libclBLAS.2.dylib
Referenced from: /Users/brandonbrown/anaconda/lib/python3.5/site-packages/DeepCL-0.0.0-py3.5-macosx-10.5-x86_64.egg/PyDeepCL.cpython-35m-darwin.so
Reason: image not found

[hughperkins]

Hmmmm, somehow missing the libclBLAS.2.dylib library in the classpath. It looks like the PyDeepCL library references it, but it's not being found.

There seem to be two possibilities I can think of:

• libclBLAS is missing from the referenced locatoin, or not in the DYLD_LIBRARY_PATH, or
• something to do with flat namespaces and so on

The first possibility is easier to check, so let's check that first. Plesae can you do:

echo $DYLD_LIBRARY_PATH
ls /Users/brandon/DeepCL/dist/lib

[outlace]

Brandons-Air:DeepCL brandon$ echo $DYLD_LIBRARY_PATH

Brandons-Air:DeepCL brandon$ ls /Users/brandon/DeepCL/dist/lib
cmake libclBLAS.2.4.0.dylib libclew.1.0.0.dylib
libDeepCL.dylib libclBLAS.2.dylib libclew.dylib
libEasyCL.dylib libclBLAS.dylib libdeepcl_gtest.a

[hughperkins]

hmmm, can you try setting your DYLD_LIBRARY_PATH, maybe something like:

export DYLD_LIBRARY_PATH=/Users/brandon/DeepCL/dist/lib

... and then retry using python

[m-smith]

For some reason, I needed to set the DYLD_FALLBACK_LIBRARY_PATH environment variable...same as above just

export DYLD_FALLBACK_LIBRARY_PATH=/Users/brandon/DeepCL/dist/lib

[outlace]

That did it! You guys are amazing, thank you so much

Brandons-Air:DeepCL brandonbrown$ python
Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 20 2015, 14:39:26)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import PyDeepCL
cl = PyDeepCL.DeepCL()
Using Apple , OpenCL platform: Apple
Using OpenCL device: HD Graphics 5000
initializing clblas

[hughperkins]

Ah, needed the DYLD_FALLBACK_LIBRARY_PATH? Cool. Nice! (sorry about my occasional repeated postings by the way, my network is a bit iffy :-P )

[hughperkins]

Added DYLD_LIBRARY_PATH and DYLD_FALLBACK_LIBRARY_PATH in 911f53b. I think I can close this now, right?

[Dexdev08]

total n00b here...
[ FAILED ] testsinglebatch.imagesize5_filtersize3_batchsize2_10filters
[ FAILED ] testNorbLoader.load1000
[ FAILED ] testCopyLocal.basic

I have tried your instructions above... I am failing 2 more unit tests... testCopyLocal.basic sounds important...

[hughperkins]

Yes, the other two are ok to fail. norbloader fails, because it needs the norb dataset, which you dont have. The singlebatch test is a bit stochastic, and fails occasionally (I should fix that really).

testcopylocal doesnt normally fail.... in fact I dont remember it ever failing... if all the other tests are passing, it probably doesnt matter really though. I think you can probably just go ahead and use the library. However, if you do want to dig into why, I think you need to provide:

• output of clinfo command
• the full output of the testCopyLocal.basic test
• any other information about your system, eg which os, which os version, 32-bit or 64-bit, which gpu card, etc ...

[Dexdev08]

Thanks! I'm using OSX Mavericks on a Macbook Air (late 2013) with the intel HD 5000 card (opencl enabled)... There's a reason why i'm looking for opencl... as I have not exploited this feature that much.

I'm attaching clinfo here... not sure how to retrieve the output of testCopyLocal.basic (shame shame on me)
clinfo.txt

[hughperkins]

Ok. Can you paste the full output of running ./deepcl_unittests tests=testCopyLocal.* please?

On my laptop, this gives me:

DeepCL/build$ ./deepcl_unittests tests=testCopyLocal.*
args: ./deepcl_unittests --gtest_filter=testCopyLocal.*
Note: Google Test filter = testCopyLocal.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from testCopyLocal
[ RUN      ] testCopyLocal.basic
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
0 0 0 0 
1 2 3 4 
5 6 7 8 
9 10 11 12 

0 0 0 0 
[       OK ] testCopyLocal.basic (809 ms)
[----------] 1 test from testCopyLocal (809 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (809 ms total)
[  PASSED  ] 1 test.

[Dexdev08]

Hi Hugh,

Thanks for your patience. I've ran the test and dumped the error message into the text file below.
error_msg.txt

Dexter

[hughperkins]

Ok... not quite sure how to fix that to be honest. Works ok on my machine. Let's at least get rid of some of those opencl build warnings (or try to...), and those two failing tests, which should be done (or at least, started) in commit 022b6a3 , on master branch

[hughperkins]

Perhaps you can try running the EasyCL tests first, since presumably it's fairly on something fairly basic (I guess....). Please can you try the following? :

git clone --recursive https://github.com/hughperkins/EasyCL.git
cd EasyCL
mkdir build
cd build
ccmake ..
# press 'c' for configure, set:
#  BUILD_TESTS=ON, 
#  CMAKE_BUILD_TYPE=Debug
# press 'c' for configure
# press 'g' for generate
# => ccmake should exit.  Then do:
make -j 4 install
cp ../test/*.cl .
./easycl_unittests

... and provide the results.

[Dexdev08]

Hi! Sorry just came from work. I've ran this...
is the warning after pressing g in ccmake relevant --> "Policy cmp0042 is not set: macosx_rpath is enabled by default ..."
easycltests.txt

[hughperkins]

is the warning after pressing g in ccmake relevant --> "Policy cmp0042 is not set: macosx_rpath is enabled by default ..."

dont think so, because the problem is probably opencl-related. rpath is used to locate libraries, ie .sos and .dylibs. If those weren't being located the program wouldnt run at all, would crash immediately with a message like couldnt find libOpenCL.so, something like that.

easycltests.txt

Hmmm, those all pass...

[Dexdev08]

i think this library is usable... i have tried running test_deepcl.py... and it seems to be running, showing a test accuracy of 95.62%

[hughperkins]

Ok, cool :-)

[Dexdev08]

thanks!

[Dexdev08]

Hi Hugh,

I think we can compile a readme for OSX installation.

I've compiled my experience here, and lifted some of your instructions.

http://mynotebookis.blogspot.sg/2015/11/installing-deepcl-on-macbook-air-2013.html

[GOFAI]

Might it be worthwhile to put together a formula for homebrew-science, or should that wait until the next official release?

[hughperkins]

@Dexdev08 Thats awesome! Thanks! :-)

[hughperkins]

@GOFAI That sounds cool. Maybe I will fire up Jenkins and compile the latest code, and we use that?

[hughperkins]

Ok. Builds working now. Created v8.1.2 release https://github.com/hughperkins/DeepCL/releases/tag/v8.1.2

[GOFAI]

I've made two Homebrew formulas. The first, head-only one worked fine with brew install -i --HEAD DeepCL:
https://gist.github.com/GOFAI/6071778631ba3d7e8ebf

It would be better to use the tarball of the versioned release, but unfortunately it turns out that if one tries that the recursive submodules EasyCL and clMathLibraries don't get cloned:
https://gist.github.com/GOFAI/6b07ecdbace369ddd4ca
When I try to step through this one in interactive mode (brew install -i DeepCL), I hit the following snags in cmake:

Setting build type to 'RelWithDebInfo'

 CMake Error at
 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:1871
 (message):
   No download info given for 'EasyCL-external' and its source directory:

    /tmp/deepcl20151120-8228-1sxnmek/DeepCL-8.1.2/EasyCL

   is not an existing non-empty directory.  Please specify one of:

    * SOURCE_DIR with an existing non-empty directory
    * URL
    * GIT_REPOSITORY
    * HG_REPOSITORY
    * CVS_REPOSITORY and CVS_MODULE
    * SVN_REVISION
    * DOWNLOAD_COMMAND
Call Stack (most recent call first):

 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:2356
 (_ep_add_download_command)
   cmake/build_EasyCL.cmake:3 (ExternalProject_Add)
   CMakeLists.txt:66 (INCLUDE)



 CMake Error at
 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:1871
 (message):
   No download info given for 'clBLAS-external' and its source directory:

    /tmp/deepcl20151120-8228-1sxnmek/DeepCL-8.1.2/clMathLibraries/clBLAS/src

    is not an existing non-empty directory.  Please specify one of:

    * SOURCE_DIR with an existing non-empty directory
    * URL
    * GIT_REPOSITORY
    * HG_REPOSITORY
    * CVS_REPOSITORY and CVS_MODULE
    * SVN_REVISION
    * DOWNLOAD_COMMAND
 Call Stack (most recent call first):

 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:2356
 (_ep_add_download_command)
   cmake/build_clBLAS.cmake:9 (ExternalProject_Add)
   CMakeLists.txt:70 (INCLUDE)

[hughperkins]

Nice! :-)

It would be better to use the tarball of the versioned release, but unfortunately it turns out that if one tries that the recursive submodules EasyCL and clMathLibraries don't get cloned

Hmmmm...ok... well... couple of ideas, not sure how doable these are?

• to use the first method, but set it to the 8.1.2 source, you could clone from the v8.1.2 tag perhaps, ie something conceptually like git clone http://some/url -b v8.1.2 ?
• to use the second method, perhaps you can also download the versioned tar files for the subcomponents, ie https://github.com/hughperkins/EasyCL/archive/b9023cdf25c10524ad06227275d3756bbe0c0ed2.zip and https://github.com/hughperkins/clBLAS/archive/bdaf1f9ef3d97168fdaff4c3ef588feeae72b171.zip ? (edit: and clew: https://github.com/hughperkins/clew/archive/1d2752564b6af98d24127cdb2d573d3f9ac98f02.zip )

[GOFAI]

Some version of the second method is canonical for brew formulae. It would be nice to be able to build against other brew formulae such as clBLAS, but if need be there is apparently a way to pull in the subcomponents as versioned tar files. I just need to untar them into the right folders before running ccmake, correct?

[hughperkins]

Yes, you will need to untar as follows:

• clBLAS => clMathLibraries directory
• EasyCL => root directory (ie conceptual root, not a directory called 'root')
• clew => EasyCL/thirdparty (so, you have to untar EasyCL first)

[GOFAI]

So I've got clBLAS, EasyCL, and clew staging correctly, but I'm not sure what arguments to pass to cmake, and using homebrew's default *std_cmake_args results in the following output:

JOHNNIAC:~ Walrus$ brew install DeepCL
==> Downloading https://github.com/hughperkins/DeepCL/archive/v8.1.2.tar.gz
Already downloaded: /Library/Caches/Homebrew/deepcl-8.1.2.tar.gz
==> Downloading https://github.com/hughperkins/EasyCL/archive/b9023cdf25c10524ad
Already downloaded: /Library/Caches/Homebrew/deepcl--EasyCL-2.zip
==> Downloading https://github.com/hughperkins/clBLAS/archive/bdaf1f9ef3d97168fd
Already downloaded: /Library/Caches/Homebrew/deepcl--clBLAS-171.zip
==> Downloading https://github.com/hughperkins/clew/archive/1d2752564b6af98d2412
Already downloaded: /Library/Caches/Homebrew/deepcl--clew-02.zip
==> mkdir build
==> cd build
==> cmake .. -DCMAKE_C_FLAGS_RELEASE= -DCMAKE_CXX_FLAGS_RELEASE= -DCMAKE_INSTALL
Last 15 lines from /Users/Walrus/Library/Logs/Homebrew/deepcl/03.cmake:
2015-11-21 12:13:36 -0800

cmake
..
-DCMAKE_C_FLAGS_RELEASE=
-DCMAKE_CXX_FLAGS_RELEASE=
-DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/deepcl/8.1.2
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_FIND_FRAMEWORK=LAST
-DCMAKE_VERBOSE_MAKEFILE=ON
-Wno-dev

CMake Error: The source directory "/tmp/deepcl20151121-14817-1dqypbk" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

READ THIS: https://git.io/brew-troubleshooting

[GOFAI]

Sorted out the problem--builds successfully and ./deepcl_unittests passes 158 of 159 tests. I just need to clean it up to meet homebrew's submission standards, including some sort of nominal test.

One issue is that using deepcl_unittests as the required test will run afoul of Brew Test Bot, because the single failed test causes the whole thing to count as a failure:

JOHNNIAC:bin Walrus$ brew test DeepCL
Testing deepcl
==> /usr/local/Cellar/deepcl/8.1.2/bin/deepcl_unittests
Last 15 lines from /Users/Walrus/Library/Logs/Homebrew/deepcl/01.deepcl_unittests:

[----------] 1 test from testjpeghelper
[ RUN      ] testjpeghelper.writeread
[       OK ] testjpeghelper.writeread (1 ms)
[----------] 1 test from testjpeghelper (1 ms total)

[----------] Global test environment tear-down
[==========] 159 tests from 30 test cases ran. (59058 ms total)
[  PASSED  ] 158 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DATA_testNorbLoader.load1000

 1 FAILED TEST
  YOU HAVE 2 DISABLED TESTS

Error: deepcl: failed

[hughperkins]

Oh.... seems I havent quite excluded the DATA_* tests (of which there are one). I'll fix that.

PS Nice work Edward :-)

[GOFAI]

Homebrew's default behavior is also to symlink everything in /bin into /usr/local/bin, including stuff like ~foo.jpg. Which of these should be symlinked?

activate.bat        deepcl_predict      easycl_activate.sh
activate.sh     deepcl_train        foo.dat
clBLAS-tune     deepcl_unittests    ~foo.jpeg

[hughperkins]

For the symlink, I think you just need:

• activate.sh
• deepcl_train
• deepcl_predict
• deepcl_unittests

maybe could add clBLAS-tune, but it's part of clBLAS, not DeepCL, probably best to leave it out? (Also, I've never used it actually....)

Hmmm.... if these are going to be symlinked into /usr/local/bin, probably no longer need activate.sh either actually. Just to confirm, is it going to do the following symlinks? :

• dist/bin/* symlinked from /usr/local/bin, and
• dist/lib/*.so symlinked from /usr/local/lib, and
• dist/lib/*.dylib symlinked from /usr/local/lib
  ?

If so, you no longer need the activate.sh, can simply include anything prefixed with deepcl_, and all the .so/.dylib files, into /usr/local/lib.

[GOFAI]

Yes, all of those automatically symlink although there are no .so files in /lib. The following are all symlinked into /usr/local/lib:

cmake           libclBLAS.2.4.0.dylib   libclew.1.0.0.dylib
libDeepCL.dylib     libclBLAS.2.dylib   libclew.dylib
libEasyCL.dylib     libclBLAS.dylib     libdeepcl_gtest.a

The cmake folder obviously shouldn't be symlinked here, but as long as the rest are can we expect everything to work without activate.sh?

[hughperkins]

v8.1.3 848118f should remove the failing DATA_norbloader unit test

[hughperkins]

The cmake folder obviously shouldn't be symlinked here, but as long as the rest are can we expect everything to work without activate.sh?

Yes... oh... what about includes? Everything in dist/include should be available eg in /usr/local/include somehow I think? Otherwise the python wrappers might not be able to build.

[GOFAI]

The includes all symlink into /usr/local/include as well. 158/158 tests pass with 8.1.3. I still haven't nailed down the problem of unnecessary symlinks, though.

[GOFAI]

I got rid of the unneeded files and everything seems to be symlinked into usr/local, but pip install --pre --upgrade DeepCL gives me the following error:

JOHNNIAC:~ Walrus$ pip install --pre --upgrade DeepCL
Collecting DeepCL
  Downloading DeepCL-8.1.3.tar.gz (139kB)
    100% |################################| 143kB 2.3MB/s 
    version:  8.1.3
Installing collected packages: DeepCL
  Running setup.py install for DeepCL
    version:  8.1.3
    building 'PyDeepCL' extension
    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Imysrc -Imysrc/lua -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c PyDeepCL.cxx -o build/temp.macosx-10.10-x86_64-2.7/PyDeepCL.o -mmacosx-version-min=10.7 -stdlib=libc++ -g -DUSE_CLEW
    PyDeepCL.cxx:250:10: fatal error: 'CppRuntimeBoundary.h' file not found
    #include "CppRuntimeBoundary.h"
             ^
    1 error generated.
    error: command 'clang' failed with exit status 1
    Complete output from command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/0h/cz_j9hc50c97dyf2cw9y1srw0000gn/T/pip-build-cHHxDH/DeepCL/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/0h/cz_j9hc50c97dyf2cw9y1srw0000gn/T/pip-wCJUuz-record/install-record.txt --single-version-externally-managed --compile:
    version:  8.1.3

    running install

    running build

    running build_ext

    building 'PyDeepCL' extension

    creating build

    creating build/temp.macosx-10.10-x86_64-2.7

    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Imysrc -Imysrc/lua -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c PyDeepCL.cxx -o build/temp.macosx-10.10-x86_64-2.7/PyDeepCL.o -mmacosx-version-min=10.7 -stdlib=libc++ -g -DUSE_CLEW

    PyDeepCL.cxx:250:10: fatal error: 'CppRuntimeBoundary.h' file not found

    #include "CppRuntimeBoundary.h"

             ^

    1 error generated.

    error: command 'clang' failed with exit status 1

    ----------------------------------------
    Command "/usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/0h/cz_j9hc50c97dyf2cw9y1srw0000gn/T/pip-build-cHHxDH/DeepCL/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/0h/cz_j9hc50c97dyf2cw9y1srw0000gn/T/pip-wCJUuz-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /var/folders/0h/cz_j9hc50c97dyf2cw9y1srw0000gn/T/pip-build-cHHxDH/DeepCL

Edit: I checked to make sure and CppRuntimeBoundary.h is symlinked into /usr/local/include/deepcl.

[hughperkins]

I got rid of the unneeded files

Nice!

everything seems to be symlinked into usr/local

Cool :-)

but pip install --pre --upgrade DeepCL gives me the following error

Ah, it's because the python setup.py isn't searching in /usr/local/include/deepcl, or /usr/local/include/easycl. Updated setup.py in 540fff3, to add these include paths.

Can you pull down the latest DeepCL source-code, and try running python setup.py install from the python subdirectory please? If this works ok, I'll push a new tag up to pypi.

[GOFAI]

How's this look?

JOHNNIAC:bin Walrus$ python -c "import PyDeepCL; cl = PyDeepCL.DeepCL()"
Using Apple , OpenCL platform: Apple
Using OpenCL device: Iris
initializing clblas
clblas teardown

[hughperkins]

Looks good :-) It's loading ok, no errors.

I will push a new tag up to pypi, with the updated setup.py.

[GOFAI]

Excellent! We're getting close--that seems to be the last functional issue with the formula. :)

A more cosmetic concern is that homebrew wants contributors to ensure that their formulas pass all brew audit --strict --online $FORMULA tests. It turns out that one of the things it doesn't like is using zipballs instead of tarballs:

JOHNNIAC:bin Walrus$ brew audit DeepCL
deepcl:
 * Stable resource "EasyCL": Use GitHub tarballs rather than zipballs (url is https://github.com/hughperkins/EasyCL/archive/b9023cdf25c10524ad06227275d3756bbe0c0ed2.zip).
 * Stable resource "clBLAS": Use GitHub tarballs rather than zipballs (url is https://github.com/hughperkins/clBLAS/archive/bdaf1f9ef3d97168fdaff4c3ef588feeae72b171.zip).
 * Stable resource "clew": Use GitHub tarballs rather than zipballs (url is https://github.com/hughperkins/clew/archive/1d2752564b6af98d24127cdb2d573d3f9ac98f02.zip).

Error: 3 problems in 1 formula

Can you point me at tarballs for these resources? (Maybe they're just the same urls with .tar.gz extensions?)

[hughperkins]

Can you point me at tarballs for these resources? (Maybe they're just the same urls with .tar.gz extensions?)

Hmmmm, I hadn't heard of this before, but as you say, seems it is sufficient to replace .zip with .tar.gz, eg:

https://github.com/hughperkins/EasyCL/archive/b9023cdf25c10524ad06227275d3756bbe0c0ed2.tar.gz

Note that this seems to create a subdirectory with a really long name when you decompress:

$ cd
$ wget https://github.com/hughperkins/EasyCL/archive/b9023cdf25c10524ad06227275d3756bbe0c0ed2.tar.gz
$ cd /tmp/foo
$ tar -xf ~/b9023cdf25c10524ad06227275d3756bbe0c0ed2.tar.gz 
$ ls
EasyCL-b9023cdf25c10524ad06227275d3756bbe0c0ed2

Might need some hacking around a bit to handle this somehow?

[hughperkins]

(new tag uploaded: https://pypi.python.org/pypi/DeepCL/8.1.4 )

[GOFAI]

Nice--and here's a version of the formula that passes the audit:
https://gist.github.com/GOFAI/53a25ff22f31ec144608

[hughperkins]

Cooolll :-) Ok, what happens next? I should link to this from the release notes? There is a brew repository somewhere that I should link to?

[GOFAI]

The next step is to submit it as a pull request on homebrew-science, I think. They'll then test it on a variety of platforms so we'll know if it builds correctly using XCode 7.1.

[hughperkins]

Ok. Sounds good. I'll add a link to the gist to the release notes for now.

Awesome!

[hughperkins]

Created v8.1.3 release notes at https://github.com/hughperkins/DeepCL/releases/tag/v8.1.3 , and added a link to your homebrew formula. Very cool! Thank you very much Edward :-)

[GOFAI]

Made the pull request, let's keep our fingers crossed!
https://github.com/Homebrew/homebrew-science/pull/3047

[GOFAI]

So it turns out the formula works for me because I had already installed its dependencies, but homebrew's tests fail on the following cmake error:

CMake Error at /usr/local/Cellar/cmake/3.4.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR)

What exactly is it looking for here? Since I almost certainly installed whatever it is using homebrew, it should be trivial to add something like depends_on jpeg to the formula.

[hughperkins]

something like libjpeg-turbo8-dev?

[GOFAI]

I have these three in /usr/local/lib:

libjpeg.8.dylib
libjpeg.a
libjpeg.dylib

I'm pretty sure that depends_on "jpeg" will fix this one, but what else should I be on the lookout for? Do we also need to depend on lua and some other things?

[hughperkins]

I think it's looking for the include files, rather than the libraries. On linux, we have two packages usually, eg:

• libjpeg-turbo8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8
• libjpeg-turbo8-dev => /usr/include/turbojpeg.h

[GOFAI]

In my /usr/local/include/ I have something called jpeglib.h.

[hughperkins]

Yes, that would probably be what it is. Although jpeglib is I think from libjpeg 6.2 or similar, but the api is forwards compatible with libjpeg-turbo, per my understanding (libjpeg-turbo is 4 times faster or so, I think, because it uses SSE extensions)

Edit: more information about libjpeg-turbo et al here: https://en.wikipedia.org/wiki/Libjpeg

Edit2: maybe http://braumeister.org/formula/jpeg-turbo ?

[GOFAI]

It doesn't seem essential, since I only have jpeg and I managed to build anyway. I pushed a version of the formula that adds the jpeg dependency and we'll see how it fares.

[hughperkins]

Cool :-) Sounds good. The impact of the jpeg speed is basically if someone reads eg imagenet jpegs directly from DeepCL, it will use a bit more processor time than otherwise.

[GOFAI]

OK, so for some reason depending on jpeg didn't fix the problem on yosemite, but on el capitan and mavericks the library built yet most of the tests aren't passing:
http://bot.brew.sh/job/Homebrew%20Science%20Pull%20Requests/3688/

[hughperkins]

Hmmmm.... it just occurred to me.... can anyone submit formulae to bot.brew.sh? And it will build them automatically, and run tests? Dont need an Xcode account or a Mac, or anything?

[GOFAI]

That's my impression--it starts building as soon as I push anything with an open pull request.

[hughperkins]

Wow, interesting :-)

[hughperkins]

Ah... well... this is the same reason why my Jenkins doesnt run unit tests actually... problem is... you need a GPU installed in order to run the tests.

Edit, basically, you can see in the output, eg http://bot.brew.sh/job/Homebrew%20Science%20Pull%20Requests/version=mavericks/3688/console :

[ RUN      ] testGpuOp.addoutofplace
unknown file: Failure
C++ exception with description "Error getting OpenCL device ids: -1" thrown in the test body.
[  FAILED  ] testGpuOp.addoutofplace (0 ms)

[hughperkins]

I suppose we could, in the worst case, create some dummy 'unit-test', and tell brew that is our 'comprehensive unit tests', and then brew sees it passes ok?

[GOFAI]

That may be appropriate in this case, if the unit tests absolutely need a GPU to run.

In fact, is there any argument that can be passed to one of the executables to show they're installed? That will do as a test for homebrew's purposes...

[hughperkins]

Ok

[hughperkins]

All of the following tests pass:

[       OK ] testfilehelper.testfilehelper (11 ms)
[       OK ] testfilehelper.testreadchunk (6 ms)
[       OK ] teststringhelper.split (0 ms)
[       OK ] teststringhelper.split2 (0 ms)
[       OK ] teststringhelper.split3 (0 ms)
[       OK ] teststringhelper.tolower (0 ms)
[       OK ] teststringhelper.replace (0 ms)
[       OK ] teststringhelper.replaceglobal (0 ms)
[       OK ] teststringhelper.strcpy_safe (0 ms)
[       OK ] testGtestGlobals.basic (0 ms)
[       OK ] testRandomSingleton.testMockRandom (0 ms)
[       OK ] testjpeghelper.writeread (2 ms)

so, eg if we run the tests as deepcl_unittests tests=teststringhelper.* ... then they will always pass :-P

(Edit: interestingly, it seems that a lot of the tests are running ok, using the CPU to run the OpenCL, so it's probably not impossible to get the tests to pass on the CPU. Probably a lot of work though. I've been taking the stance that DeepCL only supports OpenCL on GPUs, since trying to optimize OpenCL to work on different brands of GPU is already hard enough, let alone on CPUs too...)

[GOFAI]

What do you make of the continuing failure on Yosemite?

[hughperkins]

You mean, the unit test failures? http://bot.brew.sh/job/Homebrew%20Science%20Pull%20Requests/version=yosemite/3688/consoleFull ? Those are also because of lack of GPU right? I can see a few non-GPU passes here, eg:

[----------] 1 test from testjpeghelper
[ RUN      ] testjpeghelper.writeread
[       OK ] testjpeghelper.writeread (1 ms)
[----------] 1 test from testjpeghelper (1 ms total)

[hughperkins]

That's my impression--it starts building as soon as I push anything with an open pull request.

Hmmm, seems that however:

• per the instructions, supposed to build/test before pushing, ie I cant do some sort of fairly empty script, and test it by repeatedly pushing to the bot :-D
• in practice, people notice the pr really quickly, and start commenting on it, eg https://github.com/Homebrew/homebrew-science/pull/3047 , so if I used the bot as my personal 'mac dev/build/test console', people would notice pretty quickly :-D

[GOFAI]

Yeah, I wouldn't push anything before testing it. It's helpful in this case, though, because things worked fine on my machine but I could only really fix the formula by checking it on other setups.

[hughperkins]

Yes... seems like they could offer a for-pay service though, so that I could do this, using the same technology. I might email them somehow. Seems like a no-brainer way for them to raise funding.

(Edit: created an issue to ask the question: Homebrew/legacy-homebrew#46259 )

[hughperkins]

(seems some throttling issue in latest builds?

Error: GitHub API rate limit exceeded for 208.78.110.199. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

)

[GOFAI]

Yes, but I think that everything else is looking good. :) I'll just have to be patient and rerun it later...

[hughperkins]

Ok :-) Looking awesome! :-) Very cool :-)

[hughperkins]

Hmmm, neat, seems I can run Mac OS X builds on travis :-) eg https://travis-ci.org/hughperkins/homebrew-test-travis-osx/builds/92832652 (following @dunn's heads-up Homebrew/legacy-homebrew#46259 (comment) )

[hughperkins]

Copied your deepcl build into my repo, and seems to build ok :-) https://travis-ci.org/hughperkins/homebrew-test-travis-osx/builds/92940990 By the way, I've created some tags, so you can do now like:

  resource "EasyCL" do
    url "https://github.com/hughperkins/EasyCL/archive/v3.0.0.tar.gz"
    sha256 "e38fa777b4ad22e0e48ba3c5ae55bb1ecb81958806207cdeefa78f9901012e3a"
  end

  resource "clBLAS" do
    url "https://github.com/hughperkins/clBLAS/archive/hughperkins_v2.4.1.tar.gz"
    sha256 "3cbf2d0bc6eb449e3d14a57ab10220bd69d2f7df9069af6b10fbe0edd7349d30"
  end

  resource "clew" do
    url "https://github.com/hughperkins/clew/archive/hughperkins_v1.0.0.tar.gz"
    sha256 "e93d0eb1ab6ecdd08b9e79ef5e39b4370518d51ae1d62f4f826b92e5b4909d07"
  end

Per homebrew faq, authors are not allowed to submit their own projects to homebrew :-D But at least I can help out with testing/debugging Mac issues now.

Edit: by the way, swapped 'jpeg' dependency for 'jpeg-turbo', and seems to download ok, as a bottle: https://travis-ci.org/hughperkins/homebrew-test-travis-osx/builds/92944125#L876

[GOFAI]

If jpeg-turbo and jpeg are both available, will DeepCL link against that instead of jpeg? I was thinking about modifying the formula to provide an explicit --with-jpeg-turbo option that would link against jpeg-turbo.

Looking ahead, it would also be nice to be able to use the clBLAS formula as a dependency instead of staging a more recent version. I submitted a pull request for an updated formula that bumped clBLAS from 2.2 to 2.8, but closed it once it became clear that the more recent version won't build with XCode 7.1 and, moreover, clBLAS-client is experiencing weird CL_INVALID_COMMAND_QUEUE errors whose source I can't nail down.

[hughperkins]

If jpeg-turbo and jpeg are both available, will DeepCL link against that instead of jpeg?

It will link with whichever the cmake command FindJpeg finds I think.

clBLAS-client is experiencing weird CL_INVALID_COMMAND_QUEUE errors whose source I can't nail down.

Yes, so, I was briefly tempted to try debugging those via Travis (assuming they could be reproduced on a CPU OpenCL device, which is an open question), but, other than the difficulty and slowness of doing this, you know, I tried comparing the timing of neural style using the old and new clblas, and no difference. It might be that neural style is not representative, but its a VGG E network, running forward/backward propagation. Seems like should be not totally unrepresentative?

So, since you already have clblas 2.4 working, I'm not sure if it's worth the probably-high amount of time investment to do not much more than switch the '4' digit to an '8' digit :-P

If you wanted to debug it though (since I'm fairly sure I'm not going to do this for now; not unless someone somewhere demonstrates a clear speed benefit of 2.8 vs 2.4), then I would imagine the first thing to do is to find the simplest example code that causes the problem. eg, if you run one of the gemm samples in clblas samples directory, does that crash?

(Edit: for the build issue in 2.8, I think that clMathLibraries/clBLAS#190 (comment) fixes that right?)

[GOFAI]

If 2.8 doesn't offer a clear advantage, I'm perfectly happy to stick with 2.4 for the time being. :) At the moment 2.8 won't even compile on an up-to-date XCode, so why hurry?

Edit: On the other hand, might it be worthwhile to bump the current clBLAS formula from 2.2 to 2.4?

[hughperkins]

I think it is 2.4 right now right? bdaf1f9 is after the v2.4 tag? https://github.com/clMathLibraries/clBLAS/tree/v2.4 (which is 03d7e7e ) :

git/DeepCL/clMathLibraries/clBLAS$ git log -n 10 --oneline
bdaf1f9 Fix kernel crash on nvidia, caused by float4 alignemtn error, see https:
a6b3f9d Merge pull request #98 from clMathLibraries/revert-97-master
15b46fe Revert "Fix to enable successful build on MacOSX "
3107f8a Merge pull request #97 from lzamparo/master
7a9e3df changed indentation of BUILD_TEST block to increase readability
dc304d8 refactored CMakeLists.txt in BUILD_TEST block, added test for OSX detect
03d7e7e Merge pull request #80 from TimmyLiu/master