maxpumperla/betago

BetaGo: AlphaGo for the masses, live on GitHub.

Hi there, I'm Max 👋

I’m a Data Science & Engineering practitioner from Hamburg, Germany. I’m an avid open source contributor, author of machine learning & technology books, speaker and Coursera instructor.

I specialize in Deep Learning and its applications and can build machine learning solutions and data products from first prototype to production. As Ray contributor, DL4J core developer, Keras contributor, Hyperopt maintainer and author of a range of open source libraries, I have a distinct view on ML engineering and the data science ecosystem.

Here's what I'm up to right now

• 📖 Check out my new book with O'Reilly called "Learning Ray", which helps you get started with distributed Python with Ray. Most content is available on my homepage.
• 🔭 Some of the materials from my courses as Data Science professor at IU can be found here. My teaching centers around programming Python, big data technologies and reinforcement learning.
• 🔭 If you have interesting freelance opportunities, don't hesitate to get in touch with me.
• 💬 Ask me about anything Python or Data Science related. I'm happy to discuss!
• ⚡ Fun fact: I get asked for directions by total strangers all the time, even in foreign countries (and I have yet to figure out why).

Interesting things I've been doing in the past

• 📖 If you like the game of Go, or Python, or both, you might enjoy my book "Deep Learning and the Game of Go" (Manning). All the code for the book is free here on GitHub.
• 🐍 I've authored Elephas: Distributed Deep Learning with Keras & Spark
• 🐍 I'm also the author of Hyperas: simple hyperparameter optimization for Keras models.
• 🐍 We co-authored Logodetect: One-shot logo detection.
• 🐍 Hyperopt is not the most active project anymore, but I'm still trying my best to set aside some time here and there to maintain it.
• 🐍 In what feels like the very distant past, I've been a fairly active Keras contributor once.

And here's how you can get in touch with me

• My website: maxpumperla.com
• Twitter as @maxpumperla
• LinkedIn

📈 GitHub Stats

betago's Issues

game-state mismatch [bug]/[possible enhancement]

If a user refreshes the betago web page but not the server, the game does not actually reset. As a result, an error is thrown by the server any time you try to play in an area that was previously played on. The javascript does not have proper error handling for this, so it stops playing as black, and starts playing and processing as white (locally) for the next move but sends the coordinates to the server as if it were a black move.

The js error always looks like this:

And, the server side callback always looks like this:

Traceback (most recent call last):
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ubuntu/workspace/betago/betago/model.py", line 79, in next_move
    self.bot.apply_move('b', (row, col))
  File "/home/ubuntu/workspace/betago/betago/model.py", line 133, in apply_move
    self.go_board.apply_move(color, move)
  File "/home/ubuntu/workspace/betago/betago/dataloader/goboard.py", line 157, in apply_move
    raise ValueError('Move ' + str(pos) + 'is already on board.')
ValueError: Move (3, 9)is already on board.

The problem is resolved if the server is restarted but if the web page is not reloaded, too, it can lead to more confusion. The web page doesn't know to reset, so the old (and wrong) game data still shows and moves from the server can coincide with illegal moves.

A possible solution to this is to create a randomly generated session number or hash that is sent to the server with each move, as well as what move number the client thinks it is at.
If the server gets a message with a new session number, it can:

• Start a new instance of betago and keep the old one running or
• Close anything that is currently running and start a new game or
• Restart the server

If the client is getting mismatched data, it can display an alert, clear the board, or abort.

Another solution (and possible feature enhancement) could be for the server to keep track of the board and send a copy of the current game state as an array upon request of the client (i.e. when the web page is loaded or an error is suspected) or even as an additional piece of data embedded in the response (significantly enhances reliability but hurts speed and bandwidth). This would be very easy to implement client side.

Ideal for single instance server:
When UI client loads, it pulls latest game data from the server and continues from there as if it was open the whole time. The user can start a new game at any time by pressing a button.

This is a very similar to issue to #26 and the manifestation of the bugs are practically identical.

large training set management

On the kind of hardware I have access to:

1. It's not practical to encode the entire training set at once. It takes days and I run out of memory or disk
2. It's not practical to train an entire epoch uninterrupted. Either I'm using my personal laptop which I sometimes want to use for other stuff, or I'm using cheap EC2 spot instances which can be pre-empted at any time.

My proposal is:

1. Divide a corpus into manageable chunks, let's say 100K positions.
2. Build an index where, for every training chunk, you can quickly find the SGFs containing the correct board positions.
3. Encode and train on one chunk at a time, saving a progress checkpoint (including model weights) after each chunk. You can even train chunk N on the GPU while you encode chunk N + 1 on the CPU.

I started implementing this in my https://github.com/macfergus/betago/tree/large-training-set branch. It's not perfect but I have successfully completed larger training runs than I could do before

error test: AssertionError

>>> Number of zip files: 15
Traceback (most recent call last):
  File "train_generator.py", line 39, in <module>
    data_generator = processor.load_go_data(num_samples=args.sample_size)
  File "$HOME/betago/betago/dataloader/base_processor.py", line 135, in load_go_data
    self.map_to_workers(name, samples)
  File "$HOME/betago/betago/dataloader/base_processor.py", line 222, in map_to_workers
    results = p.get(0xFFFF)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
AssertionError

Keep track of score

It would be nice if either the server, or the client could keep track of the score whenever a game is in progress.

aim

Figure out python 3 support for gomill

betago uses the gomill library for processing SGF files. Unfortunately, it does not support Python 3. The author appears to have abandoned the project:

mattheww/gomill#1

traveller42 did some work on updating to Python 3, but it seems like there is still a lot left:

https://github.com/traveller42/gomill/tree/python3

So at the moment, betago supports python2 and 3 for playing bots; but only python2 for loading data and training bots.

So we could:

1. Complete the python3 porting work for gomill.
2. Pick out just the SGF reader from gomill and port that bit ourselves.
3. Find another SGF parsing library (I don't see any likely options after a quick search).
4. Drop python 3 support for betago.

I guess I like option 2 the best, and option 4 the least.

Error on BetaGo when starting

When I start Go on browser, BetaGo don't run properly.
What should I do?

taewoo@Linux-Workstation:~/Code/betago$ python run_demo.py Using TensorFlow backend.
/home/taewoo/Code/betago/betago/model.py:7: ExtDeprecationWarning: Importing flask.ext.cors is deprecated, use flask_cors instead.
from flask.ext.cors import CORS

• Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
  127.0.0.1 - - [17/Jan/2017 22:51:59] "OPTIONS /prediction HTTP/1.1" 200 -
  127.0.0.1 - - [17/Jan/2017 22:51:59] "POST /prediction HTTP/1.1" 500 -
  Traceback (most recent call last):
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1994, in call
  return self.wsgi_app(environ, start_response)
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
  response = self.handle_exception(e)
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
  return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
  reraise(exc_type, exc_value, tb)
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
  response = self.full_dispatch_request()
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
  rv = self.handle_user_exception(e)
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
  return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
  reraise(exc_type, exc_value, tb)
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
  rv = self.dispatch_request()
  File "/home/taewoo/.local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
  return self.view_functionsrule.endpoint
  File "/home/taewoo/Code/betago/betago/model.py", line 53, in next_move
  bot_row, bot_col = self.bot.select_move('b')
  File "/home/taewoo/Code/betago/betago/model.py", line 109, in select_move
  self._move_generator(bot_color))
  File "/home/taewoo/Code/betago/betago/model.py", line 218, in get_first_valid_move
  for move in move_generator:
  File "/home/taewoo/Code/betago/betago/model.py", line 133, in _model_moves
  pred = np.squeeze(self.model.predict(X))
  File "/home/taewoo/.local/lib/python2.7/site-packages/keras/models.py", line 716, in predict
  return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/home/taewoo/.local/lib/python2.7/site-packages/keras/engine/training.py", line 1216, in predict
  self._make_predict_function()
  File "/home/taewoo/.local/lib/python2.7/site-packages/keras/engine/training.py", line 751, in _make_predict_function
  **kwargs)
  File "/home/taewoo/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1621, in function
  return Function(inputs, outputs, updates=updates)
  File "/home/taewoo/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1579, in init
  with tf.control_dependencies(self.outputs):
  File "/home/taewoo/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3487, in control_dependencies
  return get_default_graph().control_dependencies(control_inputs)
  File "/home/taewoo/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3218, in control_dependencies
  c = self.as_graph_element(c)
  File "/home/taewoo/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2318, in as_graph_element
  return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/home/taewoo/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2397, in _as_graph_element_locked
  raise ValueError("Tensor %s is not an element of this graph." % obj)
  ValueError: Tensor Tensor("Softmax:0", shape=(?, 361), dtype=float32) is not an element of this graph.

end_to_end.py KerasBot has no attribute run

When running end_to_end.py the last lines with go_model.run() causes an error has kerasBot has not that attribute. The model should be used inside HTTPFrontend with that run() attribute.

Update NumPy API

Whenever I run pip install betago I get an error the error text: #warning "Using deprecated NumPy API, disable it by " \ which shows up many times right before everything freezes.

The API usage should be updated.

Handle training sets that don't fit in memory

I made two attempts at this:

1. Load a model from disk, load a small training set, run a few epochs, and write back to disk. Repeat. This actually worked kinda OK; after a couple hours it reached something like 0.15 training set accuracy, 0.20 validation set accuracy before it seems to top out. (Sorry, I wasn't very scientific and didn't track the exact number of training examples and epochs.)
2. Use fit_generator to train on a chunked data set. I ran a few epochs on 10000 samples (which yields about 1900000 training examples) and the results weren't very exciting; I'm trying to run lots more epochs on a larger set to see what happens.

I pushed some REALLY rough code to a branch in case someone wants to pick it up. (I probably won't have to time to continue on it for a few days)

https://github.com/macfergus/betago/blob/memory-hacks/examples/update.py
https://github.com/macfergus/betago/blob/memory-hacks/betago/dataloader/base_processor.py#L30

Improper handling of captured stones (server)

After you capture an area with the other player's stones (white), the board looks like this:

However, if you click in it, it places a stone but, instead of proceeding normally, you end up playing for white for several moves (and can only play where white is allowed to go)!!!
Traceback always looks something like this:

10.100.1.1 - - [23/Feb/2017 20:49:32] "POST /prediction HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ubuntu/workspace/betago/betago/model.py", line 56, in next_move
    self.bot.apply_move('b', (row, col))
  File "/home/ubuntu/workspace/betago/betago/model.py", line 110, in apply_move
    self.go_board.apply_move(color, move)
  File "/home/ubuntu/workspace/betago/betago/dataloader/goboard.py", line 157, in apply_move
    raise ValueError('Move ' + str(pos) + 'is already on board.')
ValueError: Move (5, 13)is already on board.

Main theory behind

Thank you for sharing first. So in other words, it use only DNN to predict the next move other than AlphaGo, which used also the Monte Carlo Tree Search combined with DNNs.

JS error handling [improvements needed]

I have noticed that if the server responds with any type of error, the client demobot.html freaks out. When this happens, no move is processed for white and the client switches to playing white for one (1) move. At this point, the player can only play in areas that white is allowed to play in, and locally, their moves count for white. However, the moves are still recorded and sent to the server as moves made by black.

There is currently no way to update the board with the server and there is no way (that I know of) to undo the last move (easily). The only way to avoid this issue (at this point) is to hold off actually making the move until the server resolves the move (successfully). Another option is that every time a move is made, a "snapshot" could be recorded before it is processed (and restored if it causes an error).

This lack of error handling is the partial cause for the symptoms described in #29, #26, and #27.

UnicodeDecodeError: 'gbk' codec can't decode... When running run_demo.py

I didn't change anything and simply run the run_demo.py script. And I noticed that when running the Flask app served on localhost:8080, in the static_file_large(path), when the program is openning some *.png or *.jpg files, e.g. black.png, shadow.png, shinaya.jpg, with open("ui/large/" + path).read() statement, this error popped up,
UnicodeDecodeError: 'gbk' codec can't decode byte 0x92 in position 30: illegal multibyte sequence.
I think this open statement only works with those *.js files in the ui/large/ path, not working when reading picture files. Any workaround here?

I tried from PIL import Image and use Image.open() to load the images, but doesn't work.

pip install betago fails with problems with numpy, I gess

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -DH5_USE_16_API -I/tmp/pip-build-stypprco/h5py/lzf -I/opt/local/include -I/usr/local/include -I/usr/local/lib/python3.4/dist-packages/numpy/core/include -I/usr/include/python3.4m -c /tmp/pip-build-stypprco/h5py/h5py/defs.c -o build/temp.linux-x86_64-3.4/tmp/pip-build-stypprco/h5py/h5py/defs.o
In file included from /usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1777:0,
from /usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-build-stypprco/h5py/h5py/api_compat.h:26,
from /tmp/pip-build-stypprco/h5py/h5py/defs.c:303:
/usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
In file included from /tmp/pip-build-stypprco/h5py/h5py/defs.c:303:0:
/tmp/pip-build-stypprco/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Tips for using c9.io

Hi, I was able to get the python code for c9 working but it auto-opens in lynx and I cannot figure out how to play the demo.
Any tips?

Make AJAX asynchronous

A quick look at the source code reveals that the requests to the server are being handled asynchronously. This can lock up the page and is bad for performance (and Karma).

The JS should be rewritten to work asynchronously.

Python 3

Is it Python 3 compatible ?

from betago.processor import SevenPlaneProcessor
processor = SevenPlaneProcessor()
input_channels = processor.num_planes

# Load go data and one-hot encode labels
X, y = processor.load_go_data(num_samples=1000)
X = X.astype('float32')
Y = np_utils.to_categorical(y, nb_classes)

raises

AttributeError                            Traceback (most recent call last)
<ipython-input-4-5682cb2fef10> in <module>()
      4 
      5 # Load go data and one-hot encode labels
----> 6 X, y = processor.load_go_data(num_samples=1000)
      7 X = X.astype('float32')
      8 Y = np_utils.to_categorical(y, nb_classes)

/home/hadim/Insync/Documents/Code/misc/go_game/betago/betago/dataloader/base_processor.py in load_go_data(self, types, data_dir, num_samples)
    119         num_samples: Number of Go games to load.
    120         '''
--> 121         index = KGSIndex(data_directory=self.data_dir)
    122         index.download_files()
    123 

/home/hadim/Insync/Documents/Code/misc/go_game/betago/betago/dataloader/index_processor.py in __init__(self, kgs_url, index_page, data_directory)
     41         self.file_info = []
     42         self.urls = []
---> 43         self.load_index()  # Load index on creation
     44 
     45     def download_files(self):

/home/hadim/Insync/Documents/Code/misc/go_game/betago/betago/dataloader/index_processor.py in load_index(self)
     94         Create the actual index representation from the previously downloaded or cached html.
     95         '''
---> 96         index_contents = self.create_index_page()
     97         split_page = [item for item in index_contents.split('<a href="') if item.startswith("https://")]
     98         for item in split_page:

/home/hadim/Insync/Documents/Code/misc/go_game/betago/betago/dataloader/index_processor.py in create_index_page(self)
     81         else:
     82             print('>>> Downloading index page')
---> 83             fp = urllib.urlopen(self.kgs_url)
     84             data = unicode(fp.read())
     85             fp.close()

AttributeError: module 'urllib' has no attribute 'urlopen'

How can I run it on windows

How can I run it on windows?

Get game data from server [help needed]

I am trying to get various gameplay data from the server in order to use it for the client. The data that I am looking for includes the raw data (which pieces are currently on the board, move history, and the score, komi, etc if available. My priority right now is the raw data so that I can add a sync mechanism to address possible game-state mismatches (#27).

Any help would be greatly appreciated!

No module named flask.ext.cors

When I tried to run python run_demo.py I got the following error (even though I have installed Flask)

Traceback (most recent call last):
  File "run_demo.py", line 7, in <module>
    from betago.model import KerasBot
  File "/home/taylor/Code/betago/betago/model.py", line 3, in <module>
    from flask.ext.cors import CORS, cross_origin
  File "/usr/local/lib/python2.7/dist-packages/flask/exthook.py", line 87, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named flask.ext.cors

Error on run_demo.py

just #52 shows,I changed to those following:
`
@app.route('/large/path:path')
def static_file_large(path):
try:
return open("ui/large/" + path).read()
except:
base_dir =os.getcwd()
print('here %s'%("ui/large/" + path))
image = Image.open(os.path.join(base_dir,"ui","large" , path),mode='r')
imgByteArr = BytesIO()
image.save(imgByteArr,format='PNG')
return imgByteArr
#return send_file(imgByteArr, mimetype="image/jpeg")#"ui/large/" + path
#return send_file("ui/large/" + path, mimetype='image/gif')#Image.open("ui/large/" + path)

`
but when i oepn the browser,there is nothing showed.The sceen is full of blank,what should i do

Better UI: Choose color, komi, handicap, start new game, etc.

error building 'h5py.defs' extension

$ pip install betago console output contains:

running build_ext

Autodetection skipped [libhdf5.so: cannot open shared object file: No such file or directory]

********************************************************************************

                       Summary of the h5py configuration



    Path to HDF5: None

    HDF5 Version: '1.8.4'

     MPI Enabled: False

Rebuild Required: False



********************************************************************************

Executing api_gen rebuild of defs

Executing cythonize()

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/defs.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/_errors.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/_objects.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/_proxy.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5fd.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5z.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5i.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5r.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/utils.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/_conv.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5t.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5s.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5p.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5d.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5a.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5f.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5g.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5l.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5o.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5ds.pyx because it changed.

Compiling /tmp/pip-build-rBcCvv/h5py/h5py/h5ac.pyx because it changed.

[ 1/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/_conv.pyx

[ 2/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/_errors.pyx

[ 3/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/_objects.pyx

[ 4/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/_proxy.pyx

[ 5/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/defs.pyx

[ 6/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5.pyx

[ 7/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5a.pyx

[ 8/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5ac.pyx

[ 9/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5d.pyx

[10/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5ds.pyx

[11/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5f.pyx

[12/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5fd.pyx

[13/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5g.pyx

[14/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5i.pyx

[15/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5l.pyx

[16/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5o.pyx

[17/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5p.pyx

[18/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5r.pyx

[19/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5s.pyx

[20/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5t.pyx

[21/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/h5z.pyx

[22/22] Cythonizing /tmp/pip-build-rBcCvv/h5py/h5py/utils.pyx

building 'h5py.defs' extension

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/tmp

creating build/temp.linux-x86_64-2.7/tmp/pip-build-rBcCvv

creating build/temp.linux-x86_64-2.7/tmp/pip-build-rBcCvv/h5py

creating build/temp.linux-x86_64-2.7/tmp/pip-build-rBcCvv/h5py/h5py

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DH5_USE_16_API -I/tmp/pip-build-rBcCvv/h5py/lzf -I/opt/local/include -I/usr/local/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c /tmp/pip-build-rBcCvv/h5py/h5py/defs.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-rBcCvv/h5py/h5py/defs.o

In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1781:0,

                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:18,

                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,

                 from /tmp/pip-build-rBcCvv/h5py/h5py/api_compat.h:26,

                 from /tmp/pip-build-rBcCvv/h5py/h5py/defs.c:303:

/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

 #warning "Using deprecated NumPy API, disable it by " \

  ^

In file included from /tmp/pip-build-rBcCvv/h5py/h5py/defs.c:303:0:

/tmp/pip-build-rBcCvv/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory

compilation terminated.

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

only Theano backend works

The default installation for Keras now seems to elect TensorFlow instead of Theano. With this setup, any attempt to play a move (POST /prediction) results in an error:

[03/Feb/2017 19:50:44] "POST /prediction HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1994, in __call__
    return self.wsgi_app(environ, start_response)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/betago/.betago/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/mnt/betago/betago/betago/model.py", line 54, in next_move
    bot_row, bot_col = self.bot.select_move('b')
  File "/mnt/betago/betago/betago/model.py", line 110, in select_move
    self._move_generator(bot_color))
  File "/mnt/betago/betago/betago/model.py", line 219, in get_first_valid_move
    for move in move_generator:
  File "/mnt/betago/betago/betago/model.py", line 134, in _model_moves
    pred = np.squeeze(self.model.predict(X))
  File "/mnt/betago/.betago/lib/python2.7/site-packages/keras/models.py", line 724, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/keras/engine/training.py", line 1265, in predict
    self._make_predict_function()
  File "/mnt/betago/.betago/lib/python2.7/site-packages/keras/engine/training.py", line 799, in _make_predict_function
    **kwargs)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1918, in function
    return Function(inputs, outputs, updates=updates)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1876, in __init__
    with tf.control_dependencies(self.outputs):
  File "/mnt/betago/.betago/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3487, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3218, in control_dependencies
    c = self.as_graph_element(c)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2318, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/mnt/betago/.betago/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2397, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("Softmax:0", shape=(?, 361), dtype=float32) is not an element of this graph.

Additionally, tests/model_test.py fails in the same way.

Editing ~/.keras/keras.json to specify the Theano backend works around the problem; perhaps this suggestion should be added to the README for now.

TypeError: 'NoneType' object is not iterable

I noticed that when using the default (demo) instance, near the end of the game, instead of playing valid moves, I get the following error: TypeError: 'NoneType' object is not iterable. This is odd, because there are still areas on the board that can be played by white (but they are not getting played). Once this happens, the client-side ui starts to put down white stones instead of black ones. These sysmptoms are very similar to #26 and #27.

"POST /prediction HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/workspace/.betago/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ubuntu/workspace/betago/betago/model.py", line 61, in next_move
    bot_row, bot_col = self.bot.select_move('w')
TypeError: 'NoneType' object is not iterable

error test : zipfile.BadZipfile: File is not a zip file

I was very interested in AlphaGo, and luckily I found the BetaGo. When I try to understand the working mechanism of the bot, I came across some problems when running some demos in the examples fold, such as "end_to_end.py", "train_and_store.py","train_generator.py". The error is as follows:
"total num games: 179695
Drawn 1000 samples:

Number of zip files: 103
Traceback (most recent call last):
File "train_generator.py", line 40, in
data_generator = processor.load_go_data(num_samples=args.sample_size)
File "/home/sk/.betago/local/lib/python2.7/site-packages/betago/dataloader/base_processor.py", line 132, in load_go_data
self.map_to_workers(name, samples)
File "/home/sk/.betago/local/lib/python2.7/site-packages/betago/dataloader/base_processor.py", line 219, in map_to_workers
results = p.get(0xFFFF)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
zipfile.BadZipfile: File is not a zip file"
Unfortunately, I had difficulty to solve this problem. I would be much grateful if you could give me some advice.

Total number of games used: 0

Hi,
I tried to fetch some game data, using the given code.

from betago.processor import SevenPlaneProcessor
processor = SevenPlaneProcessor()
input_channels = processor.num_planes

X, y = processor.load_go_data(num_samples=1000)

However, the program returned "total number of games used: 0", and then raised this error:

Traceback (most recent call last):
  File "/home/hung/Desktop/test.py", line 6, in <module>
    X, y = processor.load_go_data(num_samples=1000)
  File "/home/hung/tensorflow/local/lib/python2.7/site-packages/betago/dataloader/base_processor.py", line 123, in load_go_data
    sampler = Sampler(data_dir=self.data_dir)
  File "/home/hung/tensorflow/local/lib/python2.7/site-packages/betago/dataloader/sampling.py", line 24, in __init__
    self.compute_test_samples()
  File "/home/hung/tensorflow/local/lib/python2.7/site-packages/betago/dataloader/sampling.py", line 74, in compute_test_samples
    test_games = self.draw_samples(self.num_test_games)
  File "/home/hung/tensorflow/local/lib/python2.7/site-packages/betago/dataloader/sampling.py", line 45, in draw_samples
    sample = random.choice(available_games)
  File "/usr/lib/python2.7/random.py", line 275, in choice
    return seq[int(self.random() * len(seq))]  # raises IndexError if seq is empty
IndexError: list index out of range

Please help!