rohit-dua/bub

BUB : Book Uploader Bot

bub's Issues

Tell what email is used on archive.org

The interface doesn't tell what email address will be used as uploader on archive.org (presumably not mine because you don't have my credentials). It would be useful to know for clarity and to know where to search for the uploads.

Filename truncated too short

I tried to insert "Dissertatio de vnguento armario siue de naturae artisque miraculis" and it gets shortened to "Dissertatio de vnguento armario siue de ". We have 256 bytes, let's use them.

Queue processing often interrupted

Stopped again 12 hours ago, happened several times yesterday. Rohit says (quote from #wikisource, posted without permission):

20.45 < rohit-dua> Nemo_bis: i'm using r.iter_content(1024) on google-books image files. I think since w=2500 is a large image. It makes too many requests, which is caught in some gb throttle, and because of that it is again giving Network unreachable error.
20.46 < rohit-dua> maybe i should have a retry system untill 2-3 times before dropping.

Given http://docs.python-requests.org/en/latest/api/#requests.Response.iter_content I'm not sure what's the best setting, but may be worth trying 512 * 1024 for instance.

Set up bigbrother restarts

As you are already using jstart, are you ready to use bigbrother? https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Bigbrother
That should help getting all the workers continuously running without human supervision and with known-working parameters.

Check for duplicates on archive.org

The ia library can search in the metadata, let's use this before upload to Internet Archive to avoid uploading duplicates. As users increase, we can't be sure they're checking the IDs they submit; maybe the book was already uploaded by tpb or others, under a different title.

We're currently only using some of the Google Books "standard" metadata fields (#9), but the tag is good enough: let's make sure it's always added on upload, with the unique identifier in it, and then look for that identifier in all items before uploading.

Improve Gallica public domain detection

Vigneron reports that http://gallica.bnf.fr/ark:/12148/bpt6k933829z/ isn't identified as being in the public domain. I think

BUB/digi-lib/gal.py

Lines 102 to 107 in e7e0010

	for i in strong_attr:
	public_domain = get_metadata(i, specific_tag = 'Droits')
	if not public_domain:
	continue
	if 'domaine public' in public_domain.encode('utf-8'):
	return 0

is needlessly complicated; now in the HTML I see <meta name="DC.rights" content="domaine public" /> <meta name="DC.rights" content="public domain" />, BUB can just parse that.

exceptions.AttributeError

Maybe we need some try/except here?

  :<type 'exceptions.AttributeError'>, Retry in 4 seconds...--------------------
ERROR: <type 'exceptions.AttributeError'>  func:manager args:(<__main__.QueueHandler object at 0x2b2a828052d0>,){}

Set language metadata field

When uploading to archive.org, we should set the "language" field (takes three-letter ISO code or English name), or the OCR may fail. Does Google provide this information?

https://catalogd.archive.org/log/317109623 was Latin but "setting ORC language to English"

Upload to Internet Archive errors out

2015-06-01 14:20:52.279517  func:upload_to_IA  args:(<__main__.IaWorker object at 0x23a9490>, 'gb', 'qnJQAQAAIAAJ'){}
  :<type 'exceptions.TypeError'>, Retry in 1 seconds...
  :<type 'exceptions.TypeError'>, Retry in 2 seconds...  :<type 'exceptions.TypeError'>, Retry in 4 seconds...  :<type 'exceptions.TypeError'>, Retry in 8 seconds...--------------------
ERROR: <type 'exceptions.TypeError'>  func:upload_to_IA args:(<__main__.IaWorker object at 0x23a9490>, 'gb', 'qnJQAQAAIAAJ'){}

Convert ISO language codes to Library of Congress codes

https://archive.org/post/1021681/bookop-threw-exception-google-language-code-rm-not-found-in-lookup-table-check-for-update :

generally we prefer the library of congress code. you can see them at http://www.loc.gov/marc/languages/language_code.html

For instance, not rm (ISO 639-1) or roh (ISO 639-3), but roa.
Sounds quite silly to degrade standard codes to something non-standard, but whatever. :-) If the full English names are the same for ISO/Unicode and LOC, maybe it's easier to use those.

Align Google attribution to tpb's

It would be nice to follow tpb's model in metadata. For instance: https://ia600400.us.archive.org/23/items/southwesternrep06compgoog/southwesternrep06compgoog_meta.xml

<google-id>RF8LAAAAYAAJ</google-id>
<scanner>google</scanner>
<source>http://books.google.com/books?id=RF8LAAAAYAAJ&oe=UTF-8</source>
<sponsor>Google</sponsor>

Should be trivial I hope.

Preserve a copy of all book metadata by the Google Books API

It would potentially be nice to save any metadata that we don't convert to archive.org metadata, simply by storing the json on the archive.org item. It would come handy if the original book was removed and additional info were needed, for instance the library of origin of the book etc.

However I'm filing this just for the records: we can't, because it's clearly forbidden by the API terms of service:

Prohibitions on Content. Unless expressly permitted by the content owner or by applicable law, you agree that you will not, and will not permit your end users to, do the following with content returned from the APIs:
Scrape, build databases or otherwise create permanent copies of such content, or keep cached copies longer than permitted by the cache header;

Declare expected time

After how long should I give up? Just saying "wait for an email" is insufficient for impatient wikimedians, it would be useful to know if it normally takes minutes, hours or days.

Cleanup of tar files which were not uploaded/deleted

We have some tar files which don't get cleaned up. I'm not sure how to check what is in the redis queue.

$ ls -lht /data/scratch/BUB_downloads/*tar*
-rw-r--r-- 1 tools.bub tools.bub  28M Feb 14 15:03 /data/scratch/BUB_downloads/bub_gal_ark_12148_bpt6k80993x_images.tar
-rw-r--r-- 1 tools.bub tools.bub 8.8M Feb 14 14:54 /data/scratch/BUB_downloads/bub_gal_ark_12148_bpt6k375846s_images.tar
-rw-r--r-- 1 tools.bub tools.bub 417M Feb 14 14:31 /data/scratch/BUB_downloads/bub_gb_c9SIbcxGToMC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 128M Feb 14 14:14 /data/scratch/BUB_downloads/bub_gb_c9r_bNd_C_4C_images.tar
-rw-r--r-- 1 tools.bub tools.bub 5.0M Feb 14 11:05 /data/scratch/BUB_downloads/bub_gb_C9Pgt1EPWbwC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  76M Feb 10 01:57 /data/scratch/BUB_downloads/bub_gb_Nu8pAAAAYAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 105M Feb  9 01:03 /data/scratch/BUB_downloads/bub_gb_bM25rYaSBxYC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 200M Feb  8 08:53 /data/scratch/BUB_downloads/bub_gb_BjMKAwDKwAMC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 131M Feb  7 22:48 /data/scratch/BUB_downloads/bub_gb_PokUAAAAYAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 160M Feb  7 22:45 /data/scratch/BUB_downloads/bub_gb_bIC6hga9NnUC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  62M Feb  6 16:27 /data/scratch/BUB_downloads/bub_gb_B7svnAknY3gC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  53M Feb  6 16:26 /data/scratch/BUB_downloads/bub_gb_b7q3O2VIy98C_images.tar
-rw-r--r-- 1 tools.bub tools.bub 7.9M Feb  6 16:23 /data/scratch/BUB_downloads/bub_gb_b7nF8S6BmosC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 182M Feb  5 08:16 /data/scratch/BUB_downloads/bub_gb_aXNa4u56zBgC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 457M Feb  4 23:22 /data/scratch/BUB_downloads/bub_gb_u6pEAAAAcAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 154M Feb  3 02:06 /data/scratch/BUB_downloads/bub_gb_AQnLJHyv4c8C_images.tar
-rw-r--r-- 1 tools.bub tools.bub  14M Feb  2 18:13 /data/scratch/BUB_downloads/bub_gb_trJBAAAAcAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub  86M Feb  2 18:09 /data/scratch/BUB_downloads/bub_gb_FtFAAQAAMAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 103M Feb  2 18:05 /data/scratch/BUB_downloads/bub_gb_aPmkoYrG6WQC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 286M Feb  2 18:02 /data/scratch/BUB_downloads/bub_gb_trIpAQAAMAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 129M Feb  2 12:24 /data/scratch/BUB_downloads/bub_gb_aOvO8o1vxyIC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 3.5M Feb  2 12:18 /data/scratch/BUB_downloads/bub_gb_aOVc1tWFatgC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 201M Feb  2 12:18 /data/scratch/BUB_downloads/bub_gb_tpYTAAAAQAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub  18M Feb  2 12:17 /data/scratch/BUB_downloads/bub_gb_AOuTrLs-eS4C_images.tar
-rw-r--r-- 1 tools.bub tools.bub 152M Feb  1 19:28 /data/scratch/BUB_downloads/bub_gb_am_GLeB7lXMC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  19M Feb  1 19:26 /data/scratch/BUB_downloads/bub_gb_tjYyAQAAMAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 138M Feb  1 19:25 /data/scratch/BUB_downloads/bub_gb_tjYsAAAAYAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 5.4M Feb  1 19:23 /data/scratch/BUB_downloads/bub_gb_AmfxS0CFIyAC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 146M Feb  1 19:22 /data/scratch/BUB_downloads/bub_gb_AMFVomPaOlsC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  73M Feb  1 19:20 /data/scratch/BUB_downloads/bub_gb_tjYYAAAAYAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 141M Feb  1 19:17 /data/scratch/BUB_downloads/bub_gb_tjVMAQAAIAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 180M Feb  1 19:16 /data/scratch/BUB_downloads/bub_gb_AMfpVKxANl8C_images.tar
-rw-r--r-- 1 tools.bub tools.bub 155M Feb  1 15:21 /data/scratch/BUB_downloads/bub_gb_Alxu4AhT0lcC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 412M Jan 29 23:35 /data/scratch/BUB_downloads/bub_gb_acUprmtsMGcC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 278M Jan 29 15:53 /data/scratch/BUB_downloads/bub_gb_bF8-AQAAMAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 114M Jan 27 21:35 /data/scratch/BUB_downloads/bub_gb_t70UAAAAYAAJ_images.tar
-rw-r--r-- 1 tools.bub tools.bub 439M Jan 27 21:35 /data/scratch/BUB_downloads/bub_gb_A1lCoy8JHkUC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 165M Jan 26 08:37 /data/scratch/BUB_downloads/bub_gb_9S4qXE7GU-UC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 540M Jan 25 22:11 /data/scratch/BUB_downloads/bub_gb_9PecqfzyzVMC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  88M Jan 24 23:05 /data/scratch/BUB_downloads/bub_gb_9Htph5DDY58C_images.tar
-rw-r--r-- 1 tools.bub tools.bub 536M Jan 24 21:58 /data/scratch/BUB_downloads/bub_gb_-9gOuMLrSFMC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  27M Jan 24 11:15 /data/scratch/BUB_downloads/bub_gb_97lJnk6si_wC_images.tar
-rw-r--r-- 1 tools.bub tools.bub 187M Jan 23 02:53 /data/scratch/BUB_downloads/bub_gb_8kDERUFC4o4C_images.tar
-rw-r--r-- 1 tools.bub tools.bub 621M Jan 22 21:10 /data/scratch/BUB_downloads/bub_gb_8In_BgYkO0MC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  90M Jan 22 19:44 /data/scratch/BUB_downloads/bub_gb_8icca7yTZwIC_images.tar
-rw-r--r-- 1 tools.bub tools.bub  23M Jan 22 19:37 /data/scratch/BUB_downloads/bub_gb_8ibOuCI-yusC_images.tar

I checked two books of those:

• one gives 404 with the API http://googleapis.com/books/v1/volumes/tpYTAAAAQAAJ
• the other is available and PD https://www.googleapis.com/books/v1/volumes/am_GLeB7lXMC but was already uploaded in 2014 https://archive.org/details/bub_gb_am_GLeB7lXMC , weird that it got queued in the first place.

So I don't have any reason to think there is something (seriously) going wrong yet.

150 ppi is too little

I just uploaded https://archive.org/details/bub_gb_WpFZQBLDA2YC and it's fantastic! Thanks for setting ppi, it avoids some big headaches in rare cases; but 150 ppi seems unnecessarily low, it compressed the PDF almost tenfold. I think 300 ppi is more realistic.

Include OUDL

I here by request you to add Osmania University digital library (http://oudl.osmania.ac.in) to bub tool available at https://tools.wmflabs.org/bub/

This library has got quite a lot of PDFs for wide number of Indic languages. I have 2400 books for Kannada alone.

We would like to upload the out of copyright books to internet archive, push it to commons and use OCR tool to take digitization to next level.

Looking forward for a quick help in this regard.

bub is down past few days

bub is down past few days.

Include DLI and WBPLN

Please include Digital Library of India (DLI) and West Bengal Public Library Network (WBPLN) in the tool.
West Bengal Public Library Network link : http://dspace.wbpublibnet.gov.in:8080/jspui/
and
The Digital Library of India links are:

1. www.dli.ernet.in/
2. www.dli.serc.iisc.ernet.in/
3. www.dli.gov.in/

Authors sometimes leaking into subject field

Author: Burckhardt, Albert, 1854-1911; Freunde vaterländischer Geschichte, Basel
Subject: bub_upload; Joris, David, 1501 or 1502-

https://archive.org/details/bub_gb_J7pDAAAAYAAJ

From timestamp it seems it was like that in the metadata submitted on upload, rather than added by the deriver (which is still working).

Proposal:Upload at IA should done by user's accounts

Proposal:Upload at IA should done by user's account during through Wildard entry, So the use may later change meta data as necessary.

Bot sometime crash with Gallica books

Importing books from Gallica make sometime crash the bot with a ''400 - Bad Request'' error.

Some books who have crashed :
http://gallica.bnf.fr/ark:/12148/bpt6k857503f
http://gallica.bnf.fr/ark:/12148/bpt6k57483432

Install missing BUB dependencies

Looks like a number of dependencies went missing. pip isn't available, either. Hopefully I installed them all locally:

tools.bub@tools-bastion-01:~/public_html/BUB$ easy_install --user `cat requirements.txt` 
Searching for args==0.1.0
Best match: args 0.1.0
Processing args-0.1.0-py2.7.egg
args 0.1.0 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/args-0.1.0-py2.7.egg
Processing dependencies for args==0.1.0
Finished processing dependencies for args==0.1.0
Searching for Babel==1.3
Best match: Babel 1.3
Adding Babel 1.3 to easy-install.pth file
Installing pybabel script to /data/project/bub/.local/bin

Using /usr/lib/python2.7/dist-packages
Processing dependencies for Babel==1.3
Finished processing dependencies for Babel==1.3
Searching for BeautifulSoup==3.2.1
Best match: BeautifulSoup 3.2.1
BeautifulSoup 3.2.1 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for BeautifulSoup==3.2.1
Finished processing dependencies for BeautifulSoup==3.2.1
Searching for beautifulsoup4==4.3.2
Reading https://pypi.python.org/simple/beautifulsoup4/
Best match: beautifulsoup4 4.3.2
Downloading https://pypi.python.org/packages/source/b/beautifulsoup4/beautifulsoup4-4.3.2.tar.gz#md5=b8d157a204d56512a4cc196e53e7d8ee
Processing beautifulsoup4-4.3.2.tar.gz
Writing /tmp/easy_install-WALC7l/beautifulsoup4-4.3.2/setup.cfg
Running beautifulsoup4-4.3.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-WALC7l/beautifulsoup4-4.3.2/egg-dist-tmp-1TrYDW
zip_safe flag not set; analyzing archive contents...
Adding beautifulsoup4 4.3.2 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/beautifulsoup4-4.3.2-py2.7.egg
Processing dependencies for beautifulsoup4==4.3.2
Finished processing dependencies for beautifulsoup4==4.3.2
Searching for bitarray==0.8.1
Reading https://pypi.python.org/simple/bitarray/
Best match: bitarray 0.8.1
Downloading https://pypi.python.org/packages/source/b/bitarray/bitarray-0.8.1.tar.gz#md5=3825184f54f4d93508a28031b4c65d3b
Processing bitarray-0.8.1.tar.gz
Writing /tmp/easy_install-3Th_Fe/bitarray-0.8.1/setup.cfg
Running bitarray-0.8.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3Th_Fe/bitarray-0.8.1/egg-dist-tmp-wBXcDt
zip_safe flag not set; analyzing archive contents...
bitarray.test_bitarray: module references __file__
Adding bitarray 0.8.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/bitarray-0.8.1-py2.7-linux-x86_64.egg
Processing dependencies for bitarray==0.8.1
Finished processing dependencies for bitarray==0.8.1
Searching for blinker==1.3
Reading https://pypi.python.org/simple/blinker/
Best match: blinker 1.3
Downloading https://pypi.python.org/packages/source/b/blinker/blinker-1.3.tar.gz#md5=66e9688f2d287593a0e698cd8a5fbc57
Processing blinker-1.3.tar.gz
Writing /tmp/easy_install-U9egrB/blinker-1.3/setup.cfg
Running blinker-1.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-U9egrB/blinker-1.3/egg-dist-tmp-rNwijL
zip_safe flag not set; analyzing archive contents...
Adding blinker 1.3 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/blinker-1.3-py2.7.egg
Processing dependencies for blinker==1.3
Finished processing dependencies for blinker==1.3
Searching for clint==0.3.3
Best match: clint 0.3.3
Processing clint-0.3.3-py2.7.egg
clint 0.3.3 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/clint-0.3.3-py2.7.egg
Processing dependencies for clint==0.3.3
Finished processing dependencies for clint==0.3.3
Searching for cssmin==0.2.0
Reading https://pypi.python.org/simple/cssmin/
Best match: cssmin 0.2.0
Downloading https://pypi.python.org/packages/source/c/cssmin/cssmin-0.2.0.tar.gz#md5=c2798658a4f69663365a3e70c3b8250b
Processing cssmin-0.2.0.tar.gz
Writing /tmp/easy_install-Sg1qfX/cssmin-0.2.0/setup.cfg
Running cssmin-0.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Sg1qfX/cssmin-0.2.0/egg-dist-tmp-j2qCxI
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
zip_safe flag not set; analyzing archive contents...
Adding cssmin 0.2.0 to easy-install.pth file
Installing cssmin script to /data/project/bub/.local/bin

Installed /data/project/bub/.local/lib/python2.7/site-packages/cssmin-0.2.0-py2.7.egg
Processing dependencies for cssmin==0.2.0
Finished processing dependencies for cssmin==0.2.0
Searching for DateTime==4.0.1
Reading https://pypi.python.org/simple/DateTime/
Best match: DateTime 4.0.1
Downloading https://pypi.python.org/packages/source/D/DateTime/DateTime-4.0.1.zip#md5=2241f4a18a7cf16f1d6e8e324fca0ef0
Processing DateTime-4.0.1.zip
Writing /tmp/easy_install-2xjeKi/DateTime-4.0.1/setup.cfg
Running DateTime-4.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-2xjeKi/DateTime-4.0.1/egg-dist-tmp-wLNn4t
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
Adding DateTime 4.0.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/DateTime-4.0.1-py2.7.egg
Processing dependencies for DateTime==4.0.1
Finished processing dependencies for DateTime==4.0.1
Searching for decorator==3.4.0
Best match: decorator 3.4.0
decorator 3.4.0 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for decorator==3.4.0
Finished processing dependencies for decorator==3.4.0
Searching for docopt==0.6.1
Best match: docopt 0.6.1
Processing docopt-0.6.1-py2.7.egg
docopt 0.6.1 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/docopt-0.6.1-py2.7.egg
Processing dependencies for docopt==0.6.1
Finished processing dependencies for docopt==0.6.1
Searching for drench==0.0.13
Reading https://pypi.python.org/simple/drench/
Best match: drench 0.0.13
Downloading https://pypi.python.org/packages/2.7/d/drench/drench-0.0.13-py2.7.egg#md5=f65756eb1f1f2148413b89f7e8142aee
Processing drench-0.0.13-py2.7.egg
Moving drench-0.0.13-py2.7.egg to /data/project/bub/.local/lib/python2.7/site-packages
Adding drench 0.0.13 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/drench-0.0.13-py2.7.egg
Processing dependencies for drench==0.0.13
Finished processing dependencies for drench==0.0.13
Searching for Flask==0.10.1
Best match: Flask 0.10.1
Flask 0.10.1 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for Flask==0.10.1
Finished processing dependencies for Flask==0.10.1
Searching for Flask-Babel==0.9
Reading https://pypi.python.org/simple/Flask-Babel/
Best match: Flask-Babel 0.9
Downloading https://pypi.python.org/packages/source/F/Flask-Babel/Flask-Babel-0.9.tar.gz#md5=4762e0392303f464d53cbebedfb87ded
Processing Flask-Babel-0.9.tar.gz
Writing /tmp/easy_install-iBz1MC/Flask-Babel-0.9/setup.cfg
Running Flask-Babel-0.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iBz1MC/Flask-Babel-0.9/egg-dist-tmp-ajtBSR
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Adding Flask-Babel 0.9 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_Babel-0.9-py2.7.egg
Processing dependencies for Flask-Babel==0.9
Searching for speaklater>=1.2
Reading https://pypi.python.org/simple/speaklater/
Best match: speaklater 1.3
Downloading https://pypi.python.org/packages/source/s/speaklater/speaklater-1.3.tar.gz#md5=e8d5dbe36e53d5a35cff227e795e8bbf
Processing speaklater-1.3.tar.gz
Writing /tmp/easy_install-GZ9ypN/speaklater-1.3/setup.cfg
Running speaklater-1.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GZ9ypN/speaklater-1.3/egg-dist-tmp-IvcW0h
Adding speaklater 1.3 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/speaklater-1.3-py2.7.egg
Finished processing dependencies for Flask-Babel==0.9
Searching for Flask-Login==0.2.11
Reading https://pypi.python.org/simple/Flask-Login/
Best match: Flask-Login 0.2.11
Downloading https://pypi.python.org/packages/source/F/Flask-Login/Flask-Login-0.2.11.tar.gz#md5=c0a7eaf28623f0aeac4929dc05a7a064
Processing Flask-Login-0.2.11.tar.gz
Writing /tmp/easy_install-QcMGTv/Flask-Login-0.2.11/setup.cfg
Running Flask-Login-0.2.11/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QcMGTv/Flask-Login-0.2.11/egg-dist-tmp-NOb0tr
Adding Flask-Login 0.2.11 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_Login-0.2.11-py2.7.egg
Processing dependencies for Flask-Login==0.2.11
Finished processing dependencies for Flask-Login==0.2.11
Searching for Flask-Mail==0.7.6
Reading https://pypi.python.org/simple/Flask-Mail/
Best match: Flask-Mail 0.7.6
Downloading https://pypi.python.org/packages/source/F/Flask-Mail/Flask-Mail-0.7.6.tar.gz#md5=81a28b4965af72a9977a4a875ca6e068
Processing Flask-Mail-0.7.6.tar.gz
Writing /tmp/easy_install-4zw9vB/Flask-Mail-0.7.6/setup.cfg
Running Flask-Mail-0.7.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-4zw9vB/Flask-Mail-0.7.6/egg-dist-tmp-YqDb6T
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Adding Flask-Mail 0.7.6 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_Mail-0.7.6-py2.7.egg
Processing dependencies for Flask-Mail==0.7.6
Finished processing dependencies for Flask-Mail==0.7.6
Searching for Flask-OpenID==1.2.1
Reading https://pypi.python.org/simple/Flask-OpenID/
Best match: Flask-OpenID 1.2.1
Downloading https://pypi.python.org/packages/source/F/Flask-OpenID/Flask-OpenID-1.2.1.tar.gz#md5=8044847ca72003da3d3e41a677341b4f
Processing Flask-OpenID-1.2.1.tar.gz
Writing /tmp/easy_install-U0YVdV/Flask-OpenID-1.2.1/setup.cfg
Running Flask-OpenID-1.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-U0YVdV/Flask-OpenID-1.2.1/egg-dist-tmp-PCzUJj
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'example'
warning: no previously-included files matching '*.pyo' found under directory 'example'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Adding Flask-OpenID 1.2.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_OpenID-1.2.1-py2.7.egg
Processing dependencies for Flask-OpenID==1.2.1
Searching for python-openid>=2.0
Reading https://pypi.python.org/simple/python-openid/
Best match: python-openid 2.2.5
Downloading https://pypi.python.org/packages/source/p/python-openid/python-openid-2.2.5.zip#md5=f89d9d4f4dccfd33b5ce34eb4725f751
Processing python-openid-2.2.5.zip
Writing /tmp/easy_install-PE7iZJ/python-openid-2.2.5/setup.cfg
Running python-openid-2.2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-PE7iZJ/python-openid-2.2.5/egg-dist-tmp-j4tKnS
warning: no files found matching 'CHANGELOG'
warning: no files found matching '*.css' under directory 'doc'
warning: no files found matching '*.html' under directory 'doc'
zip_safe flag not set; analyzing archive contents...
Adding python-openid 2.2.5 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/python_openid-2.2.5-py2.7.egg
Finished processing dependencies for Flask-OpenID==1.2.1
Searching for Flask-SQLAlchemy==0.16
Reading https://pypi.python.org/simple/Flask-SQLAlchemy/
Best match: Flask-SQLAlchemy 0.16
Downloading https://pypi.python.org/packages/source/F/Flask-SQLAlchemy/Flask-SQLAlchemy-0.16.tar.gz#md5=9de0ddfa045a47b9f7e7084cddb68cbe
Processing Flask-SQLAlchemy-0.16.tar.gz
Writing /tmp/easy_install-_CE4nU/Flask-SQLAlchemy-0.16/setup.cfg
Running Flask-SQLAlchemy-0.16/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_CE4nU/Flask-SQLAlchemy-0.16/egg-dist-tmp-KDM4MU
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Adding Flask-SQLAlchemy 0.16 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_SQLAlchemy-0.16-py2.7.egg
Processing dependencies for Flask-SQLAlchemy==0.16
Finished processing dependencies for Flask-SQLAlchemy==0.16
Searching for Flask-WhooshAlchemy==0.55a0
Reading https://pypi.python.org/simple/Flask-WhooshAlchemy/
Best match: Flask-WhooshAlchemy 0.55a
Downloading https://pypi.python.org/packages/source/F/Flask-WhooshAlchemy/Flask-WhooshAlchemy-0.55a.tar.gz#md5=a081eeff4450f3ddeedbbca540847f74
Processing Flask-WhooshAlchemy-0.55a.tar.gz
Writing /tmp/easy_install-xRDD9q/Flask-WhooshAlchemy-0.55a/setup.cfg
Running Flask-WhooshAlchemy-0.55a/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xRDD9q/Flask-WhooshAlchemy-0.55a/egg-dist-tmp-8ubpG4
Adding Flask-WhooshAlchemy 0.55a to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_WhooshAlchemy-0.55a-py2.7.egg
Processing dependencies for Flask-WhooshAlchemy==0.55a0
Searching for Whoosh
Reading https://pypi.python.org/simple/Whoosh/
Best match: Whoosh 2.7.0
Downloading https://pypi.python.org/packages/source/W/Whoosh/Whoosh-2.7.0.zip#md5=7abfd970f16fadc7311960f3fa0bc7a9
Processing Whoosh-2.7.0.zip
Writing /tmp/easy_install-3o5NvD/Whoosh-2.7.0/setup.cfg
Running Whoosh-2.7.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3o5NvD/Whoosh-2.7.0/egg-dist-tmp-KJ58e3
warning: no files found matching '*.txt' under directory 'tests'
warning: no files found matching '*.txt' under directory 'benchmark'
warning: no files found matching '*.txt' under directory 'docs'
warning: no files found matching '*.txt' under directory 'files'
warning: no files found matching '*.py' under directory 'files'
warning: no files found matching '*.jpg' under directory 'files'
Adding Whoosh 2.7.0 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Whoosh-2.7.0-py2.7.egg
Finished processing dependencies for Flask-WhooshAlchemy==0.55a0
Searching for Flask-WTF==0.10.0
Reading https://pypi.python.org/simple/Flask-WTF/
Best match: Flask-WTF 0.10.0
Downloading https://pypi.python.org/packages/source/F/Flask-WTF/Flask-WTF-0.10.0.tar.gz#md5=b9191796dd7153e72f685d6353b1d708
Processing Flask-WTF-0.10.0.tar.gz
Writing /tmp/easy_install-cwQLdR/Flask-WTF-0.10.0/setup.cfg
Running Flask-WTF-0.10.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-cwQLdR/Flask-WTF-0.10.0/egg-dist-tmp-YZ1OyK
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Adding Flask-WTF 0.10.0 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Flask_WTF-0.10.0-py2.7.egg
Processing dependencies for Flask-WTF==0.10.0
Searching for WTForms>=1.0.5
Reading https://pypi.python.org/simple/WTForms/
Best match: WTForms 2.0.2
Downloading https://pypi.python.org/packages/source/W/WTForms/WTForms-2.0.2.zip#md5=613cf723ab40537705bec02733c78d95
Processing WTForms-2.0.2.zip
Writing /tmp/easy_install-XdkqL3/WTForms-2.0.2/setup.cfg
Running WTForms-2.0.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-XdkqL3/WTForms-2.0.2/egg-dist-tmp-rmxAQ4
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
zip_safe flag not set; analyzing archive contents...
wtforms.i18n: module references __file__
Adding WTForms 2.0.2 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/WTForms-2.0.2-py2.7.egg
Finished processing dependencies for Flask-WTF==0.10.0
Searching for flup==1.0.2
Best match: flup 1.0.2
flup 1.0.2 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for flup==1.0.2
Finished processing dependencies for flup==1.0.2
Searching for hashlib==20081119
Reading https://pypi.python.org/simple/hashlib/
Reading http://code.krypto.org/python/hashlib/
Best match: hashlib 20081119
Downloading https://pypi.python.org/packages/source/h/hashlib/hashlib-20081119.zip#md5=46e59db6d5cac34b3344ef0c30d0897f
Processing hashlib-20081119.zip
Writing /tmp/easy_install-Msdr31/hashlib-20081119/setup.cfg
Running hashlib-20081119/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Msdr31/hashlib-20081119/egg-dist-tmp-mZC3t0
no previously-included directories found matching 'build'
no previously-included directories found matching 'dist'
no previously-included directories found matching 'RCS'
no previously-included directories found matching 'CVS'
no previously-included directories found matching '.svn'
zip_safe flag not set; analyzing archive contents...
Adding hashlib 20081119 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/hashlib-20081119-py2.7-linux-x86_64.egg
Processing dependencies for hashlib==20081119
Finished processing dependencies for hashlib==20081119
Searching for htmlmin==0.1.5
Reading https://pypi.python.org/simple/htmlmin/
Best match: htmlmin 0.1.5
Downloading https://pypi.python.org/packages/2.7/h/htmlmin/htmlmin-0.1.5-py2.7.egg#md5=2d61442036bd39cf3176a765621f188f
Processing htmlmin-0.1.5-py2.7.egg
Moving htmlmin-0.1.5-py2.7.egg to /data/project/bub/.local/lib/python2.7/site-packages
Adding htmlmin 0.1.5 to easy-install.pth file
Installing htmlmin script to /data/project/bub/.local/bin

Installed /data/project/bub/.local/lib/python2.7/site-packages/htmlmin-0.1.5-py2.7.egg
Processing dependencies for htmlmin==0.1.5
Finished processing dependencies for htmlmin==0.1.5
Searching for internetarchive==0.6.9
Reading https://pypi.python.org/simple/internetarchive/
Best match: internetarchive 0.6.9
Downloading https://pypi.python.org/packages/source/i/internetarchive/internetarchive-0.6.9.tar.gz#md5=aef86811fd6ad6d45613e43dd6c16389
Processing internetarchive-0.6.9.tar.gz
Writing /tmp/easy_install-gwUR_g/internetarchive-0.6.9/setup.cfg
Running internetarchive-0.6.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gwUR_g/internetarchive-0.6.9/egg-dist-tmp-HYFYuD
zip_safe flag not set; analyzing archive contents...
Removing internetarchive 0.8.3 from easy-install.pth file
Adding internetarchive 0.6.9 to easy-install.pth file
Installing ia script to /data/project/bub/.local/bin

Installed /data/project/bub/.local/lib/python2.7/site-packages/internetarchive-0.6.9-py2.7.egg
Processing dependencies for internetarchive==0.6.9
Searching for requests==2.2.0
Reading https://pypi.python.org/simple/requests/
Best match: requests 2.2.0
Downloading https://pypi.python.org/packages/source/r/requests/requests-2.2.0.tar.gz#md5=4d2e17221d478ece045e2e81cdb177f5
Processing requests-2.2.0.tar.gz
Writing /tmp/easy_install-cG3lR2/requests-2.2.0/setup.cfg
Running requests-2.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-cG3lR2/requests-2.2.0/egg-dist-tmp-SwDLmT
Removing requests 2.3.0 from easy-install.pth file
Adding requests 2.2.0 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/requests-2.2.0-py2.7.egg
Finished processing dependencies for internetarchive==0.6.9
Searching for itsdangerous==0.24
Reading https://pypi.python.org/simple/itsdangerous/
Best match: itsdangerous 0.24
Downloading https://pypi.python.org/packages/source/i/itsdangerous/itsdangerous-0.24.tar.gz#md5=a3d55aa79369aef5345c036a8a26307f
Processing itsdangerous-0.24.tar.gz
Writing /tmp/easy_install-tXQSsU/itsdangerous-0.24/setup.cfg
Running itsdangerous-0.24/setup.py -q bdist_egg --dist-dir /tmp/easy_install-tXQSsU/itsdangerous-0.24/egg-dist-tmp-ZSrf9z
warning: no previously-included files matching '*' found under directory 'docs/_build'
Adding itsdangerous 0.24 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg
Processing dependencies for itsdangerous==0.24
Finished processing dependencies for itsdangerous==0.24
Searching for Jinja2==2.7.3
Reading https://pypi.python.org/simple/Jinja2/
Best match: Jinja2 2.7.3
Downloading https://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.7.3.tar.gz#md5=b9dffd2f3b43d673802fe857c8445b1a
Processing Jinja2-2.7.3.tar.gz
Writing /tmp/easy_install-0vKXsO/Jinja2-2.7.3/setup.cfg
Running Jinja2-2.7.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0vKXsO/Jinja2-2.7.3/egg-dist-tmp-Y6d8gP
warning: no files found matching '*' under directory 'custom_fixers'
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
Adding Jinja2 2.7.3 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Jinja2-2.7.3-py2.7.egg
Processing dependencies for Jinja2==2.7.3
Finished processing dependencies for Jinja2==2.7.3
Searching for jsmin==2.0.11
Reading https://pypi.python.org/simple/jsmin/
Best match: jsmin 2.0.11
Downloading https://pypi.python.org/packages/source/j/jsmin/jsmin-2.0.11.tar.gz#md5=dc0cebc0cd488238b3e92b64c7d54abf
Processing jsmin-2.0.11.tar.gz
Writing /tmp/easy_install-awF8Cp/jsmin-2.0.11/setup.cfg
Running jsmin-2.0.11/setup.py -q bdist_egg --dist-dir /tmp/easy_install-awF8Cp/jsmin-2.0.11/egg-dist-tmp-6hCCDR
zip_safe flag not set; analyzing archive contents...
Adding jsmin 2.0.11 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/jsmin-2.0.11-py2.7.egg
Processing dependencies for jsmin==2.0.11
Finished processing dependencies for jsmin==2.0.11
Searching for jsonpatch==0.4
Best match: jsonpatch 0.4
Processing jsonpatch-0.4-py2.7.egg
jsonpatch 0.4 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/jsonpatch-0.4-py2.7.egg
Processing dependencies for jsonpatch==0.4
Finished processing dependencies for jsonpatch==0.4
Searching for MarkupSafe==0.23
Reading https://pypi.python.org/simple/MarkupSafe/
Best match: MarkupSafe 0.23
Downloading https://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.23.tar.gz#md5=f5ab3deee4c37cd6a922fb81e730da6e
Processing MarkupSafe-0.23.tar.gz
Writing /tmp/easy_install-DjIf4V/MarkupSafe-0.23/setup.cfg
Running MarkupSafe-0.23/setup.py -q bdist_egg --dist-dir /tmp/easy_install-DjIf4V/MarkupSafe-0.23/egg-dist-tmp-ls7M9C
Adding MarkupSafe 0.23 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg
Processing dependencies for MarkupSafe==0.23
Finished processing dependencies for MarkupSafe==0.23
Searching for MySQL-python==1.2.5
Reading https://pypi.python.org/simple/MySQL-python/
Best match: MySQL-python 1.2.5
Downloading https://pypi.python.org/packages/source/M/MySQL-python/MySQL-python-1.2.5.zip#md5=654f75b302db6ed8dc5a898c625e030c
Processing MySQL-python-1.2.5.zip
Writing /tmp/easy_install-1Oh1Sk/MySQL-python-1.2.5/setup.cfg
Running MySQL-python-1.2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-1Oh1Sk/MySQL-python-1.2.5/egg-dist-tmp-avhQbo
In file included from _mysql.c:44:0:
/usr/include/mysql/my_config.h:422:0: warning: "HAVE_WCSCOLL" redefined [enabled by default]
 #define HAVE_WCSCOLL
 ^
In file included from /usr/include/python2.7/pyconfig.h:3:0,
                 from /usr/include/python2.7/Python.h:8,
                 from _mysql.c:29:
/usr/include/x86_64-linux-gnu/python2.7/pyconfig.h:911:0: note: this is the location of the previous definition
 #define HAVE_WCSCOLL 1
 ^
zip_safe flag not set; analyzing archive contents...
Adding MySQL-python 1.2.5 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg
Processing dependencies for MySQL-python==1.2.5
Finished processing dependencies for MySQL-python==1.2.5
Searching for oauthlib==0.6.3
Reading https://pypi.python.org/simple/oauthlib/
Best match: oauthlib 0.6.3
Downloading https://pypi.python.org/packages/source/o/oauthlib/oauthlib-0.6.3.tar.gz#md5=3ecba28f991ee58db2aeda8b4e2b56b1
Processing oauthlib-0.6.3.tar.gz
Writing /tmp/easy_install-Wh5Hrs/oauthlib-0.6.3/setup.cfg
Running oauthlib-0.6.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Wh5Hrs/oauthlib-0.6.3/egg-dist-tmp-yUhomF
zip_safe flag not set; analyzing archive contents...
Adding oauthlib 0.6.3 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/oauthlib-0.6.3-py2.7.egg
Processing dependencies for oauthlib==0.6.3
Finished processing dependencies for oauthlib==0.6.3
Searching for pbr==0.10.0
Reading https://pypi.python.org/simple/pbr/
Best match: pbr 0.10.0
Downloading https://pypi.python.org/packages/source/p/pbr/pbr-0.10.0.tar.gz#md5=9e02dbfb5e49210c381fd4eea00cf7b7
Processing pbr-0.10.0.tar.gz
Writing /tmp/easy_install-7kZ25l/pbr-0.10.0/setup.cfg
Running pbr-0.10.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-7kZ25l/pbr-0.10.0/egg-dist-tmp-4Kjx40
Adding pbr 0.10.0 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/pbr-0.10.0-py2.7.egg
Processing dependencies for pbr==0.10.0
Finished processing dependencies for pbr==0.10.0
Searching for Pillow==2.8.1
Reading https://pypi.python.org/simple/Pillow/
Best match: Pillow 2.8.1
Downloading https://pypi.python.org/packages/source/P/Pillow/Pillow-2.8.1.zip#md5=c9596d3676800ade486463120a0c2e18
Processing Pillow-2.8.1.zip
Writing /tmp/easy_install-lPoouG/Pillow-2.8.1/setup.cfg
Running Pillow-2.8.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-lPoouG/Pillow-2.8.1/egg-dist-tmp-czLfM5
warning: no files found matching 'LICENSE' under directory 'docs'
Building using 4 processes
libImaging/Resample.c:87:45: warning: always_inline function might not be inlinable [-Wattributes]
 static float __attribute__((always_inline)) i2f(int v) {
                                             ^
Building using 4 processes
Building using 4 processes
Building using 4 processes
Building using 4 processes
Building using 4 processes
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 2.8.1
platform     linux2 2.7.6 (default, Mar 22 2014, 22:59:56)
             [GCC 4.8.2]
--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
--- LIBTIFF support available
--- FREETYPE2 support available
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.

To check the build, run the selftest.py script.

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 357, in _handle_tasks
    debug('task handler got sentinel')
TypeError: 'NoneType' object is not callable

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 330, in _handle_workers
    debug('worker handler exiting')
TypeError: 'NoneType' object is not callable

Adding Pillow 2.8.1 to easy-install.pth file
Installing pildriver.py script to /data/project/bub/.local/bin
Installing pilconvert.py script to /data/project/bub/.local/bin
Installing pilfont.py script to /data/project/bub/.local/bin
Installing pilfile.py script to /data/project/bub/.local/bin
Installing pilprint.py script to /data/project/bub/.local/bin

Installed /data/project/bub/.local/lib/python2.7/site-packages/Pillow-2.8.1-py2.7-linux-x86_64.egg
Processing dependencies for Pillow==2.8.1
Finished processing dependencies for Pillow==2.8.1
Searching for ply==3.4
Reading https://pypi.python.org/simple/ply/
Reading http://www.dabeaz.com/ply/
Best match: ply 3.4
Downloading https://pypi.python.org/packages/source/p/ply/ply-3.4.tar.gz#md5=ffdc95858819347bf92d7c2acc074894
Processing ply-3.4.tar.gz
Writing /tmp/easy_install-wOXlhX/ply-3.4/setup.cfg
Running ply-3.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-wOXlhX/ply-3.4/egg-dist-tmp-snNB0R
zip_safe flag not set; analyzing archive contents...
Adding ply 3.4 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/ply-3.4-py2.7.egg
Processing dependencies for ply==3.4
Finished processing dependencies for ply==3.4
Searching for py==1.4.22
Reading https://pypi.python.org/simple/py/
Best match: py 1.4.22
Downloading https://pypi.python.org/packages/source/p/py/py-1.4.22.tar.gz#md5=1af93ed9a00bc38385142ae0eb7cf3ff
Processing py-1.4.22.tar.gz
Writing /tmp/easy_install-yckmvJ/py-1.4.22/setup.cfg
Running py-1.4.22/setup.py -q bdist_egg --dist-dir /tmp/easy_install-yckmvJ/py-1.4.22/egg-dist-tmp-nBqaO_
Adding py 1.4.22 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/py-1.4.22-py2.7.egg
Processing dependencies for py==1.4.22
Finished processing dependencies for py==1.4.22
Searching for pytest==2.3.4
Best match: pytest 2.3.4
Processing pytest-2.3.4-py2.7.egg
pytest 2.3.4 is already the active version in easy-install.pth
Installing py.test script to /data/project/bub/.local/bin
Installing py.test-2.7 script to /data/project/bub/.local/bin

Using /data/project/bub/.local/lib/python2.7/site-packages/pytest-2.3.4-py2.7.egg
Processing dependencies for pytest==2.3.4
Finished processing dependencies for pytest==2.3.4
Searching for python-dateutil==2.2
Reading https://pypi.python.org/simple/python-dateutil/
Reading http://labix.org/python-dateutil
Reading https://dateutil.readthedocs.org
Best match: python-dateutil 2.2
Downloading https://pypi.python.org/packages/source/p/python-dateutil/python-dateutil-2.2.tar.gz#md5=c1f654d0ff7e33999380a8ba9783fd5c
Processing python-dateutil-2.2.tar.gz
Writing /tmp/easy_install-KnEKSu/python-dateutil-2.2/setup.cfg
Running python-dateutil-2.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-KnEKSu/python-dateutil-2.2/egg-dist-tmp-wM9IKH
Adding python-dateutil 2.2 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/python_dateutil-2.2-py2.7.egg
Processing dependencies for python-dateutil==2.2
Finished processing dependencies for python-dateutil==2.2
Searching for python-openid==2.2.5
Best match: python-openid 2.2.5
Processing python_openid-2.2.5-py2.7.egg
python-openid 2.2.5 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/python_openid-2.2.5-py2.7.egg
Processing dependencies for python-openid==2.2.5
Finished processing dependencies for python-openid==2.2.5
Searching for pytz==2015.4
Reading https://pypi.python.org/simple/pytz/
Best match: pytz 2015.4
Downloading https://pypi.python.org/packages/2.7/p/pytz/pytz-2015.4-py2.7.egg#md5=4698c08f18f5cecba09b7e85d8e9328e
Processing pytz-2015.4-py2.7.egg
Moving pytz-2015.4-py2.7.egg to /data/project/bub/.local/lib/python2.7/site-packages
Exception TypeError: TypeError("'NoneType' object does not support item deletion",) in <Finalize object, dead> ignored
Adding pytz 2015.4 to easy-install.pth file
Traceback (most recent call last):
  File "/usr/bin/easy_install", line 9, in <module>
    load_entry_point('setuptools==3.3', 'console_scripts', 'easy_install')()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1963, in main
    with_ei_usage(lambda:
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1950, in with_ei_usage
    return f()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1967, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 381, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 616, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 648, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 671, in process_distribution
    self.install_egg_scripts(dist)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 553, in install_egg_scripts
    self.install_wrapper_scripts(dist)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 746, in install_wrapper_scripts
    for args in get_script_args(dist):
  File "/tmp/easy_install-7kZ25l/pbr-0.10.0/pbr/packaging.py", line 512, in override_get_script_args
AttributeError: 'NoneType' object has no attribute 'get_script_header'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 138, in apport_excepthook
    os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), 'wb') as f:
OSError: [Errno 2] No such file or directory: '/var/crash/_usr_bin_easy_install.51909.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/bin/easy_install", line 9, in <module>
    load_entry_point('setuptools==3.3', 'console_scripts', 'easy_install')()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1963, in main
    with_ei_usage(lambda:
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1950, in with_ei_usage
    return f()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1967, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 381, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 616, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 648, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 671, in process_distribution
    self.install_egg_scripts(dist)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 553, in install_egg_scripts
    self.install_wrapper_scripts(dist)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 746, in install_wrapper_scripts
    for args in get_script_args(dist):
  File "/tmp/easy_install-7kZ25l/pbr-0.10.0/pbr/packaging.py", line 512, in override_get_script_args
AttributeError: 'NoneType' object has no attribute 'get_script_header'
tools.bub@tools-bastion-01:~/public_html/BUB$ easy_install --user `cat requirements.txt` 
Searching for args==0.1.0
Best match: args 0.1.0
Processing args-0.1.0-py2.7.egg
args 0.1.0 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/args-0.1.0-py2.7.egg
Processing dependencies for args==0.1.0
Finished processing dependencies for args==0.1.0
Searching for Babel==1.3
Best match: Babel 1.3
Babel 1.3 is already the active version in easy-install.pth
Installing pybabel script to /data/project/bub/.local/bin

Using /usr/lib/python2.7/dist-packages
Processing dependencies for Babel==1.3
Finished processing dependencies for Babel==1.3
Searching for BeautifulSoup==3.2.1
Best match: BeautifulSoup 3.2.1
BeautifulSoup 3.2.1 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for BeautifulSoup==3.2.1
Finished processing dependencies for BeautifulSoup==3.2.1
Searching for beautifulsoup4==4.3.2
Best match: beautifulsoup4 4.3.2
Processing beautifulsoup4-4.3.2-py2.7.egg
beautifulsoup4 4.3.2 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/beautifulsoup4-4.3.2-py2.7.egg
Processing dependencies for beautifulsoup4==4.3.2
Finished processing dependencies for beautifulsoup4==4.3.2
Searching for bitarray==0.8.1
Best match: bitarray 0.8.1
Processing bitarray-0.8.1-py2.7-linux-x86_64.egg
bitarray 0.8.1 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/bitarray-0.8.1-py2.7-linux-x86_64.egg
Processing dependencies for bitarray==0.8.1
Finished processing dependencies for bitarray==0.8.1
Searching for blinker==1.3
Best match: blinker 1.3
Processing blinker-1.3-py2.7.egg
blinker 1.3 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/blinker-1.3-py2.7.egg
Processing dependencies for blinker==1.3
Finished processing dependencies for blinker==1.3
Searching for clint==0.3.3
Best match: clint 0.3.3
Processing clint-0.3.3-py2.7.egg
clint 0.3.3 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/clint-0.3.3-py2.7.egg
Processing dependencies for clint==0.3.3
Finished processing dependencies for clint==0.3.3
Searching for cssmin==0.2.0
Best match: cssmin 0.2.0
Processing cssmin-0.2.0-py2.7.egg
cssmin 0.2.0 is already the active version in easy-install.pth
Installing cssmin script to /data/project/bub/.local/bin

Using /data/project/bub/.local/lib/python2.7/site-packages/cssmin-0.2.0-py2.7.egg
Processing dependencies for cssmin==0.2.0
Finished processing dependencies for cssmin==0.2.0
Searching for DateTime==4.0.1
Best match: DateTime 4.0.1
Processing DateTime-4.0.1-py2.7.egg
DateTime 4.0.1 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/DateTime-4.0.1-py2.7.egg
Processing dependencies for DateTime==4.0.1
Finished processing dependencies for DateTime==4.0.1
Searching for decorator==3.4.0
Best match: decorator 3.4.0
decorator 3.4.0 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for decorator==3.4.0
Finished processing dependencies for decorator==3.4.0
Searching for docopt==0.6.1
Best match: docopt 0.6.1
Processing docopt-0.6.1-py2.7.egg
docopt 0.6.1 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/docopt-0.6.1-py2.7.egg
Processing dependencies for docopt==0.6.1
Finished processing dependencies for docopt==0.6.1
Searching for drench==0.0.13
Best match: drench 0.0.13
Processing drench-0.0.13-py2.7.egg
drench 0.0.13 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/drench-0.0.13-py2.7.egg
Processing dependencies for drench==0.0.13
Finished processing dependencies for drench==0.0.13
Searching for Flask==0.10.1
Best match: Flask 0.10.1
Flask 0.10.1 is already the active version in easy-install.pth

Using /usr/lib/python2.7/dist-packages
Processing dependencies for Flask==0.10.1
Finished processing dependencies for Flask==0.10.1
Searching for Flask-Babel==0.9
Best match: Flask-Babel 0.9
Processing Flask_Babel-0.9-py2.7.egg
Flask-Babel 0.9 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_Babel-0.9-py2.7.egg
Processing dependencies for Flask-Babel==0.9
Finished processing dependencies for Flask-Babel==0.9
Searching for Flask-Login==0.2.11
Best match: Flask-Login 0.2.11
Processing Flask_Login-0.2.11-py2.7.egg
Removing Flask-Login 0.2.6 from easy-install.pth file
Flask-Login 0.2.11 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_Login-0.2.11-py2.7.egg
Processing dependencies for Flask-Login==0.2.11
Finished processing dependencies for Flask-Login==0.2.11
Searching for Flask-Mail==0.7.6
Best match: Flask-Mail 0.7.6
Processing Flask_Mail-0.7.6-py2.7.egg
Flask-Mail 0.7.6 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_Mail-0.7.6-py2.7.egg
Processing dependencies for Flask-Mail==0.7.6
Finished processing dependencies for Flask-Mail==0.7.6
Searching for Flask-OpenID==1.2.1
Best match: Flask-OpenID 1.2.1
Processing Flask_OpenID-1.2.1-py2.7.egg
Flask-OpenID 1.2.1 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_OpenID-1.2.1-py2.7.egg
Processing dependencies for Flask-OpenID==1.2.1
Finished processing dependencies for Flask-OpenID==1.2.1
Searching for Flask-SQLAlchemy==0.16
Best match: Flask-SQLAlchemy 0.16
Processing Flask_SQLAlchemy-0.16-py2.7.egg
Flask-SQLAlchemy 0.16 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_SQLAlchemy-0.16-py2.7.egg
Processing dependencies for Flask-SQLAlchemy==0.16
Finished processing dependencies for Flask-SQLAlchemy==0.16
Searching for Flask-WhooshAlchemy==0.55a0
Best match: Flask-WhooshAlchemy 0.55a
Processing Flask_WhooshAlchemy-0.55a-py2.7.egg
Flask-WhooshAlchemy 0.55a is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_WhooshAlchemy-0.55a-py2.7.egg
Processing dependencies for Flask-WhooshAlchemy==0.55a0
Finished processing dependencies for Flask-WhooshAlchemy==0.55a0
Searching for Flask-WTF==0.10.0
Best match: Flask-WTF 0.10.0
Processing Flask_WTF-0.10.0-py2.7.egg
Flask-WTF 0.10.0 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Flask_WTF-0.10.0-py2.7.egg
Processing dependencies for Flask-WTF==0.10.0
Finished processing dependencies for Flask-WTF==0.10.0
Searching for flup==1.0.2
Best match: flup 1.0.2
Adding flup 1.0.2 to easy-install.pth file

Using /usr/lib/python2.7/dist-packages
Processing dependencies for flup==1.0.2
Finished processing dependencies for flup==1.0.2
Searching for hashlib==20081119
Best match: hashlib 20081119
Processing hashlib-20081119-py2.7-linux-x86_64.egg
hashlib 20081119 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/hashlib-20081119-py2.7-linux-x86_64.egg
Processing dependencies for hashlib==20081119
Finished processing dependencies for hashlib==20081119
Searching for htmlmin==0.1.5
Best match: htmlmin 0.1.5
Processing htmlmin-0.1.5-py2.7.egg
htmlmin 0.1.5 is already the active version in easy-install.pth
Installing htmlmin script to /data/project/bub/.local/bin

Using /data/project/bub/.local/lib/python2.7/site-packages/htmlmin-0.1.5-py2.7.egg
Processing dependencies for htmlmin==0.1.5
Finished processing dependencies for htmlmin==0.1.5
Searching for internetarchive==0.6.9
Best match: internetarchive 0.6.9
Processing internetarchive-0.6.9-py2.7.egg
internetarchive 0.6.9 is already the active version in easy-install.pth
Installing ia script to /data/project/bub/.local/bin

Using /data/project/bub/.local/lib/python2.7/site-packages/internetarchive-0.6.9-py2.7.egg
Processing dependencies for internetarchive==0.6.9
Finished processing dependencies for internetarchive==0.6.9
Searching for itsdangerous==0.24
Best match: itsdangerous 0.24
Processing itsdangerous-0.24-py2.7.egg
Removing itsdangerous 0.22 from easy-install.pth file
itsdangerous 0.24 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg
Processing dependencies for itsdangerous==0.24
Finished processing dependencies for itsdangerous==0.24
Searching for Jinja2==2.7.3
Best match: Jinja2 2.7.3
Processing Jinja2-2.7.3-py2.7.egg
Removing Jinja2 2.7.2 from easy-install.pth file
Jinja2 2.7.3 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/Jinja2-2.7.3-py2.7.egg
Processing dependencies for Jinja2==2.7.3
Finished processing dependencies for Jinja2==2.7.3
Searching for jsmin==2.0.11
Best match: jsmin 2.0.11
Processing jsmin-2.0.11-py2.7.egg
jsmin 2.0.11 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/jsmin-2.0.11-py2.7.egg
Processing dependencies for jsmin==2.0.11
Finished processing dependencies for jsmin==2.0.11
Searching for jsonpatch==0.4
Best match: jsonpatch 0.4
Processing jsonpatch-0.4-py2.7.egg
Removing jsonpatch 1.3 from easy-install.pth file
jsonpatch 0.4 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/jsonpatch-0.4-py2.7.egg
Processing dependencies for jsonpatch==0.4
Finished processing dependencies for jsonpatch==0.4
Searching for MarkupSafe==0.23
Best match: MarkupSafe 0.23
Processing MarkupSafe-0.23-py2.7-linux-x86_64.egg
Removing MarkupSafe 0.18 from easy-install.pth file
MarkupSafe 0.23 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg
Processing dependencies for MarkupSafe==0.23
Finished processing dependencies for MarkupSafe==0.23
Searching for MySQL-python==1.2.5
Best match: MySQL-python 1.2.5
Processing MySQL_python-1.2.5-py2.7-linux-x86_64.egg
Removing MySQL-python 1.2.3 from easy-install.pth file
MySQL-python 1.2.5 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg
Processing dependencies for MySQL-python==1.2.5
Finished processing dependencies for MySQL-python==1.2.5
Searching for oauthlib==0.6.3
Best match: oauthlib 0.6.3
Processing oauthlib-0.6.3-py2.7.egg
Removing oauthlib 0.6.1 from easy-install.pth file
oauthlib 0.6.3 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/oauthlib-0.6.3-py2.7.egg
Processing dependencies for oauthlib==0.6.3
Finished processing dependencies for oauthlib==0.6.3
Searching for pbr==0.10.0
Best match: pbr 0.10.0
Processing pbr-0.10.0-py2.7.egg
pbr 0.10.0 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/pbr-0.10.0-py2.7.egg
Processing dependencies for pbr==0.10.0
Finished processing dependencies for pbr==0.10.0
Searching for Pillow==2.8.1
Best match: Pillow 2.8.1
Processing Pillow-2.8.1-py2.7-linux-x86_64.egg
Removing Pillow 2.3.0 from easy-install.pth file
Pillow 2.8.1 is already the active version in easy-install.pth
Installing pildriver.py script to /data/project/bub/.local/bin
Installing pilconvert.py script to /data/project/bub/.local/bin
Installing pilfont.py script to /data/project/bub/.local/bin
Installing pilfile.py script to /data/project/bub/.local/bin
Installing pilprint.py script to /data/project/bub/.local/bin

Using /data/project/bub/.local/lib/python2.7/site-packages/Pillow-2.8.1-py2.7-linux-x86_64.egg
Processing dependencies for Pillow==2.8.1
Finished processing dependencies for Pillow==2.8.1
Searching for ply==3.4
Best match: ply 3.4
Processing ply-3.4-py2.7.egg
ply 3.4 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/ply-3.4-py2.7.egg
Processing dependencies for ply==3.4
Finished processing dependencies for ply==3.4
Searching for py==1.4.22
Best match: py 1.4.22
Processing py-1.4.22-py2.7.egg
Removing py 1.4.20 from easy-install.pth file
py 1.4.22 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/py-1.4.22-py2.7.egg
Processing dependencies for py==1.4.22
Finished processing dependencies for py==1.4.22
Searching for pytest==2.3.4
Best match: pytest 2.3.4
Processing pytest-2.3.4-py2.7.egg
pytest 2.3.4 is already the active version in easy-install.pth
Installing py.test script to /data/project/bub/.local/bin
Installing py.test-2.7 script to /data/project/bub/.local/bin

Using /data/project/bub/.local/lib/python2.7/site-packages/pytest-2.3.4-py2.7.egg
Processing dependencies for pytest==2.3.4
Finished processing dependencies for pytest==2.3.4
Searching for python-dateutil==2.2
Best match: python-dateutil 2.2
Processing python_dateutil-2.2-py2.7.egg
Removing python-dateutil 1.5 from easy-install.pth file
python-dateutil 2.2 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/python_dateutil-2.2-py2.7.egg
Processing dependencies for python-dateutil==2.2
Finished processing dependencies for python-dateutil==2.2
Searching for python-openid==2.2.5
Best match: python-openid 2.2.5
Processing python_openid-2.2.5-py2.7.egg
python-openid 2.2.5 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/python_openid-2.2.5-py2.7.egg
Processing dependencies for python-openid==2.2.5
Finished processing dependencies for python-openid==2.2.5
Searching for pytz==2015.4
Best match: pytz 2015.4
Processing pytz-2015.4-py2.7.egg
Removing pytz 2012c from easy-install.pth file
pytz 2015.4 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/pytz-2015.4-py2.7.egg
Processing dependencies for pytz==2015.4
Finished processing dependencies for pytz==2015.4
Searching for PyYAML==3.10
Best match: PyYAML 3.10
Adding PyYAML 3.10 to easy-install.pth file

Using /usr/lib/python2.7/dist-packages
Processing dependencies for PyYAML==3.10
Finished processing dependencies for PyYAML==3.10
Searching for redis==2.10.1
Reading https://pypi.python.org/simple/redis/
Best match: redis 2.10.1
Downloading https://pypi.python.org/packages/source/r/redis/redis-2.10.1.tar.gz#md5=6ffdea9c476b4815ddfee9a7362819ea
Processing redis-2.10.1.tar.gz
Writing /tmp/easy_install-vHLl7u/redis-2.10.1/setup.cfg
Running redis-2.10.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vHLl7u/redis-2.10.1/egg-dist-tmp-iGCqSQ
warning: no previously-included files found matching '__pycache__'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
zip_safe flag not set; analyzing archive contents...
Removing redis 2.7.2 from easy-install.pth file
Adding redis 2.10.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/redis-2.10.1-py2.7.egg
Processing dependencies for redis==2.10.1
Finished processing dependencies for redis==2.10.1
Searching for requests==2.3.0
Best match: requests 2.3.0
Processing requests-2.3.0-py2.7.egg
Removing requests 2.2.1 from easy-install.pth file
Adding requests 2.3.0 to easy-install.pth file

Using /data/project/bub/.local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg
Processing dependencies for requests==2.3.0
Finished processing dependencies for requests==2.3.0
Searching for requests-oauthlib==0.4.1
Reading https://pypi.python.org/simple/requests-oauthlib/
Best match: requests-oauthlib 0.4.1
Downloading https://pypi.python.org/packages/source/r/requests-oauthlib/requests-oauthlib-0.4.1.tar.gz#md5=4e5125f249580ae4f815d7291f505124
Processing requests-oauthlib-0.4.1.tar.gz
Writing /tmp/easy_install-D0vzD2/requests-oauthlib-0.4.1/setup.cfg
Running requests-oauthlib-0.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-D0vzD2/requests-oauthlib-0.4.1/egg-dist-tmp-VR8zQN
warning: no directories found matching 'tests/'
warning: no directories found matching 'docs/'
Removing requests-oauthlib 0.4.0 from easy-install.pth file
Adding requests-oauthlib 0.4.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/requests_oauthlib-0.4.1-py2.7.egg
Processing dependencies for requests-oauthlib==0.4.1
Finished processing dependencies for requests-oauthlib==0.4.1
Searching for simhash==1.4.6
Reading https://pypi.python.org/simple/simhash/
Best match: simhash 1.4.6
Downloading https://pypi.python.org/packages/2.7/s/simhash/simhash-1.4.6-py2.7.egg#md5=3c6166b545377009620a94a958e78e27
Processing simhash-1.4.6-py2.7.egg
Moving simhash-1.4.6-py2.7.egg to /data/project/bub/.local/lib/python2.7/site-packages
Adding simhash 1.4.6 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/simhash-1.4.6-py2.7.egg
Processing dependencies for simhash==1.4.6
Finished processing dependencies for simhash==1.4.6
Searching for six==1.4.1
Best match: six 1.4.1
Processing six-1.4.1-py2.7.egg
Removing six 1.5.2 from easy-install.pth file
six 1.4.1 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/six-1.4.1-py2.7.egg
Processing dependencies for six==1.4.1
Finished processing dependencies for six==1.4.1
Searching for slimit==0.8.1
Reading https://pypi.python.org/simple/slimit/
Best match: slimit 0.8.1
Downloading https://pypi.python.org/packages/source/s/slimit/slimit-0.8.1.zip#md5=5f5f86e98940a1e71a6a30d501e0d733
Processing slimit-0.8.1.zip
Writing /tmp/easy_install-iwoUhf/slimit-0.8.1/setup.cfg
Running slimit-0.8.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iwoUhf/slimit-0.8.1/egg-dist-tmp-1jJWW5
Adding slimit 0.8.1 to easy-install.pth file
Installing slimit script to /data/project/bub/.local/bin

Installed /data/project/bub/.local/lib/python2.7/site-packages/slimit-0.8.1-py2.7.egg
Processing dependencies for slimit==0.8.1
Finished processing dependencies for slimit==0.8.1
Searching for speaklater==1.3
Best match: speaklater 1.3
Processing speaklater-1.3-py2.7.egg
speaklater 1.3 is already the active version in easy-install.pth

Using /data/project/bub/.local/lib/python2.7/site-packages/speaklater-1.3-py2.7.egg
Processing dependencies for speaklater==1.3
Finished processing dependencies for speaklater==1.3
Searching for SQLAlchemy==0.7.9
Reading https://pypi.python.org/simple/SQLAlchemy/
Best match: SQLAlchemy 0.7.9
Downloading https://pypi.python.org/packages/source/S/SQLAlchemy/SQLAlchemy-0.7.9.tar.gz#md5=a0b58defc5ad0c7e1baeb932f62d08dd
Processing SQLAlchemy-0.7.9.tar.gz
Writing /tmp/easy_install-EKCzdD/SQLAlchemy-0.7.9/setup.cfg
Running SQLAlchemy-0.7.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-EKCzdD/SQLAlchemy-0.7.9/egg-dist-tmp-WKhDEb
warning: no files found matching '*.jpg' under directory 'doc'
no previously-included directories found matching 'doc/build/output'
zip_safe flag not set; analyzing archive contents...
Removing SQLAlchemy 0.8.4 from easy-install.pth file
Adding SQLAlchemy 0.7.9 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg
Processing dependencies for SQLAlchemy==0.7.9
Finished processing dependencies for SQLAlchemy==0.7.9
Searching for sqlalchemy-migrate==0.7.2
Reading https://pypi.python.org/simple/sqlalchemy-migrate/
Reading http://code.google.com/p/sqlalchemy-migrate/
Reading http://erosson.com/migrate
Reading http://trac.erosson.com/migrate
Reading http://www.openstack.org/
Best match: sqlalchemy-migrate 0.7.2
Downloading https://pypi.python.org/packages/source/s/sqlalchemy-migrate/sqlalchemy-migrate-0.7.2.tar.gz#md5=10382fda16f056491e671b5307dd6713
Processing sqlalchemy-migrate-0.7.2.tar.gz
Writing /tmp/easy_install-uFAC_k/sqlalchemy-migrate-0.7.2/setup.cfg
Running sqlalchemy-migrate-0.7.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uFAC_k/sqlalchemy-migrate-0.7.2/egg-dist-tmp-uuiTlN
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*pyc' found anywhere in distribution
warning: no previously-included files found matching '.hgtags'
warning: no previously-included files matching '*' found under directory 'docs/_build'
zip_safe flag not set; analyzing archive contents...
migrate.versioning.api: module MAY be using inspect.getsource
migrate.tests.__init__: module references __file__
migrate.tests.versioning.test_template: module references __path__
migrate.tests.integrated.test_docs: module references __file__
Adding sqlalchemy-migrate 0.7.2 to easy-install.pth file
Installing migrate script to /data/project/bub/.local/bin
Installing migrate-repository script to /data/project/bub/.local/bin

Installed /data/project/bub/.local/lib/python2.7/site-packages/sqlalchemy_migrate-0.7.2-py2.7.egg
Processing dependencies for sqlalchemy-migrate==0.7.2
Searching for Tempita>=0.4
Reading https://pypi.python.org/simple/Tempita/
Best match: Tempita 0.5.3dev
Downloading https://pypi.python.org/packages/source/T/Tempita/Tempita-0.5.3dev.tar.gz#md5=365c3b4f36435e2178902d5619301140
Processing Tempita-0.5.3dev.tar.gz
Writing /tmp/easy_install-Lv_V6B/Tempita-0.5.3dev/setup.cfg
Running Tempita-0.5.3dev/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Lv_V6B/Tempita-0.5.3dev/egg-dist-tmp-YaUR8u
Adding Tempita 0.5.3dev to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Tempita-0.5.3dev-py2.7.egg
Finished processing dependencies for sqlalchemy-migrate==0.7.2
Searching for Tempita==0.5.2
Best match: Tempita 0.5.2
Downloading https://pypi.python.org/packages/source/T/Tempita/Tempita-0.5.2.tar.gz#md5=4c2f17bb9d481821c41b6fbee904cea1
Processing Tempita-0.5.2.tar.gz
Writing /tmp/easy_install-9Ete4b/Tempita-0.5.2/setup.cfg
Running Tempita-0.5.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-9Ete4b/Tempita-0.5.2/egg-dist-tmp-w54ZFI
Removing Tempita 0.5.3dev from easy-install.pth file
Adding Tempita 0.5.2 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Tempita-0.5.2-py2.7.egg
Processing dependencies for Tempita==0.5.2
Finished processing dependencies for Tempita==0.5.2
Searching for tld==0.6.4
Reading https://pypi.python.org/simple/tld/
Best match: tld 0.6.4
Downloading https://pypi.python.org/packages/source/t/tld/tld-0.6.4.tar.gz#md5=0c9832eb78ba244c5b94bf2e0e81acb7
Processing tld-0.6.4.tar.gz
Writing /tmp/easy_install-wKZpvK/tld-0.6.4/setup.cfg
Running tld-0.6.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-wKZpvK/tld-0.6.4/egg-dist-tmp-uG7YjG
zip_safe flag not set; analyzing archive contents...
tld.utils: module references __file__
Adding tld 0.6.4 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/tld-0.6.4-py2.7.egg
Processing dependencies for tld==0.6.4
Finished processing dependencies for tld==0.6.4
Searching for validate-email==1.2
Reading https://pypi.python.org/simple/validate-email/
Best match: validate-email 1.2
Downloading https://pypi.python.org/packages/source/v/validate_email/validate_email-1.2.tar.gz#md5=b7f41a08dc4366b176cb95ea1997cee2
Processing validate_email-1.2.tar.gz
Writing /tmp/easy_install-7MjOVZ/validate_email-1.2/setup.cfg
Running validate_email-1.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-7MjOVZ/validate_email-1.2/egg-dist-tmp-0r9hOk
zip_safe flag not set; analyzing archive contents...
Adding validate-email 1.2 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/validate_email-1.2-py2.7.egg
Processing dependencies for validate-email==1.2
Finished processing dependencies for validate-email==1.2
Searching for Werkzeug==0.9.6
Reading https://pypi.python.org/simple/Werkzeug/
Best match: Werkzeug 0.9.6
Downloading https://pypi.python.org/packages/source/W/Werkzeug/Werkzeug-0.9.6.tar.gz#md5=f7afcadc03b0f2267bdc156c34586043
Processing Werkzeug-0.9.6.tar.gz
Writing /tmp/easy_install-Ci7ERP/Werkzeug-0.9.6/setup.cfg
Running Werkzeug-0.9.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Ci7ERP/Werkzeug-0.9.6/egg-dist-tmp-KDYMPM
warning: no files found matching '*' under directory 'werkzeug/debug/templates'
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
Removing Werkzeug 0.9.4 from easy-install.pth file
Adding Werkzeug 0.9.6 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Werkzeug-0.9.6-py2.7.egg
Processing dependencies for Werkzeug==0.9.6
Finished processing dependencies for Werkzeug==0.9.6
Searching for Whoosh==2.6.0
Reading https://pypi.python.org/simple/Whoosh/
Best match: Whoosh 2.6.0
Downloading https://pypi.python.org/packages/source/W/Whoosh/Whoosh-2.6.0.zip#md5=deb9be020263c20462e6bbb8053f6a0a
Processing Whoosh-2.6.0.zip
Writing /tmp/easy_install-rCU3J3/Whoosh-2.6.0/setup.cfg
Running Whoosh-2.6.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rCU3J3/Whoosh-2.6.0/egg-dist-tmp-HY41L5
warning: no files found matching '*.txt' under directory 'tests'
warning: no files found matching '*.txt' under directory 'benchmark'
warning: no files found matching '*.txt' under directory 'files'
warning: no files found matching '*.py' under directory 'files'
warning: no files found matching '*.jpg' under directory 'files'
Removing Whoosh 2.7.0 from easy-install.pth file
Adding Whoosh 2.6.0 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/Whoosh-2.6.0-py2.7.egg
Processing dependencies for Whoosh==2.6.0
Finished processing dependencies for Whoosh==2.6.0
Searching for WTForms==2.0.1
Reading https://pypi.python.org/simple/WTForms/
Best match: WTForms 2.0.1
Downloading https://pypi.python.org/packages/source/W/WTForms/WTForms-2.0.1.zip#md5=aaac94c137b3cd63efeb3ded51082cde
Processing WTForms-2.0.1.zip
Writing /tmp/easy_install-oDHScB/WTForms-2.0.1/setup.cfg
Running WTForms-2.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-oDHScB/WTForms-2.0.1/egg-dist-tmp-PBvZ_t
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
zip_safe flag not set; analyzing archive contents...
wtforms.i18n: module references __file__
Removing WTForms 2.0.2 from easy-install.pth file
Adding WTForms 2.0.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/WTForms-2.0.1-py2.7.egg
Processing dependencies for WTForms==2.0.1
Finished processing dependencies for WTForms==2.0.1
Searching for zope.interface==4.1.1
Reading https://pypi.python.org/simple/zope.interface/
Best match: zope.interface 4.1.1
Downloading https://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.1.1.tar.gz#md5=edcd5f719c5eb2e18894c4d06e29b6c6
Processing zope.interface-4.1.1.tar.gz
Writing /tmp/easy_install-hwrJAb/zope.interface-4.1.1/setup.cfg
Running zope.interface-4.1.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-hwrJAb/zope.interface-4.1.1/egg-dist-tmp-Usi9xZ
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
Removing zope.interface 4.0.5 from easy-install.pth file
Adding zope.interface 4.1.1 to easy-install.pth file

Installed /data/project/bub/.local/lib/python2.7/site-packages/zope.interface-4.1.1-py2.7-linux-x86_64.egg
Processing dependencies for zope.interface==4.1.1
Finished processing dependencies for zope.interface==4.1.1

Partial book uploaded after stat/tar failure

https://archive.org/details/bub_gb_c5aAlglfjLIC was uploaded with 11 pages while the source PDF has 65. The error log has:

2016-02-13 22:11:30.059577  func:manager  args:(<__main__.QueueHandler object at 0x2ab7e58e12d0>,){}
  :<class 'requests.exceptions.ChunkedEncodingError'>, Retry in 1 seconds...
2016-02-13 23:25:19.771508  c56nNe8GkyUC  Not Public Domain
tar: gb_c5aAlglfjLIC_000001.jpeg: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000002.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000003.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000004.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000005.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000006.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000007.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000008.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000009.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000010.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000011.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000012.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000013.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000014.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000015.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000016.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000017.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000018.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000019.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000020.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000021.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000022.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000023.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000024.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000025.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000026.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000027.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000028.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000029.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000030.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000031.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000032.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000033.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000034.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000035.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000036.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000037.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000038.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000039.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000040.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000041.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000042.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000043.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000044.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000045.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000046.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000047.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000048.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000049.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000050.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000051.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000052.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000054.png: Cannot stat: No such file or directory
tar: gb_c5aAlglfjLIC_000055.png: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
2016-02-13 23:34:47.117864  func:manager  args:(<__main__.QueueHandler object at 0x2aea276262d0>,){}
  :<class 'subprocess.CalledProcessError'>, Retry in 1 seconds...
rm: cannot remove '/data/scratch/BUB_downloads/bub_gb_c5exO45AUIMC_images.tar': No such file or directory
2016-02-13 23:45:09.380048  Command rm /data/scratch/BUB_downloads/bub_gb_c5exO45AUIMC_images.tar failed2016-02-14 00:16:59.047814  C5jTv9jLZ_YC  Not Public Domain

Comedy_of_the_Mistake_By_Sir_John_Vanbru.pdf

Download from Google Books at higher resolution

@PhE points out that we're downloading images at just 575 px width, e.g. https://archive.org/download/bub_gb_wO_IoyWTD7UC/bub_gb_wO_IoyWTD7UC_images.tar/gb_wO_IoyWTD7UC_000013.png; the images themselves are ok, but compressed as DjVu and PDF become rather hard to read (as yannf said previously). https://archive.org/download/bub_gb_wO_IoyWTD7UC

The scraper should also zoom as much as possible, before downloading. For instance, instead of http://books.google.fr/books?id=wO_IoyWTD7UC&amp;hl=fr&amp;hl=fr&amp;pg=PP12&amp;img=1&amp;zoom=3&amp;sig=ACfU3U36KATA17T_LaY5INpu6F8gse1NiQ&amp;w=575 download http://books.google.fr/books?id=wO_IoyWTD7UC&amp;hl=fr&amp;hl=fr&amp;pg=PP12&amp;img=1&amp;zoom=3&amp;sig=ACfU3U36KATA17T_LaY5INpu6F8gse1NiQ&amp;w=1324 (1324 px), or iterate zooming until the biggest resolution is found.

Allow to specify number of workers/concurrency

Similar to https://github.com/ArchiveTeam/seesaw-kit , now that bub has a proper queue, it would be nice to be able to have multiple items claimed/worked on at the same time. Then you could either specify concurrency for a worker (i.e. the process which downloads and uploads images) or just run more workers on the same queue, maybe with a different API key each.

Define multiple workers in config and spawn one job per item

Currently we have

tools.bub@tools-bastion-01:~/public_html/BUB/bot$ ls *py
mass_worker_1.py  mass_worker_2.py  mass_worker_3.py  mass_worker.py  upload_checker.py  worker.py

This duplicates code, which is just ugly. What's risky is that each worker runs indefinitely. It would be better to have a single "permanent" worker, which reads configuration for concurrency etc. from some configuration file and then spawns a labs grid job for each item, or at least for each download, so that they always use different IPs and are less likely to be blocked.

mass_worker.py: TypeError: expected string or buffer

I think this killed a worker:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/data/project/bub/public_html/BUB/bot/mass_worker.py", line 392, in wait_and_add_to_queue
    info = json.loads(q_bulk_order.pop_and_remove())
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

I guess enclose everything in a try and return proper status?

Corrupt PNG pages from Google Books

15.52 < Nemo_bis> PNG is perfectly fine
20.24 < Nemo_bis> Warning: Corrupted PNG image - /t/_image_test_005/RawImagesBookPreprocessor-image_test_005/gb_test_14.png
20.26 < Nemo_bis> And it indeed is: https://archive.org/download/image_test_005/gb_test_images.tar/gb_test_14.png
20.26 < Nemo_bis> Please check if the image is corrupted in Google Books too
20.26 < Nemo_bis> But that looks like a classic partial download

This was discovered while testing for #13 but doesn't block it, rather makes it more urgent because the PDF conversion is probably masking such errors and we'll never know how many corrupt books BUB uploaded.

If we're lucky, it's just an interrupted connection which can be resumed with standard HTTP methods; if not, the downloaded images will need to be validated one by one and retried where corrupt.

Allow overriding previous uploads

As requested by phe: at least admins will need an option to delete all files, reupload images and overwrite all metadata for older uploads. The early imports from Google Books have partial metadata coverage and are sometimes very hard to read due to low resolution.

Empty tar file uploaded

I see new books coming in, great!

https://catalogd.archive.org/log/397899878 :

tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Module threw exception:
tar -xf '/t/derive/bub_gb_pTAPAAAAQAAJ_2/bub_gb_pTAPAAAAQAAJ_images.tar' --directory '/t/_bub_gb_pTAPAAAAQAAJ_2/RawImagesBookPreprocessor-bub_gb_pTAPAAAAQAAJ_2/' failed with exit code: 2

https://archive.org/download/bub_gb_pTAPAAAAQAAJ_2 says it's a 0 byte file, so it seems a wrong upload. An error in the tar production, in the upload or in the receiving end?

Add bsb-muenchen.de

They apparently have over a million books: http://www.muenchener-digitalisierungszentrum.de/

• Example search
• Example document linking Google but also their own viewer
• The PDF downloader from their viewer asks an easy captcha and then allows to download a rather good PDF (no watermark, 180 dpi jpg) with a possibly predictable URL.

[DOC] Add docs on how to run BUB on your own

Following the example of tpb and ArchiveTeam, we shouldn't rely on few IP addresses and machines, in case we don't have enough capacity on Tool Labs or its API keys. In the documentation phase, please add instructions on how one can set up the tool locally or on their own server.

Upload images as is to archive.org, don't create PDF

PDF download requires captcha so BUB downloads the images and converts them to PDF (individually and then merging the PDFs). This is not needed, just put them all in a $IAID_images.tar file and IA will do all the conversions, assuming the images have filenames that sort the pages in correct order. We'll get a better resolution and save most of the time currently spent in processing.

See https://en.wikisource.org/wiki/Help:DjVu_files#The_Internet_Archive for more info if needed.

Year "18??" is converted to "18"

https://www.googleapis.com/books/v1/volumes/kyVDz2NJAqsC became https://archive.org/details/bub_gb_kyVDz2NJAqsC (18). If the ia library doesn't support approximate dates, let's just use 1800.

Queue stands still

The queue is standing still and books completed are not being removed from the list.

[ENH] Allow library-based Google Books IDs

"Whoops! Book ID/URL: IBNR:CR102002811 for Google-Books is invalid !
Recheck it."

That's the ID I got from http://193.206.215.17/BVE/result.php?bid=SBLE001430 ; to get the "usual" ID I only have to click the red "ebook" button which links z61y7bACShcC, can this be automated?

Not very important of course. :)

Wrong language set for man/dli.serc.iisc.ernet.in books

https://archive.org/search.php?query=sponsor%3Adli.serc.iisc.ernet.in contains many books which are marked as being in "English", while they are in other languages. The field dc.language.iso (e.g. http://dli.serc.iisc.ernet.in/handle/2015/291019?show=full ) should be fetched instead.

Set the public domain mark

When uploading to archive.org, please set the following metadata field:

licenseurl:http://creativecommons.org/publicdomain/mark/1.0/

Smarter public domain detection

A bunch of clearly PD books are erroring out as "not public domain", even a book from 1714 for instance: https://books.google.it/books?id=bDoj5_TLj2wC

If the Google API lies or is too reticent, we need to add some logic ourselves. Anything published before 1866 is clearly safe, for instance.

Google Books uploads contain post-processing artifacts like simulated hyperlinks

Example:

https://archive.org/stream/bub_gb_rmk-AAAAYAAJ_2/bub_gb_rmk-AAAAYAAJ#page/n9/mode/1up

This is because Google does post-processing on scans to simulate hyperlinks in the table of contents for the Google Book Search view, which is where BUB pulls its images from. For example:

https://books.google.com/books?id=rmk-AAAAYAAJ&pg=PP15

These artifacts don't exist in the PDF (click the button with the gear icon at the top right on that page and select "Download PDF"), or the scans in Play Books reader view:

https://books.google.com/books/reader?id=rmk-AAAAYAAJ&output=reader&pg=GBS.PP15

Corrupt PNG pages from Google Books, again

https://ia902301.us.archive.org/tarview.php?tar=/21/items/bub_gb_8BNsUnpVtiEC/bub_gb_8BNsUnpVtiEC_images.tar&file=gb_8BNsUnpVtiEC_000601.png

caused:

---

Warning: Corrupted PNG image - /t/_bub_gb_8BNsUnpVtiEC/RawImagesBookPreprocessor-bub_gb_8BNsUnpVtiEC/gb_8BNsUnpVtiEC_000601.png
0500 pppjpppppp ppppppjppp jpppjpjppp jpppppjppp ppppjppppp ppjppjpppp jpppppppjp pppppppppp pjpppppjpp pppppppjpp
non-zero exit (1) from PnmOp "/usr/bin/pngtopnm '/t/_bub_gb_8BNsUnpVtiEC/RawImagesBookPreprocessor-bub_gb_8BNsUnpVtiEC/gb_8BNsUnpVtiEC_000601.png'"; trying again via ImageMagick
convert.im6: Expected 8192 bytes; found 7644 bytes /t/_bub_gb_8BNsUnpVtiEC/RawImagesBookPreprocessor-bub_gb_8BNsUnpVtiEC/gb_8BNsUnpVtiEC_000601.png' @ warning/png.c/MagickPNGWarningHandler/1754. convert.im6: Read Exception/t/_bub_gb_8BNsUnpVtiEC/RawImagesBookPreprocessor-bub_gb_8BNsUnpVtiEC/gb_8BNsUnpVtiEC_000601.png' @ error/png.c/MagickPNGErrorHandler/1728.
convert.im6: corrupt image /t/_bub_gb_8BNsUnpVtiEC/RawImagesBookPreprocessor-bub_gb_8BNsUnpVtiEC/gb_8BNsUnpVtiEC_000601.png' @ error/png.c/ReadPNGImage/3723. convert.im6: no images defined/f/_bub_gb_8BNsUnpVtiEC//tmp.pnm' @ error/convert.c/ConvertImageCommand/3044.

Module threw exception:
convert '/t/_bub_gb_8BNsUnpVtiEC/RawImagesBookPreprocessor-bub_gb_8BNsUnpVtiEC/gb_8BNsUnpVtiEC_000601.png' +compress '/f/_bub_gb_8BNsUnpVtiEC//tmp.pnm' failed with exit code: 1

Derivation failed!
[ PDT: 2014-08-14 07:33:53 ] Executing: rm -rf '/t/_bub_gb_8BNsUnpVtiEC/' '/f/_bub_gb_8BNsUnpVtiEC/'

--------------- Module ProcessJP2 Finished PDT: 2014-08-14 07:33:53 (Took 4.2 minutes) ------------------->

---

https://catalogd.archive.org/log/328371226

Filename needs to be same as identifier

If the identifier is https://archive.org/details/bub_gb_06CrKB76Y7QC then the PDF needs to be uploaded as bub_gb_06CrKB76Y7QC.pdf so that thumbs show up in search like https://archive.org/search.php?query=publisher%3A%22C.+Coen%22

BUB: 502 Bad Gateway

Dunno whose fault.

$ curl -I https://tools.wmflabs.org/bub/
HTTP/1.1 502 Bad Gateway
Server: nginx/1.5.0
Date: Wed, 08 Oct 2014 07:15:02 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.3.10-1ubuntu3.10+wmf1

1669 book "not public domain"

I got an error that http://books.google.com/books/about/Breve_et_vniversale_cronistoria_del_mont.html?id=43y967I-pPcC is not public domain. Hard to believe, is Google giving erroneous data?

BUB got and uploaded an 52bytes PDF file

http://www2.senado.leg.br/bdsf/handle/id/221709 being selected as Dspace-based-library resulted in
http://tools.wmflabs.org/bub/progress/usp:id_221709/
that resulted in
https://ia802508.us.archive.org/17/items/bub_usp_id_221709/

Queue seems to be stuck atm

https://tools.wmflabs.org/bub/queue/ is stuck - with weirdy 1st process

Don't download files in home directory

Currently, images are downloaded all together in ~/public_html/BUB/bot/downloads and this is probably not optimal.

1. Tools sysadmins suggest to use /data/scratch for bigger I/O operations. I'm not sure what's the performance difference, but if we overload the /home filesystem there might be some angry daemon beating us.
2. Each book should be downloaded in its own directory. I suspect it's quite easy they might be interfering with each other, occasionally.

503

Bub is down for quite a long time. It shows No webservice.

Support DSpace / brasiliana.usp.br

555 asked for https://tools.wmflabs.org/bub/ to support upload from http://www.brasiliana.usp.br/bbd/ It's a DSpace site with direct PDF download, so it should be easy.

Quoting from @lugusto, «APIs URLs for all those mentioned by me: http://dspace.wbpublibnet.gov.in:8080/oai/request, http://www2.senado.leg.br/bdsf-oai/request, http://bibdig.biblioteca.unesp.br/oai/request; Brasiliana USP still don't have a public API URL but harvesting directly from the description pages shouldn't be a very difficult task, see the HTML source for http://www.brasiliana.usp.br/bbd/handle/1918/01861600 as an example»

Ideally, we'd

1. let the user enter any kind of URL to a DSpace item, for instance http://www.brasiliana.usp.br/bbd/handle/1918/7879 or http://www.brasiliana.usp.br/bbd/bitstream/handle/1918/7879/Output.o.pdf or any other representation and automatically confirm/find out it's a DSpace URL,
2. automatically extract the appropriate OAI-PMH entry (or, if not available, the microdata in HTML??),
3. get all the information we need from the standard entry chosen in (2) for the item provided in (1), in particular author, title, date, copyright status.
   @Aubreymcfato, @atomotic, @bram-atmire, what's the most general reliable way to do what above? Rohit is almost done with supporting Google Books and wants to add Brasiliana / DSpace in the next few days, help very appreciated. :-)

queue standing still

The BUB queue is standing still and books are not at all uploaded in IA