Malleefowl has WPS processes for climate data access and workflows.
Home Page: http://malleefowl.readthedocs.io/en/latest/
License: Other
A Web Processing Service for Climate Data Access and Workflows.
This package was created with Cookiecutter and the bird-house/cookiecutter-birdhouse project template.
happens because of using:
from malleefowl import WPSProcess
error message:
WARNING:root:Could not import process [<class malleefowl.process.WPSProcess at 0x2ab3bd28ef58>]: TypeError('__init__() takes at least 4 arguments (1 given)',)
Is it not possible to install malleefowl on a 32 bit system?
Using custom.cfg for buildout ...
'custom.cfg.example' -> 'custom.cfg'
Using DOWNLOAD_CACHE = /home/fklemme/malleefowl/downloads
mkdir: created directory '/home/fklemme/malleefowl/downloads'
Installing Anaconda ...
WARNING:
Your operating system appears not to be 64-bit, but you are trying to
install a 64-bit version of Miniconda.
Are sure you want to continue the installation? [yes|no]
[no] >>>
A fetch process for 'fx' time aggregation (Orography) failed.
here is the log of malleefowl:
PyWPS [2016-03-24 13:26:27,161] INFO: Execution status=ProcessStarted
PyWPS [2016-03-24 13:26:27,161] INFO: Percent completed=0
PyWPS [2016-03-24 13:26:27,161] INFO: Status message=Process esgsearch started
PyWPS [2016-03-24 13:26:27,162] INFO: Sleeping 3 seconds...
PyWPS [2016-03-24 13:26:27,371] ERROR: Search job failed! Could not retrieve files/aggregations.
Traceback (most recent call last):
File "/homel/birdhouse/malleefowl/malleefowl/esgf/search.py", line 245, in threader
self._file_search_job(**worker)
File "/homel/birdhouse/malleefowl/malleefowl/esgf/search.py", line 255, in _file_search_job
if not temporal_filter(f.filename, start_date, end_date):
File "/homel/birdhouse/malleefowl/malleefowl/esgf/search.py", line 60, in temporal_filter
start_year, end_year = date_from_filename(filename)
File "/homel/birdhouse/malleefowl/malleefowl/esgf/search.py", line 21, in date_from_filename
start_year = int(value[0][:4]) # keep only the year
ValueError: invalid literal for int() with base 10: 'fx'
... currently it always takes the default url.
for example:
... used in solr workflow.
logs are configured to send output to stderr which should be captured by supervisor. In async mode no logs are written in the log file.
examples:
the new pywps-4 supports external storage for outputs:
... or a timeout in workflow process.
search anaconda/var/cache/mako but should be ./conda/envs/birdhouse/var/cache/mako
there are already ideas and examples:
malleefowl cant be used as common module (#11).
pywps has restricted chars for literal input values: restrictedCharacters = ['\\',"#",";", "&","!"]
malleefowl is installed as a conda package which is used by other "birds" like flyingpigeon.
When you work on malleefowl processes and have other "birds" installed you might see an unexpected behavior. When a malleefowl process is called asynchronously then the process will choose the malleefowl module from the conda package ... not the malleefowl code you just edited.
Workaround: remove the conda package
$ source activate birdhouse
$ conda remove malleefowlNeeds a more convenient solution ;)
Haven't I seen this problem before? Maybe we just added the dependencies to bootstrap.sh, but that's not enough as wget is used in Makefile before.
"/home/sthao/anaconda/bin/conda" install -y -n malleefowl setuptools=36.5.0
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.4.11
latest version: 4.5.5
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: /home/sthao/.conda/envs/malleefowl
added / updated specs:
- setuptools=36.5.0
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2018.4.16 | py27_0 142 KB
The following packages will be UPDATED:
certifi: 2018.4.16-py27_0 conda-forge --> 2018.4.16-py27_0
openssl: 1.0.2o-0 conda-forge --> 1.0.2o-h20670df_0
The following packages will be DOWNGRADED:
ca-certificates: 2018.4.16-0 conda-forge --> 2018.03.07-0
Downloading and Extracting Packages
certifi 2018.4.16: ################################################################################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Update pinned conda packages ...
Update buildout bootstrap-buildout.py ...
Bootstrap buildout ...
ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check https://github.com/pypa/setuptools/issues/581 for more info; use pip to install setuptools
Traceback (most recent call last):
File "bootstrap-buildout.py", line 200, in <module>
import zc.buildout.buildout
File "/tmp/bootstrap-bGD7T9/zc.buildout-2.10.0-py2.7.egg/zc/buildout/buildout.py", line 18, in <module>
import zc.buildout.easy_install
File "/tmp/bootstrap-bGD7T9/zc.buildout-2.10.0-py2.7.egg/zc/buildout/easy_install.py", line 45, in <module>
pkg_resources.SetuptoolsVersion(setuptools_version) >=
AttributeError: 'module' object has no attribute 'SetuptoolsVersion'
Makefile:184: recipe for target 'bootstrap' failed
make: *** [bootstrap] Error 1
May be related to: buildout/buildout#442
But works fine on ubuntu16 but not with ubuntu18...
The malleefowl installation routine fails on a fresh CentOS 7 minimal installation. The package bzip2 is no longer installed by default which lets tar fail at some point.
As far as I understood the setup, this issue might be moved to https://github.com/bird-house/birdhousebuilder.bootstrap. I assume, adding the bzip2 package over there should be sufficient to fix this problem. I haven't tested if this issue also affects other birdhouse components.
... regenerate malleefowl with cookiecutter template.
... here the error message:
Version and requirements information containing pluggy:
[versions] constraint on pluggy: 0.6.0
Requirement of pytest: pluggy>=0.7
While:
Installing malleefowl.
Error: The requirement ('pluggy>=0.7') is not allowed by your [versions] constraint (0.6.0)
Current (and even older) versions of malleefowl and phoenix are no longer going to be build successfully!
See e.g. https://travis-ci.org/bird-house/pyramid-phoenix/builds/69434202:
Bootstrap buildout ...
discarding /home/travis/anaconda/bin from PATH
prepending /home/travis/.conda/envs/birdhouse/bin to PATH
bootstrap.py version 2015-07-01
Installing application with buildout ...
bash -c "source /home/travis/anaconda/bin/activate birdhouse;bin/buildout -c custom.cfg"
discarding /home/travis/anaconda/bin from PATH
prepending /home/travis/.conda/envs/birdhouse/bin to PATH
bash: bin/buildout: No such file or directory
make: *** [install] Error 127
The problem here is that a call to curl in bootstrap-buildout.py fails with code 77 (CURLE_SSL_CACERT_BADFILE). A problem could be, that old certs that may come with this installation have been expired already. In Makefile this script is called with --setuptools-version 14.3 --version 2.3.1. If you omit these parameters, buildout will work again, but the the installation will fail at a later point:
Installing application with buildout ...
bash -c "source /home/kruecke/anaconda/bin/activate birdhouse;bin/buildout -c custom.cfg"
discarding /home/kruecke/anaconda/bin from PATH
prepending /home/kruecke/.conda/envs/birdhouse/bin to PATH
Getting distribution for 'buildout.locallib'.
Got buildout.locallib 0.3.1.
Develop: '/home/kruecke/Repositories/malleefowl/.'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.*~' found anywhere in distribution
Getting distribution for 'birdhousebuilder.recipe.docker==0.2.1'.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
Got birdhousebuilder.recipe.docker 0.2.1.
Getting distribution for 'birdhousebuilder.recipe.conda==0.2.2'.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
Got birdhousebuilder.recipe.conda 0.2.2.
Getting distribution for 'zc.recipe.egg>=2.0.0a3'.
Got zc.recipe.egg 2.0.2.
Getting distribution for 'birdhousebuilder.recipe.pywps==0.2.6'.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
Got birdhousebuilder.recipe.pywps 0.2.6.
Getting distribution for 'birdhousebuilder.recipe.nginx==0.2.2'.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
Got birdhousebuilder.recipe.nginx 0.2.2.
While:
Installing.
Getting section malleefowl_wps.
Initializing section malleefowl_wps.
Installing recipe birdhousebuilder.recipe.pywps.
Getting distribution for 'birdhousebuilder.recipe.supervisor==0.2.2'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/buildout.py", line 1987, in main
getattr(buildout, command)(args)
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/buildout.py", line 552, in install
[self[part]['recipe'] for part in install_parts]
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/buildout.py", line 1137, in __getitem__
options._initialize()
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/buildout.py", line 1241, in _initialize
self.initialize()
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/buildout.py", line 1247, in initialize
recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/buildout.py", line 1201, in _install_and_load
allow_hosts=buildout._allow_hosts
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/easy_install.py", line 848, in install
return installer.install(specs, working_set)
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/easy_install.py", line 685, in install
for_buildout_run=for_buildout_run):
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/easy_install.py", line 477, in _get_dist
dist, avail = self._satisfied(requirement)
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/easy_install.py", line 234, in _satisfied
return None, self._obtain(req, source)
File "/home/kruecke/Repositories/malleefowl/eggs/zc.buildout-2.4.0-py2.7.egg/zc/buildout/easy_install.py", line 402, in _obtain
if index.obtain(requirement) is None:
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 443, in obtain
self.find_packages(requirement)
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 427, in find_packages
self.scan_url(self.index_url + requirement.unsafe_name+'/')
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 761, in scan_url
self.process_url(url, True)
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 304, in process_url
f = self.open_url(url, "Download error on %s: %%s -- Some packages may not be found!" % url)
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 700, in open_url
return open_with_auth(url, self.opener)
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 894, in _socket_timeout
return func(*args, **kwargs)
File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line 1007, in open_with_auth
fp = opener(request)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/urllib2.py", line 1200, in do_open
r = h.getresponse(buffering=True)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/httplib.py", line 1132, in getresponse
response.begin()
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/home/kruecke/.conda/envs/birdhouse/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 104] Connection reset by peer
make: *** [install] Error 1
supervisor==0.2.2 doesn't seem to be available in malleefowl_wps. Maybe an upgrade to a newer version of supervisor is required here?
in some case you want to keep the name of the output file or know the output url. pywps currently does not allow this. The output name is generated (uuid). pywps needs to be extended/patched for this.
wget=1.15 still works ...
see:
http://jachym.github.io/pywps-tutorial/build/html/testing/index.html
this can also be used for nosetests.
postgres setup needs to be fixed (buildout postgres recipe).
In which way will it be used?
See also PyWPS/mapserver integration.
curl is already part of conda.
See related issue: bird-house/flyingpigeon#240
downloading several cmip5 files. Got exception:
2017-05-30 15:49:21,515] [ERROR] line=153 module=download download failed!
Traceback (most recent call last):
File "/home/Shared/pingu/sandbox/birdhouse/malleefowl/malleefowl/download.py", line 149, in threader
self.download_job(**worker)
File "/home/Shared/pingu/sandbox/birdhouse/malleefowl/malleefowl/download.py", line 166, in download_job
progress)
File "/home/Shared/pingu/sandbox/birdhouse/malleefowl/malleefowl/download.py", line 138, in show_status
self.monitor(message, progress)
File "/home/Shared/pingu/sandbox/birdhouse/malleefowl/malleefowl/processes/wps_download.py", line 65, in monitor
response.update_status(msg, progress)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/pywps/app/WPSResponse.py", line 75, in update_status
update_response(self.uuid, self)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/pywps/dblog.py", line 138, in update_response
if requests.count():
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3089, in count
return self.from_self(col).scalar()
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2843, in scalar
ret = self.one()
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2814, in one
ret = self.one_or_none()
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none
ret = list(self)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
return self._execute_and_instances(context)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
exc_info
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/home/pingu/.conda/envs/malleefowl/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such table: pywps_requests [SQL: u'SELECT count(*) AS count_1 \nFROM (SELECT pywps_requests.uuid AS pywps_requests_uuid, pywps_requests.pid AS pywps_requests_pid, pywps_requests.operation AS pywps_requests_operation, pywps_requests.version AS pywps_requests_version, pywps_requests.time_start AS pywps_requests_time_start, pywps_requests.time_end AS pywps_requests_time_end, pywps_requests.identifier AS pywps_requests_identifier, pywps_requests.message AS pywps_requests_message, pywps_requests.percent_done AS pywps_requests_percent_done, pywps_requests.status AS pywps_requests_status \nFROM pywps_requests \nWHERE pywps_requests.uuid = ?) AS anon_1'] [parameters: ('c89dd80e-453e-11e7-8502-68f72837e1b4',)]
default url should be set in custom.cfg
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [constraints] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [limit] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [search_type] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [distrib] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [replica] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [latest] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [temporal] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [start] is not defined
PyWPS [2017-01-06 08:30:27,320] ERROR: Input [end] is not defined
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.