Giter Site home page Giter Site logo

proycon / lamachine Goto Github PK

View Code? Open in Web Editor NEW
67.0 16.0 20.0 45.14 MB

LaMachine - A software distribution of our in-house as well as some 3rd party NLP software - Virtual Machine, Docker, or local compilation/installation script

Home Page: https://proycon.github.io/LaMachine

License: GNU General Public License v3.0

Shell 52.44% HTML 0.16% Python 38.74% Ruby 4.03% JavaScript 1.22% Dockerfile 3.03% Singularity 0.38%
nlp virtual-machine docker-image software-distribution installer natural-language-processing computational-linguistics flat folia frog clam python webservices vagrant linux linux-distribution

lamachine's Introduction

Language Machines Badge Build Status Docker Pulls Project Status: Unsupported - The project has reached a stable, usable state but the author(s) have ceased all work on it

IMPORTANT NOTE: LaMachine is end-of-life and deprecated. There will be no further development and its usage is no longer recommended. See this post for reasons and alternative solutions

LaMachine is a unified software distribution for Natural Language Processing. We integrate numerous open-source NLP tools, programming libraries, web-services, and web-applications in a single Virtual Research Environment that can be installed on a wide variety of machines.

The software included in LaMachine tends to be highly specialised and generally depends on a lot of other interdependent software. Installing all this software can be a daunting task, compiling it from scratch even more so. LaMachine attempts to make this process easier by offering pre-built recipes for a wide variety of systems, whether it is on your home computer or whether you are setting up a dedicated production environment, LaMachine will safe you a lot of work.

We address various audiences; the bulk of the software is geared towards data scientists who are not afraid of the command line and some programming. We give you the instruments and it is up to you to yield them. However, we also attempt to accommodate researchers that require more high-level interfaces by incorporating webservices and websites that expose some of the functionality to a larger audience.

Installation

A) Guided installation with custom build option (recommended)

To build your own LaMachine instance, in any of the possible flavours, or to download a pre-built image, open a terminal on your Linux, BSD or MacOS system and run the following command:

bash <(curl -s https://raw.githubusercontent.com/proycon/LaMachine/master/bootstrap.sh)

This will prompt you for some questions on how you would like your LaMachine installation and allows you to include precisely the software you want or need and ensures that all is up to date. A screenshot is shown at the end of this subsection.

Are you on Windows 10 or 2016? Then you need to run this command in the Windows Linux subsystem, we do not support Windows natively. To do this you must first install the Linux Subsystem with a distribution of your choice (we recommend Ubuntu) from the Microsoft Store. Follow the instructions here. Alternatively, you may want to choose for a pre-built Virtual Machine image as explained in installation path C.

Building LaMachine can take quite some time, depending also on your computer's resources, internet connection, and the amount of software you selected to install. Half an hour to an hour is a normal build time. The bootstrap script alternatively also offers the option to download pre-built images (installation path B & C).

LaMachine bootstrap

B) Pre-built container image for Docker

We regularly build a basic LaMachine image and publish it to Docker Hub. The above installation path A also offers access to this, but you may opt to do it directly:

To download and use it, run:

docker pull proycon/lamachine
docker run  -p 8080:80 -h latest -t -i proycon/lamachine

This requires you to already have Docker installed and running on your system.

The pre-built image contains the stable version with only a basic set of common software rather than the full set, run lamachine-add inside the container to select extra software to install. Alternatively, other specialised LaMachine builds may be available on Docker Hub.

If you want another release, specify its tag explicitly:

docker pull proycon/lamachine:develop
docker run  -p 8080:80 -h develop -t -i proycon/lamachine:develop

C) Pre-built Virtual Machine image for Vagrant (recommended for Windows users)

We regularly build a basic LaMachine image and publish it to the Vagrant Cloud. The above installation path A also offers (simplified) access to this (except on Windows), but you may opt to do it directly.

To download and use a LaMachine prebuilt image:

  • Ensure you have Vagrant and VirtualBox installed on your system. Windows users also have to make sure that Hyper-V is disabled in Control Panel → Programs → Turn Windows features on or off → Hyper-V
  • Open a terminal or command prompt
  • Navigate to a folder of your choice (using cd); this will be the base folder, files inside will be shared within the VM under /vagrant
  • Download this example vagrant file into that same folder. If you are on linux or macOS you can download directly from command line like this: wget https://raw.githubusercontent.com/proycon/LaMachine/master/Vagrantfile.prebuilt.erb
  • Run vagrant init --template Vagrantfile.prebuilt.erb proycon/lamachine from the terminal.
  • Open Vagrantfile in a text editor and change the memory and CPU options to suit your system (the more resources the better!).
    • On an up-to-date windows 10 installation (at least version 1809), you can use Notepad as a text editor, but on older Windows versions this won't work and you need a better text editor!
  • Run vagrant up from the terminal to boot your VM
  • Run vagrant ssh from the terminal to connect to the VM

The pre-built image contains only a basic set of common software rather than the full set, run lamachine-stable-update --edit inside the virtual machine to select extra software to install.

To stop the VM when you're done, run: vagrant halt. Next time, navigate to the same base folder in your terminal and run vagrant up and vagrant ssh again.

Included Software

LaMachine includes a wide variety of open-source NLP software. You can select which software you want to include during the installation procedure (or any subsequent update).

  • by the Centre of Language and Speech Technology, Radboud University Nijmegen (CLST, RU)
    • Timbl - Tilburg Memory Based Learner
    • Ucto - Tokenizer
    • Frog - Frog is an integration of various memory-based natural language processing (NLP) modules developed for Dutch. It can do Part-of-Speech tagging, lemmatisation, named entity recogniton, shallow parsing, dependency parsing and morphological analysis.
    • Mbt - Memory-based Tagger
    • Wopr - Memory-based Word Predictor
    • FoLiA-tools - Command line tools for working with the FoLiA format
    • PyNLPl - Python Natural Language Processing Library
    • Colibri Core - Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way.
    • C++ libraries - ticcutils, libfolia
    • Python bindings - python-ucto, python-frog, python-timbl
    • CLAM - Quickly build RESTful webservices
    • Gecco - Generic Environment for Context-Aware Correction of Orthography
    • Valkuil - A context-aware spelling corrector for Dutch
    • Toad - Trainer Of All Data, training tools for Frog
    • foliadocserve - FoLiA Document Server
    • FLAT - FoLiA Linguistic Annotation Tool
    • TICCLTools - Tools that together constitute the bulk of TICCL: Text Induced Corpus-Cleanup.
    • PICCL - PICCL: A set of workflows for corpus building through OCR, post-correction (using TICCL) and Natural Language Processing.
    • Labirinto - A web-based portal listing all available tools in LaMachine, an ideal starting point for LaMachine
    • Oersetter - A Frisian<->Dutch Machine Translation system in collaboration with the Fryske Akademy
  • by the University of Groningen
    • Alpino - a dependency parser and tagger for Dutch
  • by the Vrije Universiteit Amsterdam (VU)
  • by Utrecht University (UU)
    • T-scan - T-scan is a Dutch text analytics tool for readability prediction (initially developed at TiCC, Tilburg University).
  • by Meertens Instituut
    • Python Course for the Humanities - Interactive tutorial and introduction into programming with Python for the humanities by Folgert Karsdorp & Maarten van Gompel (CLST, Nijmegen)
  • Major third party software (not exhaustive!):
    • Python
      • NumPy and SciPy - Python libraries for scientific computing
      • Matplotlib - A Python 2D plotting library producing publication quality figures
      • Scikit-learn - Machine learning in Python
      • IPython and Jupyter - A rich architecture for interactive computing.
        • Jupyter Lab - The successor of the popular Jupyter Notebooks, offers notebooks, a web-based IDE, terminals. An ideal entry point to get started with LaMachine and all it contains!
      • Pandas - Python Data Analysis Library
      • NLTK - Natural Language Toolkit for Python
      • PyTorch - Deep-learning library for Python
      • Spacy - Industrial-Strength NLP in Python
      • FLAIR - Framework for state-of-the-art sequence modelling through word embeddings
    • fastText - Library for efficient text classification and representation learning (has a Python binding)
    • R
    • Java
      • NextFlow - A system and language for writing parallel and scalable pipelines in a portable manner.
      • Stanford CoreNLP - Various types of linguistic enrichment
    • Hunspell - A spell checker
    • Tesseract - Open Source Optical Character Recognition (OCR)
    • Tensorflow - Open-source machine learning framework
    • Kaldi - Speech Recognition Framework (ASR)
    • Moses - Statistical Machine Translation system

Note that some software may not be available on certain platforms/distributions (most notably macOS).

For a verbose list of installed software and its metadata, run lamachine-list once you are inside your LaMachine installation. For more information regarding software metadata, check the corresponding section in the the contributor documentation.

If you enabled and started the webserver in LaMachine, then you have access to a rich portal page giving an overview of all installed software and providing access to any software with a web-based interface. This portal is powered by Labirinto.

Contribute

LaMachine is open for contributions by other software projects, please read the contributor documentation.

Architecture

LaMachine can be installed in multiple flavours:

  • Local installation - Installs LaMachine locally (natively) in a user environment on Linux or macOS machine (multiple per machine possible)
  • Global installation - Installs LaMachine globally (natively) on a Linux machine. (only one per machine)
  • Docker container - Installs LaMachine in a docker container
  • Virtual Machine - Installs LaMachine in a Virtual Machine
  • LXC container - Installs LaMachine in an LXC/LXD container.
  • Remote installation - Installs LaMachine globally (natively) on another Linux machine. (only one per machine)

In all cases, the installation is mediated through Ansible, providing a level of abstraction over whatever underlying technology is employed. Containerisation uses Docker or LXD. Virtualisation is made possible through Vagrant and VirtualBox. The local installation variant uses virtualenv with some custom extensions.

Initially, the user executes a bootstrap.sh script that acts as a single point of entry for all flavours. It will automatically download LaMachine and create the necessary configuration files for your LaMachine build, guiding you through all the options. It will eventually invoke a so-called ansible playbook that executes installation steps for all of the individual software projects included in LaMachine, depending on your distribution and flavour.

LaMachine uses Debian as primary Linux distribution (for virtualisation and containerisation), we support the distributions/platforms listed below for a native installation of LaMachine (i.e. compiled against the libraries of that distribution). We distinguish three categories of support (and for all we only support the x86-64 architecture):

  • Gold support - All participating software should work on these platforms and things are tested frequently.

    • Ubuntu 20.04 LTS (next: Ubuntu 22.04 LTS), this is the default option for all pre-built containers and VMs since LaMachine v2.25
    • Debian 10 (buster) (next: Debian 11), this was the default for docker containers and VMs before LaMachine v2.25
  • Silver support - Most software should work.

    • Debian 9 (stretch) - The previous stable release
    • Ubuntu 18.04 LTS - The previous LTS release
    • CentOS 8 / RedHat Enterprise Linux 8 - This is offered because it is a popular choice in enterprise environments. Testing is less frequent though.
  • Bronze support - Certain software is known not to work and/or things are more prone to breakage and have not been tested.

    • Debian testing (bullseye) and debian unstable (sid) - Should work but not tested.
    • Ubuntu (a non-LTS version) - Should work as long as it's newer than the one mentioned under silver support, but not tested.
    • macOS (a recent version) - Not all software is supported on macOS by definition, but a considerable portion does work. Things are a bit more prone to break if the user's environment has been heavily tweaked and differs from the stock experience.
    • Arch Linux (rolling release; things tend to work fine for most software but the nature of a rolling release makes breakages more common, e.g. on each major Python upgrade)
    • Linux Mint (recent version) - Supported in principle due to being an Ubuntu derivative, but not really tested so there could be surprises
    • Fedora (latest version); supported in principle but not really tested.
  • Deprecated:

    • Ubuntu 14.04 LTS
    • Ubuntu 16.04 LTS
    • CentOS 7 / RedHat Enterprise Linux 7
  • Unsupported (not exhaustive!) - We can not support these because our effort reached its limits:

    • FreeBSD and other BSDs
    • openSuSE / SuSE
    • Alpine
    • Gentoo
    • Void Linux
    • nixOS
    • Solaris
    • Windows

Note that this concerns the platforms LaMachine runs on natively or on which you can bootstrap your own build (installation path A). The options for host platforms for simply running a pre-built LaMachine Virtual Machine or Docker container, are much larger, and also include Windows (see installation paths B & C).

In addition to a flavour, users can opt for one of three versions of LaMachine:

  • stable - Installs the latest official releases of all participating software
  • development - Installs the latest development versions of participating software, this often means they are installed straight from the latest git version.
  • custom - Installs explicitly defined versions for all software (for e.g. scientific reproducibility).

Read more about the technical details in the contributor documentation.

Usage

How to start LaMachine differs a bit depending on your flavour.

Local Environment

Run the generated activation script to activate the local environment (here we assume your LaMachine VM is called stable!):

  • Run source lamachine-stable-activate or lamachine-stable-activate, this script should be located in your ~/bin directory.

LaMachine terminal screenshot

Virtual Machine

If you built your own LaMachine you have various scripts at your disposal (here we assume your LaMachine VM is called stable! The script names will be different for other names, replace as needed):

  • Run lamachine-stable-start to start the VM
  • Run lamachine-stable-connect to connect to a running VM and obtain a command line shell (over ssh)
  • Run lamachine-stable-stop to stop the VM
  • Run lamachine-stable-destroy to completely delete the VM again
  • lamachine-stable-activate is a shortcut that starts the VM and connects automatically, and stops the VM when you disconnect again.

If you used a prebuilt image you have to invoke vagrant yourself from the proper directory where you did vagrant init proycon/lamachine:stable:

  • Run vagrant up to start the VM
  • Run vagrant halt to stop the VM
  • Run vagrant ssh to connect to the VM and obtain a command line shell
  • Run vagrant destroy to remove the VM

Command line access to your LaMachine Virtual Machine through vagrant or lamachine-*-connect should be passwordless, other methods may require a login; use username vagrant and password vagrant. The root password is also vagrant. Change these in any exposed environments!

If you enabled a webserver in your LaMachine build, you can connect your web browser to http://127.0.0.1:8080 after having started the VM.

Port Mapping to the Virtual Machine

If you want to connect to a particular special-purpose server (not a webservice) inside the VM from your host system, then you often need to forward a port from your host system into the LaMachine VM, as for all intents and purposes, they should be considered two separated systems. This applies for instance when you want to use the server mode offered by software such as Frog or Alpino (again, this is completely different and independent from the webservices that LaMachine also offers).

From LaMachine 2.6.2 onward, the port 9999 is forwarded by default for the VM, meaning that if you connect to port 9999 on your local machine (IP 127.0.0.1), it will be forwarded to port 9999 in the LaMachine VM.

If you want to open any additional ports, you need to do so in Virtualbox for your LaMachine VM. Consult this guide for easy and illustrated instructions on how to set this up in the VirtualBox interface, or alternatively consult the relevant chapter in the Virtualbox Manual itself.

Docker Container

If you used the LaMachine bootstrap script, you will have several scripts at your disposition (we assume that your LaMachine VM is called stable, adapt the script names to your own situation). If you instead issued a docker pull proycon/lamachine manually you will need to run the docker commands yourself:

  • Run lamachine-stable-activate to start a new interactive container
    • This corresponds to docker run -i -t proycon/lamachine
  • Run lamachine-stable-run to start the command specified as a parameter in a new container (e.g. lamachine-stable-run frog)
    • This corresponds to : docker run -i -t proycon/lamachine:latest frog
      • You can omit the -i flag if the tool is not an interactive tool that reads from standard input (i.e. keyboard input).
  • Run lamachine-stable-start to start a webserver and all enabled webservices in a new LaMachine container:
    • This corresponds to: docker run -p 8080:80 -h hostname -t proycon/lamachine:latest lamachine-start-webserver -f
      • The numbers values for -p are the port numbers on the host side and on the container side respectively, the latter must always match with the http_port setting LaMachine has been built with (defaults to 80).
      • Set -h with the desired hostname, this too must match the setting LaMachine has been built with!
      • The -f argument to lamachine-start-webserver ensures the script waits in the foreground and doesn't exit after starting. In a docker context, this also makes the script a valid entrypoint (PID 1).
    • If started in this way, you can connect your webbrowser on the host system to http://127.0.0.1:8080 .

The scripts will automatically share your designated data directory (your home directory by default) with the container, mounted at /data by default. To manually make persistent storage available in the container, e.g. for sharing data, use docker parameters like: --mount type=bind,source=/path/on/host,target=/data

If you use LaMachine with docker, we expect you to actually be familiar with docker and understand the non-persistent nature of containers, understand the difference between images and containers. Be aware that new containers are created every time you run any of the above commands. If you want a more VM-like container experience, you can consider LXD instead of Docker.

Updating LaMachine

When you are inside LaMachine, you can update it by running lamachine-update, if you want to add extra software packages to your installation, run lamachine-add first (add --list for a list of installable packages). You can also edit LaMachine's settings and/or directly edit the list of packages to be installed/updated with lamachine-update --edit. Do note that this can't be used to uninstall software.

The update normally only updates what has changed, if you want to force an update of everything instead, run lamachine-update force=1. You can also use the even stronger force=2, which will forcibly remove all downloaded sources and start from scratch.

For Docker and the Virtual Machine flavours, when a SUDO password is being asked by the update script, you can simply press ENTER and leave it empty, do not run the entire script with escalated privileges.

Updating everything can be a time-consuming endeavour. If you know what you are doing then you can limit your update to certain packages, you can specify these packages (as a comma separated list) to the --only parameter, e.g: lamachine-update --only python-core,java-core. Do be aware that this could result in your LaMachine ending up in an unusable state (in which case a normal update should remedy the problem again).

Configuring LaMachine

If you want to view the LaMachine configuration, simply issue a lamachine-config from within. If you want to edit it interactively, add the --edit flag. Always run lamachine-update afterwards to apply the new configuration.

The lamachine-config tool can also be used to quickly edit a configuration setting through the command line, see lamachine-config --help for details.

Webservices and web applications

LaMachine comes with several webservices and web applications out of the box. Most are RESTful webservices served using CLAM, which also offer a generic web-interface for human end-users. The webserver provides a generic portal to all available services, powered by Labirinto, as shown in the screenshot below:

portal_screenshot

To start (or restart) the webserver and webservices, run lamachine-start-webserver from within your LaMachine installation. You can then connect your browser (on the host system) to http://localhost:8080 (the port may differ if you changed the default value). On virtual machines, the webserver will be automatically started at boot. For docker you can do: docker run -p 8080:80 -h hostname -t proycon/lamachine:latest lamachine-start-webserver -f

Warning: There is currently no or poor authentication enabled on the webservices, so do not expose them to the outside world!

Jupyter Lab

LaMachine comes with an installation of Jupyter Lab, which provides an excellent entry-point to LaMachine as it provides a web-based scripting environment or IDE (for Python and R), web-based terminal access, and especially access to the ubiquitous Jupyter Notebooks that enjoy great popularity in data science and beyond.

You can access your Jupyter Lab installation from the portal website of your LaMachine installation. By default LaMachine also preinstalls the interactive Python Course for the Humanities for you, so you can get started right away.

Jupyter Lab in LaMachine

The default password for the Lab environment is lamachine, you can change this with lamachine-passwd lab.

Warning: Do not expose this service to the world without a strong customised password as it allows arbitrary code execution and full access to your system!

Privacy

Unless you explicitly opt-out, LaMachine sends a few details to us regarding your installation of LaMachine whenever you build a new one or update an existing one. This is to help us keep track of its usage and improve it.

The following information is sent:

  • The form in which you run LaMachine (vagrant/local/docker)
  • Is it a new LaMachine installation or an update
  • Stable or Development?
  • The OS you are running on and its version
  • Your Python version

Your IP address will only be used to identify your country and not used in any other way. No personally identifiable information whatsoever will be included in any reports we generate from this and it will never be used for advertisement purposes.

To opt-out of this behaviour, set private: true in your LaMachine settings.

During build and upgrade, LaMachine downloads software from a wide variety of external sources.

Security

For a secure experience using LaMachine, take all of the following into account:

  • Our recommended bootstrap procedure downloads a script and immediately executes it. This is offered as a convenience but carries some inherent risks and is generally not a secure practice. It implies a trust relation between you and us, as well as the hoster (github). Prudent users are encouraged to download the script, inspect it, and only then execute it. We may provide PGP-signed releases in the future.
  • The bootstrap script asks for and requires root privileges for certain installation steps, this will always be asked and the user may confirm. The Ansible provisioning scripts also generally requires sudo, this will only be asked once per run, and the privileges will only be used when needed.
  • Running either the bootstrap procedure or the subsequent ansible provisioning entirely as root is forbidden for security reasons.
  • The current webserver configuration does not yet enable authentication for any of the webservices, so do NOT expose it directly to the internet without setting up authentication yourself. If you want authentication, consult the OpenID Connect section below.
  • If you are sure you don't need a webserver/webservices, disable it in the configuration upon first build.
  • The virtual machines tend to come with a preset username and password (vagrant:vagrant), the lamachine user in Docker containers has the password lamachine, you will need to change this.
  • Do not run development versions in a production environment, always use the stable release.
  • Do not run an outdated LaMachine installation, ensure you regularly run lamachine-update for updates! Bugs and potential vulnerabilities may have been patched in the meantime.
  • Only if your setup is otherwise secure (i.e. authentication on webservices), then make sure to always open only the necessary ports (80/443) to the internet, do not expose any of the UWSGI services to the world (this would allow arbitrary code execution).
  • As per the GNU General Public Licence, we do not offer any warranty despite doing our best.

OpenID Connect Authentication

LaMachine supports OpenID Connect, which is an extension on top of OAuth2, as a means to authenticate against an external single-sign-on authentication provider. You can configure OpenID Connect in the LaMachine configuration and LaMachine will attempt to propagate these parameters to all underlying software that supports OpenID Connect.

Please consult the LaMachine as a service documentation for further instructions.

Versioning

LaMachine comes in three versions, stable installs the latest stable releases of all software, development installs the latest development releases and custom installs explicitly specified versions. This section is about the latter and is for advanced users.

LaMachine itself also carries a version number, this number corresponds to the version of all the installation scripts that make up LaMachine. It is not tied to the versions of any underlying software.

In any LaMachine installation (v2.3.0 and above), you can do lamachine-list -v to obtain a customversions.yml file that explicitly states what software versions are installed. When bootstrapping a new LaMachine installation, you can place this customversions.yml file in the directory where you run the bootstrap, and opt for the custom version. LaMachine will then install the exact versions specified.

You can edit this customversions.yml file if you have good reason to opt for very specific versions of certain packages. Instead of an appropriate version number, you can also use the strings. Do be be aware that choosing version numbers that do not exist or combining versions of different packages that are not compatible will surely break things. If things fail, most software providers, us included, will not deliver support on older software versions.

The purpose of this custom versioning feature of LaMachine is to aid scientific reproducibility, with it you can build an environment consisting of older software, corresponding to the versions at the time you ran your experiments. In such cases you should publish a version of customversions.yml along with your data (and a copy of the installation manifest ideally).

This custom versioning is limited, it only pertains to software that is 1) not provided by the linux distribution itself, and 2) explicitly installed by LaMachine, rather than dependencies that are pulled in automatically by package managers. Even then, certain sofware is excluded from this scheme as the upstream provider does not provide the necessary facilities for obtaining older versions, LaMachine should output a warning in the log if that is the case. It is also not supported on MacOS.

If a strict reproduction environment is desired, we strongly recommend to use the docker or virtual machine flavour of LaMachine and archive the entire resulting image.

Frequently Asked Questioned & Troubleshooting

Q: Do I need LaMachine?

A: This depends on the software you are interested in and the kind of system you are on. LaMachine is offered as a convenience but draws from other software repositories which you can also access directly.

You may want to first check if our software packages are available for your Linux distribution. For C++ software such as Frog, ucto and Timbl, we provide packages for:

Python software is generally provided through the Python Package Index and can be installed using pip install.

LaMachine shines as it combines a lot of software, includes complex set-ups, and handles some default configuration.

Q: Why is my LaMachine installation so big??

A LaMachine installation quickly reaches 6GB, and even more if you enable software that is not enabled by default. LaMachine is a research environment that can be used in a wide variety of ways which we can't predict in advance, so we by default include a lot of popular software for maximum flexibility. When building your LaMachine, you can disable software groups you don't want and save space, or opt for extra dikspace (see the next question).

You can also limit the size somewhat by setting minimal: true in your LaMachine configuration, but this may mean that certains tools don't fully work.

Disk space is also, by far, the cheapest resource, in contrast to memory or CPU.

Q: I get an error "no space left on device" in the VM or Docker flavour of LaMachine (Issue #152)

This means the virtual disk used by the virtual machine or container is full. This may especially occur if you select some of the larger optional software packages. There is only limited space available in the VM or Docker container (roughly 9GB). For the VM, when you bootstrap your own LaMachine image from scratch (an option currently not available for Windows users though), you can opt to create extra diskspace (an extra volume).

For Docker, you may need to increase the base size of your containers (depending on the storage driver you use for docker). Consult the docker documentation at https://docs.docker.com/storage/storagedriver/ and do so now if you need this.

Advanced VM users can resolve the problem on their existing LaMachine VM by adding another virtual disk and moving some of the data, but this requires a fair amount of Linux administration expertise on their part. The procedure is roughly as follows:

  • Create an extra disk for the LaMachine VM in the VirtualBox interface (see for instance this tutorial up to step 11).
  • From within the LaMachine VM:
    • Partition the new disk (with fdisk or parted)
    • Format the new disk (with mkfs.ext4)
    • Add the new disk to /etc/fstab
    • Move /usr/local (which is where most of LaMachine is installed) to the new disk
    • Symlink the old /usr/local to the new path on the new disk

Q: Can I run LaMachine in a 32-bit environment?

No

Q: Can I run LaMachine with Python 2.7 instead of 3?

No

Q: Can I run LaMachine on an old Linux distribution?

No, your Linux distribution needs to be up to date and supported.

Q: Can I include my own software in LaMachine?

Yes! See the contribution guidelines

Q: Can I run a graphical desktop environment in the LaMachine Virtual Machine? (X.org)

Though LaMachine does not provide this out-of-the-box, you can easily install a fully fledged desktop environment as follows (do make sure you opted for extra diskspace during the bootstrap):

apt-get install task-gnome-desktop (See https://wiki.debian.org/DesktopEnvironment)

To access the graphical desktop you will want to start LaMachine from the VirtualBox interface.

Q: Can I run LaMachine on Kubernetes?

Yes, see [docs/kubernetes/README.md](these instructions) and templates.

Q: Docker gives an error: "flag provided but not defined: --build-arg"

Your Docker is too old, upgrade to at least 1.9

Q: lamachine-update gives an error: error 'fragment_class is None' (Issue #144)

This error may appear when LaMachine updates from ansible 2.7 to 2.8, if this occurs, simply rerun the update.

Q: Someone provided me with a pre-build LaMachine VM image in the form of a *.box file, how do I use it?

This is a Vagrant box file. You will need to follow the instruction as specified in Installation section C, with the following differences:

  • Prior to running vagrant init, you will need to run vagrant box add --name custom-lamachine /path/to/your/image.box (adapt the path to point to the box file you were given). You may change the name custom-lamachine to anything you like to identify this LaMachine image.
  • Instead of vagrant init --template Vagrantbox.prebuilt.erb proycon/lamachine, do vagrant init --template Vagrantbox.prebuilt.erb custom-lamachine (or another name if you changed it in the first step)

Q: I have another problem, can I report it?

Yes! Please report it in our Issue Tracker after checking that the problem has not already been reported (and solved perhaps) by someone else. Note that this is only for problems relating to the installation and availability of the software; for bugs or feature requests on any of the participating software (including our own), you should use the issue trackers pertaining to those software projects.

lamachine's People

Contributors

irishx avatar lbiemans avatar proycon avatar rahiel avatar zeusttu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lamachine's Issues

LaMachine stable fails on compilation of toad!

Since the latest releases of ticcutils et al, LaMachine stable fails on compilation of toad:

/bin/bash ../libtool  --tag=CXX   --mode=link g++ -DSYSCONF_PATH=\"/vol/customopt/lamachine/etc\" -std=c++0x -g -O2 -I/usr/include  -fopenmp -I/usr/include/libxml2   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -fopenmp -I/vol/customopt/lamachine/include   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -I/vol/customopt/lamachine/include   -I/vol/customopt/lamachine/include -I/usr/include/libxml2     -o testmbma testmbma.o  -L/vol/customopt/lamachine/lib -lfrog   -L/vol/customopt/lamachine/lib -lucto   -L/vol/customopt/lamachine/lib -lfolia   -L/vol/customopt/lamachine/lib -lticcutils   -L/vol/customopt/lamachine/lib -lmbt   -L/vol/customopt/lamachine/lib -ltimbl   -lxml2   -L/usr/lib/x86_64-linux-gnu  -licui18n -licuuc -licudata   -licuio  
libtool: link: g++ -DSYSCONF_PATH=\"/vol/customopt/lamachine/etc\" -std=c++0x -g -O2 -I/usr/include -fopenmp -I/usr/include/libxml2 -I/vol/customopt/lamachine/include -I/usr/include/libxml2 -I/vol/customopt/lamachine/include -I/usr/include/libxml2 -fopenmp -I/vol/customopt/lamachine/include -I/vol/customopt/lamachine/include -I/usr/include/libxml2 -I/vol/customopt/lamachine/include -I/vol/customopt/lamachine/include -I/usr/include/libxml2 -o testmbma testmbma.o  -L/vol/customopt/lamachine/lib /vol/customopt/lamachine/lib/libfrog.so /vol/customopt/lamachine/lib/libucto.so /vol/customopt/lamachine/lib/libfolia.so /vol/customopt/lamachine/lib/libticcutils.so /vol/customopt/lamachine/lib/libmbt.so /vol/customopt/lamachine/lib/libtimbl.so -lxml2 -L/usr/lib/x86_64-linux-gnu -licui18n -licuuc -licudata -licuio -pthread -fopenmp -Wl,-rpath -Wl,/vol/customopt/lamachine/lib -Wl,-rpath -Wl,/vol/customopt/lamachine/lib
g++ -DHAVE_CONFIG_H -I. -I..  -I../include  -DSYSCONF_PATH=\"/vol/customopt/lamachine/etc\" -std=c++0x -g -O2 -I/usr/include  -fopenmp -I/usr/include/libxml2   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -fopenmp -I/vol/customopt/lamachine/include   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -I/vol/customopt/lamachine/include   -I/vol/customopt/lamachine/include -I/usr/include/libxml2   -MT froggen.o -MD -MP -MF .deps/froggen.Tpo -c -o froggen.o froggen.cxx
froggen.cxx: In function ‘void create_mblem_trainfile(const std::multimap<icu_52::UnicodeString, std::map<icu_52::UnicodeString, std::set<icu_52::UnicodeString> > >&, const string&)’:
froggen.cxx:238:41: error: call of overloaded ‘UnicodeToUTF8(icu_52::UnicodeString&)’ is ambiguous
       string out = UnicodeToUTF8(outLine);
                                         ^
froggen.cxx:238:41: note: candidates are:
froggen.cxx:70:8: note: std::string UnicodeToUTF8(const icu_52::UnicodeString&)
 string UnicodeToUTF8( const UnicodeString& s ){
        ^
In file included from /vol/customopt/lamachine/include/ucto/tokenize.h:39:0,
                 from froggen.cxx:40:
/vol/customopt/lamachine/include/ticcutils/Unicode.h:36:15: note: std::string TiCC::UnicodeToUTF8(const icu_52::UnicodeString&)
   std::string UnicodeToUTF8( const UnicodeString&  );
               ^
froggen.cxx:269:41: error: call of overloaded ‘UnicodeToUTF8(icu_52::UnicodeString&)’ is ambiguous
       string out = UnicodeToUTF8(outLine);
                                         ^
froggen.cxx:269:41: note: candidates are:
froggen.cxx:70:8: note: std::string UnicodeToUTF8(const icu_52::UnicodeString&)
 string UnicodeToUTF8( const UnicodeString& s ){
        ^
In file included from /vol/customopt/lamachine/include/ucto/tokenize.h:39:0,
                 from froggen.cxx:40:
/vol/customopt/lamachine/include/ticcutils/Unicode.h:36:15: note: std::string TiCC::UnicodeToUTF8(const icu_52::UnicodeString&)
   std::string UnicodeToUTF8( const UnicodeString&  );
               ^
froggen.cxx:375:39: error: call of overloaded ‘UnicodeToUTF8(icu_52::UnicodeString&)’ is ambiguous
     string out = UnicodeToUTF8(outLine);
                                       ^
froggen.cxx:375:39: note: candidates are:
froggen.cxx:70:8: note: std::string UnicodeToUTF8(const icu_52::UnicodeString&)
 string UnicodeToUTF8( const UnicodeString& s ){
        ^
In file included from /vol/customopt/lamachine/include/ucto/tokenize.h:39:0,
                 from froggen.cxx:40:
/vol/customopt/lamachine/include/ticcutils/Unicode.h:36:15: note: std::string TiCC::UnicodeToUTF8(const icu_52::UnicodeString&)
   std::string UnicodeToUTF8( const UnicodeString&  );
               ^
make[2]: *** [froggen.o] Error 1
make[2]: Leaving directory `/vol/customopt/lamachine/src/toad/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/vol/customopt/lamachine/src/toad'
make: *** [all] Error 2
================ FATAL ERROR ==============
An error occurred during installation!!
toad make failed
===========================================

Vagrant build fails due to Arch linux upgrade issue with ca-certificates-utils

==> default: failed to commit transaction (conflicting files)
==> default: ca-certificates-utils: /etc/ssl/certs/ca-certificates.crt exists in filesystem
==> default: Errors occurred, no packages were upgraded.
==> default: ================ FATAL ERROR ==============
==> default: An error occured during installation!!
==> default: Unable to install global dependencies
==> default: ===========================================

Reported upstream: terrywang/vagrantboxes#25

Arch issue: https://www.archlinux.org/news/ca-certificates-utils-20170307-1-upgrade-requires-manual-intervention/

Error during building of docker image

For a project I'm working on, I need to create a slightly modified version of this project.

Therefore I'm running docker build to create the customized version of the project. For a few weeks now it was working fine up until today where the installation of libfolia fails. The error message that is shown is the following:

screen shot 2016-05-10 at 15 27 18

The first thing I did was to check if there were any updates to this package, but there weren't. I also checked out a fresh copy of this project and tried to build the docker image again. I keep getting the same error message.

Do you have insights on this issue?

crossdomain problem localhost/127.0.0.1

Using the webservices with docker on port 8080, for example frog, when using url
http://localhost:8080/frog
the reference to the stylesheet is
<?xml-stylesheet type="text/xsl" href="http://127.0.0.1:8080/frog//static/interface.xsl"?>
causing cross-domain problems. On Firefox 57.0.4:
Error loading stylesheet: An unknown error has occurred (805303f4)
Everything works when instead url
http://127.0.0.1:8080/frog
is used. It would be nice if the url for the stylesheet uses the domain name used in the request.

Changed libraries on ubuntu 17.10

I just tried to install LaMachine (the latest stable version, I assume) on a mostly fresh install of Ubuntu 17.10. It works, but the included list of dependencies don't all exist any more. I had to change the following ones to make it install:

libgnutils-dev -> libgnutils28-dev
libatlas-dev -> libatlas-base-dev

I also got an error due to not having virtualenv.

After manually installing the following list of dependencies, the bootstrap script got things running, passing all tests:

pkg-config git-core make gcc g++ autoconf automake autoconf-archive libtool autotools-dev libicu-dev libxml2-dev libxslt1-dev libbz2-dev zlib1g-dev libtar-dev libaspell-dev libhunspell-dev libboost-all-dev python3 python3-dev python3-pip python3-virtualenv libgnutls28-dev libcurl4-gnutls-dev wget curl libexttextcat-dev libatlas-base-dev libblas-dev gfortran libsuitesparse-dev libfreetype6-dev myspell-nl perl default-jre tesseract-ocr tesseract-ocr-eng tesseract-ocr-nld tesseract-ocr-deu tesseract-ocr-deu-frak tesseract-ocr-fra poppler-utils djvulibre-bin libdjvulibre-text imagemagick virtualenv

VM upgrade error, conflicting files in package python-requests-oauthlib

As reported by @faustusdotbe :

vagrant@lamachine:~$ sudo lamachine-update.sh ticcl
=====================================================================
           ,              LaMachine - NLP Software distribution
          ~)                     (http://proycon.github.io/LaMachine)
           (----í         Language Machines research group
            /| |\         & Centre of Language and Speech Technology
           / / /|	        Radboud University Nijmegen 
=====================================================================

Bootstrapping Virtual Machine or Docker image....

--------------------------------------------------------
[LaMachine] Installing global dependencies
--------------------------------------------------------
Conflict prevention...
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: virtualbox-guest-utils: removing virtualbox-guest-dkms breaks dependency 'VIRTUALBOX-GUEST-MODULES'
:: Synchronizing package databases...
 core                                                                                                                 123.4 KiB   289K/s 00:00 [#######################################################################################] 100%
 extra                                                                                                               1677.7 KiB   973K/s 00:02 [#######################################################################################] 100%
 community                                                                                                              3.7 MiB   816K/s 00:05 [#######################################################################################] 100%
 archlinuxfr is up to date
warning: archlinux-keyring-20170104-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) archlinux-keyring-20170104-1

Total Installed Size:  0.81 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                                                                                                                 [#######################################################################################] 100%
(1/1) checking package integrity                                                                                                               [#######################################################################################] 100%
(1/1) loading package files                                                                                                                    [#######################################################################################] 100%
(1/1) checking for file conflicts                                                                                                              [#######################################################################################] 100%
(1/1) checking available disk space                                                                                                            [#######################################################################################] 100%
:: Processing package changes...
(1/1) reinstalling archlinux-keyring                                                                                                           [#######################################################################################] 100%
==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key 684148BB25B49E986A4944C55184252D824B18E8...
  -> Locally signing key 91FFE0700E80619CEB73235CA88E23E377514E00...
  -> Locally signing key AB19265E5D7D20687D303246BA1DFB64FFF979E7...
  -> Locally signing key 0E8B644079F599DFC1DDC3973348882F6AC6A4C2...
  -> Locally signing key 44D4A033AC140143927397D47EFD567D4C7EA887...
  -> Locally signing key 27FFC4769E19F096D41D9265A04F9397CDFD6BB0...
==> Importing owner trust values...
==> Disabling revoked keys in keyring...
  -> Disabling key 7FA647CD89891DEDC060287BB9113D1ED21E1A55...
  -> Disabling key D4DE5ABDE2A7287644EAC7E36D1A9E70E19DAA50...
  -> Disabling key BC1FBE4D2826A0B51E47ED62E2539214C6C11350...
  -> Disabling key 63F395DE2D6398BBE458F281F2DBB4931985A992...
  -> Disabling key 8F76BEEA0289F9E1D3E229C05F946DED983D4366...
  -> Disabling key 81D7F8241DB38BC759C80FCE3A726C6170E80477...
  -> Disabling key E7210A59715F6940CF9A4E36A001876699AD6E84...
  -> Disabling key F5A361A3A13554B85E57DDDAAF7EF7873CFD4BB6...
  -> Disabling key 9515D8A8EAB88E49BB65EDBCE6B456CAF15447D5...
  -> Disabling key 4A8B17E20B88ACA61860009B5CED81B7C2E5C0D2...
  -> Disabling key 0B20CA1931F5DA3A70D0F8D2EA6836E1AB441196...
  -> Disabling key 66BD74A036D522F51DD70A3C7F2A16726521E06D...
==> Updating trust database...
gpg: next trustdb check due at 2017-09-07
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
Installing base-devel....
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 archlinuxfr is up to date
:: There are 25 members in group base-devel:
:: Repository core
   1) autoconf  2) automake  3) binutils  4) bison  5) fakeroot  6) file  7) findutils  8) flex  9) gawk  10) gcc  11) gettext  12) grep  13) groff  14) gzip  15) libtool  16) m4  17) make  18) pacman  19) patch  20) pkg-config  21) sed
   22) sudo  23) texinfo  24) util-linux  25) which

Enter a selection (default=all): 
warning: autoconf-2.69-4 is up to date -- skipping
warning: automake-1.15-2 is up to date -- skipping
warning: binutils-2.27-1 is up to date -- skipping
warning: bison-3.0.4-2 is up to date -- skipping
warning: fakeroot-1.21-2 is up to date -- skipping
warning: file-5.30-1 is up to date -- skipping
warning: findutils-4.6.0-2 is up to date -- skipping
warning: flex-2.6.3-1 is up to date -- skipping
warning: gawk-4.1.4-2 is up to date -- skipping
warning: gcc-6.3.1-1 is up to date -- skipping
warning: gettext-0.19.8.1-2 is up to date -- skipping
warning: grep-3.0-1 is up to date -- skipping
warning: groff-1.22.3-7 is up to date -- skipping
warning: gzip-1.8-2 is up to date -- skipping
warning: libtool-2.4.6-7 is up to date -- skipping
warning: m4-1.4.18-1 is up to date -- skipping
warning: make-4.2.1-1 is up to date -- skipping
warning: pacman-5.0.1-4 is up to date -- skipping
warning: patch-2.7.5-1 is up to date -- skipping
warning: pkg-config-0.29.1-2 is up to date -- skipping
warning: sed-4.4-1 is up to date -- skipping
warning: sudo-1.8.19.p2-1 is up to date -- skipping
warning: texinfo-6.3-1 is up to date -- skipping
warning: util-linux-2.29.1-1 is up to date -- skipping
warning: which-2.21-2 is up to date -- skipping
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (20) bzip2-1.0.6-6  dbus-1.10.16-1  fixesproto-5.0+9+g4292ec1-1  glib2-2.50.3-1  gssproxy-0.6.2-1  linux-4.9.9-1  linux-headers-4.9.9-1  mesa-17.0.0-1  mesa-libgl-17.0.0-1  poppler-0.52.0-1  pyqt5-common-5.8-1
              python-chardet-2.3.0-4  python-lxml-3.7.3-1  python-pyqt5-5.8-1  python-requests-oauthlib-0.8.0-1  python-sip-4.19.1-1  sip-4.19.1-1  tesseract-3.05.00-1  xorg-server-1.19.1-5  xorg-server-common-1.19.1-5

Total Download Size:    89.47 MiB
Total Installed Size:  223.27 MiB
Net Upgrade Size:        0.95 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 bzip2-1.0.6-6-x86_64                                                                                                  52.8 KiB   187K/s 00:00 [#######################################################################################] 100%
 dbus-1.10.16-1-x86_64                                                                                                273.1 KiB   853K/s 00:00 [#######################################################################################] 100%
 glib2-2.50.3-1-x86_64                                                                                                  2.2 MiB  1107K/s 00:02 [#######################################################################################] 100%
 gssproxy-0.6.2-1-x86_64                                                                                               77.6 KiB  0.00B/s 00:00 [#######################################################################################] 100%
 linux-4.9.9-1-x86_64                                                                                                  60.1 MiB  1673K/s 00:37 [#######################################################################################] 100%
 linux-headers-4.9.9-1-x86_64                                                                                           7.2 MiB  1147K/s 00:06 [#######################################################################################] 100%
 fixesproto-5.0+9+g4292ec1-1-any                                                                                       11.8 KiB  0.00B/s 00:00 [#######################################################################################] 100%
 mesa-17.0.0-1-x86_64                                                                                                   8.2 MiB   775K/s 00:11 [#######################################################################################] 100%
 mesa-libgl-17.0.0-1-x86_64                                                                                             4.3 KiB  0.00B/s 00:00 [#######################################################################################] 100%
 poppler-0.52.0-1-x86_64                                                                                             1009.8 KiB   809K/s 00:01 [#######################################################################################] 100%
 pyqt5-common-5.8-1-x86_64                                                                                            492.1 KiB   710K/s 00:01 [#######################################################################################] 100%
 python-chardet-2.3.0-4-any                                                                                           179.8 KiB   649K/s 00:00 [#######################################################################################] 100%
 python-lxml-3.7.3-1-x86_64                                                                                           911.8 KiB   707K/s 00:01 [#######################################################################################] 100%
 sip-4.19.1-1-x86_64                                                                                                  140.8 KiB  1006K/s 00:00 [#######################################################################################] 100%
 python-sip-4.19.1-1-x86_64                                                                                            80.4 KiB  11.2M/s 00:00 [#######################################################################################] 100%
 python-pyqt5-5.8-1-x86_64                                                                                              3.2 MiB   756K/s 00:04 [#######################################################################################] 100%
 xorg-server-common-1.19.1-5-x86_64                                                                                    26.7 KiB  8.69M/s 00:00 [#######################################################################################] 100%
 xorg-server-1.19.1-5-x86_64                                                                                         1282.2 KiB   712K/s 00:02 [#######################################################################################] 100%
 python-requests-oauthlib-0.8.0-1-any                                                                                  27.1 KiB  0.00B/s 00:00 [#######################################################################################] 100%
 tesseract-3.05.00-1-x86_64                                                                                             4.1 MiB   823K/s 00:05 [#######################################################################################] 100%
(20/20) checking keys in keyring                                                                                                               [#######################################################################################] 100%
(20/20) checking package integrity                                                                                                             [#######################################################################################] 100%
(20/20) loading package files                                                                                                                  [#######################################################################################] 100%
(20/20) checking for file conflicts                                                                                                            [#######################################################################################] 100%
error: failed to commit transaction (conflicting files)
python-requests-oauthlib: /usr/lib/python3.6/site-packages/requests_oauthlib/compliance_fixes/__pycache__/fitbit.cpython-36.pyc exists in filesystem
python-requests-oauthlib: /usr/lib/python3.6/site-packages/requests_oauthlib/compliance_fixes/fitbit.py exists in filesystem
Errors occurred, no packages were upgraded.
================ FATAL ERROR ==============
An error occured during installation!!
Unable to install global dependencies
===========================================

Might be a potential packaging problem upstream? Investigating...

how to run frog through virtual box?

Hey,

Ive been trying to get the LaMachine to work, but i cant figure out the install instructions. I'm running windows, and downloaded vagrant & virtual box.

Do I

  1. boot up virtual box and install ubuntu,
  2. run the $ git ... && laMachine in terminal
  3. ??
  4. profit

Honestly i'm completely lost.

Do I boot up the terminal in virtualbox (can i do that)? Does LaMachine run inside the virtualbox or do i need to install an operating system first?

If you could provide a detailed guide, or at least a couple more pointers, would be forever grateful.

[lamachine2] Provide a single point-of-entry bootstrap script

Single bootstrap command as an entry point to all possible installations.

Users will be able to bootstrap LaMachine v2 entirely through one command:

bash <(curl -s https://raw.githubusercontent.com/proycon/LaMachine/master/bootstrap.sh)

It will start an automated wizard asking the user for his choices for his LaMachine
build. The script will provide a single entry point and will install necessary initial dependencies (e.g. Vagrant, Ansible). It is geared for a wide variety of unix-like platforms (including Windows 10 with the Linux Subsystem).

python-frog doesn't install on proycon/lamachine image

On the docker image, python-frog is not yet installed (could be usefull to include), and it doesn't install either:

with pip install python-frog:

[...]
Error compiling Cython file:
    ------------------------------------------------------------
    ...
            pass



    cdef class Document:
        cdef frog_classes.Document capi
            ^
    ------------------------------------------------------------

    frog_wrapper.pyx:34:9: 'Document' is not a type identifier
    building 'frog' extension
    creating build
    creating build/temp.linux-x86_64-3.5
    gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fPIC -I/root/local/include/ -I/usr/include/ -I/usr/include/libxml2 -I/usr/local/include/ -I/usr/include/python3.5m -c frog_wrapper.cpp -o build/temp.linux-x86_64-3.5/frog_wrapper.o --std=c++0x
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    frog_wrapper.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
     #error Do not use this file, it is the result of a failed Cython compilation.
      ^
    error: command 'gcc' failed with exit status 1

from source:

[...]
frog_wrapper.cpp:364:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
     #define __Pyx_PyBool_FromLong(b) ((b) ? __Pyx_NewRef(Py_True) : __Pyx_NewRef(Py_False))
                                        ^
    frog_wrapper.cpp: In function ‘int __pyx_pf_4frog_11FrogOptions_4__setitem__(__pyx_obj_4frog_FrogOptions*, PyObject*, PyObject*)’:
    frog_wrapper.cpp:2220:24: error: ‘class FrogOptions’ has no member named ‘doDeepMorph’
         __pyx_v_self->capi.doDeepMorph = ((bool)__pyx_t_6);
                            ^
    error: command 'gcc' failed with exit status 1

Docker build fails

Ucto and frog are not installed properly. And docker didn't correctly report my test as failed.

Error when deploying virtual environment

This is what happens at step 3 of the virtual environment method, the bootstrapping part. It happens with and without sudo.

The log file does not show the error message, so I'm pasting the terminal output.

sigmund@debian:~$ sudo /tmp/LaMachine/virtualenv-bootstrap.sh
(logging to /tmp/virtualenv-bootstrap-1487588652.log)
=====================================================================
           ,              LaMachine - NLP Software distribution
          ~)                     (https://proycon.github.io/LaMachine)
           (----í         Language Machines research group
            /| |\         & Centre for Language and Speech Technology
           / / /|	        Radboud University Nijmegen 
=====================================================================

Bootstrapping VirtualEnv....

================================================================================
      LaMachine will install the latest stable releases where possible
================================================================================
Detecting package manager...
Detected OS: debian
Detected distribution ID: debian
Detected distribution release: 8

-------------------------------
Updating global dependencies 
-------------------------------
 (this step, and only this step, may require root access, skip it with CTRL-C if you do not have it)
Command: sudo apt-get -m  install pkg-config git-core make gcc g++ autoconf automake autoconf-archive libtool autotools-dev libicu-dev libxml2-dev libxslt1-dev libbz2-dev zlib1g-dev libtar-dev libaspell-dev libhunspell-dev libboost-all-dev python3 python3-dev python3-pip python-virtualenv libgnutls28-dev libcurl4-gnutls-dev wget libexttextcat-dev libatlas-dev libblas-dev gfortran libsuitesparse-dev libfreetype6-dev myspell-nl perl libsort-naturally-perl tesseract-ocr tesseract-ocr-eng tesseract-ocr-nld poppler-utils
Ign http://ftp.belnet.be jessie InRelease
Hit http://ftp.belnet.be jessie-updates InRelease
(...)
Reading package lists...
Building dependency tree...
Reading state information...
libaspell-dev is already the newest version.
libatlas-dev is already the newest version.
autoconf is already the newest version.
autoconf-archive is already the newest version.
automake is already the newest version.
autotools-dev is already the newest version.
libblas-dev is already the newest version.
libboost-all-dev is already the newest version.
libbz2-dev is already the newest version.
libcurl4-gnutls-dev is already the newest version.
myspell-nl is already the newest version.
libfreetype6-dev is already the newest version.
g++ is already the newest version.
gcc is already the newest version.
gfortran is already the newest version.
git-core is already the newest version.
libgnutls28-dev is already the newest version.
libhunspell-dev is already the newest version.
libicu-dev is already the newest version.
libexttextcat-dev is already the newest version.
libsort-naturally-perl is already the newest version.
libtar-dev is already the newest version.
libtool is already the newest version.
libxml2-dev is already the newest version.
libxslt1-dev is already the newest version.
make is already the newest version.
perl is already the newest version.
pkg-config is already the newest version.
poppler-utils is already the newest version.
python3-pip is already the newest version.
python-virtualenv is already the newest version.
python3 is already the newest version.
python3-dev is already the newest version.
libsuitesparse-dev is already the newest version.
tesseract-ocr is already the newest version.
tesseract-ocr-eng is already the newest version.
tesseract-ocr-nld is already the newest version.
wget is already the newest version.
zlib1g-dev is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.

-----------------------------------------
Creating virtual environment
-----------------------------------------
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 70, in <module>
    import packaging.version
ImportError: No module named 'packaging'
================ FATAL ERROR ==============
An error occurred during installation!!
Unable to create virtual environment
===========================================

pip and pip3 are up to date, packaging exists in pip but throws an error in pip3:

sigmund@debian:~$ sudo pip install packaging
Requirement already satisfied: packaging in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages/six-1.10.0-py2.7.egg (from packaging)
Requirement already satisfied: pyparsing in /usr/lib/python2.7/dist-packages (from packaging)
sigmund@debian:~$ sudo pip3 install packaging
Traceback (most recent call last):
  File "/usr/bin/pip3", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 70, in <module>
    import packaging.version
ImportError: No module named 'packaging'

Passing the python2 argument when bootstrapping the virtual environment throws the same error.

Running ``lamachine-update.sh`` sometimes fails with weird syntax error, but upon rerunning everything is fine

I often find that the virtualenv lamachine-update.sh fails with a strange syntax error. I wonder if it might be caused by the running script being updated whilst it runs? Isn't it entirely in memory yet?

Simply rerunning lamachine-update.sh after this error always fixes it.

--------------------------------------------------------
Updating LaMachine itself (branch: master)
--------------------------------------------------------
/scratch2/www/lamachine/src/LaMachine
Already on 'master'
Your branch is up-to-date with 'origin/master'.
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From https://github.com/proycon/LaMachine
   9686aa0..19ca62f  master     -> origin/master
Updating 9686aa0..19ca62f
Fast-forward
 virtualenv-bootstrap.sh | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
/scratch2/www/lamachine/bin/lamachine-update.sh: line 783: syntax error near unexpected token `('
/scratch2/www/lamachine/bin/lamachine-update.sh: line 783: `echo "Updating LaMachine itself (branch: $BRANCH)"

problem with using ticcl.nf with polish and english lexicon in Docker relase

Might be problem with reading .txt stead of folia.xml

`nextflow run LanguageMachines/PICCL/ticcl.nf --inputdir /home/projects/Kaggle_denoise/ICDAR-2017-Post-OCR-Correction/text/ --lexicon data/int/pol/pol.aspell.dict --alphabet data/int/pol/pol.aspell.dict.lc.chars --charconfus data/int/pol/pol.aspell.dict.c0.d2.confusion --inputtype 'text'

N E X T F L O W ~ version 0.27.4
Launching LanguageMachines/PICCL [mad_legentil] - revision: a006ed747c [master]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [e9754ef2e1]

TICCL Pipeline

[warm up] executor > local
[f4/ce2b3e] Submitted process > txt2folia (1)
[3f/e17ce8] Submitted process > corpusfrequency (1)
ERROR ~ Error executing process > 'corpusfrequency (1)'

Caused by:
Missing output file(s) corpus.wordfreqlist.tsv expected by process corpusfrequency (1)

Command executed:

set +u
if [ ! -z "" ]; then
source /bin/activate
fi
set -u

FoLiA-stats --class "OCR" -s -t 1 -e folia.xml --lang=none --ngram 1 -o corpus .

Command exit status:
0

Command output:
start processing of 1 files
done processsing directory '.'
start calculating the results
in total 0 n-grams were found.

Command error:

XML-error: PCDATA invalid Char value 12

FoLiA-stats: failed to load document './doc.folia.xml'
FoLiA-stats: reason: XML error: No XML document read

Work dir:
/usr/src/LaMachine/work/3f/e17ce8cb5ee9d497957609b34fdc29

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

-- Check '.nextflow.log' file for details`

CLAM refuses to answer to Nginx

Hello,
we are trying to kickstart the LaMachine in form of a Docker deployment in AWS Cloud. Our purpose is to get Frog working with RESTful wrapper.
we've modified the Dockerfile with nginx setting to run in foreground, so it would not close the running image. (added one line with RUN echo "daemon off;" >> /usr/src/LaMachine/nginx.conf and replaced CMD from bash to CMD /usr/src/LaMachine/startwebservices.sh)

i've built the image on windows pc and it builds and starts and runs...
when i run it locally /frog is responding with a decent output of a clam service (i suppose,.. we are investigating the tool).

however, when i run it in the Cloud - i see a "default" page, but upstreams are failing with nginx error of connecting to upstream. or, actually, responding with a blank page.

2017/03/08 15:12:57 [error] 13#13: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 172.17.0.1, server: localhost, request: "GET /frog HTTP/1.1", upstream: "uwsgi://127.0.0.1:3032", host: "blabla.eu-west-1.elasticbeanstalk.com", referrer: "http://blabla.eu-west-1.elasticbeanstalk.com/"

and later

[pid: 27|app: 0|req: 10/10] 172.17.0.1 () {52 vars in 909 bytes} [Wed Mar 8 16:09:36 2017] GET /frog => generated 5469 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 80 bytes (1 switches on core 1)
[pid: 27|app: 0|req: 11/11] 172.17.0.1 () {52 vars in 909 bytes} [Wed Mar 8 16:09:36 2017] GET /frog => generated 5469 bytes in 1 msecs (HTTP/1.1 200)

can someone point us to a right direction of what could be the reason of this failure and how to fix it?

64bit Amazon Linux 2016.09 v2.5.0 running Docker 1.12.6

macOS frog make error

op macOS High Sierra, 10.13.2

--------------------------------------------------------
Installing/updating frog
--------------------------------------------------------
Cloning into 'frog'...
remote: Counting objects: 5884, done.
remote: Total 5884 (delta 0), reused 0 (delta 0), pack-reused 5884
Receiving objects: 100% (5884/5884), 41.72 MiB | 5.12 MiB/s, done.
Resolving deltas: 100% (4419/4419), done.
Fetching origin
   Upgrading from  to latest stable release v0.13.9 ...
Note: checking out 'tags/v0.13.9'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 403d06e... NEWS about release
aclocal: installing 'm4/ax_lib_readline.m4' from '/usr/local/share/aclocal/ax_lib_readline.m4'
aclocal: installing 'm4/ax_pthread.m4' from '/usr/local/share/aclocal/ax_pthread.m4'
aclocal: installing 'm4/libtool.m4' from '/usr/local/share/aclocal/libtool.m4'
aclocal: installing 'm4/ltoptions.m4' from '/usr/local/share/aclocal/ltoptions.m4'
aclocal: installing 'm4/ltsugar.m4' from '/usr/local/share/aclocal/ltsugar.m4'
aclocal: installing 'm4/ltversion.m4' from '/usr/local/share/aclocal/ltversion.m4'
aclocal: installing 'm4/lt~obsolete.m4' from '/usr/local/share/aclocal/lt~obsolete.m4'
aclocal: installing 'm4/pkg.m4' from '/usr/local/share/aclocal/pkg.m4'
glibtoolize: putting auxiliary files in '.'.
glibtoolize: linking file './ltmain.sh'
configure.ac:26: installing './compile'
configure.ac:26: installing './config.guess'
configure.ac:26: installing './config.sub'
configure.ac:6: installing './install-sh'
configure.ac:6: installing './missing'
Makefile.am: installing './INSTALL'
src/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking build system type... x86_64-apple-darwin17.3.0
checking host system type... x86_64-apple-darwin17.3.0
checking how to print strings... printf
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 196608
checking how to convert x86_64-apple-darwin17.3.0 file names to x86_64-apple-darwin17.3.0 format... func_convert_file_noop
checking how to convert x86_64-apple-darwin17.3.0 file names to toolchain format... func_convert_file_noop
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin17.3.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking whether the g++ linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fno-common -DPIC
checking if g++ PIC flag -fno-common -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin17.3.0 dyld
checking how to hardcode library paths into programs... immediate
checking for stdbool.h that conforms to C99... no
checking for _Bool... no
checking for inline... inline
checking for size_t... yes
checking for pid_t... yes
checking vfork.h usability... no
checking vfork.h presence... no
checking for vfork.h... no
checking for fork... yes
checking for vfork... yes
checking for working fork... yes
checking for working vfork... (cached) yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for localtime_r... yes
checking whether gcc is Clang... yes
checking whether Clang needs flag to prevent "argument unused" warning when linking with -pthread... -Qunused-arguments
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking whether more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for g++ option to support OpenMP... unsupported
configure: We don't have OpenMP. Multithreaded operation is disabled
checking for pkg-config... /usr/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libxml-2.0 >= 2.6.16 ... yes
checking for icu-uc >= 50 icu-io ... yes
checking for a readline compatible library... -lreadline
checking readline.h usability... no
checking readline.h presence... no
checking for readline.h... no
checking readline/readline.h usability... yes
checking readline/readline.h presence... yes
checking for readline/readline.h... yes
checking whether readline supports history... yes
checking history.h usability... no
checking history.h presence... no
checking for history.h... no
checking readline/history.h usability... yes
checking readline/history.h presence... yes
checking for readline/history.h... yes
checking for ticcutils >= 0.9 ... yes
checking for timbl >= 6.4.4 ... yes
checking for mbt >= 3.2.14 ... yes
checking for folia >= 1.10... yes
checking for ucto >= 0.9.7 ... yes
checking for frogdata >= 0.15 ... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating frog.pc
config.status: creating m4/Makefile
config.status: creating docs/Makefile
config.status: creating tests/Makefile
config.status: creating src/Makefile
config.status: creating include/Makefile
config.status: creating include/frog/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in src
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../include  -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3   -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include  -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2  -MT FrogAPI.lo -MD -MP -MF .deps/FrogAPI.Tpo -c -o FrogAPI.lo FrogAPI.cxx
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT FrogAPI.lo -MD -MP -MF .deps/FrogAPI.Tpo -c FrogAPI.cxx  -fno-common -DPIC -o .libs/FrogAPI.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT FrogAPI.lo -MD -MP -MF .deps/FrogAPI.Tpo -c FrogAPI.cxx -o FrogAPI.o >/dev/null 2>&1
mv -f .deps/FrogAPI.Tpo .deps/FrogAPI.Plo
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../include  -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3   -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include  -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2  -MT mbma_rule.lo -MD -MP -MF .deps/mbma_rule.Tpo -c -o mbma_rule.lo mbma_rule.cxx
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT mbma_rule.lo -MD -MP -MF .deps/mbma_rule.Tpo -c mbma_rule.cxx  -fno-common -DPIC -o .libs/mbma_rule.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT mbma_rule.lo -MD -MP -MF .deps/mbma_rule.Tpo -c mbma_rule.cxx -o mbma_rule.o >/dev/null 2>&1
mv -f .deps/mbma_rule.Tpo .deps/mbma_rule.Plo
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../include  -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3   -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include  -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2  -MT mbma_mod.lo -MD -MP -MF .deps/mbma_mod.Tpo -c -o mbma_mod.lo mbma_mod.cxx
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT mbma_mod.lo -MD -MP -MF .deps/mbma_mod.Tpo -c mbma_mod.cxx  -fno-common -DPIC -o .libs/mbma_mod.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT mbma_mod.lo -MD -MP -MF .deps/mbma_mod.Tpo -c mbma_mod.cxx -o mbma_mod.o >/dev/null 2>&1
mv -f .deps/mbma_mod.Tpo .deps/mbma_mod.Plo
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../include  -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3   -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include  -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2  -MT mbma_brackets.lo -MD -MP -MF .deps/mbma_brackets.Tpo -c -o mbma_brackets.lo mbma_brackets.cxx
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT mbma_brackets.lo -MD -MP -MF .deps/mbma_brackets.Tpo -c mbma_brackets.cxx  -fno-common -DPIC -o .libs/mbma_brackets.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT mbma_brackets.lo -MD -MP -MF .deps/mbma_brackets.Tpo -c mbma_brackets.cxx -o mbma_brackets.o >/dev/null 2>&1
mv -f .deps/mbma_brackets.Tpo .deps/mbma_brackets.Plo
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../include  -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3   -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include  -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2  -MT clex.lo -MD -MP -MF .deps/clex.Tpo -c -o clex.lo clex.cxx
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../include -DSYSCONF_PATH=\"/Users/antalb/Software/lamachine/share\" -std=c++11 -W -Wall -pedantic -g -O3 -g -O2 -I/usr/include/libxml2 -I/usr/local/Cellar/icu4c/60.2/include -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/include/libxml2 -MT clex.lo -MD -MP -MF .deps/clex.Tpo -c clex.cxx  -fno-common -DPIC -o .libs/clex.o
In file included from clex.cxx:32:
../include/frog/clex.h:47:22: error: implicit instantiation of undefined
      template 'std::__1::basic_string<char, std::__1::char_traits<char>,
      std::__1::allocator<char> >'
  inline std::string get_iDescr( char c ) { return iNames[c]; }
                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iosfwd:193:32: note: 
      template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_string;
                               ^
In file included from clex.cxx:32:
../include/frog/clex.h:47:52: error: implicit instantiation of undefined
      template 'std::__1::basic_string<char, std::__1::char_traits<char>,
      std::__1::allocator<char> >'
  inline std::string get_iDescr( char c ) { return iNames[c]; }
                                                   ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iosfwd:193:32: note: 
      template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_string;
                               ^
In file included from clex.cxx:32:
../include/frog/clex.h:48:22: error: implicit instantiation of undefined
      template 'std::__1::basic_string<char, std::__1::char_traits<char>,
      std::__1::allocator<char> >'
  inline std::string get_tDescr( CLEX::Type t ) {  return tagNames[t]; }
                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iosfwd:193:32: note: 
      template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_string;
                               ^
In file included from clex.cxx:32:
../include/frog/clex.h:48:59: error: implicit instantiation of undefined
      template 'std::__1::basic_string<char, std::__1::char_traits<char>,
      std::__1::allocator<char> >'
  inline std::string get_tDescr( CLEX::Type t ) {  return tagNames[t]; }
                                                          ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iosfwd:193:32: note: 
      template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_string;
                               ^
4 errors generated.
make[2]: *** [clex.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
================ FATAL ERROR ==============
An error occurred during installation!!
frog make failed
===========================================

[lamachine2] Test other distributions

  • Switch the base Linux distribution from Arch Linux to Debian Linux (stable/stretch), as this makes more sense in production environments (e.g wrt security) and is what more users
    are accustomed too. (DONE)
  • Allow for easy expansion to other distributions, allowing to accomodate production environments that run on e.g. CentOS.

libgnutls-dev not in debian for local installation

The local installation tries to install libgnutls-dev, but it looks like it has been removed after Debian 8 (Wheezy), so apt shows the error E: Package 'libgnutls-dev' has no installation candidate , the LaMachine installation gives an error and continues. I've manually installed all the packages except the gnutls one and the LaMachine installation succeeded (except gecco and [python] timble, but that's because I used python2).

frog server stops running in Docker image

After building the docker image, starting the frog server doesn't work because after booting the server stops immediately with error message Killed. I didn't kill the frog though.

[root@413e5da503d2 LaMachine]# frog -S 40000
frog 0.13.1 (c) CLTS, ILK 1998 - 2016
[...]
frog-parser-:initiating parser ...
frog-parser-:reading /etc/frog//Frog.mbdp.1.0.pairs.sampled.ibase
frog-parser-:reading /etc/frog//Frog.mbdp.1.0.dir.ibase
Killed
[root@413e5da503d2 LaMachine]#

Using frog, for example through the python bindings, works fine.

[virtualenv] Some packages don't work on Mac OS X (hunspell, python-frog)

During install:

Python-timbl compilation, no pthread on Mac OS X?

building 'timblapi' extension
creating build
creating build/temp.macosx-10.10-x86_64-3.4
creating build/temp.macosx-10.10-x86_64-3.4/src
clang -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/boost/1.58.0/include/ -I/Users/proycon/work/LaMachine/lamachine/include -I/usr/include/libxml2 -I/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c src/timblapi.cc -o build/temp.macosx-10.10-x86_64-3.4/src/timblapi.o
In file included from src/timblapi.cc:47:
src/timblapi.h:72:165: error: expected expression
        TimblApiWrapper(const std::string& args, const std::string& name="") : Timbl::TimblAPI(args, name) { detachedexp = NULL; debug = false; runningthreads = 0; lock = PTHREAD_MUTEX_INITIALIZER;}
                                                                                                                                                                           ^
/usr/include/pthread.h:180:35: note: expanded from macro 'PTHREAD_MUTEX_INITIALIZER'
#define PTHREAD_MUTEX_INITIALIZER {_PTHREAD_MUTEX_SIG_init, {0}}

Python-frog compilation:

clang++ -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.4/frog_wrapper.o -L/Users/proycon/work/LaMachine/lamachine/lib -L/Users/proycon/local/lib/ -L/usr/lib -L/usr/local/lib -lfrog -lucto -lfolia -o build/lib.macosx-10.10-x86_64-3.4/frog.so
ld: warning: directory not found for option '-L/Users/proycon/local/lib/'
running install_lib
copying build/lib.macosx-10.10-x86_64-3.4/frog.so -> /Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages
running install_egg_info
Writing /Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages/python_frog-0.2.5.1-py3.4.egg-info

Gecco setup, error in 3rd party dependency hunspell.

rocessing dependencies for Gecco==0.2
Searching for hunspell
Reading https://pypi.python.org/simple/hunspell/
Best match: hunspell 0.3.3
Downloading https://pypi.python.org/packages/source/h/hunspell/hunspell-0.3.3.tar.gz#md5=25ceaf642f16e623a857689b2323bd65
Processing hunspell-0.3.3.tar.gz
Writing /var/folders/16/2x7n_xyn0nl64z08qk695c440000gn/T/easy_install-egn1e49c/hunspell-0.3.3/setup.cfg
Running hunspell-0.3.3/setup.py -q bdist_egg --dist-dir /var/folders/16/2x7n_xyn0nl64z08qk695c440000gn/T/easy_install-egn1e49c/hunspell-0.3.3/egg-dist-tmp-awkhvry5
hunspell.c:21:10: fatal error: 'hunspell.h' file not found
#include <hunspell.h>
         ^
1 error generated.
error: Setup script exited with error: command 'clang' failed with exit status 1
================= ERROR ===================
setup.py install gecco failed
===========================================

From python interpreter:

Python 3.4.3 (default, May  1 2015, 19:14:18)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.49)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import frog
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages/frog.so, 2): Symbol not found: __ZNSt3__1lsIcNS_11char_traitsIcEENS_9allocatorIcEEEERNS_13basic_ostreamIT_T0_EES9_RKNS_12basic_stringIS6_S7_T1_EE
  Referenced from: /Users/proycon/work/LaMachine/lamachine/lib/libfrog.1.dylib
  Expected in: flat namespace
 in /Users/proycon/work/LaMachine/lamachine/lib/libfrog.1.dylib
>>> import timbl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages/timbl.py", line 24, in <module>
    import timblapi
ImportError: dlopen(/Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages/timblapi.so, 2): Symbol not found: __ZN5boost6python6detail11init_moduleER11PyModuleDefPFvvE
  Referenced from: /Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages/timblapi.so
  Expected in: flat namespace
 in /Users/proycon/work/LaMachine/lamachine/lib/python3.4/site-packages/timblapi.so
>>>

Virtualenv-bootstrap virtualenv/pip not found

I'm trying to run LaMachine in a CentOS 7 LXC container.
I cloned LaMachine and runned ./virtualenv-bootstrap.sh python2,
but I get No suitable package manage detected, virtualenv not found and pip3 or pip not found. I have virtualenv and pip installed, and I can run those commands in the terminal. Do I have to do something extra?

Compilation of python3-timbl fails inside of Docker

I just ran lamachine-update.sh inside of the proycon/lamachine container. The process failed when trying to install python3-timbl:

Searching for python3-timbl
Reading https://pypi.python.org/simple/python3-timbl/
Best match: python3-timbl 2016.3.13
Downloading https://pypi.python.org/packages/cb/51/6ca37d2098d75f6c4f76e8355cffc38c4ef1828974006ae69fad8b80d433/python3-timbl-2016.03.13.tar.gz#md5=834cb3a035d472e9aad64119da525cb8
Processing python3-timbl-2016.03.13.tar.gz
Writing /tmp/easy_install-0zu3ilpp/python3-timbl-2016.03.13/setup.cfg
Running python3-timbl-2016.03.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0zu3ilpp/python3-timbl-2016.03.13/egg-dist-tmp-riy_lvts
/usr/lib/python3.5/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '2016.03.13' to '2016.3.13'
  normalized_version,
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/boost/type_traits/ice.hpp:15:0,
                 from /usr/include/boost/python/detail/def_helper.hpp:9,
                 from /usr/include/boost/python/class.hpp:29,
                 from /usr/include/boost/python.hpp:18,
                 from src/timblapi.h:53,
                 from src/timblapi.cc:47:
/usr/include/boost/type_traits/detail/ice_or.hpp:17:71: note: #pragma message: NOTE: Use of this header (ice_or.hpp) is deprecated
 # pragma message("NOTE: Use of this header (ice_or.hpp) is deprecated")
                                                                       ^
In file included from /usr/include/boost/type_traits/ice.hpp:16:0,
                 from /usr/include/boost/python/detail/def_helper.hpp:9,
                 from /usr/include/boost/python/class.hpp:29,
                 from /usr/include/boost/python.hpp:18,
                 from src/timblapi.h:53,
                 from src/timblapi.cc:47:
/usr/include/boost/type_traits/detail/ice_and.hpp:18:72: note: #pragma message: NOTE: Use of this header (ice_and.hpp) is deprecated
 # pragma message("NOTE: Use of this header (ice_and.hpp) is deprecated")
                                                                        ^
In file included from /usr/include/boost/type_traits/ice.hpp:17:0,
                 from /usr/include/boost/python/detail/def_helper.hpp:9,
                 from /usr/include/boost/python/class.hpp:29,
                 from /usr/include/boost/python.hpp:18,
                 from src/timblapi.h:53,
                 from src/timblapi.cc:47:
/usr/include/boost/type_traits/detail/ice_not.hpp:17:72: note: #pragma message: NOTE: Use of this header (ice_not.hpp) is deprecated
 # pragma message("NOTE: Use of this header (ice_not.hpp) is deprecated")
                                                                        ^
In file included from /usr/include/boost/type_traits/ice.hpp:18:0,
                 from /usr/include/boost/python/detail/def_helper.hpp:9,
                 from /usr/include/boost/python/class.hpp:29,
                 from /usr/include/boost/python.hpp:18,
                 from src/timblapi.h:53,
                 from src/timblapi.cc:47:
/usr/include/boost/type_traits/detail/ice_eq.hpp:17:71: note: #pragma message: NOTE: Use of this header (ice_eq.hpp) is deprecated
 # pragma message("NOTE: Use of this header (ice_eq.hpp) is deprecated")
                                                                       ^
src/timblapi.cc: In member function ‘boost::python::tuple TimblApiWrapper::classify(const string&)’:
src/timblapi.cc:72:19: error: could not convert ‘std::make_tuple(_Elements&& ...) [with _Elements = {bool&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}](cls)’ from ‘std::tuple<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >’ to ‘boost::python::tuple’
  return make_tuple(result, cls);
         ~~~~~~~~~~^~~~~~~~~~~~~
src/timblapi.cc: In member function ‘boost::python::tuple TimblApiWrapper::classify2(const string&)’:
src/timblapi.cc:81:19: error: could not convert ‘std::make_tuple(_Elements&& ...) [with _Elements = {bool&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, double&}](cls, distance)’ from ‘std::tuple<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, double>’ to ‘boost::python::tuple’
  return make_tuple(result, cls, distance);
         ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
src/timblapi.cc: In member function ‘boost::python::tuple TimblApiWrapper::classify3(const string&, bool, unsigned char)’:
src/timblapi.cc:96:30: error: could not convert ‘std::make_tuple(_Elements&& ...) [with _Elements = {bool, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::python::dict, double&}](cls, TimblApiWrapper::dist2dict(const Timbl::ValueDistribution*, bool, double) const(distrib, ((int)normalize), (double)0), distance)’ from ‘std::tuple<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::python::dict, double>’ to ‘boost::python::tuple’
             return make_tuple(true, cls, dist2dict(distrib, normalize), distance);
                    ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/timblapi.cc: In member function ‘boost::python::tuple TimblApiWrapper::classify3safe(const string&, bool, unsigned char)’:
src/timblapi.cc:158:30: error: could not convert ‘std::make_tuple(_Elements&& ...) [with _Elements = {bool, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::python::dict, double&}](cls, TimblApiWrapper::dist2dict(const Timbl::ValueDistribution*, bool, double) const(distrib, ((int)normalize), (double)0), distance)’ from ‘std::tuple<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::python::dict, double>’ to ‘boost::python::tuple’
             return make_tuple(true, cls, dist2dict(distrib, normalize), distance);
                    ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: Setup script exited with error: command 'gcc' failed with exit status 1
================= ERROR ===================
setup.py install gecco failed

docker startwebservices doesn't work

Creating clam webservices with
docker run -p 8080:80 -t -i proycon/lamachine
and then starting them with
/usr/src/LaMachine/startwebservices.sh
doesn't work, no service available on port 8080.
On testing nginx with
nginx -t
I did get
2018/01/26 13:43:15 [emerg] 6#6: getpwnam("http") failed
When a user 'http' is added with
useradd --no-create-home http
before the call to startwebservices.sh, at least ucto, frog, timbl and colibri-core seem to work

[lamachine2] Rewrite entire provisioning infrastructure through Ansible

One of the core changes in LaMachine v2 is the use of Ansible as the primary provisioning engine instead of custom shell scripts.

  • Supported by Vagrant for provisioning of Virtual Machines
  • Supports Docker for provisioning containers
  • Makes deployment in production environments easier; allows setting up multiple hosts/containers/vms at once (cf. Docker Compose)
  • Improves modularity

frogwrapper.py seems to be making a wrong assumption about location of the frog config files

We're trying to use frog via clam. When starting the project it dies with the following in the error.log:

[CLAM Dispatcher] Adding to PYTHONPATH: /usr/lib/python3.5/site-packages/clam/config
[CLAM Dispatcher] Started CLAM Dispatcher v2.1 with clam.config.frog (2016-05-19 14:19:52)
[CLAM Dispatcher] Running python3 "/usr/lib/python3.5/site-packages/clam/wrappers/frogwrapper.py" "/usr/bin/" "/clamdata/frog.clam/projects/anonymous/test1/clam.xml" "/clamdata/frog.clam/projects/anonymous/test1/.status" "/clamdata/frog.clam/projects/anonymous/test1/output/" > "/clamdata/frog.clam/projects/anonymous/test1/output//log"
[CLAM Dispatcher] Running with pid 31 (2016-05-19 14:19:52)
frog 0.13.3 (c) CLTS, ILK 1998 - 2016
CLST  - Centre for Language and Speech Technology,Radboud University
ILK   - Induction of Linguistic Knowledge Research Group,Tilburg University
based on [ucto 0.8.4, libfolia 1.0.1, timbl 6.4.7, ticcutils 0.11, mbt 3.2.15]
unable to read configuration from /usr/bin/../etc/frog/frog.cfg
failed to read configuration from '/usr/bin/../etc/frog/frog.cfg' !!
Did you correctly install the frogdata package?
frog-:fatal error: init failed
Processing 9851.txt...
    DocID from filename: 9851
Invoking Frog
[CLAM Dispatcher] Process ended (2016-05-19 14:19:53, 0.401692s) 
[CLAM Dispatcher] Removing temporary files
[CLAM Dispatcher] Status code out of range (256), setting to 127
[CLAM Dispatcher] Finished (2016-05-19 14:19:53), exit code 127, dispatcher wait time 0.39999999999999997s, duration 0.40246s

As far as I can see, the frogwrapper script assumes that /etc/frog folder can be reached one level above the /usr/bin (so it would be /usr/etc/frog), but there isn't any there. There is /etc/frog under the container root though. It seems like frogwrapper expected frog to be in something like /usr/local/ with the binary in /usr/local/bin and configuration in /usr/local/etc/frog?

read-only filesystem error with docker

On a Debian jessie system:

docker pull proycon/lamachine
latest: Pulling from proycon/lamachine

7fa28f78c23a: Pull complete
5631ac34d1dc: Pull complete
7fa28f78c23a: Download complete
5631ac34d1dc: Download complete
5b9f0c0a871a: Download complete
23f01f465de5: Download complete
054e500f560c: Download complete
0a859bdf1639: Download complete
8a2ff8d51adc: Download complete
339ea9783423: Download complete
172cdc63c6d5: Download complete
15be8d3646fb: Download complete
6b40b18485d5: Download complete
99053ec2352c: Download complete
9fe344299f32: Download complete
a0e6544c7147: Download complete
01c228d19253: Download complete
e684b8450f1b: Download complete
4e4e8331bc7b: Download complete
440a385aadbf: Error downloading dependent layers                                               9db285bdfc13: Error pulling image (latest) from proycon/lamachine, endpoint: https://registry-1.docker.io/v1/, ApplyLayer exit status 1 stdout:  stderr: write /var/cache/pacman/pkg/aspell-0.9db285bdfc13: Error pulling image (latest) from proycon/lamachine, ApplyLayer exit status 1 stdout:  stderr: write /var/cache/pacman/pkg/aspell-0.60.6.1-3-x86_64.pkg.tar.xz: read-only file system
FATA[0354] Error pulling image (latest) from proycon/lamachine, ApplyLayer exit status 1 stdout:  stderr: write /var/cache/pacman/pkg/aspell-0.60.6.1-3-x86_64.pkg.tar.xz: read-only file system

error with virtualenv-bootstrap.sh and ticcutils

On a Debian jessie system, I get the following error:

$ ~/src/LaMachine/% ./virtualenv-bootstrap.sh tscan
[...]
--------------------------------------------------------
Installing/updating ticcutils
--------------------------------------------------------
Cloning into 'ticcutils'...
remote: Counting objects: 1181, done.
remote: Total 1181 (delta 0), reused 0 (delta 0), pack-reused 1181
Receiving objects: 100% (1181/1181), 236.65 KiB | 122.00 KiB/s, done.
Resolving deltas: 100% (867/867), done.
Checking connectivity... done.
aclocal: installing 'm4/libtool.m4' from '/usr/share/aclocal/libtool.m4'
aclocal: installing 'm4/ltoptions.m4' from '/usr/share/aclocal/ltoptions.m4'
aclocal: installing 'm4/ltsugar.m4' from '/usr/share/aclocal/ltsugar.m4'
aclocal: installing 'm4/ltversion.m4' from '/usr/share/aclocal/ltversion.m4'
aclocal: installing 'm4/lt~obsolete.m4' from '/usr/share/aclocal/lt~obsolete.m4'
aclocal: installing 'm4/pkg.m4' from '/usr/share/aclocal/pkg.m4'
libtoolize: putting auxiliary files in `.'.
libtoolize: linking file `./ltmain.sh'
configure.ac:68: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1
================ FATAL ERROR ==============
An error occured during installation!!
ticcutils bootstrap failed
===========================================

Characters sometimes dissapear when using Frog

When using lamachine (frog) I noticed that sometimes characters dissapear (in tokenization?). A few examples are:

met een kreatininespiegel tussen de 120 en de 140mmol/L.
Becomes:
met een kreatininespiegel tussen de 120 en de 140mmol L.

Berkhof, arts-assistente longziekten Berkhof arts-assistente longziekten/dr.
Becomes:
Berkhof , arts-assistente longziekten Berkhof arts-assistente longziekten dr.

Lengte 1,85m.
Becomes:
Lengte 1,8 m.

ECG: sinusritme 70 slagen per minuut, smal QRS, oud voorwandinfarct, QTc 426ms.
Becomes:
ECG : sinusritme 70 slagen per minuut , smal QRS , oud voorwandinfarct , QTc 42 ms.

cyste, astma of mengbeeld met COPD, FEV1 0.98L.
Becomes:
cyste , astma of mengbeeld met COPD , FEV1 0.9 L.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.