totara/totara-docker-dev

A docker setup to create a development environment for Totara Learn

totara-docker-dev's Issues

Use directory permissions defined in config.php when using chmod

When setting the dataroot directory permissions, tdb always sets them to 0777. This may not be in sync with what is defined in the config.php, which can cause phpunit failures. We should instead grab the $CFG->directorypermissions value and use in within the script.

Make nodejs container compatible with CI system

Is your feature request related to a problem? Please describe.
Currently our CI system runs on an older node version with NPM 5.6.0. Unfortunately updating the package-lock.json file with the latest NPM version changes the lock file in a way which lets our build fail.

Describe the solution you'd like
Until the CI system supports newer NPM versions we should make sure the container version correlates with the one on the CI system to avoid those problems.

Use correct db collation for MySQL/MariaDB

For mysql 5.7/mariadb, the correct DB collation isn't specified in the readme or in the tdb command. This can cause issues when running automated tests etc.
For mysql 5.7/mariadb, they should be updated to use utf8mb4_bin.

Set mariadb version to 10.4

Set mariadb image version to 10.4

Fix behat config for Chrome 95

Currently the configuration for behat in the config.php is not 100% compatible with Chrome 95.

We need to make sure it includes the following:

            "goog:chromeOptions":
              args:
                - '--disable-infobars'
                - '--disable-background-timer-throttling'
                - '--disable-backgrounding-occluded-windows'
              prefs:
                credentials_enable_service: false
              w3c: false
              excludeSwitches:
                - enable-automation

Add proper tests for the docker image

Ideally the docker images should be covered by tests so that if something breaks after a change we know immediately and can fix it before someone starts using a broken image.

There are some tools out there which make testing docker images easy:
https://medium.com/@aelsabbahy/tutorial-how-to-test-your-docker-image-in-half-a-second-bbd13e06a4a9
https://medium.com/@rajatvig/test-driven-development-of-docker-images-with-goss-3b6d23a16eaf
https://github.com/aelsabbahy/goss#manual-installation

Find a way to get xdebug working without setting IP address in .env

Is your feature request related to a problem? Please describe.
Currently you need to set the IP address of the host manually in the .env file to get xdebug working properly. If you are always on the same network with the same IP address this is no problem but as soon as you switch you might need to change it manually which is annoying.

Describe the solution you'd like
Ideally you don't need to set the IP address but the system automatically detects it and xdebug works out of the box.

Additional context
One possible way could be to use xdebug.remote_connect_back as described here: https://xdebug.org/docs/remote

PHP exceptions on the php-8.0 container

When visiting pages (such as any admin page) with php-8.0, we get the following exception:

Exception - Object of class admin_category could not be converted to string

This seems to only happen on the php-8.0 container. Interestingly, the php-8.0-debug container doesn't experience this issue. There might be something missing in the php-8.0 dockerfile or something.

Full stack trace:

line 227 of /lib/classes/collator.php: Error thrown
line 227 of /lib/classes/collator.php: call to Collator->asort()
line 305 of /lib/classes/collator.php: call to core_collator::asort()
line 1285 of /lib/adminlib.php: call to core_collator::asort_objects_by_property()
line 836 of /lib/adminlib.php: call to admin_category->get_children()
line 4095 of /lib/navigationlib.php: call to admin_category->__get()
line 4048 of /lib/navigationlib.php: call to settings_navigation->load_administration_settings()
line 3900 of /lib/navigationlib.php: call to settings_navigation->load_administration_settings()
line 3838 of /lib/navigationlib.php: call to settings_navigation->legacy_initialise()
line 758 of /lib/pagelib.php: call to settings_navigation->initialise()
line 815 of /lib/pagelib.php: call to moodle_page->magic_get_settingsnav()
line 8084 of /lib/adminlib.php: call to moodle_page->__get()
line 149 of /course/management.php: call to admin_externalpage_setup()

Document HTTPS support

Having HTTPS support with our sites may be useful to have. It would give the advantage of letting us save common login password info, marginally increase page load speed, and would stop that warning message when installing/upgrading totara via the web UI.

To implement this we would need to add a server {} directive for the 443 port in the nginx server.conf, and would need to create a root certificate that supports all the totaraXX/totaraXX.debug hosts. Then the dev using docker-dev would need to manually add the root cert to their system's trusted certs.

Add multiple site support to tbash/tzsh commands

Currently if you run tbash or tzsh, it will start the shell from the directory defined as LOCAL_SRC in your .env. It would be more convenient if it could intelligently work out if you were running multiple sites, and if you were, then cd into the directory for the site you were currently in on your host terminal.

Upgrade xdebug for PHP 7.3

Currently there's the RC1 up: https://xdebug.org/
Either we wait until it's fully stable or we could uprade earlier. I don't mind.

Our setup currently seem to work fine with the beta.

'mssql' db type not supported by tdb

tdb does not support the db_type mssql, despite it being a valid db type in older versions of totara. Currently if you want to run mssql, you need to specify the db type as sqlsvr. We should really change it to accept either mssql OR sqlsvr.

Is torara LMS open or close source?

Describe the bug

Different sources from the official website seem to conflict with each other regarding the open source policy of torara.

To Reproduce

Steps to reproduce the behavior:

1. Visit this page from the official web site, which says totara learn lms is open sourced.
2. Visit this official getting code page, which says only parters and subscribers can access the code.

Expected behavior

The open source policy should be clear and consistent.

Allow force overwriting tdb backups

Currently if there is an existing backup made with the tdb command, then we force the dev to delete that backup file before they can backup again. This is intentional - the inconvenience of having to manually do this is far less than the inconvenience caused by accidentally overwriting an important backup.
However, we could still improve the usability a bit here. We should allow specifying an argument like -f/--force to force devs to double check.

Detect active database sessions before attempting backup/restore

If you have an active database session then you can't backup/restore (limitation with the dbms). We should show a helpful error message if there are active sessions when attempting to backup/restore via tdb, instead of attempting it anyway and then having the dbms spit out a bunch of errors like this:

psql:/tdb_backups/test_data.pgsql11:48836: ERROR:  duplicate key value violates unique constraint "mdl_mnetremorpc_id_pk"
DETAIL:  Key (id)=(1) already exists.
CONTEXT:  COPY mdl_mnet_remote_rpc, line 1
psql:/tdb_backups/test_data.pgsql11:48860: ERROR:  duplicate key value violates unique constraint "mdl_mnetremoserv_id_pk"
DETAIL:  Key (id)=(1) already exists.
CONTEXT:  COPY mdl_mnet_remote_service2rpc, line 1
psql:/tdb_backups/test_data.pgsql11:48883: ERROR:  duplicate key value violates unique constraint "mdl_mnetrpc_id_pk"
DETAIL:  Key (id)=(1) already exists.
CONTEXT:  COPY mdl_mnet_rpc, line 1

Cannot call tbash/tzsh from outside a Totara project directory

Describe the bug
Can no longer run tbash or tzsh from any directory. If you're not in the REMOTE_SRC or a child of it, you'll run into the following:

OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: chdir to cwd ("/var/www/totara/src//home/cody/Projects") set in config.json failed: no such file or directory: unknown

To Reproduce
Steps to reproduce the behavior:

1. Open a terminal
2. CD into a non-Totara folder
3. Try running tzsh php-7.3

Expected behavior
Expected to connect to the PHP container via zsh, defaulting to the "/var/www/totara/src" folder

Additional context
Seems to be introduced in 52b5ae7

sub_path="${PWD//$LOCAL_SRC/}"
remote_path="$REMOTE_SRC/$sub_path"

if $PWD isn't a child of $LOCAL_SRC, you'll end up with a malformed path such as /var/www/totara/src//home/cody/Projects
There used to be a further check that caught it, but that's been removed:

if [ ! -f "$local_path/version.php" ]; then
    remote_path="$REMOTE_SRC"
fi

Add XDebug container for PHP 8.0

We currently don't have a way of running XDebug for PHP 8.0 - we should add a container for it

Add pgsql13 container

Add container for PHP 5.3

Early Totara 2.x versions have a minimum php version of 5.3, so it would be good to have a container for it

Replace docker-sync with mutagen

Is your feature request related to a problem? Please describe.
On Mac OS we use docker-sync to sync files from host to the container volumes. This is needed as a normal mounted volume performance is really slow on Mac. Using a normal named volume gives us almost native speed. But docker-sync (or rather Docker and unison) have problems syncing when a larger number of files need to by synced. The syncing just stops and you need to restart docker-sync to get it back working. There's no solution in sight. See: EugenMayer/docker-sync#646

Describe the solution you'd like
I came across a new tool called mutagen (https://mutagen.io/). It's a two-way cross-platform syncing tool. Very lightweight, easy to use. One thing mutagen promises is support for windows. This could make this project really usable on Windows as well.

Mutagen is still in a beta state but it works quite well already. I would like to replace docker-sync or at least add mutagen as a second option to the setup. Mutagen needs little setup. Should not be hard to integrate.

Describe alternatives you've considered
I've looked at some syncing tools which sync via network (Like Resilio Sync or syncthing) but the lag and performance of the sync was not that great.

Make it compatible with mutagen 0.10.0

A new mutagen version came out with some great improvements. But it needs adjustments as there are some changes in the commands.

MSSQL is not working for PHP 7.0+

On a new build of the PHP 7 containers mssql drivers do not work anymore.

For PHP 7.0 the issue is that in the most recent sqlsrv drivers they dropped support for PHP 7.0 (see https://github.com/Microsoft/msphpsql/releases).
In PHP 7.1+ the issue is that pecl install ... does not enable the sqlsrv extensions anymore.

Specify the version in the names of the DB containers

Currently there exists containers for our databases that don't actually define what version they are, e.g. pgsql (should be pgsql12), mysql (should be mysql57), mariadb (should be mariadb105), and mssql (should be mssql2017)
We should rename them to reflect what version they are, as otherwise it makes it more difficult when adding new containers. It also raises more awareness around what version you are actually using.

Improve dataroot handling

In the tdb command, the dataroot can be backed up and restored. However, often when setting up a new site we might want to create a fresh dataroot. This still has to be done independently of docker-dev.

I'm thinking it might be nice to have a tdataroot command that handles creating the dataroot directory - i.e takes in the config.php variables, and does an action in a similar way to tdb. Once that is done, then we can abstract out the backup/restoring of the dataroot from tdb into tdataroot.

With this working, it wouldn't really be necessary to ship the php containers with predefined dataroots. Instead they could just be created/deleted on-demand.

Confusing shell aliases override behaviour

I wanted to change the default email/password for the install command so I copied shell/default-aliases.sh to shell/aliases.sh. However I found that it was not getting picked up within the shell container.

Turns out that it sources all .sh files in the shell folder in alphabetical order. I didn't remove the original default-aliases.sh because it's in GIT, so that version replaced my install command. If I renamed the new file to my-aliases.sh it worked fine as it comes later alphabetically.

Not sure if there is a good fix for this or if it's just a documentation issue as it's obviously handy for the default aliases to work out of the box.

Add example config.php

With the recent directory changes on top of the config values required to get your site running (like the complex logic in order to support totaraXX/totaraXX.debug/totaraXX.behat), I think we should provide an example config.php that docker-dev's can use to get their sites up and running with ease. Ideally it would have if/else conditions in order to be compatible with totara versions 13+ while simultaneously being compatible with t12 and older.

Add support for ML recommender

It would be great to add a container setup to run the ML recommender system used by Totara Engage. System requirements and installation instructions here:

https://help.totaralearning.com/display/DEV/Recommender+installation+and+configuration

'tnpm' command does not work with sites in sub-directories on macOS

Describe the bug
The tnpm command was made with the intention of supporting a docker setup that has multiple sites installed in different sub-directories. When tnpm is run while in a sub-directory it will run the command in the corresponding sub-directory inside the node container. This works fine on linux, but it was not tested on macOS and it does not work.

To Reproduce
Steps to reproduce the behaviour:

1. Run an npm command with tnpm while in a site installed in a subdirectory - e.g. tnpm run tui-dev
2. You will get an npm error like this: enoent ENOENT: no such file or directory, open '/var/www/totara/src/package.json', complaining there is no package.json, because it did not correctly cd into the appropriate subdirectory.

Expected behaviour
The command should be run in the appropriate folder without error.

Xdebug with PHP 7.3 not working

Describe the bug
When trying to open Totara with PHP 7.3 and xdebug the PHP 7.3 fpm backend just terminates.

To Reproduce
Steps to reproduce the behavior:

1. Point nginx to 'php-7.3-debug'
2. Try to open a page
3. See a 503 error
4. See something like WARNING: [pool www] child 10 exited on signal 11 (SIGSEGV) after 45.864850 seconds from start in the logs

Expected behavior
xdebug works

Additional context
I think PHP 7.3 with xdebug 2.7.0beta1 is still unstable and not working. For now I changed the nginx backend for totara73 to the one without xdebug to be able to open the site.

Behat tests with javascript fail due to $_COOKIE['BEHAT'] not set

Describe the bug
Behat tests containing @javascript tags fail due to $_COOKIE['BEHAT'] not set

To Reproduce
Steps to reproduce the behavior:

1. Setup Totara 11 to run on mysql / mariadb using PHP 7.2 / 7.3
2. Add additional parameters for behat to config.php e.g.
   $CFG->behat_wwwroot = 'http://totara72.behat'; $CFG->behat_dataroot = '/var/www/totara/data/ver10.mysql.behat'; $CFG->behat_prefix = 'behat_';
3. Start up containers e.g. bin/tup mariadb php-7.3 selenium-hub bin/tup mysql php-7.2 selenium-hub`
4. Log into php container and run the behat init.php script
5. Run a behat test containing an @javascript tag e.g. vendor/bin/behat --tags="@block_accessibility"
6. Visit http://totara72.behat or http://totara73.behat in the browser

Expected behavior
The behat test should execute when it contains javascript tags
The acceptance test site should be accessible through the browser

Actual behaviour
The behat test fails with:

 Failed hooks:

   BeforeScenario # behat_hooks::before_scenario()
   Visiting of the main page failed (Exception)

The acceptance test site is not accessible through the browser. The user is redirected to the http://totara72or http://totara73 site (due to empty($_COOKIE['BEHAT']) ).

Screenshots

Desktop (please complete the following information):

• OS: MAC OS High Sierra 10.3.6
• Browser: Chrome
• Version: 73.0.3683.103 (Official Build) (64-bit)

Additional context

1. When visiting http://totara72.behat or http://totara73.behat in the browser it redirects to http://totara72 or http://totara73
2. I can access the Selenium console ok in http://totara72.behat:4444/ and http://totara73.behat:4444/
3. It seems as though Totara is redirecting to $CFG->wwwroot because it thinks its not a behat test site i.e. behat_is_test_site() in lib/setup.php returns false because $_COOKIE['BEHAT'] is empty.

Fix file permissions issues when running commands inside containers

If you run a command such as git checkout integration/t13-release whilst inside a container, then the correct permissions will not be applied to the affected files. This can get annoying - especially with files such as the tui build files which are build using the tnpm command - as you aren't able to edit or delete them on your host machine's file system.

To fix this, we need to make sure the correct UID and GID values are passed to the php and nodejs containers so that when they run the commands, the run it as the same user as the host machine (ie you). This is done by adding user: "${UID}:${GID}" to each container compose yml entry.

Document PHPStorm integration

Using docker for running tests inside PHPStorm is incredibly useful, it would be great to have some instructions on how to get it set up.

Upgrade XDebug to 3.0

XDebug 3.0 is considerably quicker than the 2.x versions we are using currently. There are changes to the configuration that are required so this may not be simple to do.

Nginx crashes when turning page editing on with PHP 7.3

Describe the bug
On PHP 7.3 only turning page editing on in a course crashed Nginx. It does not happen on PHP 7.2

To Reproduce
Steps to reproduce the behavior:

1. Create a course
2. Click on the 'Turn editing on' button
3. You'll get a white page
4. Check the error logs, you'll see a SIGSEGV error

Screenshots

PHP 5.4 and 5.5 not working anymore

Describe the bug
Building the images for PHP 5.4 and 5.5 result in the following error messages:

W: Failed to fetch http://security.debian.org/debian-security/dists/wheezy/updates/main/binary-amd64/Packages  404  Not Found

W: Failed to fetch http://deb.debian.org/debian/dists/wheezy/main/binary-amd64/Packages  404  Not Found

W: Failed to fetch http://deb.debian.org/debian/dists/wheezy-updates/main/binary-amd64/Packages  404  Not Found

Apparently debian wheezy support is dropped. Either get it working with different repositories or alternatively build PHP 5.4 and 5.5 from scratch.

Expected behavior
Build succeeds

Add container for MariaDB 10.5

Use proper release cycle

As more people are using this project there should be a proper release cycle making sure everyone can work on a stable version.

Use standard GitHub release tags and flow.

Container bash window not resizing according host window size

When connecting to the php containers via Ubuntu then container bash not resizes based on host window size.

Set tdb backup directory in .env

At the moment you have to specify the full backup path on your local filesystem for where you want to backup/restore from for the tdb command if you have a custom backup name. (E.g tdb restore totara13 ~/tdb_backups/1000_users.pgsql11)

This is a little inconvenient, as in most cases you probably wouldn't have multiple different directories for your backups. Really we should just make it so you set a variable like TDB_BACKUP_DIR=/path/to/backup/dir in the .env file, and then you could just go something like: tdb restore totara13 1000users.pgsql11

Add openssl extension to PHP containers

We need to add ext-openssl for php-7.3 and newer to support the MS Teams gateway properly

Fix out of the box MSSQL support

Before you can install totara on MSSQL, you must run

EXEC sp_configure 'show advanced options', 1;
EXEC sp_configure 'clr strict security', 0;
RECONFIGURE;
GO

on the command line. We should make it so it automatically does it in the build.
In mssql/custom/setup.sql and mssql/custom/setup.sh it theoretically should do this, but it doesn't actually work in my experience.

README improvements

• Add section on how to update containers after pulling new changes from this repo
• Add details to Running Behat section
  ** Need to run 'tup selenium-hub' before running behat
  ** Need to point to selenium-hub in config.php
• Include dynamic wwwroot for config.php to allow access via totaraXX, totaraXX.debug and totaraXX.behat simultaneously

Symbolic links created by behat halt mutagen sync

Describe the bug
When initiating a parallel behat run mutagen stops syncing with an error.

Invalid symbolic link: ... target is absolute

To Reproduce
Steps to reproduce the behavior:

1. Set up everything for a parallel behat run
2. Run init.php with the parallel option
3. Check mutagen monitor

Expected behavior
No errors whule syncing

Additional context
By default mutagen tries to sync symbolic links and throws errors for links it cannot handle (see https://mutagen.io/documentation/symbolic-links/).

Setting --symlink-mode=ignore on the mutagen sync point might work.

Can't tbash into non-php containers

I get the following message:

OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "chdir to cwd (\"/var/www/totara/src\") set in config.json failed: no such file or directory": unknown

When I run tbash pgsql
This is presumably because tbash cds into the site directory you are in in your home terminal session, but of course non-php containers don't have your site directory as a volume

Add option to create custom docker-compose for local changes

Currently it's not easily possible to make changes to the local setup without modifying the code checked in to this project.

We should allow for a custom docker-compose file anyone could create to overwrite default containers, or add new containers if needed.

We should also find solutions for config files which someone might want to adjust locally.

totara/ is not an allowed web root for Totara 13+, needs to be totara/server/

Currently totara-docker-dev sites use the source code root as a web root, but this might be prevented in the future for security reasons.

The web root for Totara 13+ must be set to server/

Document Windows usage

Add section for using totara-docker-dev in Windows.

Following the steps here: https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly

Support multiple sites by default and add proper docs

Having multiple totara sites running simultaneously is quite useful in my experience and supporting it out of the box would be good as it can be a bit tricky to set up. There is also no proper documentation on how to do it, so that would be good to add also.

Can't set up remote interpreters in PHPStorm on Windows

Currently it isn't possible to integrate PHPStorm to work with PHPUnit by remote interpreter on Windows when following the instructions here: https://github.com/totara/totara-docker-dev/wiki/PHPStorm%20Integration#running-unit-tests

Need to work out how to get it working and then update the Wiki accordingly.

selenium-chrome-debug VNC port conflicts with Virtualbox or other VNC Server

Currently we expose the default VNC port to the host when running the selenium-chrome-debug container.

To avoid conflicts with any running VNC Server we should change this port to something like 5901