humanmade/altis-local-server

Local Server module for Altis

altis-local-server's Issues

Add database connection helper command

It's easy to connect to a chassis db for example so it'd be useful to be able to do the same here. The command should be named composer server db.

Acceptance criteria:

• composer server db should log the user into mysql on the db server
• composer server db spf should generate and open an SPF file for sequel pro (also add an alias of sequel)
• composer server db info should print out db connection details
• Update command help output
• Document db command in a new file docs/database.md

composer server db info output template:

Root password: wordpress

Database: wordpress
Username: wordpress
Password: wordpress

Host: 0.0.0.0
Port: 31232

Version: MySQL 5.7

Bonus

• composer server db workbench Windows support for MySQL Workbench

Running cli command db import with STDIN doesn't work

Our documented solution to import a db here https://www.altis-dxp.com/resources/docs/local-server/cli/#wp-cli doesn't work currently.

If you run composer server cli -- db import - < database.sql you get logged into mysql on the db container instead. This isn't the expected behaviour! The WP CLI import command should run properly instead.

Add debug.log to the logs command

I'm currently ssh'ing into the machine to view wp logs. Can debug.log be added to the composer local-server logs command?

Add method to run WP CLI commands

An extension to composer local-server to run WP CLI will be invaluable as part of the dev toolkit. composer local-server wp ... seems a logical entrypoint.

Elasticsearch Warning message on install

Setting up ElasticPress... Adding mapping... Deleting index... Warning: Error in ElasticPress request: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"sandboxsitealtisdev-1","index_uuid":"_na_","index":"sandboxsitealtisdev-1"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"sandboxsitealtisdev-1","index_uuid":"_na_","index":"sandboxsitealtisdev-1"},"status":404} (404) in /usr/src/app/vendor/altis/enhanced-search/inc/namespace.php on line 143 Success: Index deleted Success: Mapping sent Indexing posts... Processed 3/3 entries. . . Number of posts indexed on site 1: 3

Mailhog sometimes fails to start

Sometimes when trying to start the local server I get:

Starting dev_mailhog_1 ... error

ERROR: for dev_mailhog_1  Cannot start service mailhog: b'Ports are not available: /forwards/expose/port returned unexpected status: 500'

Restarting Docker appears to fix it temporarily.

File permission issue on Linux

Can't seem to be able to install themes/plugins via CLI with this error:

>>> $ composer server cli -- theme install twentynineteen
Installing Twenty Nineteen (1.6)
Downloading installation package from https://downloads.wordpress.org/theme/twentynineteen.1.6.zip...
The authenticity of twentynineteen.1.6.zip could not be verified as no signature was found.
Unpacking the package...
Warning: Could not create directory.
Warning: The 'twentynineteen' theme could not be found.
Error: No themes installed.

On 4.0.0 RC 1, the uploads directory was created with the owner set to root. Changing the ownership to my user didn't fix it. On v3, the uploads directory wasn't created at all.

Commands don't properly account for directories with spaces

When running start, I get this:

sh: Made/projects/example: No such file or directory
sh: Made/projects/example: No such file or directory
Installed database.

Output username and password when starting

If the site hasn't been installed, we should output the default username and password in the terminal along with the site URL.

Shell command is not documented

Document a workaround for: Cannot run Local Server while something else is listening on port 443 / 80

Updated:

I have Nginx running locally. When I start Local Server I get the following error:

> composer server
Starting...
Starting docker_proxy_1 ... error

ERROR: for docker_proxy_1  Cannot start service proxy: Ports are not available: listen tcp 0.0.0.0:443: bind: address already in use

ERROR: for proxy  Cannot start service proxy: Ports are not available: listen tcp 0.0.0.0:443: bind: address already in use
ERROR: Encountered errors while bringing up the project.
Could not start traefik proxy.

I thought that I could fix this by removing the host ports from docker/proxy.yml:

       - "$PWD/sni:/etc/traefik/sni"
       - /var/run/docker.sock:/var/run/docker.sock
     ports:
-      - '8080:8080'
-      - '80:80'
-      - '443:443'
+      - '8080'
+      - '80'
+      - '443'
 
 networks:
   proxy:

This allows the containers to initialise using ephemeral ports on the host, however my-project.altis.dev is still being served by 127.0.0.1. Stopping my local Nginx server has no effect.

What can I do to allow an actual local server (eg. Nginx on the host machine) to run at the same time as Altis Local Server?

Task:
We are going to document the workaround or troubleshooting guide to overcome this issue in the troubleshooting guide.

Tag docker image version

We should tag the images with the altis version, rather than relying on latest.

Error on initial install

Warning: Error in ElasticPress request: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"demoprojectaltisdev-1","index_uuid":"_na_","index":"demoprojectaltisdev-1"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"demoprojectaltisdev-1","index_uuid":"_na_","index":"demoprojectaltisdev-1"},"status":404} (404) in /usr/src/app/vendor/altis/enhanced-search/inc/namespace.php on line 143

Add sequel pro spf file generator command

While it's possible to use the CLI command to interact with the database it'd be good to support connecting sequel pro in an easy way.

We could also consider Windows alternatives like MySQL workbench.

Props @ntwb for the suggestion.

Confirmation prompt for destructive commands

Describe the solution you'd like
When I run composer server destroy or any other destructive command, I'd like to be presented with a "Are you sure (y/n)" prompt in order to be able to cancel that.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Start up time of "exec" is slow

time composer local-server exec -- whoami
nobody

real	0m3.336s
user	0m0.855s
sys	0m0.322s

This is a bit slow, and is a overhead to all commands that interact with the PHP container, let's see if we can speed it up.

Apply start, stop & destroy to traefik proxy as well

Currently the traefik proxy container once started is never stopped or removed on destroy along with the application containers. This is because it can be shared by other instances of local server.

We should look into how we can check for other running instances of local server and if none are found on stop or destroy also stop or destroy the proxy.

On start the proxy container should also be restarted if it's already running. This will help avoid the problem of certificates getting out of sync too.

Add links to kibana and mailhog to WP admin tools menu

We should make it easier to find mailhog and kibana by linking to them from the tools menu in WP admin.

Document Windows 10 Home Edition limitation

Docker Desktop cannot run on Windows 10 Home edition so we should mention this in the docs and point people to the chassis set up guide.

Change default password to "password"

To match the Chassis environment, we should change the default password to password.

Escaping issues with exec command arguments containing spaces

If you run something like composer server cli -- db query 'select * from wp_posts;' you get an error saying "Too many positional arguments". You have to add backslashes before the spaces for it to work.

This should just work ideally - I'm not sure if there are cases other SQL queries where this behaviour would be good to have but it's likely. If we can at least handle escaping spaces in anything between quotes it would fix this issue and for other commands that accept strings as arguments.

Hostname fails if current dir has a .

If the current DIR is not a valid subdomain, things fail. E.g. example.com.altis.dev. We should sanitize this.

Deleting a site via network admin fails

There's an error where the S3 API throws an exception while trying to remove the site's uploads directory stating that it doesn't exist.

This is possibly an issue with it not creating the directory when a site is added.

Remove restart on failure directive on Cavalcade container

There's a problem that was previously worked around by setting the Cavalcade container to always restart.

This is because it would exit if WP wasn't installed yet which isn't possible because the containers must all be started before WP can be installed.

See #81 for background.

The changes in Cavalcade v2 should help to mitigate this issue so we need to test that and if successful remove the restart on failure configuration.

@kadamwhite does this more or less cover the issue you noticed with the Cavalcade container not stopping? I thought restart on failure should allow it to be stopped successfully.

Add ability to map uploads folder

@roborourke Let me see if I can explain this properly for what I think is needed with local-server.

So the other day I needed to access the uploads folder inside of s3 docker, however because this isn't mapped to any folder on the host machine. I had to go and change the docker yml file locally, to map it to the uploads folder on the host machine.

My recommendation here is that do the same thing by default: Map the S3 docker instance to the uploads folder, so we have access to the contents inside the uploads folder. This gives the developer the ability to add assets when using assets from production etc.

Hopefully this makes sense, happy to discuss further.

Add signals to stdout for triggering composer events / scripts

We echo the output buffer to the console right now. If possible it'd be handy to find the specific lines that let us know when individual containers are ready and perhaps when all containers are ready so that follow up actions such as installing WP can be taken. We may need to add the output to the appropriate docker containers.

Because the PHP container requires all the others and assuming the db container setup doesn't happen async perhaps waiting for that container is enough.

Define HM_ENV_REGION

This is required for the get_aws_sdk() function in altis/core.

Original issue humanmade/altis-core#31

Alias "local-server" command to "server"

Writing composer local-server <command> is needlessly long and the hyphen isn't great for quick tab completion.

Let's do two things:

• add an alias so we can write composer server start
• make start the default command so it's just composer server

Elasticsearch can take more than the healthcheck timeout

When starting the local server, I quite frequently get:

Starting...
Recreating docker_proxy_1 ... done
Starting vantage-backend_db_1            ... done
Starting vantage-backend_elasticsearch_1 ... done
Starting vantage-backend_redis_1         ... done
Starting vantage-backend_pinpoint_1      ... done
Starting vantage-backend_s3_1            ... done
Starting vantage-backend_xray_1          ... done
Starting vantage-backend_mailhog_1       ... done
Starting vantage-backend_cognito_1       ... done
Starting vantage-backend_tachyon_1       ... done

ERROR: for php  Container "2f76cd13081e" is unhealthy.

ERROR: for kibana  Container "2f76cd13081e" is unhealthy.

ERROR: for cavalcade  Container "2f76cd13081e" is unhealthy.
ERROR: Encountered errors while bringing up the project.

php, kibana and cavalcade are not actually not-starting, they just depend on Elasticsearch have a status of healthy. There's a timeout for I think 5 checks, if a dependency is unhealthy for that long, the service that depends on it will show as unhealthy and fail to start.

This is similar to ES not having enough memory, and hence doesn't start -- but this issue is when Elasticsearch takes a long time to start up (and becomem healthy) that the depending services all fail. After 30s or more, you can just run composer server start again, where PHP, kibana etc will now start fine as Elasticsearch has now completed starting up.

Note: if you keep running composer server restart it will fail to get better, as that shuts down ES too -- so if you do get this, always run start more times, not restart!

Expose ES port

In a similar vein to #1 you often need to connect to and play with ES via the head extension for chrome or similar to debug and develop queries.

Elasticsearch memory limit

Currently we give the elasticsearch service 1gb of memory for the docker container. This often results in start failures, where increasing it to 2gb will usually resolve it.

Per the docs on https://elk-docker.readthedocs.io/

A minimum of 4GB RAM assigned to Docker
Elasticsearch alone needs at least 2GB of RAM to run.

Apart from the fact that I think it's crazy you need 2GB of RAM to run ES, we should either increase our mem size to 2GB for ES, or look into ways to run ES with less memory.

Document importing SQL files

Add docs on importing a database file into local-server. I think either piping like cat file.sql | composer server cli db import - or maybe composer server cli db import file.sql will work.

Renew *.altis.dev certificate

The certificate for *.altis.dev will expire on March 7, 2020. This cert is distributed with Local Server, so we need to register a new one, but also backport the update, as existing local installations will stop working if they don't update.

@rmccue how do you handle renewing the certs in Chassis, or is it a super long lived self-signed cert?

Document issues with proxies

We should add to the troubleshooting section a bit about issues you may hit using proxies.

The specific error looks like this:

Get: registry-1.docker.io/v2/: EOF

The fix is to configure your proxy settings for Docker Desktop to be the same as for your web connection.

On OSX Catalina you may also need to set your proxy environment variables too eg:

export HTTP_PROXY=127.0.0.1:7070
export HTTPS_PROXY=127.0.0.1:7070

props @jazzsequence for reporting this one!

Acceptance criteria:

• Above information is added to the troubleshooting docs page under the heading
  "Issues when using a proxy"

Error on install with elasticsearch

I think this when installing for the first time:

Warning: Error in ElasticPress request: cURL error 7: Failed to connect to elasticsearch port 9200: Connection refused (http_request_failed) in /usr/src/app/vendor/altis/enhanced-search/inc/namespace.php on line 127

CORS headers for tachyon

Right now the tachyon service is using a different domain name to deliver the images. There are no CORS headers that allow loading of those images via javascript which breaks the smart media cropping.

Possible options are:

• Add more permissive CORS headers
• Use a sub-sub domain eg. ..altis.dev
• Use a sub folder path and proxy_pass via nginx

Turn display_errors on by default

Currently we use the php.ini from production, which has display_errors = Off. This means we don't get errors showing by default for fatals, which makes things like wp cli commands failing super difficult to debug.

I thinl we should change this to default on. I think we need to do this in the actual php.ini because setting it in code falls into the "error happens before this code" problem.

Acceptance criteria:

• Remove the line

  altis-local-server/inc/namespace.php

  Line 52 in 4619532

  ini_set( 'display_errors', 'on' );

• In the module's wp-config.php file check if it's the CLI environment and if so call ini_set( 'display_errors', 'on' );

To test restart the local server to apply the changes and confirm this works by calling an undefined function at the top of the wp-config.php file and running a CLI command. You can also var_dump( ini_get( 'display_errors' ) ) to confirm the new default value.

SPIKE: Update HTTPS certificate to support *.*.altis.dev

One of the limitations we have with local server is the inability to support sub-sub domains of altis.dev. This makes it hard to migrate data from production or to test out sites that employ logic around having subdomains or the SUBDOMAIN_INSTALL constant.

Related to #126

Acceptance criteria:

• sub-subdomains of altis.dev are supported and served over https automatically

Timebox: 5 days

Add support to install wp-cli packages

When trying to install a wp-cli package this morning I encounted the following:

❯ composer server cli -- package install foo/bar
Error: Composer directory '/.wp-cli/packages' for packages couldn't 
be created: mkdir(): Permission denied

It would be great if this folder was created during setup, or made writable for the cli command to create on demand.

Aside: It would also be great to be able to define wp-cli packages in a projects composer.json file, though this might be out of scope for this issue

Consider merging $_ENV with the env vars passed to new Process calls

We hit an issue recently on #108, I was unaware the global environment variables would not be available to the process called through new Process(). This could catch people out as it did me and may well be expected and very useful on CI runs.

Traefik redirects to wrong backend

When there are two sites, foo.example.dev and bar.example.dev, and submitting 2 requests to foo.example.dev, one of the requests will be served by bar.example.dev's backend but the url will be foo.example.dev. I've noticed this locally but haven't logged it anywhere, but it's something to figure out!

Release non RC container versions and pin everything

We have some containers pinned to RC versions and some not pinned at all. Let's fix that.

• Release & pin PHP 1.0.0 & 3.0.0
• Pin Cognito
• Pin Pinpoint
• Pin Mailhog
• Release & pin elasticsearch 1.0.0
• Release & pin latest Tachyon

PHP warning on install

Warning: count(): Parameter must be an array or an object that implements Countable in /usr/src/app/wordpress/wp-includes/meta.php on line 210

Allowing opening shell as root / sudo user

To allow people to debug / install packages, they need a root permissions level user in the shell command. Passing -u root to the command would suffice.

Acceptance criteria:

• Add an option flag to the shell command called --root, optional, alias as -r
• When flag is set add -u root to the docker exec call
• Update the commands documentation with this argument
• Add a note to the command docs on why you would use the root user:
  • To test modifications to the PHP container eg. installing PHP libraries or other packages
  • To carry out any other tasks that require sudo privileges such as making directories outside of the app directory

Link to new Altis project docs

The following paragraph https://www.altis-dxp.com/resources/docs/local-server/#installing should link to https://www.altis-dxp.com/resources/docs/getting-started/#creating-a-new-altis-project

WordPress Install Failed

I'm consistently hitting this issue when trying to spin up my local environment:

Starting...
docker_proxy_1 is up-to-date
Starting snopes_xray_1          ... done
Starting snopes_mailhog_1       ... done
Starting snopes_pinpoint_1      ... done
Starting snopes_redis_1         ... done
Starting snopes_s3_1            ... done
Starting snopes_cognito_1       ... done
Starting snopes_db_1            ... done
Starting snopes_elasticsearch_1 ... done
Starting snopes_tachyon_1       ... done
Starting snopes_kibana_1        ... done
Creating snopes_cavalcade_1     ... done
Starting snopes_php_1           ... done
Starting snopes_nginx_1         ... done
<div id="error"><p class="wpdberror"><strong>WordPress database error:</strong> [Table &#039;wordpress.wp_terms&#039; doesn&#039;t exist]<br /><code>SELECT tt.term_id, tt.term_taxonomy_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy as tt ON tt.term_id = t.term_id WHERE t.slug = &#039;smg-generated-pages&#039; AND tt.taxonomy = &#039;smg-attributes&#039; ORDER BY t.term_id ASC LIMIT 1</code></p></div><div id="error"><p class="wpdberror"><strong>WordPress database error:</strong> [Table &#039;wordpress.wp_terms&#039; doesn&#039;t exist]<br /><code>SELECT tt.term_id, tt.term_taxonomy_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy as tt ON tt.term_id = t.term_id WHERE t.name = &#039;smg-generated-pages&#039; AND tt.taxonomy = &#039;smg-attributes&#039; ORDER BY t.term_id ASC LIMIT 1</code></p></div>Notice: Trying to get property 'capabilities' of non-object in /usr/src/app/content/mu-plugins/smg/inc/setup/roles.php on line 18
<div id="error"><p class="wpdberror"><strong>WordPress database error:</strong> [Table &#039;wordpress.wp_options&#039; doesn&#039;t exist]<br /><code>INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES (&#039;wp_user_roles&#039;, &#039;a:1:{s:8:\&quot;reporter\&quot;;a:2:{s:4:\&quot;name\&quot;;s:8:\&quot;Reporter\&quot;;s:12:\&quot;capabilities\&quot;;N;}}&#039;, &#039;yes&#039;) ON DUPLICATE KEY UPDATE `option_name` = VALUES(`option_name`), `option_value` = VALUES(`option_value`), `autoload` = VALUES(`autoload`)</code></p></div>Error: One or more database tables are unavailable. The database may need to be repaired.
<div id="error"><p class="wpdberror"><strong>WordPress database error:</strong> [Table &#039;wordpress.wp_terms&#039; doesn&#039;t exist]<br /><code>SELECT tt.term_id, tt.term_taxonomy_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy as tt ON tt.term_id = t.term_id WHERE t.slug = &#039;smg-generated-pages&#039; AND tt.taxonomy = &#039;smg-attributes&#039; ORDER BY t.term_id ASC LIMIT 1</code></p></div><div id="error"><p class="wpdberror"><strong>WordPress database error:</strong> [Table &#039;wordpress.wp_terms&#039; doesn&#039;t exist]<br /><code>SELECT tt.term_id, tt.term_taxonomy_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy as tt ON tt.term_id = t.term_id WHERE t.name = &#039;smg-generated-pages&#039; AND tt.taxonomy = &#039;smg-attributes&#039; ORDER BY t.term_id ASC LIMIT 1</code></p></div>Notice: Trying to get property 'capabilities' of non-object in /usr/src/app/content/mu-plugins/smg/inc/setup/roles.php on line 18
<div id="error"><p class="wpdberror"><strong>WordPress database error:</strong> [Table &#039;wordpress.wp_options&#039; doesn&#039;t exist]<br /><code>INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES (&#039;wp_user_roles&#039;, &#039;a:1:{s:8:\&quot;reporter\&quot;;a:2:{s:4:\&quot;name\&quot;;s:8:\&quot;Reporter\&quot;;s:12:\&quot;capabilities\&quot;;N;}}&#039;, &#039;yes&#039;) ON DUPLICATE KEY UPDATE `option_name` = VALUES(`option_name`), `option_value` = VALUES(`option_value`), `autoload` = VALUES(`autoload`)</code></p></div>Error: One or more database tables are unavailable. The database may need to be repaired.
WordPress install failed.

A completely fresh install at the latest version results in this every time.

Visiting the site gives you this:

I cannot establish connection to the db or a recreated db.

Command for viewing / tailing / grepping logs

Logs are key to the dev experience. In a similar way to how zeit's now allows you run now logs it'd cool to have a command like composer local-server logs <container> where container is one of php, nginx, es, db, tachyon, cavalcade etc...

This is low priority as you can configure WP to output debug.log in the content directory.

Document the process of importing uploads to FakeS3 for use in Tachyon

I asked on Slack earlier about help troubleshooting Tachyon images in my local-server install, where a uploads directory I had copied into my /content directory from a production export were not available in Tachyon.

The part of the process which I didn't understand beforehand was that the content/uploads directory isn't automatically synced to the fake s3 service container which tachyon accesses. I was able to "upload" my uploads directory by running s3-uploads upload-directory:

composer server cli -- s3-uploads upload-directory content/uploads

This would be good to explain in the documentation. I'll come back to do this soon, just leaving a note here for myself.

---

Acceptance criteria:

• Section added to troubleshooting with the title "Files in uploads directory not available via Tachyon or direct URL" with an explanation of why and the command to run

Install fails when SUBDOMAIN_INSTALL is set

Because the install command is not including --subdomain, it will install the blog as if it wernt, but if the constant is set to true, "weird" things happen.

Seems to be that for one, tables like wp_1_options etc get installed, and then the site can not be found too.

Add a mock email handler

Mailhog is well established tool for this job.