larkro/crosstie-ror

Crossties, small boxes that keeps the Rails in place. A metaphor of using containers as a supportive tool to develop Rails.

Crosstie RoR

Contain your development. Crossties, small boxes that keeps the Rails in place. A metaphore of using containers as a supportive tool to develop Rails.

A railroad tie, crosstie, railway tie or railway sleeper is a rectangular support for the rails in railroad tracks. Generally laid perpendicular to the rails, ties transfer loads to the track ballast and subgrade, hold the rails upright and keep them spaced to the correct gauge. -- Wikipedia

• This PoC is a fun summer pet project, WORK IN PROGRESS and built on Mac.

Background

Insprired by RailsConf 2022 - Keynote: The Success of Ruby on Rails by Eileen Uchitelle. In the talk there is a part about contributing to Rails and looking at the development_dependencies_install there are two ways, The easy way. "We supply the container that runs Codespaces and you can use that with VSCode" And the hard way. "It took me more than 12 hours to get Ruby 3.1 and MySQL to play nice!".

Could I contribute? Maybe I can make it easier, less scary, for others to contribute? Inspired by the talk: You don't need to invent something major

Goals

• work with the code locally with your favorite IDE, run everything in containes.
• minimal install on your local machine (brew, git, make).
• easily switch versions of ruby, postgres, mariadb/mysql, memcached and redis.

Prerequisite

You need brew, git and make.

Getting started, TL;DR

If you have the prerequiste you can:

make install            # Use brew to install: colima kubectl docker docker-compose
make colima             # Start colima with kubernetes and $HOME writable
make clone              # Clone the rails repo to directory rails
make docker-build       # Build the rails docker-image
make docker-compose-up  # Start dependency services: memcached, redis, mariadb, postgresql. Variables from .env

Getting started, how to work with this

After TL;DR make docker-compose-up you got an environment with all needed services running in containers and a docker image rails-dev where your code can be tested.

As is, some database things need to be setup run make setup-mysql-user and make setup-db. Give mariadb a few sec to boot between docker-compose-up and setup-mysql-user. These two commands are partly redundant, and will be updated once I have had the time to look into the existing tasks and documentation about setting up the databases.

Edit .env for specific versions.

Use you favorite IDE to edit files in directory rails.

Start an instance of rails-dev using make run-command (will mount directory rails) where you can manually run the tests for the thing you are working on. Adding gems or other dependencies requires rebuilding the make docker-build step so it's available in the running container.

Example:

$ make run-command
docker run -it --network rails-net --env-file .env \
                --volumes-from=postgres \
                --volumes-from=mariadb \
                -v `pwd`/rails:/usr/src/rails rails-dev:3.1 "/bin/bash"
root@hostname:/usr/src/rails# cd activerecord
root@hostname:/usr/src/rails# bin/test test/cases/binary_test.rb -n test_load_save
Using sqlite3
Run options: -n test_load_save --seed 19719

# Running:

.

Finished in 0.104350s, 9.5831 runs/s, 86.2482 assertions/s.
1 runs, 9 assertions, 0 failures, 0 errors, 0 skips

Recommended Workflow

The recommended workflow is

• pick the versions of each dependency in .env

• edit rails files in the host computer with your favorite IDE and

• run a container with a shell where you run tests.

What it is build on

A container engine that can run docker images, we have used colima. Instructions to use Colima

note: colima must be started with write permissions, add:

--mount ${HOME}:w`

Default mount is read only, write is needed when running rails tests.

The tools we have used can easliy be installed using HomeBrew.

brew install colima
brew install kubectl
brew install docker
brew install docker-compose

Read more about it: colima - Container runtimes on macOS (and Linux) with minimal setup

For extra fun start it with kubernetes support, add some cpu and ram. Yes, future plans includes running Crossties in kubernetes, running multple versions at the same time, hosted in different places, gather metrics/logs/stuff using that ecosystem.

colima start --with-kubernetes --cpu 6 --memory 6 --mount ${HOME}:w

How to

Things are "documented" in Makefile. Run make to see the help.

The short story.

install              Use brew to install: colima kubectl docker docker-compose
colima               Start colima with kubernetes and $HOME writable
lint                 Run hadolint on Dockerfile
clone                Clone the rails repo to directory rails
docker-build         Build the rails docker-image
docker-convert       Display docker compose convert
docker-compose-up    Start dependency services: memcached, redis, mariadb, postgresql. Variables from .env
docker-clean-up      Stop dependency services: memcached, redis, mariadb, postgresql. Prune containers and volumes
setup-mysql-user     Run the rails-dev docker-image to setup the mysql db (mariadb)
setup-db             Run the rails-dev docker-image to create and build databases
drop-create-test-db  Drop and create test dbs
run-command          Run command with rails-dev image, default: /bin/bash
run-command-without-env Run command with rails-dev image, without env vars, default: /bin/bash
run-test             Run rails tests in the rails-dev docker-image towards services in docker-compose
run-test-testops     Run rails tests in the rails-dev docker-image ... with testopts, default: --verbose
help                 Display this output.

Example

Edit files locally in your prefered IDE.

sed -i".bak" 's/Spammer layout We do not spam/This should show up as fail/' rails/actionmailer/test/mail_layout_test.rb

Run test, often easier to start a container with a shell and manually run it in another terminal.

make RUN_COMMAND="/bin/bash -c 'cd actionmailer && bin/test test/mail_layout_test.rb -n test_explicit_class_layout'" run-command

See the error and iterate until pass.

# Running:

F

Failure:
LayoutMailerTest#test_explicit_class_layout [/usr/src/rails/actionmailer/test/mail_layout_test.rb:90]:
--- expected
+++ actual
@@ -1 +1 @@
-"This should show up as fail"
+"Spammer layout We do not spam"

Issues

Still a lot. :-) Just started. Create Issues in git for better tracking. Some general things are:

Too much is running with too high privilege and access is without proper authentication. First step is to make it work. Then make it work in a more correct way.

Look into why some tests are skipped and good ways of handling version deps and such.

crosstie-ror's Issues

Failed RedisCacheStoreTests

Failure:
ActiveSupport::Cache::RedisCacheStoreTests::RedisCacheStoreCommonBehaviorTest#test_sjis_kddi_encoded_values [/usr/src/rails/activesupport/test/cache/behaviors/encoded_key_cache_behavior.rb:11]:
Expected: "1"
Actual: nil

rails test usr/src/rails/activesupport/test/cache/behaviors/encoded_key_cache_behavior.rb:8

Pick postgres version

Got a few postgres version dependencies:

rails tests use pg_dump from host OS (our built docker-image) which based on Ubuntu bullseye and has postgres 13.7.
docker-compose starts the database where we can pick image version.

would be nice to enable running rails tests on different postgres versions.

How do we add and pick postgres repo and version when building the docker-image?
Keep it in sync with image in docker-compose (and later helm charts)?

Postgres : user, access, permissions

• Start postgres db container
• Setup correct user, pass, access and databases needed for rails.
• Start rails-dev container with the correct variables.

Got env vars like POSTGRES_USER POSTGRES_PASSWORD and POSTGRES_DB in docker-compose.yml and in Makefile.

Permissions and databases needs to be created.
Access is from an external host (since it's containers).

Example of error:
Postgres as "foreign_server" fails ForeignTableTest#test_update_record: RuntimeError: Wrapped undumpable exception for: ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR: could not connect to server "foreign_server" DETAIL: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: role "postgres" does not exist

At a first glance it looks like multiple roles/access types are needed. Looks like some access is via socket and some via TCP. Can not use DATABASE_URL since it breaks other things and running docker-image as USER postgres breaks something else.

Mount of $PWD/rails fails after switch to colima

The plan was something like this:
1 clone the rails repo to $PWD/rails
2 make the images with gems and stuff installed.
3 run the image and mount the same directory so a local IDE can update the code and be tested by reusing all installed stuff on the image as well as all services running in other containers.

After switching to colima the mount "-v pwd/rails:/usr/src/rails" gives an empty directory.
Tested some other mount options and found:
touch: cannot touch 'f': Read-only file system

I think it has to do with something like this.
https://blog.dustinrue.com/2021/10/getting-started-with-colima/
Out of the box colima will mount your entire home directory as a read only volume within the colima VM which makes it easily accessible to Docker.

well, right now, the plan doesn't work. :-(

libxml version

WARNING: Nokogiri was built against libxml version 2.9.13, but has dynamically loaded 2.9.10

Connection issues to postgres, foreign_server

Connection issues to postgres.

Example errors:
Error:
ForeignTableTest#test_delete_record:
ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR: could not connect to server "foreign_server"
DETAIL: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: role "postgres" does not exist

cut
/usr/src/rails/activerecord/test/cases/adapters/postgresql/foreign_table_test.rb:105:in `test_delete_record'

rails test usr/src/rails/activerecord/test/cases/adapters/postgresql/foreign_table_test.rb:104

Error:
ForeignTableTest#test_attributes:
ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR: could not connect to server "foreign_server"
DETAIL: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: role "postgres" does not exist

cut
/usr/src/rails/activerecord/test/cases/adapters/postgresql/foreign_table_test.rb:80:in `test_attributes'

rails test usr/src/rails/activerecord/test/cases/adapters/postgresql/foreign_table_test.rb:79

mariadb or mysql?

Which database should be used default?
Ubuntu, which the containers is built on has mariadb in default package repo.
Variables used by Rails is named MYSQL_*

As is, we get a not so pretty naming with mariadb specific variables like MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and rails specific like MYSQL_HOST in both .env and docker-compose.yml

Ubuntu development dependencies install instructions

Started from https://guides.rubyonrails.org/development_dependencies_install.html#ubuntu

Installed poppler-utils in Dockerfile
because of error msg: ActiveStorage::Previewer::PopplerPDFPreviewerTest#test_previewing_a_PDF_that_can't_be_previewed [/usr/src/rails/activestorage/test/previewer/poppler_pdf_previewer_test.rb:38]: [ActiveStorage::PreviewError] exception expected, not Class: <Errno::ENOENT> Message: <"No such file or directory - pdftoppm">

And the same with libvips42 because of error msg: ActiveStorage::Representations::RedirectControllerWithVariantsTest#test_showing_variant_inline: LoadError: Could not open library 'vips.so.42': vips.so.42: cannot open shared object file: No such file or directory. Could not open library 'libvips.so.42': libvips.so.42: cannot open shared object file: No such file or directory

Instructions development_dependencies_install for 2.3.2 Ubuntu talk about mysql, but it's no longer the standard app so instead Dockerfile is updated with packages
mariadb-server libmariadb-dev-compat libmariadb-dev

Pick ruby version

Skipped:
ActiveRecord::ConnectionAdapters::ConnectionPoolFiberTest#test_disconnect_and_clear_reloadable_connections_are_able_to_preempt_other_waiting_threads [/usr/src/rails/activerecord/test/cases/connection_pool_test.rb:858]:
Can't test isolation_level=fiber without a Ruby 3.1+ Fiber Scheduler

Pick mysql (mariadb) version

Skipped:
ActiveRecord::Migration::ChangeSchemaWithDependentObjectsTest#test_create_table_with_force_cascade_drops_dependent_objects [/usr/src/rails/activerecord/test/cases/migration/change_schema_test.rb:522]:
MySQL > 5.5 does not drop dependent objects with DROP TABLE CASCADE

Mysql (mariadb) : user, access, permissions

• Start mariadb db container
• Setup correct user, pass, access and databases needed for rails.
• Start rails-dev container with the correct variables.

Setup instructions in mysql-setup-database.sql are from https://guides.rubyonrails.org/development_dependencies_install.html#database-configuration but probably not needed as related things in https://guides.rubyonrails.org/contributing_to_ruby_on_rails.html says "For MySQL and PostgreSQL, it is sufficient to run:".
As is right now, some db creating things are twice.

Added MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=true and MARIADB_MYSQL_LOCALHOST_GRANTS=true to docker-compose
because of access between different pods. Error was: Access denied for user 'rails'@'172.18.0.6' (using password: NO) (Mysql2::Error::ConnectionError).

Added both --env MYSQL_HOST=mariadb and --env MYSQL_SOCK="/run/mysqld/mysqld.sock" when starting the rails-dev container.

Skipped, UniquenessValidationTest

Skipped:
UniquenessValidationTest#test_validate_uniqueness_uuid [/usr/src/rails/activerecord/test/cases/validations/uniqueness_validation_test.rb:605]:
Skipped, no message given