heroku/heroku-buildpack-ci-postgresql

An experimental buildpack to provide Postgresql running inside CI dynos

Warning this is an experimental buildpack and is provided as-is without any promise of support.

Heroku CI buildpack: Postgresql

This experimental Heroku buildpack vendors Postgresql into the dyno. It is intended for use with Heroku CI or any other environment where data retention is not important.

Please note that Postgresql will lose all data each time a dyno restarts.

Usage

This is intended to be transparent to your application. Connect to the database in the same way as you would for Heroku Postgresql by reading the value of the DATABASE_URL environment variable into your application.

By default, the buildpack provides the latest Postgres version that is generally available on Heroku. You can specify a POSTGRESQL_VERSION in the env section of your app.json to use a different major (e.g., "10" or "9.6") version. This feature is experimental and subject to change.

PostgreSQL fsync

If you want to turn off fsync during your test, you can set HEROKU_CI_POSTGRES_FSYNC_OFF in your app.json env section.

heroku_ext extension schema

By default, this buildpack will create the heroku_ext schema to better reflect production databases.

To disable this, you can set HEROKU_CI_POSTGRES_HEROKU_EXT_OFF in your app.json env section.

Releasing a new version

Follow the playbook.

heroku-buildpack-ci-postgresql's Issues

Builds failing

I am getting the following on a couple Heroku projects since the updates to Postgres 12:

-----> Copying .profile.d/00-pg-path.sh to add postgresql binaries to PATH
-----> Postgresql done
-----> Timed out running buildpack Postgresql
 !     Push failed

It hangs there for awhile and then eventually fails.

Unable to install Postgres 12 or 13 on a new app

I am getting the following when trying to install Postgres 12 or 13 on a new app:

-----> Copying .profile.d/00-pg-path.sh to add postgresql binaries to PATH
-----> Postgresql done
-----> Timed out running buildpack Postgresql
 !     Push failed

It will hang on Postgresql done until the Heroku build timeout kicks in. I am using the POSTGRESQL_VERSION config var to specify the version. If I switch it to Postgres 10 or 11 on the same app it works as expected.

Change default PostgreSQL version from 13 to 14

PostgreSQL 14 is now the default for Heroku PostgreSQL addons:
https://devcenter.heroku.com/changelog-items/2445

...so this buildpack's default should be updated to match:

heroku-buildpack-ci-postgresql/bin/compile

Line 21 in 79d3bc4

POSTGRESQL_MAJOR_VERSION=13

Ideally this would be in the playbook for new PG version releases (I don't see this work tracked in the GUS epic, so presume it's not in the playbook) :-)

cc @kbarber @beanieboi

Remove support for Heroku-16

The Heroku-16 stack reached end-of-life on May 1st, 2021, and in a few days (June 1st, 2021) the Heroku-16 build system will be decommissioned, meaning no further builds will be possible:
https://help.heroku.com/0S5P41DC/heroku-16-end-of-life-faq

As such, support for Heroku-16 can be dropped:
https://github.com/heroku/heroku-buildpack-ci-postgresql/search?q=16

There are also some Cedar-14 references that can be removed :-)

Configure Postgres for UTF-8

When setting up this buildpack for Heroku Connect, I initially had trouble with some of our Unicode tests, which we eventually tracked down to the fact that the default Postgres database was using C for its locale settings (LC_CTYPE and LC_COLLATE, for example), while the databases created by the Heroku Postgres addon use en_us.UTF-8. We were able to get the database to use the correct encoding by setting the following env vars in our CI configuration:

• LANG=en_us.UTF-8
• LC_CTYPE=en_us.UTF-8
• LC_COLLATE=en_us.UTF-8

It might've been more than we need, or we might need to do more in the long run, but this was at least enough to get our tests working again. It'd be nice if this could be set correctly via the buildpack, directly.

Add support for additional extensions (pgtap)

I filed a support for Heroku PostgreSQL to add pgtap (http://pgtap.org/documentation.html)

but maybe this is easier and more relevant to do in this buildpack.

• How would I go about adding extensions to the buildpack?

Add support for more locales

I would like to use this but run into problems since I need to support collation sv_SE on some columns. I have added a rake task:

namespace :db do
  task add_collation: [:environment] do
    puts "Adding sv_SE collation..."
    ActiveRecord::Base.connection.execute("SET search_path TO 'public'; CREATE COLLATION sv_SE (locale = 'sv_SE.UTF-8');")
  end
end

Rake::Task['db:structure:load_if_sql'].enhance [:add_collation]

But when using heroku-postgres:in-dyno I'm getting:

ActiveRecord::StatementInvalid: PG::InvalidParameterValue: ERROR:  could not create locale "sv_SE.UTF-8": No such file or directory
       DETAIL:  The operating system could not find any locale data for the locale name "sv_SE.UTF-8".

Is there an easy way to add more locales in the buildpack?

Does not support TLS

The build of Postgres used in the buildpack does not support TLS. If we're building from source, we need to pass the --with-openssl flag. (this is inaccurate; see comment below.)

A build that supports TLS is nice to have for dev/prod parity.

Getting DATABASE_URL undefined in my CI

Here is my app.json. Note that the only non-standard thing I am doing is using the subdir-heroku-buildpack

{
  "environments": {
    "test": {
      "scripts": {
        "test": "pytest"
      },
      "env": {
        "DJANGO_SETTINGS_MODULE": "myapp.settings",
        "DJANGO_SECRET_KEY": "###"
      },
      "buildpacks": [
        {"url": "https://github.com/timanovsky/subdir-heroku-buildpack#6d1c9021ebccd1d55681fad1c22e244a86908fe2"},
        {"url": "heroku/python"}
      ],
      "addons": [
        "heroku-postgresql:in-dyno"
      ]
    }
  }
}

Then I get TESTS FAILED with an error from my django settings file :

    "default": dj_database_url.config(default=os.environ["DATABASE_URL"])
                                              ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "<frozen os>", line 679, in __getitem__
KeyError: 'DATABASE_URL'

Any idea what's going on ? Is that a bug ?

Allow configuring POSTGRESQL_VERSION

Right now, it looks like this is configured for 9.6.2, with no way to use anything else, although a lot of the supporting infrastructure is in place. Any thoughts on supporting a user-configurable version (via config var or... buildpack source control tag or...)? It would be a great way to, e.g., try out the Postgres 10 betas.