Giter Site home page Giter Site logo

cloudfoundry-community / postgresql-docker-boshrelease Goto Github PK

View Code? Open in Web Editor NEW
4.0 14.0 11.0 490 KB

BOSH Release for PostgreSQL in Docker container

License: Apache License 2.0

Ruby 24.20% Shell 5.96% HTML 0.83% CSS 68.03% Dockerfile 0.97% SCSS 0.01%

postgresql-docker-boshrelease's Introduction

BOSH Release for PostgreSQL in Docker container

This BOSH release has three use cases:

  • run a single Docker container of a PostgreSQL Docker image on a single BOSH VM
  • run a Cloud Foundry service broker that itself runs containers of PostgreSQL Docker image on a BOSH VM based on user requests
  • embedded PostgreSQL Docker image that could be used by another BOSH release

As a Cloud Foundry service broker, there are two version of PostgreSQL that can be offered:

$ cf marketplace
Getting services from marketplace in org system / space dev as admin...
OK

service        plans   description
postgresql93   free    postgresql 9.3 service for application development and testing
postgresql94   free    postgresql 9.4 service for application development and testing

NOTE: if you're deploying the broker for the first time, it is suggested to only offer the latest database to minimize the operations upset of deprecating and disabling the older one in the future.

The PostgreSQL image can be referenced either:

  • from an embebbed/bundled image stored with each BOSH release version
  • from upstream and/or private registries

Spiff deployment templates are included for:

  • bosh-lite/garden
  • bosh-lite/warden (older bosh-lites, deprecated)
  • bosh/aws

Learn more about embedding Docker images in BOSH releases.

Installation

To use this BOSH release, first upload it to your bosh and the docker release

bosh upload release https://bosh.io/d/github.com/cf-platform-eng/docker-boshrelease
bosh upload release https://bosh.io/d/github.com/cloudfoundry-community/postgresql-docker-boshrelease

For the various Usage cases below you will need this git repo's templates folder:

git clone https://github.com/cloudfoundry-community/postgresql-docker-boshrelease.git
cd postgresql-docker-boshrelease

Usage

Run a single container of PostgreSQL

For bosh-lite, you can quickly create a deployment manifest & deploy a single VM:

templates/make_manifest warden container embedded
bosh -n deploy

This deployment will look like:

$ bosh vms postgresql-docker-warden
+------------------------+---------+---------------+--------------+
| Job/index              | State   | Resource Pool | IPs          |
+------------------------+---------+---------------+--------------+
| postgresql_docker_z1/0 | running | small_z1      | 10.244.20.6  |
+------------------------+---------+---------------+--------------+

If you want to use the upstream version of the Docker image, reconfigure the deployment manifest:

templates/make_manifest warden container upstream
bosh -n deploy

To register your Logstash with a Cloud Foundry application on bosh-lite/warden:

cf cups postgresql -l syslog://10.244.20.6:514

Now bind it to your applications and their STDOUT/STDERR logs will automatically stream to your PostgreSQL.

cf bs my-app postgresql

Run a Cloud Foundry service broker for PostgreSQL

For bosh-lite, you can quickly create a deployment manifest & deploy a single VM that also includes a service broker for Cloud Foundry

templates/make_manifest warden broker embedded
bosh -n deploy

This deployment will also look like:

$ bosh vms postgresql-docker-warden
+------------------------+---------+---------------+--------------+
| Job/index              | State   | Resource Pool | IPs          |
+------------------------+---------+---------------+--------------+
| postgresql_docker_z1/0 | running | small_z1      | 10.244.20.6 |
+------------------------+---------+---------------+--------------+

As a Cloud Foundry admin, you can register the broker and the service it provides:

cf create-service-broker postgresql-docker containers containers http://10.244.20.6
cf enable-service-access postgresql93
cf marketplace

If you want to use the upstream version of the Docker image, reconfigure the deployment manifest:

templates/make_manifest warden container upstream
bosh -n deploy

Using Cloud Foundry Service Broker

Users can now provision PostgreSQL services and bind them to their apps.

cf cs postgresql93 free my-pg
cf bs my-app my-pg

Versions & configuration

The version of PostgreSQL is determined by the Docker image bundled with the release being used. The source for building the Docker images is in the images/ folders of this repo. See below for instructions.

Development of postgresql configuration

To push new ideas/new PostgreSQL versions to an alternate Docker Hub image name:

cd images/postgresql95
export DOCKER_USER=<your user>
docker build -t $DOCKER_USER/postgresql .
docker push $DOCKER_USER/postgresql:9.5

This will create a new Docker image, based upon the upstream cfcommunity/postgresql-base:9.5.

Create an override YAML file, say my-docker-image.yml

---
meta:
  postgresql_images:
    image: USERNAME/postgresql
    tag: 9.5

To deploy this change into BOSH, add the my-docker-image.yml file to the end of the make_manifest command:

./templates/make_manifest warden container upstream my-docker-image.yml
bosh deploy

postgresql-docker-boshrelease's People

Contributors

drnic avatar starkandwayne-bot avatar jhunt avatar samdai avatar

Stargazers

Lakshman Diwaakar avatar James Yang avatar  avatar Alexander Egorov avatar

Watchers

Tyler Poland avatar  avatar Shawn Neal avatar James Cloos avatar Bertrand Lupart avatar Quintessence avatar Sean Keery avatar Benjamin Gandon avatar Xiujiao Gao avatar DigitalEagle avatar  avatar  avatar Ibrahim Wu avatar  avatar

postgresql-docker-boshrelease's Issues

How to install a Postgres extension at database creation

I want to install a Postgres extension "ltree" at database creation, but don't know how to configure it. In file /images/postgresql93/scripts/first_run.sh, if we can pass value to system environment variable "POSTGRES_EXTENSIONS", can install extensions at database creation, but don't know how to pass value to it.

Can't generate a manifest for openstack

./templates/make_manifest openstack broker embedded

2 error(s) detected:

  • $.meta.subnets: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
  • $.networks.0.subnets: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)

Unable to connect to the Docker Remote API

i get the following error

/var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/docker_manager.rb:268:in 
`rescue in validate_docker_remote_api': Unable to connect to the Docker Remote API `unix:///var/vcap/sys/run/docker/docker.sock': 
Connection refused - connect(2) for /var/vcap/sys/run/docker/docker.sock (Errno::ECONNREFUSED) (Exceptions::BackendError)
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/docker_manager.rb:258:in `validate_docker_remote_api'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/docker_manager.rb:20:in `initialize'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/plan.rb:67:in `new'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/plan.rb:67:in `build_container_manager'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/plan.rb:26:in `initialize'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/plan.rb:11:in `new'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/plan.rb:11:in `build'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/service.rb:11:in `block in build'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/service.rb:11:in `map'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/service.rb:11:in `build'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/catalog.rb:13:in `block in services'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/catalog.rb:13:in `map'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/catalog.rb:13:in `services'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/app/models/catalog.rb:21:in `plans'
        from /var/vcap/data/packages/cf-containers-broker/0876194f52cc409e4d55b4163f86910d8a2ade5d.1-ba7d9f71b4ba79ef935c221e3da147b32175fb55/lib/container_lifecycle.rb:10:in `update_all'
        from /var/vcap/packages/cf-containers-broker/bin/update_all_containers:7:in `<main>'

i noticed this error because the route never got registered
when i do

curl -L index.docker.io

i can connect to it.
any other suggestions?

postgresql_docker_z1/0' is not running after update

Hello, I am having a clone of this repo, and it works well before.(about 1,2 month ago I think)

These days, I updated stemcell and also the cf/diego version

bosh-aws-xen-hvm-ubuntu-trusty-go_agent | ubuntu-trusty | 3215*   | ami-958f66f5 light

+-------------------+-----------+-------------+
| Name              | Versions  | Commit Hash |
+-------------------+-----------+-------------+
| cf                | 230       | 3de108d9+   |
|                   | 231*      | f67ebb73+   |
| diego             | 0.1450.0  | a0a9f2cc    |
|                   | 0.1454.0* | 72b65532    |
| docker            | 23        | 82346881+   |
|                   | 25*       | 0c96d2d9+   |
| etcd              | 27        | 5937325d    |
|                   | 36*       | 993cbd05    |
| file-server       | 3         | ba2ba922+   |
| garden-linux      | 0.330.0*  | a92518a2    |
| postgresql-docker | 4+dev.1*  | 89cfa910    |
+-------------------+-----------+-------------+

It failed to start up.

ubuntu@ip-172-31-28-108:~/bosh_workspace/max-services-docker-boshrelease/tt/docker$ cat docker.stderr.log
time="2016-03-23T09:53:41.299283226Z" level=warning msg="/!\\ DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING /!\\"
time="2016-03-23T09:53:41.599187072Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2016-03-23T09:53:42.736214452Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2016-03-23T09:53:43.334475864Z" level=fatal msg="Error starting daemon: AppArmor enabled on system but the docker-default profile could not be loaded."
time="2016-03-23T09:53:46.073428052Z" level=warning msg="/!\\ DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING /!\\"
time="2016-03-23T09:53:46.075740174Z" level=info msg="[graphdriver] using prior storage driver \"aufs\""
time="2016-03-23T09:53:46.076009848Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2016-03-23T09:53:46.183688206Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2016-03-23T09:53:46.266488604Z" level=fatal msg="Error starting daemon: AppArmor enabled on system but the docker-default profile could not be loaded."

And here is monit log for installing docker

time="2016-03-23T09:53:46.075740174Z" level=info msg="[graphdriver] using prior storage driver \"aufs\""
Selecting previously unselected package aufs-tools.
(Reading database ... 52688 files and directories currently installed.)
Preparing to unpack .../aufs-tools_20120411-3_amd64.deb ...
Unpacking aufs-tools (1:3.0+20120411-3ubuntu1) ...
Setting up aufs-tools (1:3.0+20120411-3ubuntu1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.7) ...
(Reading database ... 52742 files and directories currently installed.)
Preparing to unpack .../aufs-tools_20120411-3_amd64.deb ...
Unpacking aufs-tools (1:3.0+20120411-3ubuntu1) over (1:3.0+20120411-3ubuntu1) ...
Setting up aufs-tools (1:3.0+20120411-3ubuntu1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.7) ...

Any hint on it, or I need to sync with repo?

"virtual_network_name required for manual network" during broker deployment

Hi guys,

When following the service broker example from the README

templates/make_manifest warden broker embedded
bosh -n deploy

I ran into an error during the deployment:

  Started preparing deployment > Preparing deployment. Done (00:00:00)

  Started preparing package compilation > Finding packages to compile. Done (00:00:00)

  Started compiling packages
  Started compiling packages > ruby/1702980750298e853f3ccc6521e0d91cb6a5bf94
  Started compiling packages > golang/5519bcd760fbf15e8c2a0562da4cf2e00d846df9
  Started compiling packages > cfcommunity_postgresql_9_5_image/7688cf04da18878d5d96c847d6ff9eecda58cc68
  Started compiling packages > cfcommunity_postgresql_9_4_image/4fa9df2fc1f1dd2c155fee4677ae159167185349
  Started compiling packages > cfcommunity_postgresql_9_3_image/c3a8ffa69c622124e7e01f79e630cb8fd759dca3
  Started compiling packages > docker/cef2cbc7b3e2898f19426f4a78556ecff9eb9bb0
   Failed compiling packages > cfcommunity_postgresql_9_3_image/c3a8ffa69c622124e7e01f79e630cb8fd759dca3: virtual_network_name required for manual network (00:00:06)
   Failed compiling packages > cfcommunity_postgresql_9_5_image/7688cf04da18878d5d96c847d6ff9eecda58cc68: virtual_network_name required for manual network (00:00:06)
   Failed compiling packages > golang/5519bcd760fbf15e8c2a0562da4cf2e00d846df9: virtual_network_name required for manual network (00:00:06)
   Failed compiling packages > docker/cef2cbc7b3e2898f19426f4a78556ecff9eb9bb0: virtual_network_name required for manual network (00:00:06)
   Failed compiling packages > cfcommunity_postgresql_9_4_image/4fa9df2fc1f1dd2c155fee4677ae159167185349: virtual_network_name required for manual network (00:00:06)
   Failed compiling packages > ruby/1702980750298e853f3ccc6521e0d91cb6a5bf94: virtual_network_name required for manual network (00:00:06)

Error 100: virtual_network_name required for manual network

Task 85 error

Where should I add this key in the generated manifest file?

Change default value of checkpoint_segments and checkpoint_completion_target for better performance?

Thanks for providing this useful container.

I have a question about two postgre config: It seems you keep the default setting of checkpoint_segments to 5min and checkpoint_completion_target to 0.5.

However, The official document recommends to set checkpoint_segments to at least 10, which improves the performance: "Unless you're running on a very small configuration, you'll almost certainly be better setting this to at least 10, which also allows usefully increasing the completion target."

The document also recommends to set checkpoint_completion_target to 0.9, which is the more useful value and improves the performance: "You can spread those writes out further, lowering the average write overhead, by increasing the checkpoint_completion_target parameter to its useful maximum of 0.9"

Shall we change the default of checkpoint_segments to 10 and checkpoint_completion_target to 0.9 for better performance?
Thanks.

make_manifest aws container not working

root@ip-10-0-0-9:~/postgresql-docker-boshrelease/templates# ./make_manifest aws container embedded
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Acting as user 'admin' on 'my-bosh'
my-bosh aws_cpi 99e40f7e-2859-4fb3-9374-93969adda76d bosh-aws-xen-hvm-ubuntu-trusty-go_agent
2 error(s) detected:

  • $.meta.subnets: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
  • $.networks.0.subnets: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)

root@ip-10-0-0-9:~/postgresql-docker-boshrelease/templates#

Fix shipit task

Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
/opt/resource/out: cd: line 69: can't cd to final-release/boshrelease

How to persist database data?

Currently when I use postgresql-docker-boshrelease to create postgresql service, the PostgreSQL server is configured to store data in the /data directory inside the container. how can I map the container's /data volume to a volume on the host so the data becomes independent of the running container?

embedded containers doesn't work without manual intervention

./templates/make_manifest warden container embedded
bosh -n deploy

$ psql -h 10.244.20.6 -U user -d starkandwayne
psql: could not connect to server: Connection refused
    Is the server running on host "10.244.20.6" and accepting
    TCP/IP connections on port 5432?

Not immediately sure of the root cause - the containers job template should be running after the postgresql_images template to install the image; but its screwed up somehow. Needs investigation.

CI testflight failing - docker_meta.txt: No such file or directory

As discovered in #19

Seeing the following log errors on failing job:

==> /var/vcap/sys/log/monit/postgresql_images.err.log <==
+ for image in '/var/vcap/packages/*_image/*.tgz'
++ dirname /var/vcap/packages/cfcommunity_postgresql_9_3_image/image.tgz
+ img_dir=/var/vcap/packages/cfcommunity_postgresql_9_3_image
++ cat /var/vcap/packages/cfcommunity_postgresql_9_3_image/docker_meta.txt
cat: /var/vcap/packages/cfcommunity_postgresql_9_3_image/docker_meta.txt: No such file or directory
+ docker_tag=

==> /var/vcap/sys/log/monit/monit_debugger.install_ctl.log <==
MONIT-DEBUG exit code 1

==> /var/vcap/sys/log/cf-containers-broker/route-registrar.stdout.log <==
{"timestamp":"1459638444.436142683","source":"Route Registrar","message":"Route Registrar.Error connecting to NATS","log_level":1,"data":{"error":"dial tcp 10.244.0.6:4222: i/o timeout"}}

==> /var/vcap/sys/log/cf-containers-broker/route-registrar.stderr.log <==
panic: Failed to connect to NATS bus.

bosh-lite Instructions Don't Work

$ templates/make_manifest warden container embedded
Acting as user 'admin' on 'Bosh Lite Director'
Bosh cpi 9bca5701-66c0-43ee-a4d0-64813b4b5ece bosh-warden-boshlite-ubuntu-trusty-go_agent
12 error(s) detected:
 - $.compilation.cloud_properties: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.director_uuid: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.jobs: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.meta.job_templates: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.meta.name: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.meta.stemcell: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.meta.subnets: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.networks: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.networks: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.properties: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.releases: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)
 - $.resource_pools.0.cloud_properties: inappropriate use of (( merge )) operator outside of a list (this is spruce, after all)


$ spruce --version
spruce - Version 1.0.1 (release)
$ git rev-parse HEAD
85f48710145fbec4f5504c595c20b545f0cfe2d1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.