Comments (8)
We don't backup postgresql.conf in order preserve it for the future: there are only 2 reasons why are we doing this:
- wal-e removes all .conf files when it restores the backup.
- pg_basebackup does not copy symlinks, in order to create the replica out of the master with symlinked .conf files we have to copy the symlinks into normal files inside PGDATA, then produce the base backup and then copy them back.
The first reason is the primary reason Patroni produces the .backups, the second is just a side effect that solves the issue.
I don't have any issues with supporting conf.d configuration (you didn't specify any details, so I guess it would be implemented by supplying command-line options to pg_ctl in order to specify the location of the configuration files), as long as it is not a default option and Patroni can still locate the recovery.conf in order to rewrite it during promition/demotion.
from patroni.
Ah, OK, I didn't understand this!
- should then be part of the WAL-E backup module, not part of main Patroni, no?
- Do people really use symlinked .conf files?
My main point is that backing up the .conf files is currently not optional in the code; it happens whether you actually need it or not. Since most installations don't need it, it should be optional, and off by default.
For conf.d, we need to support that by appending to the pg.conf file. You cannot supply a pg_ctl option to add a conf directory; that can only be done in the .conf file. That's why I haven't implemented it yet; writing to pg.conf is a new thing for us.
from patroni.
I don't understand what is the problem with conf.backup files: we don't overwrite existing configuration files in PGDATA, so the only case Patroni decides to copy over the conf.backup files is when the .conf files are actually missing from the PGDATA, but present in the backup: the latter means they were in the original PGDATA that were copied into the backup, therefore that PostgreSQL cluster likely used configuration files in PGDATA. If configuration files are stored outside of PGDATA - no con.backup files will be created.
WAL-E restore will be broken without copying/restoring the configuration files (as the replica created from WAL-E won't be able to start), so it should be turned on at least when wal-e is chosen as a replica method. We also symlink configuration files (we keep configuration files in git for all PostgreSQL clusters in our data centre and link them to actual PGDATA) internally, so removal of this code will break Patroni for us.
Nevertheless, we can live with this functionality being optional, but I want to learn if there is an actual use case where it breaks the production system? Perhaps it would make sense to enable it automatically when WAL-E is set as a replication method, or when configuration files are symlinked?
For the conf.d, one can already point patroni to the existing postgresql.conf with the options described at #53. The only step is to create postgresql.conf, with include_dir pointing somewhere, but why should it be the task of Patroni? A docker container containing the postgresql.conf with include_dir should be good enough to make sure the same configuration is propagated to the replica.
from patroni.
I've had failures because of permissions issues with the backup file location. When the user doesn't need conf file backups to begin with, they shouldn't be asked to troubleshoot those failures.
The reason why patroni needs to add the conf.d line to pg.conf is that on clean deploys, patroni is running initdb to begin with, which means that patroni is generating the pg.conf. There is no opportunity for any other utility to modify pg.conf in any way.
from patroni.
@jberkus @feikesteenbergen @alexeyklyukin What would be the next step here?
from patroni.
@LappleApple @jberkus I think it's a valid request. Let's keep this ticket until someone of us has time to implement it.
from patroni.
I believe this topic is pretty much covered by #466
from patroni.
yep, thanks!
from patroni.
Related Issues (20)
- Patroni overwrite synchronous_standby_names on primary in async mode
- Failsafe mode when master doesn't have access to DCS HOT 1
- TypeError: string argument without an encoding HOT 1
- Patroni Does Not Failover on Data Disk Full Shutdown HOT 3
- Missing cdiff in requirements HOT 2
- switchover pg cluster,but master not failover HOT 3
- Unable to deploy haproxy after deploy Patroni on K8s HOT 1
- [3.1.0] synchronous_mode updating synchronous_standby_names with leader node HOT 12
- recovery settings in postgresql parameters will cause recovery_conf ignored while building config HOT 1
- master postgres crashed and rejoin failed HOT 1
- 'psycopg2.extensions.connection' object has no attribute 'info'
- WAL Files are not deleted after 20 GB pg_restore on primary node HOT 3
- Patroni lost connection and restarted after restarting the etcd-Server HOT 3
- doing unnecessary crash recovery when primary_start_timeout=0 and failover is impossible HOT 1
- Patronictl edit-config throws an exception but updates the config file even when less and more are present on the host HOT 1
- FATAL: could not receive data from WAL stream: HOT 12
- Failover issue url: /patroni ('Connection aborted.', 'Connection reset by peer' HOT 1
- When patroni executes pg_basebackup, the maxfailures parameter is written as 2. Can this parameter be changed? HOT 7
- Incorrect host in .pgpass file for post_init script when using unix_socket_directories HOT 2
- Duplicate patroni.yml log_filename config lines can cause issues HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from patroni.