zeuswpi / ansible-config Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 1.0 2.11 MB

Development version of Ansible config.

License: MIT License

Shell 6.66% Python 12.02% Jinja 14.39% Berry 66.93%

ansible hacktoberfest

ansible-config's Introduction

ZeusWPI

Zeus Extraordinary Uploading Service With Productive (Prolog) Interface

Development

Install SWI-Prolog using asdf.

asdf install

This project relies on Redis for performant session storage. You can start an instance using the provided docker-compose.devel.yml file.

Next create the required directories.

mkdir -p {data,files}

Then start the development (Live reload) server using:

swipl src/main.pl devel 5000

Or run the above steps by running make devel.

Visit your browser at http://localhost:5000.

Deployment

The deployment setup is contained in the provided docker-compose.yml file.

There are 3 environment variables to change:

Key	Value
ZEUSWPI_CLIENT_ID	The OAUTH Client ID
ZEUSWPI_CLIENT_SECRET	The OAUTH Secret
ZEUSWPI_REDIRECT_URI	The OAUTH Redirect URI (Set the correct domain)

You also need to create the required directories in this mode. As these will be mounted as volumes in the container.

mkdir -p {data,files}

Then start with:

docker compose up -d

Adding admins

First the person you want make an admin has to log in to the application once. Then on the server find the user fact in the database data/user.db

assert(user(69,"the_admin",user)).

and update the record to

assert(user(69,"the_admin",admin)).

then restart the application.

ansible-config's People

Contributors

Stargazers

Watchers

Forkers

bondroitstef

ansible-config's Issues

Relay access denied; from=<> to=<[email protected]> proto=ESMTP helo=<smtp1.ugent.be>

Found this in the logs, no idea if this is a problem.

Aug  1 14:17:09 king postfix/smtpd[14425]: connect from smtp1.ugent.be[157.193.71.182]
Aug  1 14:17:09 king postfix/smtpd[14425]: NOQUEUE: reject: RCPT from smtp1.ugent.be[157.193.71.182]: 454 4.7.1 <[email protected]>: Relay access denied; from=<> to=<[email protected]> proto=ESMTP helo=<smtp1.ugent.be>
Aug  1 14:17:09 king postfix/smtpd[14425]: disconnect from smtp1.ugent.be[157.193.71.182]

When migrated

Things we have to do when we're officialy making the switch

Write a blogpost

People like to know what's going on. We should write one.

postfix is spewing out errors

Jul  5 19:08:54 king postfix/qmgr[4245]: 69BD26018F: from=<[email protected]>, size=4813, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: warning: connect to transport private/amavisfeed: No such file or directory
Jul  5 19:08:54 king postfix/error[10350]: 69BD26018F: to=<[email protected]>, orig_to=<root>, relay=none, delay=25673, delays=25673/0.01/0/0.02, dsn=4.3.0, status=deferred (mail transport unavailable)
Jul  5 19:08:54 king postfix/qmgr[4245]: B773C60023: from=<>, size=2354, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: error: open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf is unavailable. open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf lookup error for "*"
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf is unavailable. open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf lookup error for "*"
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf is unavailable. open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf lookup error for "[email protected]"
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: transport_maps lookup failure
Jul  5 19:08:54 king postfix/qmgr[4245]: 08056600DA: from=<>, size=6570, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf is unavailable. open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf lookup error for "[email protected]"
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: transport_maps lookup failure
Jul  5 19:08:54 king postfix/qmgr[4245]: 578FD601EB: from=<[email protected]>, size=590, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: 2F8B760195: from=<[email protected]>, size=590, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: 7858A6012C: from=<>, size=2348, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf is unavailable. open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf lookup error for "[email protected]"
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: transport_maps lookup failure
Jul  5 19:08:54 king postfix/qmgr[4245]: 8E8576017B: from=<[email protected]>, size=740, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: E592060083: from=<>, size=2354, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: E7074600A2: from=<>, size=2616, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf is unavailable. open database /etc/postfix/transport.cf.db: No such file or directory
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: hash:/etc/postfix/transport.cf lookup error for "[email protected]"
Jul  5 19:08:54 king postfix/trivial-rewrite[10354]: warning: transport_maps lookup failure
Jul  5 19:08:54 king postfix/qmgr[4245]: 064AC6025F: from=<[email protected]>, size=590, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: 0049B60261: from=<[email protected]>, size=590, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: A036660234: from=<[email protected]>, size=4813, nrcpt=1 (queue active)
Jul  5 19:08:54 king postfix/qmgr[4245]: 1DFDC6013A: from=<>, size=2240, nrcpt=1 (queue active)

Define authorized keys

Not sure how to do this the best way possible as each user needs different keys

Migrate from fcgi to uwsgi for Python

Reflect the zeusquotes->slackintegrations changes

Caveat: mysql users don't allow such long names, so the name of the mysql-use ris slackintegration, not slackintegrations

Switch up passwords in the vault

Some still contain the old values, these should be switched up.

Fix passenger

[....] Restarting nginx: nginxnginx: [alert] Unable to start the Phusion Passenger watchdog because its executable (/usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini/buildout/agents/PassengerWatchdog) does not exist. This probably means that your Phusion Passenger installation is broken or incomplete, or that your 'passenger_root' directive is set to the wrong value. Please reinstall Phusion Passenger or fix your 'passenger_root' directive, whichever is applicable. To learn how to fix 'passenger_root', please read https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#PassengerRoot (-1: Unknown error)

Add capistrano secrets.yml and database.yml

These are 2 shared files in ~/production/shared/config which contain passwords and secrets, these should be added to the vault and automaticly deployed.

Check out clamav failures

Clamav doesn't seem to have a daemon running:

Sep  5 10:42:48 king amavis[2939]: (02939-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory
Sep  5 10:42:49 king amavis[2939]: (02939-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory
Sep  5 10:42:49 king amavis[2939]: (02939-01) (!)ClamAV-clamd: All attempts (1) failed connecting to /var/run/clamav/clamd.ctl, retrying (2)
Sep  5 10:42:55 king amavis[2939]: (02939-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory
Sep  5 10:42:55 king amavis[2939]: (02939-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamav/clamd.ctl (All attempts (1) failed connecting to /var/run/clamav/clamd.ctl) at (eval 96) line 613.\n
Sep  5 10:42:55 king amavis[2939]: (02939-01) (!)WARN: all primary virus scanners failed, considering backups

We do have clamav-freshclam for virus database updates, but there doesn't seem to be a daemon for the actually scanning enabled?

Finish configuration and installation

Software

Configuration

app-users
create databases
configure nginx configs
...

Check out mysql encoding

We're seeing problems in gamification and gandalf with the utf8 support. We should probably swith stuff to the utf8_general_ci collocation

Decently protect dokuwiki

there are warnings under the admin interface that the data dir is not protected
remove install from nginx

Remove mediawiki when dokuwiki has been installed

Things to remove:

mediawiki mysql database (leave, see comment)
mediawiki mysql user (leave, see comment)
/var/lib/mediawiki (moved and deleted)
/usr/share/mediawiki (moved and deleted)
/home/dokuwiki/MediaWiki-to-DokuWiki-Importer

Remove rsnapshot

Move all files to abyss/ branch temporarily

Check .forwards and /etc/aliases duplicates

We have two email configurations for apps at the moment:

.forward files in king home dirs
/etc/aliases on Clarke

Some apps (eg gandalf) have both a .forward file and an entry in etc/aliases entry. What happens when [email protected] is mailed? Does postfix check the .forward first, does he check the /etc/aliases first?

Either way, these duplicates should be merged and put in one place, preferably in the home dirs on King imo.

Checkboxes for checked and deduplicated apps:

fail2ban doesn't run, doesn't produces any output

Add slotmachien

Number-six

Perhaps number six should run a few times on King, at least once for Slack and perhaps the one on wina.ugent.be should run there, too?

Fix dokuwiki URL rewrite

see https://www.dokuwiki.org/rewrite#nginx

Add /blok

Add zeus.ugent.be/blok if still needed

Send a mail to Dieter from the datacenter

To ask if he has a spare server of the same model lying around (dell poweredge 2850) we can salvage raid controller hardware from

Adress: [email protected]

install matrix

create new ansible matrix install role, make it work

Install pip3 and pyquery: dependencies

Needed for the new resto scraper

Switch to mailman 3.1 when it's there, or to an alternative

Discussion: what should be included in this repo?

The topic of what should be handled by Ansible and what by backups, recently arose. It mainly concerns user files:

.forward files
.ssh/authorized_keys
secrets.yml & database.yml needed for Capistrano

My opinion is that:

the functions of backups and Ansible should be disjunct: if backups can easily restore something, Ansible shouldn't handle it. This way, stuff from backups doesn't get overridden by Ansible and vice versa (which would lead to some frowning).
user-based stuff should be handled solely by backups because users can change this at anytime, which would result in an inconsistency between the repo and the machine state.

@silox' opinion -- correct me if I'm wrong -- is that:

Ansible should be used as a management & deployment platform, with the purpose of not having to login directly to the server to change minor stuff. Kind of like Capistrano-style.
Ansible should create these files if they don't exist, but not overwrite them if they do.

Discuss.

Backup restore script / checklist

Now that the migration is fresh in memory, we should soon write a backup script or a checklist.
Inspiration can be extracted from #27 and #43.

Migrate from SysV to systemd

[email protected] mails are broken

Transcript of session follows. 

Out: 220 zeus.ugent.be ESMTP Postfix (Debian/GNU) 
In: EHLO relay2.ugent.be 
Out: 250-zeus.ugent.be 
Out: 250-PIPELINING 
Out: 250-SIZE 20480000 
Out: 250-VRFY 
Out: 250-ETRN 
Out: 250-STARTTLS 
Out: 250-ENHANCEDSTATUSCODES 
Out: 250-8BITMIME 
Out: 250 DSN 
In: MAIL FROM:<[email protected]> SIZE=6725 BODY=7BIT 
Out: 452 4.3.1 Insufficient system storage 
In: RCPT TO:<[email protected]> ORCPT=rfc822;[email protected] 
Out: 503 5.5.1 Error: need MAIL command 
In: DATA 
Out: 503 5.5.1 Error: need RCPT command 
In: RSET 
Out: 250 2.0.0 Ok 
In: QUIT 
Out: 221 2.0.0 Bye 


For other details, see the local mail logfile

Add mysql to users

Users can rely on other databases too (such as mongodb for errbit). We should add mongo=yes and mysql=yes to the users file to make this distinction.

Run dokuwiki as a user

Dokuwiki now runs in /srv which is inconsistent with our other users. We should create a user and move the site there.

Actually switch the servers

We want both the DNS and IP swapped here, so zeus.ugent.be still is our "main" domain and so we don't have to switch all the DNS A-records pointing to our servers. I mailed DICT to switch, but they'll do it after the GF.

Mail everyone who we host a site from

To notice them of the migration and that the next period can be buggy

Check out sensitive data

I deleted some stuff (also from the git history):

ssl keys from nginx_configs/files/ssl: Is this backup data? Do we put this in the vault? In any case, regenerate them
rsysmond license key: I put it in the vault, but the template file should be readded