shift-org/infra

details about network/website/etc

infra's Issues

iCal / Atom footer link on calendar pages

I have heard several techie folks on rides mention that they are subscribed to the Shift Calendar events feed via their personal calendar, but the current iteration of the calendar site doesn't advertise a feed.

Current status

• My guess is that a previous iteration of the calendar advertised a feed, and the feed URL is still live only not being generated by Hugo. Is that about right?
• The front page advertises an Atom feed in the source code: <link rel="alternate" href="/index.xml" type="application/rss+xml">, which I believe is sufficient for traversal, however the Atom feed entries generated by Hugo lack event fields.

Desired changes

• The Hugo-generated Atom feed could be made conformant (by modifying feed entry template).
• Alternatively, use the existing legacy feed. Someone knowledgable about the infra would have to make that call.
• Advertise the feed link to humans via a standard Atom feed badge graphic in the page footer. This will cue folks who know about subscribing to event feeds that they can do so with Shift Cal.

I would like to contribute this feature

I'm a long time member of portland bike fun communities and a web developer, and I'd like to contribute to the shift project. I know a lot about web calendars, having worked on substantial chunks of Calagator back in the day.

create and document backups of database

set up certbot in AWS instance

This needs to get automated so that we always renew for api.shift2bikes.org

update README.md about how to run local docker and make sure CMS instructions/setup up to date

• README.md isn't quite current
• not sure the netlify instructions work now

document AWS account setup, and calculate cost for finance comm

There seem to be AWS tools for this:

• there's a forecasting tool in the billing console (https://console.aws.amazon.com/billing/)
• You can also use AWS CloudWatch to create billing alerts. (https://console.aws.amazon.com/cloudwatch/)

monitor: disk space usage and site uptime

was lucky to accidentally notice an out of space error while watching the netlify deploy logs this time. Also would like to know if the site goes down.

We need reliable monitoring that at least emails some people.

create troubleshooting playbooks for website/email

document VM snapshot procedures and URL for console

At some point @onewheelskyward created snapshot of our prod instance at AWS.
I couldn't find it in the console, so here's a request for him to write up and maybe add to https://github.com/shift-org/shift-docs/tree/master/docs ?

Shift script - Docker CLI message about Docker Compose

Running ./shift up to start the server works as expected, but includes this console note:

Docker Compose is now in the Docker CLI, try `docker compose up`

Seems harmless right now, but if the recommended usage is equivalent then we should change it eventually.

migrate domain registration to a shift account

make sure there is a database seed for new devs

@carrythebanner mentioned that when he setup his new machine with the app, there was no database created and so the PHP barfs a bit on failures to connect. We should probably add a step to import schema at least to the setup docs, maybe some seed data without email addrs?

archive canvashost content, and cancel account

Document ec2 image location and restoration steps

So anyone can do it. I like screenshots!

investigate weird thing in syslog

Something weird is going on in docker network land on the new server. Not sure if a big problem or not, but writing here for posterity. This happens once a minute and is visible in syslog. Somewhat informed guesstimate is that it might be some automated tool that struggles with dockers virtual networking that should be tweaked to ignore the virtual interfaces. maybe?

May 13 02:53:06 newshiftapi systemd-udevd[1345426]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
May 13 02:53:06 newshiftapi kernel: [1817009.014515] br-d49621cda334: port 2(vethfa4e04d) entered blocking state
May 13 02:53:06 newshiftapi kernel: [1817009.014518] br-d49621cda334: port 2(vethfa4e04d) entered disabled state
May 13 02:53:06 newshiftapi kernel: [1817009.014582] device vethfa4e04d entered promiscuous mode
May 13 02:53:06 newshiftapi systemd-udevd[1345425]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
May 13 02:53:06 newshiftapi networkd-dispatcher[617]: WARNING:Unknown index 59873 seen, reloading interface list
May 13 02:53:06 newshiftapi systemd-udevd[1345425]: Using default interface naming scheme 'v245'.
May 13 02:53:06 newshiftapi systemd-udevd[1345425]: vethe64e57c: Could not generate persistent MAC: No data available
May 13 02:53:06 newshiftapi systemd-networkd[562]: vethfa4e04d: Link UP
May 13 02:53:06 newshiftapi dockerd[1381]: time="2021-05-13T02:53:06.083485566Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
May 13 02:53:06 newshiftapi dockerd[1381]: time="2021-05-13T02:53:06.083515758Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
May 13 02:53:06 newshiftapi systemd-udevd[1345426]: Using default interface naming scheme 'v245'.
May 13 02:53:06 newshiftapi systemd-udevd[1345426]: vethfa4e04d: Could not generate persistent MAC: No data available
May 13 02:53:06 newshiftapi containerd[638]: time="2021-05-13T02:53:06.179424367Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/118eee19458c54b81d62dd9c6b90569afd9274a7c0cfa22b5ac3137523135fb9 pid=1345457
May 13 02:53:06 newshiftapi kernel: [1817009.596414] eth0: renamed from vethe64e57c
May 13 02:53:06 newshiftapi systemd-networkd[562]: vethfa4e04d: Gained carrier
May 13 02:53:06 newshiftapi kernel: [1817009.610854] IPv6: ADDRCONF(NETDEV_CHANGE): vethfa4e04d: link becomes ready
May 13 02:53:06 newshiftapi kernel: [1817009.610892] br-d49621cda334: port 2(vethfa4e04d) entered blocking state
May 13 02:53:06 newshiftapi kernel: [1817009.610894] br-d49621cda334: port 2(vethfa4e04d) entered forwarding state
May 13 02:53:06 newshiftapi dockerd[1381]: time="2021-05-13T02:53:06.924133809Z" level=info msg="ignoring event" container=118eee19458c54b81d62dd9c6b90569afd9274a7c0cfa22b5ac3137523135fb9 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 13 02:53:06 newshiftapi containerd[638]: time="2021-05-13T02:53:06.925244942Z" level=info msg="shim disconnected" id=118eee19458c54b81d62dd9c6b90569afd9274a7c0cfa22b5ac3137523135fb9
May 13 02:53:07 newshiftapi systemd-networkd[562]: vethfa4e04d: Lost carrier
May 13 02:53:07 newshiftapi kernel: [1817009.953414] br-d49621cda334: port 2(vethfa4e04d) entered disabled state
May 13 02:53:07 newshiftapi kernel: [1817009.953478] vethe64e57c: renamed from eth0
May 13 02:53:07 newshiftapi networkd-dispatcher[617]: WARNING:Unknown index 59873 seen, reloading interface list
May 13 02:53:07 newshiftapi systemd-udevd[1345484]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
May 13 02:53:07 newshiftapi systemd-udevd[1345484]: Using default interface naming scheme 'v245'.
May 13 02:53:07 newshiftapi kernel: [1817010.013482] br-d49621cda334: port 2(vethfa4e04d) entered disabled state
May 13 02:53:07 newshiftapi systemd-networkd[562]: vethfa4e04d: Link DOWN
May 13 02:53:07 newshiftapi kernel: [1817010.019642] device vethfa4e04d left promiscuous mode
May 13 02:53:07 newshiftapi kernel: [1817010.019646] br-d49621cda334: port 2(vethfa4e04d) entered disabled state
May 13 02:53:07 newshiftapi systemd-networkd[562]: rtnl: received neighbor for link '59874' we don't know about, ignoring.
May 13 02:53:07 newshiftapi systemd-networkd[562]: rtnl: received neighbor for link '59874' we don't know about, ignoring.
May 13 02:53:07 newshiftapi systemd-udevd[1345484]: vethe64e57c: Failed to get link config: No such device
May 13 02:53:07 newshiftapi systemd[1320499]: run-docker-netns-af823a2e13ea.mount: Succeeded.
May 13 02:53:07 newshiftapi systemd[1320499]: var-lib-docker-containers-118eee19458c54b81d62dd9c6b90569afd9274a7c0cfa22b5ac3137523135fb9-mounts-shm.mount: Succeeded.
May 13 02:53:07 newshiftapi systemd[1320499]: var-lib-docker-overlay2-6cc1753d8d5f41611a3f5ab8b2199b58bbd5b7acf94c2c4b03389fb03dab4371-merged.mount: Succeeded.
May 13 02:53:07 newshiftapi systemd[1]: run-docker-netns-af823a2e13ea.mount: Succeeded.
May 13 02:53:07 newshiftapi systemd[1]: var-lib-docker-containers-118eee19458c54b81d62dd9c6b90569afd9274a7c0cfa22b5ac3137523135fb9-mounts-shm.mount: Succeeded.
May 13 02:53:07 newshiftapi systemd[1]: var-lib-docker-overlay2-6cc1753d8d5f41611a3f5ab8b2199b58bbd5b7acf94c2c4b03389fb03dab4371-merged.mount: Succeeded.

Automate backups to s3://shift2bikes-backups-us-west-2

Let's get the file backups off the server and onto another service.

migrate DNS to netlify

• shift2bikes.org
• shift2bikes.com
• shifttobikes.com
• shifttobikes.org
• pedalpalooza.org

huge graphics in the posters dir are problematic to clone. should move to LFS + netlify large media

Document useful info from Slack workspace before old messages become inaccessible

Slack is changing its policy for free accounts, effective 1 Sep 2022: https://slack.com/blog/news/pricing-and-plan-updates

After Sep 1 we'll have a rolling 90 day window of messages. Prior to that date we should extract any useful bits of info and document them somewhere more stable.

review DNS settings at canvashost

• move all web (www/bare domain) to netlify
• review all MX / email to make sure it's correct and working

create real "netlifriends" or "open source" type netlify account for shift-docs

...this will allow the site to run unmolested on Netlify in case fool quits working there.

expand disk size in AWS

cf #11 as to why this is important :)

find helpers

Need to get someone(s) ops-y who is willing to learn all the things about the server to help us administer. Write playbooks and have them replay them to establish understanding :)

document :allthethings: in the AWS console/dashboard

there should be a playbook of:

• where to find and create and decom instances, backups, etc
• budget/billing stuff
• user adding stuff
• anything else that ends up being relevant (email and DNS services?)

complete disaster recovery playbook

Can we rebuild the whole stack from a fresh AWS machine? to include:

• start from repo + backup
• document AWS setup (ses + snapshots)
• make sure shift up does the right thing after installing
• how to change DNS to point to new machine (should need only api.shift2bikes.org)
• how to change netlify to use new SSH key on server for deploying
• cron jobs and other backupy stuff

`shift up` doesn't work without some additional setup

Recently tried to start up an instance of the site for the first time and got the below error. I think there is some required config for letsencrypt these days, at least on OSX. I fixed this by:

• editing the shift script to export LETSENCRYPT_WEBROOT to be /private/tmp/letsencrypt-auto
• exporting /tmp/letsencrypt-auto in docker (on a mac, that translated to /private/tmp/letsencrypt-auto and I couldn't figure out how to avoid that)

then shift up worked. Probably wants an @sdobz - approved fix rather than making it like 3x as hard to start for the first time?

$ ./shift up
Recreating shift_hugo_1 ... done
Recreating shift_db_1   ... done
Recreating shift_php_1  ... done
Recreating shift_nginx_1 ... error
[...]
ERROR: for nginx  Cannot start service nginx: b'Mounts denied: \r\nThe path /tmp/letsencrypt-auto\r\nis not shared from OS X and is not known to Docker.\r\nYou can configure shared paths from Docker -> Preferences... -> File Sharing.\r\nSee https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.\r\n.'
ERROR: Encountered errors while bringing up the project.

admin: use Andrew's advice to migrate to a more cost-efficient AWS setup

@onewheelskyward is an AWS pro and looked quickly at our settings and suggests:

• change t2.micro to t3.micro

• use a 16G EBS

• use a reserved instance (for a year or 3)

• add swap

• save a bit of $ ($10 -> ~$5/mo).

move to real email service for @shift2bikes.org addresses

cc @sdobz I might bug you for help with this one since you navigated amazon email for sending

come up with periodic review process for server (reboot to new kernel, confirm backup functionality)

this is probably a task to be shared: #9