willnilges/shelflife

An easy to use application that automatically spins down and expires unused OKD projects

ShelfLife

An easy to use Rust application that automatically spins down and expires unused OKD projects.

This is an application I worked on during the CSH summer hackathon. It's used to increase the ease of management of old OKD projects on CSH's Openshift cluster. We don't have a good way of managing these applications, so here it is!

At its core, this tool queries the Openshift API for a particular project's info, queries a MongoDB server for data, and then reacts to that data.

Basic features:

• Operate on a cron-based schedule
• Track all OKD namespaces, their admins, and when they were last deployed (or one of a few other timestamps, if those don't exist)
  • Store metrics on Openshift project lifespans in MongoDB
• Notify admins of those namespaces when their namespace hits a certain age (according to the build timestamp)
• Spin down, back up, and delete old projects to save resources

Also check out the frontend! (In development)

Installation

Recommended distro: Ubuntu 18.04

To use this application, download the install.sh file and run it as root. It will fetch the latest release binary, as well as a env file, if you don't have one, and install your crontab. From there, you must fill out the .env file with your OKD cluster info, a mongodb, mailing info and options, a backup path, and a log path.

Usage

ShelfLife uses cronjobs to complete its tasks. The default looks like this:

0 * * * 1-3 /usr/local/bin/shelflife -a #On the hour, every hour, Sat-Wed,
0 * * * 6-7 /usr/loca/bin/shelflife -a  #query the OKD cluster for changes

0 12 * * 4 /usr/local/bin/shelflife -D  # On Thursday at noon, do a dryrun of the 
                                        # actions to be taken, and let the
                                        # admins know what is about to happen

0 12 * * 5 /usr/local/bin/shelflife -C  # On Friday at noon, do a cull,
                                        # and send a report of the cull to
                                        # the admins.

To use the shelflife command line, run shelflife and pass it flags:

CLI Arguments

USAGE:
    shelflife [FLAGS] [OPTIONS]

FLAGS:
    -a, --all                   Queries all available namespaces and adds/updates any that are missing/outdated to the
                                database.
    -c, --cull                  Checks graylist for projects that need attention. Takes appropriate course of action.
    -C, --cull_with_report      Culls, and generates and sends a report to ShelfLife admins.
    -d, --dryrun                Checks graylist for projects that need attention. Takes no action.
    -D, --dryrun_with_report    Dryruns, and generates and sends a report to ShelfLife admins.
    -h, --help                  Prints help information
    -l, --list                  Print namespaces currently tracked in the database.
    -V, --version               Prints version information
    -w, --whitelist             Enables whitelist mode for that command, performing operations on the whitelist instead
                                of the greylist.

OPTIONS:
    -k, --known <NAMESPACE>      Query API and ShelfLife Database for a known namespace. If it is missing from the
                                 database, the user is asked if they want to add it.
    -p, --project <NAMESPACE>    Query API for project info about a namespace.
    -r, --remove <NAMESPACE>     Removes a namespace from the database.

Contributing

Firstly, I just wanna say, "Thanks!"

Next, here's what you'll need:

Prerequisites

• An Openshift cluster that can be accessed via API calls
• An admin™ account for ShelfLife to view and manage namespaces
• MongoDB installed and running (https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/)
• A .env file to store Openshift cluster information, DB information, and a few other miscellaneous things. Get started by copying the provided .env.sample file to .env and then fill in the appropriate values.

Installation

Build Dependencies

• Ubuntu (Server 18.04 recommended)

• build-essential

• libssl-dev

• pkg-config

• Clone the repo and run the install script.

git clone https://www.github.com/willnilges/shelflife
./dev_install.sh

• Run the following commands on your openshift cluster:

oc create sa shelflife-dev-bot # Create a service account for shelflife to use.
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:default:shelflife-dev-bot # Make the service account an admin on your cluster.
oc get token shelflife-dev-bot # Spits out the API token.

• Copy the API token, cluster URL, and email credentials into the .env file.
• Configure the .env file with your email information.
• Go to town.

shelflife's Issues

Fix grammar

With the change to "Ignore Lists" and "Track Lists," some of the grammar in the -k option doesn't really work anymore. That should be fixed.

Handle DB being unreachable

Currently, ShelfLife will hang and then panic if it can't hit the DB server. It should display an error message promptly and exit safely.
This is somewhat related to #54 .

Don't touch anything with "system:admin" as owner

Check the admins of a project and automatically whitelist anything that has system:admin: as an owner.

Write some A̢̫̺͝u̢̻̞̫͍̘̙͇͎̙͘͡t̨̢̘͈̜͚͕͍̤̩o҉̤̠̬͖͈m̠a̤͉͓̰t҉̝̝̀e̦̫̹͈͖̫͚ͅd̡͏̣͔̱͉͓̲͈̪ ̛͇t̶̰e̶͉̰s̜͇̝̮t͏̯͈̩s̡̢̼̜

CI or smthn to make sure that emails, deletion, restoration, etc. work properly

Need a better way to parse command line args.

@nicholastmosher Suggests https://clap.rs (https://docs.rs/clap/2.33.0/clap). This needs to be implemented because the current way of doing things is not gonna work in the future.

Cause isn't accurate

I'm not sure if cause is even being stored. All projects return cause as "Deployment." Seems like it's probably an issue with get_db().

Collect Regular Backups

Backup all projects (including those not capable of being deleted) once a week and retain the last 3-4 weeks.

dryrun culling

Implement a dryrun feature that returns a list of all projects that would be culled if you were to run the culling function.

Add `weeks since` to project list option

It would be great to be able to see that to get a better idea of how long it's been since something was touched.

Add verbose mode for debugging.

Good fucking grief.
Maybe have separate levels for data manipulation all the way down to API calls and JSON.

Track projects by discovery date

What do we do for projects that are 24 weeks old when you turn on shelf life for the first time?

Throw more descriptive error when .env file is missing

This made me debug for like 3 hours. When the .env file is missing, it throws this error, which is REALLY confusing coming from a Docker image that you've just built:

root@c140184ee321:/shelflife/shelflife-src/target/release# ./shelflife                           
Error: NotPresent  

Refactor code

This code isn't very well written. While working on #54, we should definitely refactor the code as much as possible.

Support for multiple items and wildcards on command line interface

Bring back the automated installer

Now that we're not containerizing anymore, update the CLI installer and ship it with the app.

Create function for spinning down a project

Gotta stop all pods and deployments on a project when they pass the 16 week mark.

https://docs.openshift.com/container-platform/3.7/rest_api/api/v1.Pod.html

flag or env variable to turn off mail.

The RTPs freak out every time I even view this repo's github page.

Unify naming scheme.

Is it namespace or project? I know we track namespaces but like come on, Kube is wack. We gotta fix this.

[Feature Request] Administrative Digest Emails

It’s a little much to send a copy of all emails to the cluster admins, but it’d be neat to at least send a digest email so they would be aware of what’s expiring.

Something like

The following projects are expiring soon and will be spun down

The following projects are about to be nuked

The following projects have been recently nuked. Backups are in these locations

Containerize

We want this whole thing running as an Openshift project.

Lol. Inception.

Error codes don't return

If you enter a bad namespace as an attempt to query, ShelfLife will simply return without any clear output as to what happened.

Only notify original creator of project. (or don't notify the RTPs!)

This might not be a huge issue, but if the RTPs are going to get an email every time someone doesn't work on their project, then it should be fixed.

Handle errors

This code... doesn't handle errors. Like, it's fine, but it should probably handle errors using Rust's error handling instead of... whatever it's currently doing.

Investigate async

I'm working on updating all my libraries, and that breaks a lot of stuff, because it seems like they all use async and other newer rust features. Might be worth looking into those.

Implement function to make API call to get all namespaces.

The big one — Get every namespace the API Token has access to. This function will be an essential part of the application, and will run stupid early every day. It'll be a part of a system that looks for any changes to any namespaces, and updates the DB accordingly. This needs to be implemented, but first, I kinda need creds to the Openshift cluster.

https://docs.openshift.com/container-platform/3.7/rest_api/api/v1.Namespace.html#Get-api-v1-namespaces

CLI: Clap won't parse from arg vec

When using the command line, Clap won't recognize the passed arguments. See https://github.com/WillNilges/ShelfLife/blob/CLI/src/bin/main.rs

      Welcome to ShelfLife     
******We nuke old projects******
 Get a job or get D E L E T E D 

> yeet
[src/bin/main.rs:59] &args = [
    "yeet",
]
[src/bin/main.rs:101] &matches = ArgMatches {
    args: {},
    subcommand: None,
    usage: Some(
        "USAGE:\n    yeet [FLAGS] [OPTIONS]",
    ),
}
> list
[src/bin/main.rs:59] &args = [
    "list",
]
[src/bin/main.rs:101] &matches = ArgMatches {
    args: {},
    subcommand: None,
    usage: Some(
        "USAGE:\n    list [FLAGS] [OPTIONS]",
    ),
}
> "list"
[src/bin/main.rs:59] &args = [
    "list",
]
[src/bin/main.rs:101] &matches = ArgMatches {
    args: {},
    subcommand: None,
    usage: Some(
        "USAGE:\n    list [FLAGS] [OPTIONS]",
    ),
}
> 

Examine network traffic

• Whitelist projects with sufficient traffic
• Avg traffic over time
• How do we log that?
• Create a traffic bar that is required to hit in order to reset ShelfLife.

Lacking environment variables silently fails

It should not do this.

Add configuration options for time thresholds and emails

.csv support for importing projects to whitelist and greylist.

MongoDB's gotta have this fucntionality.

Yeet exports somewhere

Yea we need a more robust way of handling exports. We can't just yeet them into /tmp. But homedirs are hard at CSH. But an RTP/web interface needs to do it anyway since root access. UGH.

Let's figure out how to attach them to emails or figure out how to make a directory.

Remember, the end goal is containerization.

Need to query something other than namespaces

A pinch of research and a whole bunch of garbage test projects later I've deduced that namespaces are not a reliable way to track projects with the current implementation. I'm not quite sure how to continue at this point, but I have found a way to query by project name, which returns a bunch of useful info. This info might be used to find relevant information, and target specific namespaces within a project. Basically, a whole bunch of stuff needs to change.

JSON error: missing field `completionTimestamp`

Alright, here's one that stumped me last night 🙃 So I was running ShelfLife on the OKD cluster to fill the db. What it does is it queries the cluster for several things: Name of project, last update, and cause. It does this by asking for build and deployment dates. This is returned from the API in json. This json is then stored in a struct that Rust can understand. Problem (I think) is, sometimes there are absolutely no builds or deployments to give me a completion timestamp. This causes a Json error in Rust. Here's the code that does it:

 // Query for builds
    let builds_call = format!("https://{}/apis/build.openshift.io/v1/namespaces/{}/builds",endpoint, namespace); // Formulate the call
    let builds_resp = get_call_api(&http_client, &builds_call); // Make the call
    let builds_json: BuildlistResponse = builds_resp?.json()?; // Bind json of reply to struct.
    let mut builds = Vec::new();
    for item in builds_json.items {
        builds.push(DateTime::parse_from_rfc3339(&item.status.completion_timestamp));
    }

And here's the runtime error. This clearly is not safe, and the program just exits after this. I've included some sample outputs for reference.

Querying API for namespace clowncadante... API Response: clowncadante ["egg"] Fri, 29 Nov 2019 22:15:10 +0000 Deployment
The requested namespace is in the database. Updating entry...
clowncadante has been removed.
[src/lib.rs:501] &item.last_update = "Fri, 29 Nov 2019 22:15:10 +0000"
Entry updated.
Querying API for namespace collabora... API Response: collabora ["wgahnagl"] Fri,  6 Sep 2019 02:24:40 +0000 Deployment
The requested namespace is in the database. Updating entry...
collabora has been removed.
[src/lib.rs:501] &item.last_update = "Fri,  6 Sep 2019 02:24:40 +0000"
Entry updated.
Querying API for namespace conditional... API Response: conditional ["system:admin"] Fri, 29 Nov 2019 22:11:36 +0000 Deployment
The requested namespace is in the database. Updating entry...
conditional has been removed.
[src/lib.rs:501] &item.last_update = "Fri, 29 Nov 2019 22:11:36 +0000"
Entry updated.
Error: Error(Json(Error("missing field `completionTimestamp`", line: 1, column: 3238)))
Querying API for namespace constitution-web...wilnil@shelflife-main:~/shelflife$ 

I think this is the doc page for json() which I believe to be the issue
https://doc.rust-lang.org/1.0.0/serialize/json/

Names are added to databases with quotation marks

This causes problems:

Gotta strip those quotations when the name is added to the vector. This'll require a little re-working.

.env variable for email address domains.

Persistent login?

Right now, ShelfLife requires a token that only lasts 24 hours. Needa figure out how to make that more permanent.

Clean up lib directory

There's some stubborn-ass file in the lib dir that won't go away, I've seen it like 5 times now and im finally gonna make a damn issue about it.

it's called .lib.rs.swo

wtf is it?

Whitelisting a project does not remove it from the graylist

That should uhhhhhhhhhhh happen.

OpenShift pods keep getting evicted

It's been a hot minute since I looked at my OKD project for this app, and I noticed that pods were getting evicted (presumably due to resource constrictions) every few minutes. There were hundreds of them.

What the hell!?

You've heard of shelflife...

Get ready for

ShelfLife 2!!!!!!!

Yea.... uhhhhhhh.

What if we just.

Made this.

In Go.

And made it a Kube operator....

Or like. Put it in Kube.