cci-moc/moc-openstack-tools

Tools for OpenStack administration, user/project management, etc.

moc-openstack-tools's Issues

Resetting password is a manual process

It seems password reset was never automated completely ?

Resetting passwords:

The ticket will contain the OpenStack username and a 4 digit PIN. Log
into the helpdesk VM and type:

$ moc reset-password <USERNAME> <PIN>

addusers Kaizen OpenStack creds visible in config file

The password for Kaizen user addusers is visible in a configuration file in the moc private repo. It would be a better practice to keep a dummy password there and store the real one on the VM(s) that are running the software.

Allow users to create project names with whitespace.

Currently, when a user requests a project through the Google Form, they are required to create a project without any whitespace. This is required because addusers.py defines the project name as a function, and the Python interpreter does not allow functions that contain whitespace.

need requirements.txt

Should have a requirements.txt so we can install dependencies easily.

Request processing and emailing both depend on same cron cycle

This is another issue brought up during the meeting with @kamfonik.

check-approved-requests.py both processes new requests and sends out a reminder email at the speed of the cron frequency. It might be beneficial to separate these functions, which would allow processing to happen at faster intervals than the emailing.

make better use of the project description field

Project description field should also contain contact info for the primary project owner and the organization.

check-approved-requests.py partially fails.

The check approved requests script is failing to write to the "Helpdesk Notified" column in the requests Google Sheet. Since addusers.py will only create approved accounts when the "Helpdesk Notified" column has been filled, this would explain why addusers.py recently failed to run from the crontab.

[moc-tools@helpdesk-vm production]$ python check-approved-requests.py
Traceback (most recent call last):
  File "check-approved-requests.py", line 318, in <module>
    check_requests('Access', auth_file, worksheet_key)
  File "check-approved-requests.py", line 285, in check_requests
    timestamp_spreadsheet(sheet, timestamp, processed_rows, column=1)
  File "check-approved-requests.py", line 206, in timestamp_spreadsheet
    batch.execute()
  File "/usr/lib/python2.7/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/googleapiclient/http.py", line 842, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/1J32SU5HunV5fhff4deathB5IsAIPtCOTWRZuSQFqns0:batchUpdate?alt=json returned "The caller does not have permission">

Update: the same thing happens with addusers.py. So, it creates the user account and the project, but fails to move the spreadsheet entry to the "current users" spreadsheet. So, it appears to be the same write access failure problem.

Traceback (most recent call last):
  File "/usr/local/src/moc-tools/production/addusers.py", line 481, in <module>
    sheet.append_rows(copy_rows, target="Current Users")
  File "/usr/local/src/moc-tools/production/spreadsheet.py", line 188, in append_rows
    return req.execute()
  File "/usr/lib/python2.7/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/googleapiclient/http.py", line 842, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/1J32SU5HunV5fhff4deathB5IsAIPtCOTWRZuSQFqns0/values/Current%20Users%21A1:append?alt=json&insertDataOption=INSERT_ROWS&valueInputOption=RAW returned "The caller does not have permission">

Logging

right now we only print errors/results to console, which is not ideal

Order is not guaranteed when processing form entries

Because new projects are stored in a dictionary, and the users are stored under projects, there is no guarantee of the order in which spreadsheet lines are processed.

How this can create a problem:

• User X fills out the form asking for a new account
• User X then fills out the form again right away because he needs a second project; this time he marks his account as existing.

It's possible for the second entry to get processed before the first, and in that case it is skipped because the user doesn't exist.

The workaround is simply to run the script again and the second entry will now be processed correctly since the user exists.

But, long term we likely want to guarantee execution order of the spreadsheet rows.

set search columns in config using column headers

Currently we have USER_COLUMN and PROJECT_COLUMN hard coded, we would prefer to load the header in config and then determine the index programmatically.

HACKING.md suggests running a CentOS command on Ubuntu

On line 61 of HACKING.md in the "documentation" branch, under "Install required packages," the documentation suggests running:

$ sudo yum install python-virtualenv

However, this part of the documentation covers setting up a Ubuntu VM, so I believe it should be:

$ sudo apt install python-virtualenv

Remove dependency on Google Forms/Sheets

It would be better to use a local database (similar to Setpass) and a custom form; this would give us more flexibility over sending emails and streamline a few parts of the approval/notification project.

Need to remove preceeding and trailing whitespace from user input

When users input information such as project names, sometimes preceding and trailing whitespace is entered by accident. This causes the names to mismatch because the whitespace is processed. We need to check for this whitespace and remove it before the information is processed.

centralize exceptions into one file

Store exceptions in a central file.

user/project selection does not preserve row index

This causes issues when it is time to move/copy rows, if there are other rows that should be left behind.

Remove bare except blocks

There are bare except blocks in addusers.py and reset-password.py. It would be a better practice to write which exceptions we expect to be caught there.

Example: https://github.com/CCI-MOC/moc-openstack-tools/blob/master/addusers.py#L118

allow processing of only a single user's requests

Currently we process all approved/ticketed requests on the spreadsheet each time the script is run.

Being able to specify a single user (or project, in the case of quotas) would be useful for the helpdesk staff who might want the option of dealing with only their assigned ticket.

new project email doesn't contain the name of the project

We should include the name of the project to prevent confusion.

add additional information to the Current Users page

The script should fill out new columns in Current Users:

• a Processed timestamp
• for new users, the user ID

There was a PR a long while back in the old private repo that attempted to address this, it had some important flaws but the basics are there:
https://github.com/CCI-MOC/moc/pull/54

Remove set-quota requests from Google Sheets

Currently here's a bug on Google Sheets where processed rows are not deleted automatically.

fix directory structure so that tests work

Right now we have a hack in place to get imports in the tests working with the current directory structure, but the real solution is to fix the directory structure.

allow manual command-line input for creating users and projects

This would be useful in a number of cases:

• testing changes to the code that don't involve the form
• fixing a mistake
• simple tasks like adding an existing user to a project
• adjusting things if there was some bug in the script

There is some basic code in the TODO at the end of addusers.py that could be adapted for this.

Remove permanent resource requests

We want all resource requests to be temporary, and have some way of enforcing the end date of a resource request. To do: change the google form to reflect this, automate reminder emails to user for month/week/48hrs before end of resource, and automate reminder email to MOC staff to terminate resource after end date.

Duplicate row indices handled poorly by spreadsheet delete_rows functions

If we pass a list of rows with duplicate indices - e.g. [1,1,1] - to spreadsheet.delete_rows, it will delete row 1 three times - and since the indices update after each delete operation, you end up accidentally deleting row 2 and 3.

We need to remove duplicates from the list at some point in the process.

Email address checks should not be case sensitive

Welcome to the MOC helpdesk interface. Type moc help for instructions.
[helpdesk@helpdeskvm ~]$ moc reset-password [email protected] 9188
User [email protected] not found
[helpdesk@helpdeskvm ~]$ moc reset-password [email protected] 9188
Password successfully reset for user [email protected]

Email comparisons should be case INsensitive

Asking user to use osticketing system

Right now, the welcome email asks user to report issues and ask question by sending email to [email protected].

We want to replace this with asking the users to use https://osticket.massopen.cloud/ - our ticketing system instead.

Automation of projects visible in request form

It would be beneficial to automate the process of adding OpenStack projects to the visible drop-down list in the project request form. This would require some exploration of the Google API and some further administrative decisions.

One issue about this that was discussed during the meeting on 10/19 was about deciding which projects should be available. For example, if the automation is setup to show all OpenStack projects, it presents the possibility of a user selecting and being approved for a project that they really shouldn't be in.

Store new user information and new project information in different worksheets

Also store the user ID and project ID of the new projects when they are created.

Default quota setting is incorrect (?)

Jeremy Freudberg [email protected]
Reply all|
Yesterday, 10:35 PM
Saowarattitada, Piyanai;
MOC Team [email protected]
Actually, I just looked into this more closely now.

In my own project on Kaizen, I have one instance and the "standard"
network topology which we recommend to users. This actually takes up 4
ports:

• the project's router
• two DHCP agents
• one instance

I just added myself to Kia's project as well, and I see 3 ports in use:

• the project's router
• two DHCP agents
• zero instances

So actually we need to tweak our default quotas a bit. It should
really be 13 ports and 10 instances for the default quota, not 10 and
10. From that point onward, if a user requests more instances, then we
can just keep adding 1 to that number. But there has to be an extra 3
ports "reserved" for holding the necessary networky bits.

(Worth noting that if a user creates a "non-standard" network
topology, for example multiple interfaces on one VM, extra routers, no
DHCP, etc, then this "extra" 3 ports might be a different number. But
if they are doing that then they are probably smart enough to know
what a port is in the Neutron sense.)

10 - 3 = 7 which fits with Kia's observed problem.

But what I can't remember, from back in the old days of
Liberty/Mitaka, was it always 3 ports taken up? I mean, was there
always 2 DHCP agents there? I really can't remember.

TL;DR -- Increase the default port quota by 3 and everyone is happy.

From: kaizen [email protected] on behalf of Jeremy
Freudberg [email protected]
Sent: Tuesday, October 17, 2017 12:01:54 AM
To: [email protected]
Subject: [Kaizen] Fwd: Error - Maximum number of ports exceeded (HTTP 403)

Hmm... I thought this problem was already fixed, generally:
2c9b599

-------- Original Message --------
Subject: [Kaizen] Error - Maximum number of ports exceeded (HTTP 403)
Date: 2017-10-16 23:52
From: Kia Teymourian [email protected]
To: [email protected]

Hi,

when I try to start more than 7 instances, I get the following error

"Maximum number of ports exceeded (HTTP 403) (Request-ID:
req-339c4be6-7272-481a-b71d-2eb3263a6efb)"

My project name is "DataStreamAnalysis"

I have only two public IPs and assign only one IP to the one of the
machines (my Master machine).

I need that all ports are open inside the cluster network (inside my own
VPC).

Could you please help me with this.

Thanks,
Kia Teymourian

User account request form: Time stamps captured are incorrect

For example:

Approved | 23 Oct 2017 21:25:08 |   | 10/23/2017 21:14:54 | ...

The 2nd ("Helpdesk notified") and 4th ("Timestamp") columns above should have read something like "17:...". Appears to be 4 hours difference here.

I "think" this problem has been there on-off for a while now. It would be good to get this fixed for real.

Piyanai