nimbo-sh/nimbo

Run compute jobs on AWS as if you were running them locally.

nimbo's Issues

Migrate from AWSCLI v1 to v2

AWS CLI v2 is recommended for general use by AWS.

Improve testing

Right now testing is minimal, and the quality of the tests is not fantastic.

1. Introduce --dry-run tests for every CLI option.
2. Write more end-to-end tests.

Would be desirable to aim for 100% test coverage.

Replace AWSCLI usage with boto3 implementation

Right now awscli is only useful for s3 due to aws sync, but the usefulness of this command is limited.

By replacing awscli usage with our own implementation for interacting with s3 we will be able to:

• Implement aws sync like ability, and extend it to fix #16
• Remove a dependency
• Not need to migrate to v2 #24
• Will also resolve the issue with colorama version mismatches.

Bash and Zsh command autocompletion

To make the experience of using Nimbo more seamless, we should support Bash and Zsh autocompletion.

Renaming a subdirectory in datasets and pushing reuploads all files

If you have pushed a subdirectory in your datasets folder, then rename it locally and push again, all files will be reuploaded. It seems to me that through aws mv it should be possible to avoid this.

Allow naming / describing instances

nimbo list-active outputs a nice list of active instances. However, instance ids are pretty cryptic, so it is hard to know what is what. It would be really nice to be able to name instances, or perhaps to attach some sort of "description" field (that could for example contain hyperparameters, datasets used, etc).

Frequent results sync in remote_setup.sh causes temporary cached files to be uploaded

I was finding some strange files like some_results.pkl.H23ad4 with a hex string appended in my S3 bucket. I believe this is due to the fact that remote_setup.sh syncs the results folder frequently enough that sometimes an object is partially written to disk and we try to sync it.

create-bucket fails

Steps to reproduce

$ nimbo create-bucket BUCKET_NAME

Error

produces this stacktrace.
The final lines of the stacktrace:

  File "/nix/store/k8azdbzwsyklkm344271wacjjsi4mkp5-python3.8-botocore-1.20.52/lib/python3.8/site-packages/botocore/validate.py", line 293, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "DryRun", must be one of: ACL, Bucket, CreateBucketConfiguration, GrantFullControl, GrantRead, GrantReadACP, GrantWrite, GrantWriteACP, ObjectLockEnabledForBucket

Group CLI help options

At the moment CLI options like auxiliary commands like list-gpu-prices are mixed main commands like ssh. Group CLI help options by category.

For example:

Utilities:
  list-gpu-prices
  list-spot-gpu-prices
  allow-current-ip

s3:
  ls
  pull
  push

Package Nimbo for apt and brew

Packaging Nimbo for apt and brew will cover the majority of Nimbo usecases.

DryRun handling is inconsistent, a lot of repeated code

Cleanup DryRun

nimbo spending hangs instead of failing when user does not have permissions for cost explorer

Nimbo fails to connect via SSH / Port 22 while terminal session succeeds

Describe the bug
I'm using nimbo run "python script.py" and the script fails at "Something went wrong while connecting to the instance."
In the AWS EC2 cloud console I can see the instance up and running, plus I can connect using a separate terminal session using ssh -i {SAME_SSH_KEY_AS_IN_NIMBO_CONFIG}.pem ubuntu@{IP}.

Expected behavior
Nimbo should succeed and fail depending on the actual state of ssh / port 22 connection.

Configuration

cloud_provider: AWS

# Data paths
local_datasets_path: data  # relative to project root
local_results_path: results    # relative to project root
s3_datasets_path: s3://xyz/data
s3_results_path: s3://xyz/results

# Device, environment and regions
aws_profile: default
region_name: eu-west-1
instance_type: g4dn.2xlarge
spot: yes

image: ubuntu18-latest-drivers
disk_size: 80  # In GB
conda_env: conda-env.yml  # denotes project root

# Job options
run_in_background: no
persist: no  # whether instance persists when the job finishes or on error

# Permissions and credentials
security_group: default
instance_key: key.pem  # can be an absolute path
role: NimboFullS3AccessRole

ssh_timeout: 450

To Reproduce
Try running a python script using nimbo. Connect to IP address via separate terminal session.

nimbo ssh <instance_id> connects to the instance as well.

Any hints how to fix this or how to disable the initial up check?

Improve printing

Right now the way information output is:

• Inconsistent
• Unrefined

In addition, Nimbo depends on click and awscli which in turn depend on colorama. Right now we are using rich, investigate if it is worth reducing our dependencies.

issue creating spot instances

Hi! Thanks for this nice tool.

I'm having an issue with bringing up spot instances.

In nimbo-config.yml if spot is set to no, it works as expected, but setting to yes gives me this error:

Exception: {'Code': 'bad-parameters', 'Message': 'Your Spot request failed due to bad parameters. 
Spot request cannot be fulfilled due to invalid availability zone.', 
'UpdateTime': datetime.datetime(2021, 5, 12, 19, 8, 50, tzinfo=tzutc())}

I have tried on both ubuntu 20.04 and macos and meet the same error. On my linux box I tried both installing nimbo with pip and from source and met this availability zone error, on macos I've only tried installing nimbo with pip.

Is this a known issue? How can it be resolved?

Thanks

Add Continuous Integration

There is no reason not to run tests that do not require spinning up of instances on each PR and master.

Docker support (as alternative to conda)

Is your feature request related to a problem? Please describe.
Conda env setup takes >45 mins every time I'm starting a new instance. This gets 10x more annoying when relying on spot instances.

Describe the solution you'd like
Instead of creating the same conda environment on each new instance, why not have a docker image that can be ready to start much quicker

Additional context
Would love any pointers how to add this to nimbo. I'd be up for creating a PR depending on the capacity needed.
Is this on the feature roadmap for nimbo anyway? (Is there a feature roadmap? :) )

Thanks, would love to hear your thoughts on this

Allow passing config parameters as environment variables

It would be useful to be able to pass certain parameters as environment variables. For example, suppose I want to commit a config file, but each user of the repo has a different key. With the current setup this wouldn't work, because everyone has different entries for this field:

instance_key: /home/andre/andre-key.pem

A similar issue would arise when running a CI job, for example.

One possible solution is to be able to pass this parameter (and perhaps others) as an environment variable. So something like this:

instance_key: $NIMBO_KEY

Cheers and keep up the great work.

Automatically create instance keys

We should have a command line option to create and download instance keys. Careful, the user might not have the right permissions to create them programatically.

AWS Credential errors

Really like the concept of nimbo, it is quite lightweight for launching instances. However, I have not been able to give it a try due to AWS key errors. Would love to chat more about using this tool though.

Nimbo instance alerts

Enable users to setup email alerts for when they forget to terminate instances. This could be done by deploying a lambda function that executes at some time of the day and sends an email to the user.

Tag the source

Could you please tag the source? This allows distributions to get the complete source from GitHub if they want.

Thanks

Current IP addition improvements

At the moment, when adding current IP to a security group /16 CIDR block is used, this should be a config option. Should probably default to /32.

Nimbo should also attempt to add the current IP automatically to the current security group without the user having to run allow-current-ip group_name to reduce the number of actions the user has to perform.

nimbo unable to read pem key 'load pubkey "mykey.pem": invalid format'

Reproduced by

1. Generate a key mykey.pem in the AWS console and download it
2. Copy mykey.pem to project dir
3. Put instance_key: mykey.pem in nimbo-config.yml
4. Run nimbo test-access
5. Get a bunch of load pubkey "mykey.pem": invalid format errors

Inspecting the key shows it looks fine with correct header/footer and can be used to manually SSH onto EC2 instances. Is there something Nimbo does differently when it comes to keys?

Changing nimbo-config.yml name

Hello,

We have a docker container which runs as a microservice that uses nimbo for training a model on EC2. However, if two users in the same container were to run a job at the same time than someone may use the other person's nimbo-config.yml file.

Is there a way to change the nimbo-config.yml name so that each user can have their own nimbo configuration file.