suse-enceladus/ec2imgutils

A collection of utility scripts to manage images

ec2imgutils's Issues

Typos in ec2deprecateimg man page

A few typos in the man page

https://github.com/SUSE-Enceladus/ec2imgutils/blob/master/man/man1/ec2deprecateimg.1#L13

"The tagging mechanism is an implementation based on. "

This is not a complete sentence. Based on what?

https://github.com/SUSE-Enceladus/ec2imgutils/blob/master/man/man1/ec2deprecateimg.1#L22

"This can be considered equivalent to a deltion."

s/deltion/deletion

verify - ec2publishimg --share-with none appears to throw when the image is not shared already

[ec2imgutils-8.1.2] no ability to use --version flag

ec2uploadimg --version
usage: ec2uploadimg [-h] [-a ACCOUNT_NAME] [--access-id AWS_ACCESS_KEY]
                    [-B EC2_BACKING_STORE] [--billing-codes BILLING_CODES]
                    [--boot-kernel AWS_AKI_ID] -d IMAGE_DESCRIPTION
                    [-e AWS_AMI_ID] [--ena-support] [-f CONFIG_FILE] [--grub2]
                    [-i AWS_INSTANCE_ID] -m ARCH -n IMAGE_NAME
                    [-p PRIVATE_KEY] -r EC2_REGIONS
                    [--root-volume-size ROOT_VOLUME_SIZE]
                    [--ssh-key-pair AWS_KEY_PAIR_NAME] [-s AWS_SECRET_KEY]
                    [--security-group-ids AWS_SECURITY_GROUP_IDS]
                    [--session-token AWS_SESSION_TOKEN] [--snaponly]
                    [--sriov-support] [--ssh-timeout SSH_TIME_OUT]
                    [-t AWS_UPLOAD_INST_TYPE] [-u AWS_INSTANCE_USER]
                    [--use-private-ip] [--use-root-swap] [-V AWS_VIRT_TYPE]
                    [--verbose] [--version] [--vpc-subnet-id VPC_SUBNET_ID]
                    [--wait-count AWS_WAIT_COUNT]
                    source
ec2uploadimg: error: the following arguments are required: -d/--description, -m/--machine, -n/--name, -r/--regions, source

Multiple boto3 clients with a single botocore session is not thread safe

Whether or not temporary credentials are used with ec2imgutils there's a chance for a session to be shared across multiple clients. This is not recommended based on boto3 documentation https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-or-multiprocessing-with-resources.

This is in reference to the connect method on the EC2ImgUtils class https://github.com/SUSE-Enceladus/ec2imgutils/blob/master/lib/ec2imgutils/ec2imgutils.py#L45.

When using a temporary credential the session gets reused many times with new clients, otherwise the client is created which leaves it up to boto3/botocore to find an existing session or create a new one. As per the boto3 docs it's recommended to explicitly create a new session for every new client.

Outdate AMIs in ec2utils.conf.example file

Hello,

as we discussed with @smarlowucf the list of ec2utils.conf.example EC2 AMIs is outdated.
For example the ami-bff32ccc on eu-west-1 does not have lsblk >= 2.27 with the -J option.

This breaks upload of new EC2 on-demand images in our openQA.

cc: @cfconrad @asmorodskyi @grisu48

Return code 0 on unsuccessfull upload

The ec2uploadimg tool terminates with exit code 0, also if there are obvious configuration issues and thereby no image being uploaded. See the following output we obtained from one of our openQA test runs, where we rely on the return code of ec2uploadimg to determine, if the upload was successful.

# ec2uploadimg ... 'SLES15-SP4.aarch64-0.9.9-EC2-Build1.55.raw.xz'
Creating temporary security group
Temporary Security Group Created sg-062f66b1f33abb2bb in vpc 
Successfully allowed incoming SSH port 22 for security group sg-062f66b1f33abb2bb in 
Specified subnet subnet-b44c72df not found
Traceback (most recent call last):
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/ec2imgutils/ec2uploadimg.py", line 193, in _check_subnet_exists
    self._connect().describe_subnets(SubnetIds=[self.vpc_subnet_id])
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidSubnetID.NotFound) when calling the DescribeSubnets operation: The subnet ID 'subnet-XXXXXXXX' does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv_ec2uploadimg/bin/ec2uploadimg", line 670, in <module>
    ami = uploader.create_image_use_root_swap(args.source)
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/ec2imgutils/ec2uploadimg.py", line 1029, in create_image_use_root_swap
    target_root_volume = self._create_image_root_volume(source)
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/ec2imgutils/ec2uploadimg.py", line 332, in _create_image_root_volume
    self._check_subnet_exists()
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/ec2imgutils/ec2uploadimg.py", line 196, in _check_subnet_exists
    raise EC2UploadImgException(error_msg)
ec2imgutils.ec2imgutilsExceptions.EC2UploadImgException: Specified subnet subnet-XXXXXXXX not found
EC2UploadImgException: Specified subnet subnet-XXXXXXXX not found
Successfully deleted security group sg-062f66b1f33abb2bb

The issue is likely attributed to the except block here. I will try to create a pull request with a solution suggestion.

ec2uploadimg exit with 0 on VpcLimitExceeded

ec2uploadimg --access-id '' -s '' --backing-store ssd --grub2 --machine 'arm64' -n 'openqa-SLES15-SP2-BYOS.aarch64-1.0.19-EC2-HVM-Build1.29.raw.xz' --virt-type hvm --sriov-support --ena-support --verbose --regions 'eu-west-1' --ssh-key-pair 'openqa1631060210_0' --private-key-file QA_SSH_KEY.pem -d 'OpenQA upload image' --wait-count 3 --ec2-ami 'ami-02eae5be24d203db3' --type 'c6g.large' --user 'ec2-user' 'SLES15-SP2-BYOS.aarch64-1.0.19-EC2-HVM-Build1.29.raw.xz'; echo 4U0tD-$?-
An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The maximum number of VPCs has been reached.
Traceback (most recent call last):
  File "/root/.venv_ec2uploadimg/bin/ec2uploadimg", line 568, in <module>
    vpc_subnet_id = setup.create_vpc_subnet()
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/ec2imgutils/ec2setup.py", line 140, in create_vpc_subnet
    self._create_vpc()
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/ec2imgutils/ec2setup.py", line 175, in _create_vpc
    response = self._connect().create_vpc(CidrBlock='192.168.0.0/16')
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/.venv_ec2uploadimg/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The maximum number of VPCs has been reached.
ClientError: An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The maximum number of VPCs has been reached.
4U0tD-0-

Get regions method is inoperable

ec2imgutils/lib/ec2imgutils/ec2utils.py

Line 208 in 8cbc629

session = boto3.session.Session(

When getting a session the default region is used. For a us-gov or cn account this will fail if the default region is in aws partition. In order for this to work a default region mechanism would need to be added to ec2imgutils per account in the .ec2imgutils config file. That way the session could be created like:

session = boto3.session.Session(
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_key,
    region_name=default_region
)

support use of aws cli config file

If the access and secret keys are missing in the ec2utils config file the config file for the aws cli should be used to try and find the keys.

uefi-preferred option missing in man pages

The uefi-preferred option was added to ec2uploadimg as a boot mode but the man pages were not updated to reflect the new option.

Option to override deprecation date

An image may need to be deprecated after the fact. For this use case it would be nice to have an option to override the deprecation date so it matches the correct time of deprecation.

SUSE Package Hub out of date

SuSE Package Hub (https://packagehub.suse.com/packages/python3-ec2imgutils/) shows the latest version available for ec2imgutils as 8.0.0-bp152.1.1 updated back in May 2020.

GitHub shows the current version is v10.0.1 as of May 2022 (over 2 years out of date).

What is package update process? Is there a plan to keep packages up to date?

Some lag time is expected. But what is appropriate? Perhaps 30-60 days from GitHub release to availability on Package Hub?

Thanks much

security group clean up

When clean up is skipped because image creation timed out it appears that there is still and attempt made to clean up the created security group, this leads to a traceback.

Expects Xen based helper image

When using ec2uploadimg a Xen based instance type is expected as the helper script looks for sdf or xvdf. However on Nitro based instances with NVMe the device naming is different. The ec2uploadimg script should account for this.

ec2deprecateimg should only check regions where the deprecating image exists

When running this command:

ec2deprecateimg --image-name suse-sles-11-sp4-byos-v20180816-pv-ssd-x86_64 --replacement-name suse-sles-12-sp3-byos-v20190314-hvm-ssd-x86_64 --account resale-images --verbose

this error shown is

Region: af-south-1 -> Replacement image not found, "suse-sles-12-sp3-byos-v20190314-hvm-ssd-x86_64" did not match any image.
Region: eu-south-1 -> Replacement image not found, "suse-sles-12-sp3-byos-v20190314-hvm-ssd-x86_64" did not match any image.
Region: me-south-1 -> Replacement image not found, "suse-sles-12-sp3-byos-v20190314-hvm-ssd-x86_64" did not match any image.

but the source image suse-sles-11-sp4-byos-v20180816-pv-ssd-x86_64 does not exist in these regions and should not produce this message

Please update pypi package

the version on pypi is from 11/2018

Add ingress rules to non existent security group

In the EC2Setup class a temporary security group is created and SSH ingress rules are added. https://github.com/SUSE-Enceladus/ec2imgutils/blob/master/lib/ec2imgutils/ec2setup.py#L98 It's possible for the the rules to be added before the security group actually exists in the platform:

Exception in test: An error occurred (InvalidGroup.NotFound) when
calling the AuthorizeSecurityGroupIngress operation: The security group
'sg-abcdefg' does not exist

It's possible this is also related to an issue with sessions getting picked that were created in a different thread #76.

ec2uploadimg fails when the helper instance is m5d.xlarge

invalid literal for int() with base 10: '139.7'

Create image from existing snapshot

ec2uploadimg should have an option to create an image from and exiting snapshot. this should support root-swap as well as "regular" image registration.

Account/Profile SDK Option

When using ec2imgutils from CLI there is an option to provide an account or profile instead of the credentials. It would be helpful for this functionality to be encapsulated in the classes so it's available when using the "SDK".

Also, it would be helpful for this feature to also check/leverage the AWS CLI credentials file in case the ec2imgutils credentials file doesn't exst.

support for EC2 instance type t3

Hello,

I'm not sure whether EC2 instance type t3 is supported, but it is currently not working with ec2uploadimg. I get the following error when uploading a raw image:

Could not find disk device in helper instance with path /dev/sdf

When I deploy my raw image to an instance type t2 it dsoes work. Due certain load patterns I would prefer instance type t3.

ec2uploadimg

Should handle upload of non compressed images.

Your requested instance type (a1.large) is not supported in your requested Availability Zone

When using ec2uploadimg for uploading arm64 images to the us-east-1 region, there is a chance that the call will fail because an Availability zone without ARM instances will be picked for the VM (e.g. us-east-1f).

The issue arises only sometimes when using certain regions. We're using the us-east-1 region for arm64 testing, and the call sometimes fails, and sometimes it passes (likely depending on the internal choice of the availability zone).

Example:

ec2uploadimg --access-id <redacted> -s <redacted> --backing-store ssd --grub2 --machine 'arm64' -n 'openqa-SLES15-SP4-Micro-5-3-BYOS.aarch64-1.0.2-EC2-Build1.3.raw.xz' --virt-type hvm --sriov-support --ena-support --verbose --regions 'us-east-1' --ssh-key-pair <redacted> --private-key-file <redacted> -d 'OpenQA upload image' --wait-count 3 --ec2-ami <redacted> --type 'a1.large' --user 'ec2-user' 'SLES15-SP4-Micro-5-3-BYOS.aarch64-1.0.2-EC2-Build1.3.raw.xz'
...
Successfully allowed incoming SSH port 22 for security group sg-<redaced> in vpc-<redacted>

An error occurred (Unsupported) when calling the RunInstances operation: Your requested instance type (a1.large) is not supported in your requested Availability Zone (us-east-1f). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1b, us-east-1d.
Traceback (most recent call last):
  File "/root/.venv_ec2uploadimg/bin/ec2uploadimg", line 957, in upload_image_to_region
    ami = uploader.create_image(args.source)
  File "/root/.venv_ec2uploadimg/lib64/python3.10/site-packages/ec2imgutils/ec2uploadimg.py", line 1022, in create_image
    snapshot = self.create_snapshot(source)
  File "/root/.venv_ec2uploadimg/lib64/python3.10/site-packages/ec2imgutils/ec2uploadimg.py", line 1161, in create_snapshot
    root_volume = self._create_image_root_volume(source)
  File "/root/.venv_ec2uploadimg/lib64/python3.10/site-packages/ec2imgutils/ec2uploadimg.py", line 354, in _create_image_root_volume
    helper_instance = self._launch_helper_instance()
  File "/root/.venv_ec2uploadimg/lib64/python3.10/site-packages/ec2imgutils/ec2uploadimg.py", line 764, in _launch_helper_instance
    instance = self._connect().run_instances(
  File "/root/.venv_ec2uploadimg/lib64/python3.10/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/.venv_ec2uploadimg/lib64/python3.10/site-packages/botocore/client.py", line 911, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (Unsupported) when calling the RunInstances operation: Your requested instance type (a1.large) is not supported in your requested Availability Zone (us-east-1f). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1b, us-east-1d.
An error occurred (Unsupported) when calling the RunInstances operation: Your requested instance type (a1.large) is not supported in your requested Availability Zone (us-east-1f). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1b, us-east-1d.
Successfully deleted security group sg-<redacted>

See https://openqa.suse.de/tests/11897135#step/upload_image/86

ec2publishimg should wait for image avialability

When the image is pending it cannot be published. ec2publishimg generates a traceback

An error occurred (InvalidAMIID.Unavailable) when calling the ModifyImageAttribute operation: The AMI ID 'ami-0aef1f7c38d24d19c' is currently pending and may not be used for this operation
Traceback (most recent call last):
File "/usr/bin/ec2publishimg", line 254, in
publisher.publish_images()
File "/usr/lib/python3.6/site-packages/ec2imgutils/ec2publishimg.py", line 207, in publish_images
UserIds=self.visibility.split(',')
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAMIID.Unavailable) when calling the ModifyImageAttribute operation: The AMI ID 'ami-0aef1f7c38d24d19c' is currently pending and may not be used for this operation

Ratherthan catching the error an making the user re-submit the request ec2publishimg should wait for the image to become available.

ec2uploadimg 'local variable 'files' referenced before assignment'

First of all I want to say that it took me several hours to just figure out how to use the python script, because cli options are not correctly parsed. Maybe I will open up another issue for that.

Now I have the problem with the command execution on Version 7.0.2 or is it Version 7.04?

ec2uploadimg --version 7.0.2
~/git/ec2imgutils/lib/ec2imgutils> cat VERSION 7.0.4

ec2uploadimg --description 'XXX-KIWI-SLES-TEST' -m 'x86_64' -n 'XXX-kiwi-sles-test' --regions eu-central-1 SLES4SAP15.x86_64-0.1.0.raw -a servers --verbose --use-private-ip

This is works so far but is canceled while or after uploading the image..

Using VPC subnet: subnet-xxxx
Using Security Group IDs: sg-xxxx
Waiting for instance: i-xxxx
. .
Waiting for volume creation: vol-xxxx
.
Wait for volume attachment
.
Waiting for volume creation: vol-xxxx
.
Wait for volume attachment
.
Waiting to obtain instance IP address
Attempt ssh connection
/usr/lib/python3.6/site-packages/paramiko/client.py:822: UserWarning: Unknown ssh-ed25519 host key for 172.xx.0.xx: b'9a3c50b5df661bdd8e8bd0c677227ca1'
key.get_name(), hostname, hexlify(key.get_fingerprint())
Formating storage volume
Creating ext3 filesystem on storage volume
Uploading image file: SLES4SAP15.x86_64-0.1.0.raw
. . . . . . . . . .
local variable 'files' referenced before assignment

So where does this variable come from? It is not used in the script itself.

Deprecate image without replacement

An image may be deprecated without a replacement. This is not currently possible as replacement image is a required argument. Replacement image should instead be an optional argument.

Add waiter on image exists

It would be helpful to have a built in waiter method that blocks until a given image exists.