mludvig / aws-ssm-tools Goto Github PK
View Code? Open in Web Editor NEWHandy tools for AWS Systems Manager - ssm-session, ecs-session, ssm-ssh and ssm-tunnel
License: Other
Handy tools for AWS Systems Manager - ssm-session, ecs-session, ssm-ssh and ssm-tunnel
License: Other
This warning can be quite annoying if you have many instances not in inventory. It shows not just on a list but also ssh-ssm utils etc
$ ssm-session --list --profile admin-guideitprod
[ssm-session] WARNING: SSM inventory entity not recognised: {'Id': 'i-02a#############c4b', 'Data': {}}
....
I changed the warning to debug temporarily maybe you want to make that permanent. Or if better option let me know.
$ diff /usr/local/lib/python3.8/dist-packages/ssm_tools/resolver.py /tmp/resolver.py
39a40
>
61,62c62
< #logger.warning("SSM inventory entity not recognised: %s", entity)
< logger.debug("SSM inventory entity not recognised: %s", entity)
---
> logger.warning("SSM inventory entity not recognised: %s", entity)
Hi Michael! Thanks for some very cool tools
Unfortunately on OSX don't have the "ip" command.
Also since I'd be ok not having my local routing changed, and port-forward, connecting to local port like you get with e.g. aws ssm --document-name AWS-StartPortForwardingSession ....
I tested this out with socat + the stock AWS Document, and can do something like:
socat -v TCP4-LISTEN:9999,fork,reuseaddr TCP4:172.16.8.56:80
and am thinking, for my use case, I might take your code for ssm-tunnel, do the local forward as described, and replace the agent with socat.
I guess I could also look at making my own SSM Document just like the AWS one, but extend it to (install+) run socat on the remote ...
It's quite a change and perhaps not necessarily an improvement on the existing method, just interested if you had any thoughts or advice there.
Cheers
Nick
$ ssm-session --list
mi-abcdef1234567890a mycomputer.WORKGROUP 10.0.0.16
$ ssm-session mi-abcdef1234567890a
[ssm-session] WARNING: Could not resolve Instance ID for 'mi-abcdef1234567890a'
[ssm-session] WARNING: Perhaps the 'mi-abcdef1234567890a' is not registered in SSM?
$ aws ssm start-session --target mi-abcdef1234567890a
Starting session with SessionId: user-0724d37010e406ad6
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Windows\system32>
#
$ aws ssm get-inventory --query "Entities[?Id=='mi-abcdef1234567890a']"
[
{
"Id": "mi-abcdef1234567890a",
"Data": {
"AWS:InstanceInformation": {
"TypeName": "AWS:InstanceInformation",
"SchemaVersion": "1.0",
"CaptureTime": "2021-03-18T00:09:21Z",
"Content": [
{
"AgentType": "amazon-ssm-agent",
"AgentVersion": "3.0.655.0",
"ComputerName": "mycomputer.WORKGROUP",
"IamRole": "SSMServiceRole",
"InstanceId": "mi-abcdef1234567890a",
"IpAddress": "10.0.0.16",
"PlatformName": "Microsoft Windows 10 Pro",
"PlatformType": "Windows",
"PlatformVersion": "10.0.19042",
"ResourceType": "ManagedInstance"
}
]
}
}
}
]
$ aws --version
aws-cli/2.1.30 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off
$ /usr/local/sessionmanagerplugin/bin/session-manager-plugin --version
1.2.54.0
$ ssm-session --version
ssm-tools/1.3.2
Looks like the regex for checking instance ids doesn't handle "mi-xxxx"
if re.match('^i-[a-f0-9]+$', instance):
should be
if re.match('^m?i-[a-f0-9]+$', instance):
made the change and tested it, have a PR in a sec.
I wanted to use the new ecs-session to login to our app container but can't figure out how. Our app (kicker) is deployed with a nginx sidecar conainer in each instance.
~> ecs-session --list
dev-Cluster-mWkJ7Y4 service:kicker-service-09C2qGCVy 11f372449e0eda484b486e9b94cc9f74 kicker 172.31.21.13
dev-Cluster-mWkJ7Y4 service:kicker-service-09C2qGCVy 4b5f60b18d6cb83f9f4e7e8e30edf99e kicker 172.31.39.74
dev-Cluster-mWkJ7Y4 service:kicker-service-09C2qGCVy 11f372449e0eda484b486e9b94cc9f74 nginx 172.31.21.13
dev-Cluster-mWkJ7Y4 service:kicker-service-09C2qGCVy 4b5f60b18d6cb83f9f4e7e8e30edf99e nginx 172.31.39.74
Now ecs-session won't let me specify both the container and the task id and without that it always finds more than one container and errors.
~> ecs-session kicker
Found 4 instances for: kicker
Use Container IP or Task ID to connect to a specific one
dev-Cluster-mWkJ7Y4 service:kicker-service-09C2qGCVy 11f372449e0eda484b486e9b94cc9f74 kicker 172.31.21.13
dev-Cluster-mWkJ7Y4 service:kicker-service-09C2qGCVy 4b5f60b18d6cb83f9f4e7e8e30edf99e kicker 172.31.39.74
Same when I try the task id or the ip. How can I login to one of them?
BTW Why does it say "Found 4 instances" when there are only 2?
I'd like to contribute a change to the code-base, but since it affects the default behaviour of the tool I seek some implementation/design guidance.
The desired behaviour is such that pressing Ctrl+Z on an active SSM session should be handled by the server side, and not the client side.
(local) ➜ ~ ssm-session i-abcdefgh
Starting session with SessionId: todor-12345678efgh
(remote) $ sleep 60
[2] + 26132 suspended ssm-session i-abcdefgh
(local) ➜ ~
➜ ~ ssm-session i-abcdefgh
Starting session with SessionId: todor-12345678efgh
(remote) $ sleep 60
^Z[1] + Stopped sleep 60
(remote) $
For my personal use-case the below is sufficient, but it affects the default behaviour of the tool. Is such a change acceptable? If not - where is a better place to implement it?
diff --git a/ssm-session b/ssm-session
index f659c42..172484a 100755
--- a/ssm-session
+++ b/ssm-session
@@ -3,5 +3,8 @@
import sys
from ssm_tools.ssm_session_cli import main
+import signal
+signal.signal(signal.SIGTSTP, signal.SIG_IGN)
+
if __name__ == "__main__":
sys.exit(main())
When attempting to connect to an SSM managed instance with ssm-ssh command on Windows 10 using latest OpenSSH build, this error is received from ssm-ssh
SSM inventory entity not recognised: {'Id': '<Instance ID>', 'Data': {}}
CreateProcessW failed error:2
posix_spawnp: No such file or directory
Possible fix is per microsoft/vscode-remote-release#18 (comment), this should be resolved by running ssh.exe
on Windows instead of ssh
.
Hi, thanks for this great package!
The docs could explain authentication a little bit more. Namely, it wasn't clear to me that the password I am asked for by ssm-tunnel
is my passowd on my local machine (though I am sure it is obvious to many). It would be nice to mention that explicitly. Thank you!
Hello,
We figure out that is an instance rans without tags associated to it, it generates the following error:
Traceback (most recent call last):
File "/home/jynolen/.local/bin/ssm-session", line 8, in <module>
sys.exit(main())
File "/home/jynolen/.local/lib/python3.8/site-packages/ssm_tools/ssm_session_cli.py", line 105, in main
InstanceResolver(args).print_list()
File "/home/jynolen/.local/lib/python3.8/site-packages/ssm_tools/resolver.py", line 122, in print_list
items = self.get_list().values()
File "/home/jynolen/.local/lib/python3.8/site-packages/ssm_tools/resolver.py", line 93, in get_list
for tag in instance['Tags']:
KeyError: 'Tags'
Please help me know if anything i need to do to fix this.
C:\Users\Administrator>ec2-session --list --region ap-south-1 --profile vtest
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Scripts\ec2-session.exe\__main__.py", line 4, in <module>
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\ssm_tools\ssm_session_cli.py", line 30, in <module>
signal.signal(signal.SIGTSTP, signal.SIG_IGN) # Ignore Ctrl-Z - pass it on to the shell
^^^^^^^^^^^^^^
AttributeError: module 'signal' has no attribute 'SIGTSTP'
aws ssm start-session
supports a --reason
option that is used for documentation and change control since it's part of the API call and written to Cloudtrail.
ec2-session
doesn't support this option. It would be very helpful to support this.
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ssm/start-session.html
when I do:
ssh i-xyz
I get this error:
Pseudo-terminal will not be allocated because stdin is not a terminal.
-bash: line 1: $'SSH-2.0-OpenSSH_8.1\r': command not found
I use this ssh config:
host i-* mi-*
ProxyCommand ssm-ssh --profile prd --region eu-west-1 remoteuser@%h
Hi,
Do you know why running something like this in Ubuntu 20.04 (python 3.8) always fails with timeout?
Thanks
ivan@anecua:~$ ssm-tunnel i-7171717171717171 --profile my-profile --debug
[ssm-tunnel] DEBUG: Logging level set to DEBUG
[ssm-tunnel] DEBUG: Spawning: aws --profile my-profile ssm start-session --target i-7171717171717171
[ssm-tunnel] DEBUG: PID: 288300
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 111, in expect_loop
incoming = spawn.read_nonblocking(spawn.maxread, timeout)
File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 482, in read_nonblocking
raise TIMEOUT('Timeout exceeded.')
pexpect.exceptions.TIMEOUT: Timeout exceeded.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/ssm-tunnel", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/ssm_tools/ssm_tunnel_cli.py", line 345, in main
tunnel = SsmTunnel(instance_id, profile=args.profile, region=args.region, logger_name=logger_name)
File "/usr/local/lib/python3.8/dist-packages/ssm_tools/ssm_tunnel_cli.py", line 83, in __init__
super().__init__(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/ssm_tools/talker.py", line 9, in __init__
self.connect(instance_id, profile, region)
File "/usr/local/lib/python3.8/dist-packages/ssm_tools/talker.py", line 23, in connect
self.wait_for_prompt()
File "/usr/local/lib/python3.8/dist-packages/ssm_tools/talker.py", line 49, in wait_for_prompt
self._child.expect('sh.*\$ $')
File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 340, in expect
return self.expect_list(compiled_pattern_list,
File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 369, in expect_list
return exp.expect_loop(timeout)
File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 119, in expect_loop
return self.timeout(e)
File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x7f81cf756100>
command: /usr/local/bin/aws
args: [b'/usr/local/bin/aws', b'--profile', b'my-profile', b'ssm', b'start-session', b'--target', b'i-7171717171717171']
buffer (last 100 chars): '\r\nStarting session with SessionId: my-user@company-097fbec8ab0500248\r\n$ '
before (last 100 chars): '\r\nStarting session with SessionId: my-user@company-097fbec8ab0500248\r\n$ '
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 288300
child_fd: 5
closed: False
timeout: 10
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
0: re.compile('sh.*\\$ $')
I have a few Raspberry Pis that are registered as managed instances mi-*
in Fleet Manager. Connecting to them works fine with a command like ec2-session pi-1
:
$ ec2-session pi-1
Starting session with SessionId: strophy-0f0424c62fea5e575
$
Exiting session with sessionId: strophy-0f0424c62fea5e575.
But ec2-ssh
fails with the following behaviour:
$ ec2-ssh pi-1
INFO: Resolved instance name 'pi-1' to 'mi-048afb7f6460a110d'
ERROR: Unable to figure out the EC2 login name. Use "-l {user}" or {user}@{instance}.
$ ec2-ssh pi@pi-1
INFO: Resolved instance name 'pi-1' to 'mi-048afb7f6460a110d'
INFO: Using SSH key from SSH Agent, should be as good as any.
ERROR: An error occurred (InvalidArgsException) when calling the SendSSHPublicKey operation: 1 validation error detected: Value 'mi-048afb7f6460a110d' at 'instanceId' failed to satisfy constraint: Member must satisfy regular expression pattern: ^i-[a-f0-9]+$
Is this a restriction imposed by aws-ssm-tools
or the underlying ssm start-session
command? Are there any workarounds?
This utility is great, but ssm-copy is not working for me:
➜ ~ ssm-copy test i-05bxxxxxxxxxxx:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pexpect/expect.py", line 111, in expect_loop
incoming = spawn.read_nonblocking(spawn.maxread, timeout)
File "/usr/local/lib/python3.7/site-packages/pexpect/pty_spawn.py", line 482, in read_nonblocking
raise TIMEOUT('Timeout exceeded.')
pexpect.exceptions.TIMEOUT: Timeout exceeded.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/ssm-copy", line 204, in <module>
main()
File "/usr/local/bin/ssm-copy", line 186, in main
sender = SsmFileSender(instance_id, profile = args.profile, region = args.region, logger_name = logger_name)
File "/usr/local/lib/python3.7/site-packages/ssm_tools/talker.py", line 9, in __init__
self.connect(instance_id, profile, region)
File "/usr/local/lib/python3.7/site-packages/ssm_tools/talker.py", line 23, in connect
self.wait_for_prompt()
File "/usr/local/lib/python3.7/site-packages/ssm_tools/talker.py", line 49, in wait_for_prompt
self._child.expect('sh.*\$ $')
File "/usr/local/lib/python3.7/site-packages/pexpect/spawnbase.py", line 341, in expect
timeout, searchwindowsize, async_)
File "/usr/local/lib/python3.7/site-packages/pexpect/spawnbase.py", line 369, in expect_list
return exp.expect_loop(timeout)
File "/usr/local/lib/python3.7/site-packages/pexpect/expect.py", line 119, in expect_loop
return self.timeout(e)
File "/usr/local/lib/python3.7/site-packages/pexpect/expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x10952a390>
command: /usr/local/bin/aws
args: [b'/usr/local/bin/aws', b'ssm', b'start-session', b'--target', b'i-05b7c0ffb1bc9664a']
buffer (last 100 chars): '\r\nStarting session with SessionId: 1569835485940510000-0723a5e801a6f63f0\r\n$ '
before (last 100 chars): '\r\nStarting session with SessionId: 1569835485940510000-0723a5e801a6f63f0\r\n$ '
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 35112
child_fd: 7
closed: False
timeout: 10
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
0: re.compile('sh.*\\$ $')
ssm-session to the same instance works fine.
The instance is Debian but I'm on Mac OS locally. Installed aws-ssm-tools using pip3. My python installation is:
#!/usr/local/opt/python/bin/python3.7
aws-cli and session-manager-plugin versions:
aws --version
aws-cli/1.16.248 Python/3.7.3 Darwin/18.7.0 botocore/1.12.238
session-manager-plugin --version
1.1.33.0
Thanks
It would be awesome to be able to run ssm-tools in a docker image for us on OSX and those that prefer to keep tools off their system. This is what I tried, and it runs ssm-session
:
FROM ubuntu:bionic
RUN apt-get update -qq && \
apt-get install -y python3-pip
RUN pip3 install awscli --upgrade
# See https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-debian
ADD https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb session-manager-plugin.deb
RUN dpkg -i session-manager-plugin.deb
RUN pip3 install aws-ssm-tools
# ssm-tunnel prerequisities:
RUN apt-get update -qq && \
apt-get install -y sudo iproute2
ENTRYPOINT ["/bin/bash"]
however ssm-tunnel <instance>
fails without any clear indication of why:
root@eb4429e01043:/# ssm-tunnel --verbose --debug i-xxxxxx
[ssm-tunnel] DEBUG: Logging level set to DEBUG
[ssm-tunnel] DEBUG: Spawning: aws ssm start-session --target i-xxxxxx
[ssm-tunnel] DEBUG: PID: 241
[ssm-tunnel] DEBUG: Starting session with SessionId: john.snow@43213109573-019fead4f60ee2455
[ssm-tunnel] DEBUG: Creating tunnel
[ssm-tunnel] DEBUG: # Agent device tunSSM.122.45 is ready
[ssm-tunnel] DEBUG: command: sudo ip tuntap add tunSSM.122.44 mode tun user 0
open: No such file or directory
[ssm-tunnel] DEBUG: command: sudo ip link set tunSSM.122.44 down
Cannot find device "tunSSM.122.44"
[ssm-tunnel] DEBUG: command: sudo ip tuntap del tunSSM.122.44 mode tun
open: No such file or directory
[ssm-tunnel] INFO: Closing tunnel, please wait...
[ssm-tunnel] DEBUG: Closing session
Is it possible to set an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment instead of using a profile? I assume it should work since you're just invoking the AWS CLI under the hood, but all the documentation seems to expect that you declare a profile name to provide credentials.
Firstly, I have a pull request coming, I just wanted to create this issues for reference and justification of the pull request.
Steps to reproduce
Expected:
Brief pause while entity name is resolved, then imediately connects.
Actual:
1000's of warnings like the following before evntually connecting
[ssm-session] WARNING: SSM inventory entity not recognised: {'Id': 'i-***************', 'Data': {'AWS:InstanceInformation': {'TypeName': 'AWS:InstanceInformation', 'SchemaVersion': '1.0', 'CaptureTime': '2022-09-09T04:38:50Z', 'Content': [{'InstanceId': 'i-*************', 'InstanceStatus': 'Terminated'}]}}}
Troubleshooting:
I pulled out some of the code and started reproducing. The main thing I found was that it was choking on a KeyError
trying to check the ResourceType
attribute. A large number of terminated instances don't have a ResourceType
attribute when returned from the API.
For example, the following entity only has InstanceStatus
and InstanceId
attributes
[ssm-session] WARNING: SSM inventory entity not recognised: {'Id': 'i-*************', 'Data': {'AWS:InstanceInformation': {'TypeName': 'AWS:InstanceInformation', 'SchemaVersion': '1.0', 'CaptureTime': '2022-09-09T07:59:39Z', 'Content': [{'InstanceId': 'i-*************', 'InstanceStatus': 'Terminated'}]}}}
We don't care about terminated, stopped or disconnected hosts anyway so not much point logging/warning about them.
Recommendations
Let's utilize AWS's built in filtering so that we only pull down a subset of records that match our criteria.
Currently in my situation we are pulling down 8800 records and then filtering them downs to just 26, and then resolving the 1 instance.
Again, pull request incoming.
ec2-session instance --region us-east-1 -c "\"sudo -u ec2-user -i /bin/bash -c 'touch 1.txt'\""
Starting session with SessionId: myuser-0001eda76e4e3ee7a
and it never returns or exit. Ctrl+C also wont respond
On MacOS monterrey
When running ssm-tunnel
from my MacBook I get the following error:
$ ssm-tunnel -v i-123456789 -r 10.0.0.0/8
sudo: ip: command not found
sudo: ip: command not found
sudo: ip: command not found
[ssm-tunnel] INFO: Closing tunnel, please wait...
Is this supposed to work on macOS? The agent is installed on the target Amzn2 instance.
ssm-session --list --profile SBX-POC
Traceback (most recent call last):
File "/usr/local/bin/ssm-session", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/ssm_tools/ssm_session_cli.py", line 81, in main
InstanceResolver(args).print_list()
File "/usr/local/lib/python3.6/dist-packages/ssm_tools/resolver.py", line 88, in print_list
items = self.get_list().values()
File "/usr/local/lib/python3.6/dist-packages/ssm_tools/resolver.py", line 42, in get_list
assert content["ResourceType"] == "EC2Instance"
SSH can accept multiple arguments for a command. rsync
relies on this behaviour when you use a command as the remote shell (-e
). It should be trivial to use ssm-ssh
for that with rsync -e "ssm-ssh --region blah --profile blah
, but instead it fails with No instance-id found for destination .
.
With some debugging I can see the argument received by ssm-ssh
are:
['/home/dqm/.local/bin/ssm-ssh', '--region', 'eu-west-1', '--', '-l', 'root', 'i-0e5169d18eee8506f', 'sudo rsync', '--server', '--sender', '-e.LsfxCIvu', '.', 'my-file']
The failure is caused by ssm-ssh
erroneously assuming there is exactly one command
argument, and hence the destination must be the second-to-last argument. This is not a valid assumption.
Changing the whole argument parsing is probably not a viable option, but my problem can be solved if a --
argument is respected and any further arguments are passed directly to SSH - the destination would then be guaranteed to be the only freestanding argument before the --
.
When attempting to list available instances by running "ssm-session --list" I receive two KeyErrors intermittently on InstanceName and HostName:
Traceback (most recent call last):
File "/home/srepetski/.local/bin/ssm-session", line 8, in
sys.exit(main())
File "/home/srepetski/.local/lib/python3.6/site-packages/ssm_tools/ssm_session_cli.py", line 81, in main
InstanceResolver(args).print_list()
File "/home/srepetski/.local/lib/python3.6/site-packages/ssm_tools/resolver.py", line 107, in print_list
print(f"** Bad InstanceName is {item['InstanceName']}")
KeyError: 'InstanceName'
When I add some debugging to resolver.py to print the list item when there there are less than 4 elements, I can see that I am getting at least one item with just two elements (InstanceId and HostName), and InstanceName and Address are missing. I'm not sure if the issue is in the data SSM is returning or how the tool is parsing,
srepetski@:~$ ssm-session --list
{'InstanceId': 'i-05a6746e9bf2eeaea', 'HostName': 'uc1bplis536trau'}
I would like to use this in my organization but I am not able to as there is no licence file included which details the terms of use. Something along the lines of: https://www.apache.org/licenses/LICENSE-2.0#apply or whatever licence that you want to use.
There appears to be some latency between when an instance is terminated and when it's removed from the SSM inventory. As a result, in accounts with a large number of instances and high churn (I have ~1,000 instances in a single region, and have hundreds of them are replaced every day), it's likely that DescribeInstances will fail on one or more instances that are still in the SSM Inventory. In these cases, aws-ssm-tools' method of calling DescribeInstances with the full list of instance IDs in the SSM Inventory will render it useless as it will encounter an error.
The following is a trimmed-down excerpt of the ssm-session
debug output from an account that has 724 instances in the SSM Inventory for this region. 649 of those were properly recognized by SSM Inventory as terminated and skipped (DEBUG: Ignoring terminated instance:
), but it looks like 4 of them were terminated within a short enough period that SSM doesn't know they're gone, but DescribeInstances errors:
[ssm-session] DEBUG: Logging level set to DEBUG
[ssm-session] DEBUG: Fetching SSM inventory
...
[ssm-session] DEBUG: Added instance: i-02df27b30a98a2bf6: {'InstanceId': 'i-02df27b30a98a2bf6', 'HostName': 'ip-10-230-8-231'}
...
[ssm-session] DEBUG: Added instance: i-06a5cd08dccd6cf94: {'InstanceId': 'i-06a5cd08dccd6cf94', 'HostName': 'ip-10-230-8-70'}
...
[ssm-session] DEBUG: Added instance: i-093177e4964fd4be3: {'InstanceId': 'i-093177e4964fd4be3', 'HostName': 'ip-10-142-0-108'}
...
[ssm-session] DEBUG: Added instance: i-0f4d372b4cba55a1a: {'InstanceId': 'i-0f4d372b4cba55a1a', 'HostName': 'ip-10-230-12-196'}
...
[ssm-session] ERROR: An error occurred (InvalidInstanceID.NotFound) when calling the DescribeInstances operation: The instance IDs 'i-02df27b30a98a2bf6, i-06a5cd08dccd6cf94, i-093177e4964fd4be3, i-0f4d372b4cba55a1a' do not exist
I can think of a few possible fixes for this:
i-
string), we short-circuit InstanceResolver.resolve_instance() and directly return that Instance ID. The SSM Inventory data includes both Instance IDs and HostNames, but we make the full DescribeInstances call any time our argument isn't an Instance ID. In InstanceResolver.get_list() we could check for hostname matches, and if there are any, return those without needing to call DescribeInstances to get the IP addresses. This would both fix this specific bug in all cases except specifying an IP address, and would also avoid potentially-slow DescribeInstances calls in cases where we don't need them.EC2.Instance(instance_id)
resource for each instance. This would greatly increase the number of DescribeInstances API calls, and likely adversely effect performance, but would allow us to try/except on each instance ID itself.InvalidInstanceID.NotFound
error, parse out the Instance IDs from the error message, remove them from the list of instances, and retry the DescribeInstances operation with the updated list of instance IDs.I'd be happy to open a pull request to implement any of these options, or a combination of them, but I'll leave the choice of which up to you. Since this should probably be an edge case, I'd personally lean towards a combination of options 1 and 3.
(BTW - This is an absolutely wonderful project! Thank you so much! It's completely revolutionized how I work with EC2 instances when I need access to an actual instance, and I'm working hard to let everyone else at my company know about it.)
Hello :-)
Thanks for providing these tools, very useful!
I have a feature suggestion:
Current way to run a custom command:
ssm-session my-instance --document-name AWS-StartInteractiveCommand --parameters 'command=["echo 42"]'
I would suggest a new option --command
to simplify this use case:
ssm-session my-instance --command "echo 42"
The new option would be incompatible with --document-name
and --parameters
.
The --user
option could be made compatible with --command
(just concatenate sudo -i -u USER
with COMMAND
).
What do you think about this?
If you find it useful, I could try submitting a PR.
This feature would be even more useful once this AWS CLI / session-manager-plugin issue is fixed: aws/amazon-ssm-agent#358
I encountered some problems to use ecs-session in one of my account.
After adding --debug
arg, it appeared that one of my ECS cluster has no tasks (seems not so useful, but why not).
After digging a little in the code, the issue happens here :
It fails when using the paginator list_tasks
that returns an empty list of tasks.
Error thrown is
[ssm-tools.ecs-session] ERROR: An error occurred (InvalidParameterException) when calling the DescribeTasks operation: Tasks cannot be empty.
It would be better to first check if the paginator is empty or not, before going further.
However, I'm not sure what is the best way to check :
can_paginate
KeyCount
key on the first page (as mentionned hereI've noticed that if there is no tag called Name on an instance this error is thrown:
Traceback (most recent call last):
File "/usr/local/bin/ssm-session", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/ssm_tools/ssm_session_cli.py", line 81, in main
InstanceResolver(args).print_list()
File "/usr/local/lib/python3.7/site-packages/ssm_tools/resolver.py", line 103, in print_list
print(f"{item['InstanceId']} {item['HostName']:{hostname_len}} {item['InstanceName']:{instname_len}} {' '.join(item['Addresses'])}")
ValueError: '=' alignment not allowed in string format specifier
Might be helpful to have a more informative error message.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.