aws ec2-instance connect ssh fails on Windows

[fabiomoratti]

Describe the bug

On Windows, the following command

aws ec2-instance-connect ssh --os-user XXXX --instance-id i-XXXXXXXXX

fails because of the (temporary) key file permissions.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Connect to the remote instance.

Current Behavior

The complete error (with the user and instance id anonymized) is:

Bad permissions. Try removing permissions for user: \\OWNER RIGHTS (S-1-3-4) on file C:/Users/XXXX/AppData/Local/Temp/tmp3cja4v_s/private-key.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'C:\\Users\\XXXX\\AppData\\Local\\Temp\\tmp3cja4v_s\\private-key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "C:\\Users\\XXXX\\AppData\\Local\\Temp\\tmp3cja4v_s\\private-key": bad permissions

Note that, after the command fails, the key file (private-key) and its folder (C:\Users\XXXX\AppData\Local\Temp\tmp3cja4v_s) are missing, I presume they are created "on the fly" by the command and then removed, so there is no (easy) way to examine the file and its permission.

Reproduction Steps

• Create a VPC, with public and private subnets
• Create a EC2 Instance Connect Endpoint
• Create EC2 instance on the private subnet.
• Get the EC2 instance ID
• Issue the following command:

aws ec2-instance-connect ssh --os-user XXXX --instance-id i-XXXXXXXXX

Possible Solution

I suspect that the temporary private key created for the connection does not have the correct permissions.
The solution is to downgrade to a previous version of the CLI that does not present the regression.

Additional Information/Context

This appears to be a regression, version up to 2.17.0 work as expected, while versions 2.17.65, 2.18.0, 2.20.0 and 2.22.0 fail with the above error.
I did not check all the minor 2.17 versions, but apparently the regression appeared between 2.17.0 and 2.17.65.

CLI version used

aws-cli/2.17.65 Python/3.12.6 Windows/11 exe/AMD64

Environment details (OS name and version, etc.)

Windows 11 Pro, version 23H2 - OS Build 22631.4460

[RyanFitzSimmonsAK]

Hi @fabiomoratti, thanks for reaching out. I wasn't able to reproduce the behavior you've described on CLI version 2.17.35. Could you provide full debug logs? You can get debug logs by adding --debug to your command, and redacting any sensitive information. Thanks!

[fabiomoratti]

Hello @RyanFitzSimmonsAK I confirm that version 2.17.35 is working as expected so I tried all 2.17.x version to see where the bug emerged (that is 18 versions..., I hope the effort is appreciated...):

• versions form 2.17.35 up to 2.17.51 work as expected
• version 2.17.52 fails with the error reported above.

As requested find below the output of the aws ec2-instance-connect ssh --debug --os-user XXXX --instance-id i-XXXXXXXXX command with the --debug option turned on.
I removed or redacted all possibly sensitive information and homogenized dates and other request-specific data so you can easily diff the two files to see where the command fails.

My guess is that between version 2.17.51 and version 2.17.52 the code to generate the temp key has changed and somehow does not sets the correct permission of the newly created temp key file.

I also tried to find the code where the log "Generated temporary key file:" (line 53 in the attached file) is printed to inspect the coded there but I failed, maybe I was looking in the wrong place.

Thank you for the kind assistance.

ec2-instance-connect-out--2.17.55 (success).txt
ec2-instance-connect-out-2.17.51 (fail).txt

[Slackdow]

hi, i have the same behavior with the temp pem file that is too open, im using 2.22.14. if i downgrade to 2.17.35 its working well.

@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'C:\Users\thien\AppData\Local\Temp\tmpssmpv6wf\private-key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "C:\Users\thien\AppData\Local\Temp\tmpssmpv6wf\private-key": bad permissions
[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
PS C:\Users\thien\OneDrive\AWS\E6K> msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2-2.17.35.msi
PS C:\Users\thien\OneDrive\AWS\E6K> msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2-2.17.35.msi
PS C:\Users\thien\OneDrive\AWS\E6K> aws ec2-instance-connect ssh --instance-id i-064d8a39ad5c019ba
, #_
_ ####_ Amazon Linux 2023
~~ _#####
~~ ###|
~~ #/ ___ https://aws.amazon.com/linux/amazon-linux-2023
~~ V' '->
~~~ /
~~._. _/
_/ _/
_/m/'
Last login: Wed Dec 11 19:17:01 2024 from 10.0.137.130

[RyanFitzSimmonsAK]

Hey, thanks for following up. I was able to reproduce this behavior. While we look into this, you can specify your private key as a workaround. In my testing, using aws ec2-instance-connect ssh --instance-id i-xxx --private-key-file mykey.pem worked successfully, while omitting it failed.

[Slackdow]

indeed its what i use too.

[hssyoo]

Thanks for raising this issue with us. The root cause is that the generated key file inherits permissions from the directory created by Python's tempfile.TemporaryDirectory, which recently changed. I opened an issue with CPython to track this: python/cpython#128038. In the meantime, we recommend using the workarounds suggested by @RyanFitzSimmonsAK.