Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InternalFailure when calling the CreateCustomDbEngineVersion (RDS Custom SQL) #9059

Closed
1 task
GeorgeWilsonPGATour opened this issue Nov 7, 2024 · 3 comments
Assignees
Labels
bug This issue is a bug. p2 This is a standard priority issue rds service-api This issue is due to a problem in a service API, not the SDK implementation.

Comments

@GeorgeWilsonPGATour
Copy link

Describe the bug

Error using AWS CLI (and console) when creating a CEV for RDS Custom for SQL Server. Liscense: SQL Server Developer (BYOM)

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

Create a new CEV

Current Behavior

Failure: An error occurred (InternalFailure) when calling the CreateCustomDBEngineVersion operation (reached max retries: 2): An internal error has occurred. Please try your query again at a later time.

Sanitized Debug Log:
--debug
2024-11-07 11:27:33,070 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.2 Python/3.11.8 Windows/10 exe/AMD64
2024-11-07 11:27:33,070 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['rds', 'create-custom-db-engine-version', '--engine', 'custom-sqlserver-ee', '--engine-version', '15.00.4365.2.dev_cev', '--image-id', '<ami_id>, '--description', 'Custom SQL Server EE 15.00.4365.2 CEV Developer', '--profile', 'SLDEV_SSO', '--debug']
2024-11-07 11:27:33,386 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x0000016B99B787C0>
2024-11-07 11:27:33,387 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x0000016B999B0040>
2024-11-07 11:27:33,387 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-11-07 11:27:33,387 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x0000016B99921C60>
2024-11-07 11:27:33,388 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x0000016B999232E0>
2024-11-07 11:27:33,388 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x0000016B99B7B1A0>
2024-11-07 11:27:33,389 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x0000016B999EEA20>
2024-11-07 11:27:33,389 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-11-07 11:27:33,389 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x0000016B99B7B060>
2024-11-07 11:27:33,389 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x0000016B99C67990>>
2024-11-07 11:27:33,390 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\data\cli.json
2024-11-07 11:27:33,395 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x0000016B99AC4540>
2024-11-07 11:27:33,395 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x0000016B99AC4860>
2024-11-07 11:27:33,395 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x0000016B99AC47C0>
2024-11-07 11:27:33,397 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x0000016B99AC49A0>
2024-11-07 11:27:33,397 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x0000016B99AC4900>
2024-11-07 11:27:33,397 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x0000016B99C64D40>
2024-11-07 11:27:33,397 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'SLDEV_SSO'
2024-11-07 11:27:33,400 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.2 Python/3.11.8 Windows/10 exe/AMD64
2024-11-07 11:27:33,400 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['rds', 'create-custom-db-engine-version', '--engine', 'custom-sqlserver-ee', '--engine-version', '15.00.4365.2.dev_cev', '--image-id', '<ami_id>, '--description', 'Custom SQL Server EE 15.00.4365.2 CEV Developer', '--profile', 'SLDEV_SSO', '--debug']
2024-11-07 11:27:33,401 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x0000016B99B79080>
2024-11-07 11:27:33,401 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x0000016B99774720>
2024-11-07 11:27:33,401 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x0000016B99C1BC40>
2024-11-07 11:27:33,401 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x0000016B9962C720>
2024-11-07 11:27:33,401 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x0000016B99794040>
2024-11-07 11:27:33,404 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-11-07 11:27:33,406 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2024-11-07 11:27:33,406 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x0000016B999DB2E0>
2024-11-07 11:27:33,408 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x0000016B999A02C0>
2024-11-07 11:27:33,673 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\rds\2014-10-31\service-2.json
2024-11-07 11:27:33,695 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\rds\2014-10-31\service-2.sdk-extras.json
2024-11-07 11:27:33,706 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds: calling handler <function _building_command_table at 0x0000016B99AC7920>
2024-11-07 11:27:33,706 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds: calling handler <function _add_generate_db_auth_token at 0x0000016B99AC7740>
2024-11-07 11:27:33,706 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds: calling handler <function add_waiters at 0x0000016B99B7B060>
2024-11-07 11:27:33,913 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\rds\2014-10-31\waiters-2.json
2024-11-07 11:27:33,914 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x0000016B99C67990>>
2024-11-07 11:27:33,916 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('engine', <awscli.arguments.CLIArgument object at 0x0000016B9A22ED10>), ('engine-version', <awscli.arguments.CLIArgument object at 0x0000016B9A22EE50>), ('database-installation-files-s3-bucket-name', <awscli.arguments.CLIArgument object at 0x0000016B9A22EF10>), ('database-installation-files-s3-prefix', <awscli.arguments.CLIArgument object at 0x0000016B9A22EF90>), ('image-id', <awscli.arguments.CLIArgument object at 0x0000016B9A22F1D0>), ('kms-key-id', <awscli.arguments.CLIArgument object at 0x0000016B9A22F310>), ('description', <awscli.arguments.CLIArgument object at 0x0000016B9A22F490>), ('manifest', <awscli.arguments.CLIArgument object at 0x0000016B9A22F5D0>), ('tags', <awscli.arguments.ListArgument object at 0x0000016B9A22F710>), ('source-custom-db-engine-version-identifier', <awscli.arguments.CLIArgument object at 0x0000016B9A22F7D0>), ('use-aws-provided-latest-image', <awscli.arguments.BooleanArgument object at 0x0000016B9A22F850>), ('no-use-aws-provided-latest-image', <awscli.arguments.BooleanArgument object at 0x0000016B98EF45D0>)])
2024-11-07 11:27:33,916 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.create-custom-db-engine-version: calling handler <function add_streaming_output_arg at 0x0000016B99B79580>
2024-11-07 11:27:33,917 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.create-custom-db-engine-version: calling handler <function add_cli_input_json at 0x0000016B997949A0>
2024-11-07 11:27:33,917 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.create-custom-db-engine-version: calling handler <function add_cli_input_yaml at 0x0000016B99794A40>
2024-11-07 11:27:33,917 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.create-custom-db-engine-version: calling handler <function unify_paging_params at 0x0000016B999B0680>
2024-11-07 11:27:34,277 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\rds\2014-10-31\paginators-1.json
2024-11-07 11:27:34,282 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\rds\2014-10-31\paginators-1.sdk-extras.json
2024-11-07 11:27:34,283 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.create-custom-db-engine-version: calling handler <function add_generate_skeleton at 0x0000016B99AA2D40>
2024-11-07 11:27:34,283 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.rds.create-custom-db-engine-version: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x0000016B9A22FB10>>
2024-11-07 11:27:34,284 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.rds.create-custom-db-engine-version: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x0000016B9A219DD0>>
2024-11-07 11:27:34,284 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.rds.create-custom-db-engine-version: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x0000016B9A23FAD0>>
2024-11-07 11:27:34,284 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds_create-custom-db-engine-version: calling handler <function add_waiters at 0x0000016B99B7B060>
2024-11-07 11:27:34,284 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds_create-custom-db-engine-version: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x0000016B99C67990>>
2024-11-07 11:27:34,287 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.engine: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,288 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.rds.create-custom-db-engine-version: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000016B996D0550>
2024-11-07 11:27:34,288 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'custom-sqlserver-ee' for parameter "engine": 'custom-sqlserver-ee'
2024-11-07 11:27:34,288 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.engine-version: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,290 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.rds.create-custom-db-engine-version: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000016B996D0550>
2024-11-07 11:27:34,290 - MainThread - awscli.arguments - DEBUG - Unpacked value of '15.00.4365.2.dev_cev' for parameter "engine_version": '15.00.4365.2.dev_cev'
2024-11-07 11:27:34,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.database-installation-files-s3-bucket-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.database-installation-files-s3-prefix: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.image-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,291 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.rds.create-custom-db-engine-version: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000016B996D0550>
2024-11-07 11:27:34,291 - MainThread - awscli.arguments - DEBUG - Unpacked value of '<ami_id> for parameter "image_id": '<ami_id>
2024-11-07 11:27:34,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.kms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.description: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,292 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.rds.create-custom-db-engine-version: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000016B996D0550>
2024-11-07 11:27:34,292 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'Custom SQL Server EE 15.00.4365.2 CEV Developer' for parameter "description": 'Custom SQL Server EE 15.00.4365.2 CEV Developer'
2024-11-07 11:27:34,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.manifest: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.tags: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.source-custom-db-engine-version-identifier: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,294 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.use-aws-provided-latest-image: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,294 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,294 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,294 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.create-custom-db-engine-version.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x0000016B99C9FA50>
2024-11-07 11:27:34,295 - MainThread - botocore.hooks - DEBUG - Event calling-command.rds.create-custom-db-engine-version: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x0000016B9A22FB10>>
2024-11-07 11:27:34,295 - MainThread - botocore.hooks - DEBUG - Event calling-command.rds.create-custom-db-engine-version: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x0000016B9A219DD0>>
2024-11-07 11:27:34,295 - MainThread - botocore.hooks - DEBUG - Event calling-command.rds.create-custom-db-engine-version: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x0000016B9A23FAD0>>
2024-11-07 11:27:34,296 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-11-07 11:27:34,296 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-11-07 11:27:34,297 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-11-07 11:27:34,302 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\endpoints.json
2024-11-07 11:27:34,322 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x0000016B984F09A0>
2024-11-07 11:27:34,543 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\rds\2014-10-31\endpoint-rule-set-1.json
2024-11-07 11:27:34,548 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\partitions.json
2024-11-07 11:27:34,552 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.rds: calling handler <function add_generate_db_auth_token at 0x0000016B98349260>
2024-11-07 11:27:34,553 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.rds: calling handler <function add_generate_presigned_url at 0x0000016B98439620>
2024-11-07 11:27:34,553 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for rds via: environment_service
2024-11-07 11:27:34,554 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for rds via: environment_global
2024-11-07 11:27:34,554 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for rds via: config_service
2024-11-07 11:27:34,555 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for rds via: config_global
2024-11-07 11:27:34,555 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-11-07 11:27:34,560 - MainThread - botocore.endpoint - DEBUG - Setting rds timeout as (60, 60)
2024-11-07 11:27:34,563 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.rds.CreateCustomDBEngineVersion: calling handler <function base64_decode_input_blobs at 0x0000016B99C1BCE0>
2024-11-07 11:27:34,564 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.rds.CreateCustomDBEngineVersion: calling handler <function generate_idempotent_uuid at 0x0000016B984F2D40>
2024-11-07 11:27:34,564 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False}
2024-11-07 11:27:34,565 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://rds.us-east-1.amazonaws.com
2024-11-07 11:27:34,565 - MainThread - botocore.hooks - DEBUG - Event before-call.rds.CreateCustomDBEngineVersion: calling handler <function inject_api_version_header_if_needed at 0x0000016B98510860>
2024-11-07 11:27:34,565 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateCustomDBEngineVersion) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.17.2 md/awscrt#0.20.11 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.8 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#rds.create-custom-db-engine-version'}, 'body': {'Action': 'CreateCustomDBEngineVersion', 'Version': '2014-10-31', 'Engine': 'custom-sqlserver-ee', 'EngineVersion': '15.00.4365.2.dev_cev', 'ImageId': '<ami_id>, 'Description': 'Custom SQL Server EE 15.00.4365.2 CEV Developer'}, 'url': 'https://rds.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x0000016B9A21EF10>, 'has_streaming_input': False, 'auth_type': None}}
2024-11-07 11:27:34,567 - MainThread - botocore.hooks - DEBUG - Event request-created.rds.CreateCustomDBEngineVersion: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000016B9B524D50>>
2024-11-07 11:27:34,567 - MainThread - botocore.hooks - DEBUG - Event choose-signer.rds.CreateCustomDBEngineVersion: calling handler <function set_operation_specific_signer at 0x0000016B984F2C00>
2024-11-07 11:27:34,569 - MainThread - botocore.credentials - DEBUG - Credentials for role retrieved from cache.
2024-11-07 11:27:34,569 - MainThread - botocore.credentials - DEBUG - Retrieved credentials will expire at: 2024-11-08 06:25:34+00:00
2024-11-07 11:27:34,570 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-11-07 11:27:34,570 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:rds.us-east-1.amazonaws.com
x-amz-date:20241107T182734Z
x-amz-security-token:

content-type;host;x-amz-date;x-amz-security-token

2024-11-07 11:27:34,570 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241107T182734Z
20241107/us-east-1/rds/aws4_request

2024-11-07 11:27:34,571 - MainThread - botocore.auth - DEBUG - Signature:

2024-11-07 11:27:34,571 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://rds.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.17.2 md/awscrt#0.20.11 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.8 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#rds.create-custom-db-engine-version', 'X-Amz-Date': b'20241107T182734Z', 'X-Amz-Security-Token': b'', 'Authorization': b'AWS4-HMAC-SHA256 Credential=/20241107/us-east-1/rds/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=', 'Content-Length': '205'}>
2024-11-07 11:27:34,573 - MainThread - botocore.httpsession - DEBUG - Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem
2024-11-07 11:27:34,574 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): rds.us-east-1.amazonaws.com:443
2024-11-07 11:27:36,018 - MainThread - urllib3.connectionpool - DEBUG - https://rds.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 500 298
2024-11-07 11:27:36,020 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '3dd75dc7-1b7e-4298-a11d-b4b1b19fd989', 'Strict-Transport-Security': 'max-age=31536000', 'Content-Type': 'text/xml', 'Content-Length': '298', 'Date': 'Thu, 07 Nov 2024 18:27:37 GMT', 'Connection': 'close'}
2024-11-07 11:27:36,021 - MainThread - botocore.parsers - DEBUG - Response body:
b'ReceiverInternalFailureAn internal error has occurred. Please try your query again at a later time.3dd75dc7-1b7e-4298-a11d-b4b1b19fd989'
2024-11-07 11:27:36,025 - MainThread - botocore.hooks - DEBUG - Event needs-retry.rds.CreateCustomDBEngineVersion: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000016B9B576B10>>
2024-11-07 11:27:36,025 - MainThread - botocore.retries.standard - DEBUG - Retry needed, retrying request after delay of: 0.10785788939158802
2024-11-07 11:27:36,026 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.10785788939158802 seconds
2024-11-07 11:27:36,134 - MainThread - botocore.hooks - DEBUG - Event request-created.rds.CreateCustomDBEngineVersion: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000016B9B524D50>>
2024-11-07 11:27:36,135 - MainThread - botocore.hooks - DEBUG - Event choose-signer.rds.CreateCustomDBEngineVersion: calling handler <function set_operation_specific_signer at 0x0000016B984F2C00>
2024-11-07 11:27:36,135 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-11-07 11:27:36,136 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:rds.us-east-1.amazonaws.com
x-amz-date:20241107T182736Z
x-amz-security-token:

content-type;host;x-amz-date;x-amz-security-token

2024-11-07 11:27:36,136 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241107T182736Z
20241107/us-east-1/rds/aws4_request
a1d3b32baa2e0d597c234a7e982e06cc2ceca3d46f57225d70bfb05bffc29b12
2024-11-07 11:27:36,137 - MainThread - botocore.auth - DEBUG - Signature:

2024-11-07 11:27:36,138 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://rds.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.17.2 md/awscrt#0.20.11 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.8 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#rds.create-custom-db-engine-version', 'X-Amz-Date': b'20241107T182736Z', 'X-Amz-Security-Token': b'', 'Authorization': b'AWS4-HMAC-SHA256 Credential=/20241107/us-east-1/rds/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=', 'Content-Length': '205'}>
2024-11-07 11:27:36,138 - MainThread - botocore.httpsession - DEBUG - Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem
2024-11-07 11:27:36,139 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: rds.us-east-1.amazonaws.com
2024-11-07 11:27:37,581 - MainThread - urllib3.connectionpool - DEBUG - https://rds.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 500 298
2024-11-07 11:27:37,583 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '', 'Strict-Transport-Security': 'max-age=31536000', 'Content-Type': 'text/xml', 'Content-Length': '298', 'Date': 'Thu, 07 Nov 2024 18:27:38 GMT', 'Connection': 'close'}
2024-11-07 11:27:37,583 - MainThread - botocore.parsers - DEBUG - Response body:
b'ReceiverInternalFailureAn internal error has occurred. Please try your query again at a later time.'
2024-11-07 11:27:37,584 - MainThread - botocore.hooks - DEBUG - Event needs-retry.rds.CreateCustomDBEngineVersion: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000016B9B576B10>>
2024-11-07 11:27:37,585 - MainThread - botocore.retries.standard - DEBUG - Retry needed, retrying request after delay of: 1.5713100888085954
2024-11-07 11:27:37,585 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 1.5713100888085954 seconds
2024-11-07 11:27:39,158 - MainThread - botocore.hooks - DEBUG - Event request-created.rds.CreateCustomDBEngineVersion: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000016B9B524D50>>
2024-11-07 11:27:39,158 - MainThread - botocore.hooks - DEBUG - Event choose-signer.rds.CreateCustomDBEngineVersion: calling handler <function set_operation_specific_signer at 0x0000016B984F2C00>
2024-11-07 11:27:39,159 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-11-07 11:27:39,160 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:rds.us-east-1.amazonaws.com
x-amz-date:20241107T182739Z
x-amz-security-token:

content-type;host;x-amz-date;x-amz-security-token

2024-11-07 11:27:39,160 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241107T182739Z
20241107/us-east-1/rds/aws4_request
11a6f8d5cf5c453eb8701a6ef4a3be76e69267a93ac09349cabb603932432676
2024-11-07 11:27:39,161 - MainThread - botocore.auth - DEBUG - Signature:

2024-11-07 11:27:39,162 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://rds.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.17.2 md/awscrt#0.20.11 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.8 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#rds.create-custom-db-engine-version', 'X-Amz-Date': b'20241107T182739Z', 'X-Amz-Security-Token': b'', 'Authorization': b'AWS4-HMAC-SHA256 Credential=/20241107/us-east-1/rds/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=', 'Content-Length': '205'}>
2024-11-07 11:27:39,162 - MainThread - botocore.httpsession - DEBUG - Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem
2024-11-07 11:27:39,163 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: rds.us-east-1.amazonaws.com
2024-11-07 11:27:40,437 - MainThread - urllib3.connectionpool - DEBUG - https://rds.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 500 298
2024-11-07 11:27:40,439 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '', 'Strict-Transport-Security': 'max-age=31536000', 'Content-Type': 'text/xml', 'Content-Length': '298', 'Date': 'Thu, 07 Nov 2024 18:27:41 GMT', 'Connection': 'close'}
2024-11-07 11:27:40,439 - MainThread - botocore.parsers - DEBUG - Response body:
b'ReceiverInternalFailureAn internal error has occurred. Please try your query again at a later time.'
2024-11-07 11:27:40,440 - MainThread - botocore.hooks - DEBUG - Event needs-retry.rds.CreateCustomDBEngineVersion: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000016B9B576B10>>
2024-11-07 11:27:40,441 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.
2024-11-07 11:27:40,442 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-11-07 11:27:40,442 - MainThread - botocore.hooks - DEBUG - Event after-call.rds.CreateCustomDBEngineVersion: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x0000016B9B5026D0>>
2024-11-07 11:27:40,445 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "awscli\clidriver.py", line 499, in main
File "awscli\clidriver.py", line 634, in call
File "awscli\clidriver.py", line 837, in call
File "awscli\clidriver.py", line 963, in invoke
File "awscli\clidriver.py", line 975, in _make_client_call
File "awscli\botocore\client.py", line 360, in _api_call
File "awscli\botocore\client.py", line 739, in _make_api_call
botocore.exceptions.ClientError: An error occurred (InternalFailure) when calling the CreateCustomDBEngineVersion operation (reached max retries: 2): An internal error has occurred. Please try your query again at a later time.

Reproduction Steps

aws rds create-custom-db-engine-version --engine custom-sqlserver-ee --engine-version 15.00.4365.2.dev_cev --image-id --description "Custom SQL Server EE 15.00.4365.2 CEV Developer"

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.17.2

Environment details (OS name and version, etc.)

Windows 10

@GeorgeWilsonPGATour GeorgeWilsonPGATour added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 7, 2024
@tim-finnigan tim-finnigan self-assigned this Nov 7, 2024
@tim-finnigan tim-finnigan added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Nov 7, 2024
@tim-finnigan
Copy link
Contributor

Thanks for reaching out. This is an issue with the CreateCustomDBEngineVersion API rather than the AWS CLI directly. Upon searching internally, it looks like RDS team is already aware of an issue with this API and is working on a fix. In the meantime, the following workaround was recommended:

  1. Create an RPEV instance (needed to ensure Service-linked role "AWSServiceRoleForRDSCustom" exists [1])
  2. Retry CEV creation
  3. Clean up the RPEV instance created in Step 1

(And once the fix is deployed, steps 1 and 3 will no longer be necessary to create the CEV.)

[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAM.ServiceLinkedRoles.html#slr-permissions-custom

@tim-finnigan tim-finnigan added rds service-api This issue is due to a problem in a service API, not the SDK implementation. p2 This is a standard priority issue and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. needs-triage This issue or PR still needs to be triaged. labels Nov 7, 2024
@GeorgeWilsonPGATour
Copy link
Author

@tim-finnigan Thank you - this resolved my issue

Copy link

github-actions bot commented Nov 7, 2024

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p2 This is a standard priority issue rds service-api This issue is due to a problem in a service API, not the SDK implementation.
Projects
None yet
Development

No branches or pull requests

2 participants