Missing resource type schema definitions when using NewListTypesPaginator and DescribeType

[sk593]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

I have code that uses the AWS SDK to download AWS resource type schemas into a JSON format. Specifically, these APIs are used:

aws-sdk-go-v2/service/cloudformation/api_op_ListTypes.go

Line 246 in 8939ed0

func NewListTypesPaginator(client ListTypesAPIClient, params *ListTypesInput, optFns ...func(*ListTypesPaginatorOptions)) *ListTypesPaginator {

and

aws-sdk-go-v2/service/cloudformation/api_op_DescribeType.go

Line 20 in 8939ed0

func (c *Client) DescribeType(ctx context.Context, params *DescribeTypeInput, optFns ...func(*Options)) (*DescribeTypeOutput, error) {

The JSON format is described using a TypeRecord where the AWS resource type schema is added as the schema field of the TypeRecord.
TypeRecord definition: https://github.com/sk593/bicep-types-aws/blob/9e2142a12a3baec67a666099cc5021f2ee9c9449/src/aws-type-downloader/main.go#L170
Relevant code: https://github.com/sk593/bicep-types-aws/blob/9e2142a12a3baec67a666099cc5021f2ee9c9449/src/aws-type-downloader/main.go#L132

When the AWS resource type schema gets parsed into JSON format using these APIs, there are some definitions that are missing. For example with AWS Quicksight Analysis, I would expect there to be a definition for VisualInteractionOptions but that field is missing in the definitions: https://github.com/sk593/bicep-types-aws/blob/update-aws-types/artifacts/types/AWS%3A%3AQuickSight%3A%3AAnalysis.json

There are a few more resource types where this is the case but I'll reference Quicksight Analysis for this issue

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would expect all definitions to be present in the schema coming from DescribeType.

Current Behavior

Please see this file for an example of the missing definitions: https://github.com/sk593/bicep-types-aws/blob/update-aws-types/artifacts/types/AWS%3A%3AQuickSight%3A%3AAnalysis.json

VisualInteractionOptions should have a definition but that seems to be missing. There should be a 1 to 1 mapping between the Schema field returned by DescribeType and the schema field of TypeRecord. It seems like the definitions are missing in the returned Schema field of DescribeType and therefore also missing from the schema field of TypeRecord

Reproduction Steps

This a repro of the repository that I'm working with: https://github.com/sk593/bicep-types-aws/tree/update-aws-types

Instructions for running this code can be found here (step 3 specifically): https://github.com/sk593/bicep-types-aws/tree/update-aws-types/docs/contributing/contributing-code/contributing-code-building

This does take a while to run so you can also look at Quicksight::Analysis as an isolated case by updating the for loop on line 88 of src/aws-type-downloader/main.go to the following code. This will download the AWS::Quicksight::Analysis resource type in the expected JSON format and also print the Schema response from DescribeType into a separate file (both will have the missing fields). Run go run main.go --output ../../artifacts/types in the src/aws-type-downloader folder.

	for paginator.HasMorePages() {
		response, err := paginator.NextPage(cmd.Context())
		if err != nil {
			return err
		}

		for _, typeSummary := range response.TypeSummaries {
			if *typeSummary.TypeName == "AWS::QuickSight::Analysis" {
				fmt.Println("theres a match for quicksight")

				fmt.Printf("Processing Type: %s\n", *typeSummary.TypeName)
				filename := *typeSummary.TypeName + ".json"
				delete(existing, filename)

				response, err := client.DescribeType(cmd.Context(), &cloudformation.DescribeTypeInput{
					Arn: typeSummary.TypeArn,
				})
				if err != nil {
					return err
				}

				fmt.Printf("Quicksight: %s - checking schema\n", *typeSummary.TypeName)
				// Write response.Schema to a file
				filePath := path.Join(outputDirectory, "analysis.json")
				err = os.WriteFile(filePath, []byte(*response.Schema), 0644)
				if err != nil {
					return fmt.Errorf("failed to write schema for %s: %w", *typeSummary.TypeName, err)
				}
				fmt.Printf("Schema written to %s\n", filePath)

				fmt.Printf("Writing: %s\n", *typeSummary.TypeName)

				schema := map[string]interface{}{}
				err = json.Unmarshal([]byte(*response.Schema), &schema)
				if err != nil {
					return fmt.Errorf("type %s could not be converted to JSON JSON: %w", *typeSummary.TypeName, err)
				}

				record := TypeRecord{
					ARN:       *typeSummary.TypeArn,
					Schema:    schema,
					Type:      *typeSummary.TypeName,
					Timestamp: *typeSummary.LastUpdated,
				}

				result, err := json.MarshalIndent(&record, "", "  ")
				if err != nil {
					return err
				}

				err = os.WriteFile(path.Join(outputDirectory, filename), result, 0644)
				if err != nil {
					return err
				}

			}
		}
	}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

require (
github.com/aws/aws-sdk-go-v2 v1.32.5
github.com/aws/aws-sdk-go-v2/config v1.28.5
github.com/aws/aws-sdk-go-v2/service/cloudcontrol v1.23.1
github.com/aws/aws-sdk-go-v2/service/cloudformation v1.56.0
github.com/spf13/cobra v1.8.1
)

require (
github.com/aws/aws-sdk-go-v2/credentials v1.17.46 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.20 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.24 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.24 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.6 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.1 // indirect
github.com/aws/smithy-go v1.22.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
)

Compiler and Version used

go version go1.23.3 darwin/arm64

Operating System and version

macOS/darwin

[RanVaknin]

Hi @sk593 ,

Thanks for reaching out. I think your reproduction example is a bit broad. If I understand the issue correctly, when running a DescribeType API call on a AWS::QuickSight::Analysis the schema returned references VisualInteractionOptions in the form of:

          "$ref": "#/definitions/VisualInteractionOptions"

However, the schema actually never defines this, only references it.

My experience with CFN is limited, but if my understanding of the problem is accurate, then the issue is with the CFN API itself, and not related to the SDK.

I have created an internal ticket with the API team (ticket ID #P175451388), however I don't have any ETA / visibility as to when this would get addressed. If you have access to AWS support, feel free to create your own support ticket and reference the ticket ID mentioned in my comment.

All the best,
Ran~

[sk593]

Yes, your understanding of the problem is correct. Thanks for submitting the internal ticket.

There are additional resources that have this issue too so I'm assuming I'll need to submit support ticket for those resources too. If I don't have access to AWS support, what's the best way to get in touch with the API teams to update the schema?

[rynowak]

@RanVaknin - do you think https://github.com/aws-cloudformation/cloudformation-coverage-roadmap would be a better place for us to report this?

[RanVaknin]

Hi @sk593,

Without AWS support you won't be able to interface with the service team unless they have some open source presence.
You can post your findings here and I can update the internal ticket on your behalf.

@rynowak,

I'm not familiar with the repo you linked, but it seems like its owned by the CFN team, so it might be a step in the right direction to raise visibility to this issue.

Thanks,
Ran~

[sk593]

Hi @RanVaknin,

I'll go ahead and open an issue on the CFN repo linked. If you'd be able to submit internal tickets for the following resources, that would be amazing!

AWS::QuickSight::Analysis and AWS::QuickSight::Dashboard do not have definitions for VisualInteractionOptions
AWS::QuickSight::Template does not have a definition for GeospatialCircleSymbolStyle
AWS::SageMaker::Cluster does not have a definition for CustomMetadata