Databrew Dataset PathOptions

[synalogik-jfrench]

Hi All,

It appears that SAM does not support Databrew Dataset PathOptions, unless anyone is able to tell me otherwise?

Structure of resource documented here: https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-databrew/blob/main/aws-databrew-dataset/aws-databrew-dataset.json

Deployment
sam build
sam deploy

Error:

======================================================
Deploy this changeset? [y/N]: y

2023-04-24 22:41:43 - Waiting for stack create/update to complete

CloudFormation events from stack operations (refresh every 5.0 seconds)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                                                    ResourceType                                                                      LogicalResourceId                                                                 ResourceStatusReason                                                            
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_FAILED                                                                     AWS::DataBrew::Dataset                                                            GlueDataBrewDataset                                                               Properties validation failed for resource GlueDataBrewDataset with message:     
                                                                                                                                                                                                                                                     #/Input: extraneous key [PathOptions] is not permitted                          
UPDATE_ROLLBACK_IN_PROGRESS                                                       AWS::CloudFormation::Stack                                                        dataset-name                                                        The following resource(s) failed to update: [GlueDataBrewDataset]. 

Code:

GlueDataBrewDataset:
    Type: AWS::DataBrew::Dataset
    Properties:
      Name: test-dataset'
      Format: CSV
      Input:
        S3InputDefinition:
          Bucket:
            Ref: S3Bucket
      PathOptions:
         Parameters:
            PathParameter:
              - DatasetParameter:
                  Name: "test"
                  Type: "Datetime"
              - PathParameterName: "file_creation_date"

[hoffa]

Can you provide context on where you see that error? Is it from a CloudFormation deployment?

[synalogik-jfrench]

Hi, This is through SAM CLI deployment

sam build
sam deploy

[hoffa]

Interesting, what region is this?

It doesn't look related to either the SAM transform or the SAM CLI as it's a regular CloudFormation resource, and the failure comes from the DataBrew resource.

[synalogik-jfrench]

It is odd and preventing the creating dynamic datasets. The region is eu-west-2, where you can create the resource manually via the console

[hoffa]

So it looks like the issue is that it's the wrong syntax.

In the example you provided Parameters has a PathParameter subproperty, however, it should be a list of PathParameters.

So something like this should work:

  GlueDataBrewDataset:
    Type: AWS::DataBrew::Dataset
    Properties:
      Name: test-dataset
      Format: CSV
      Input:
        S3InputDefinition:
          Bucket:
            Ref: S3Bucket
      PathOptions:
         Parameters:
            - DatasetParameter:
                Name: "test"
                Type: "Datetime"
              PathParameterName: "file_creation_date"

[synalogik-jfrench]

Indeed you are correct, many thanks :)

[synalogik-jfrench]

Any idea's please?