bug: when configuring tflocal for remote s3 state, live AWS is used

[torenware]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When setting up remote state using dynamoDB and s3, localstack currently behaves as follows:

1. Create a bucket and dynamodb table either using awslocal or tflocal.
2. Write a terraform.backend block for a simple terraform configuration to reference the backend bucket and table.

terraform {
  backend "s3" {
    bucket = "backend-bucket-iac-terraform"
    key    = "terraform.tfstate"
    region = "us-west-2"

    dynamodb_table = "terraform-state-locking"

    # s3_use_path_style           = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    # skip_requesting_account_id  = true
  }
}

3. Now run tflocal init. This will FAIL, saying that the bucket was not found.
4. Now create the corresponding backend bucket and dynamodb table on live AWS with the same names and regions as you used for localstack.
5. Without changing your terraform configuration, run tflocal init again. IT WILL SUCCEED, and moreover, the live S3 bucket on AWS will contain your remote state (including the fake ARN refs to the created objects on localstack).

This is behavior is somewhat unexpected :-)

I have a repo containing the demonstration files for anyone who'd like to try this out for themselves.

Expected Behavior

I'd expect tflocal to look for the bucket and dynamodb table locally, on the localstack mocked db, and not on live AWS :-)

How are you starting LocalStack?

With the localstack script

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

$ localstack start

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

1. Download my demo repo
2. Follow the instructions in the repo for creating the bucket and dynamodb table set up in the `remote-state` directory.
3. `cd ../dyn1`
4. `tflocal init`
5. Observe the error returned; tflocal did not find the remote state bucket:

Initializing the backend...
╷
│ Error: Failed to get existing workspaces: S3 bucket does not exist.
│
│ The referenced S3 bucket must have been previously created. If the S3 bucket
│ was created within the last minute, please wait for a minute or two and try
│ again.
│
│ Error: NoSuchBucket: The specified bucket does not exist
│ 	status code: 404, request id: 7WRR8Q21Y5DTWVAV, host id: Zb0SMz3HOprcP8kDQo9QRMJ2YtrbjFqUW0rtWJIWqqnlykGRS1yaZTmZAfS/5aiGVyK9BLbIaV8=
│
│
│

7. Now create a bucket with the same name and region on *live AWS*.
8. Run `tflocal init` again, and observe that init works, and you can apply the configuration as well.  The resources will be created on localstash, but the bucket and dynamodb table on *live AWS* will be updated with the remote state.

Environment

- OS:  MacOS Big Sur, using Colima for docker.
- LocalStack: 2.1.0
- awscli v2 is configured for us-west-2 with a valid key and secret.

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[MarcelStranak]

Hi @torenware,

Could you please try using this terraform code and running the tflocal init, you might need to run it with -reconfigure option?

terraform {
  backend "s3" {
    bucket                      = "mybucket"
    key                         = "terraform.tfstate"
    region                      = "us-east-1"
    dynamodb_table              = "terraform-state-lock"
    endpoint                    = "http://s3.localhost.localstack.cloud:4566"
    iam_endpoint                = "http://localhost:4566"
    sts_endpoint                = "http://localhost:4566"
    dynamodb_endpoint           = "http://localhost:4566"
  }
}

[torenware]

This time, it appears that if mybucket does not exist in the localstack server, tflocal init gets a "bucket does not exist" error. If however I create the bucket, tflocal init now SUCCEEDS. I'm testing out tflocal apply now; there's an issue with the dynomodb lock table right now I'm looking into.

[torenware]

OK, if the lock table appears in the correct region, apply also works.

So the terraform.backend block also needs endpoint and *_endpoint arguments? Is this documented anywhere?

[torenware]

OK, I see it's documented here.. This is however a work-around; tflocal should not be going back to real-world AWS, yes?

[MarcelStranak]

Sorry, I forgot to mention the documentation earlier.
Yes, this is a workaround and I will move this issue under the https://github.com/localstack/terraform-local

As tflocal is a wrapper, you can create your configuration file using the same technique https://developer.hashicorp.com/terraform/language/files/override.

[whummer]

Hi @torenware @MarcelStranak . Thanks for sharing the details here. Quick update - we have now pushed a few enhancements for tflocal, which should enable the use of local S3 storage backends out of the box.

When you get a chance, can you please upgrade to the latest version and give it another try? Thanks!

pip install --upgrade terraform-local

[torenware]

When you get a chance, can you please upgrade to the latest version and give it another try? Thanks!

@whummer I'll give it a try later today. Thanks for implementing this!

[torenware]

@whummer : definitely a problem with the new code. On tflocal init with my files (see above for link to the repo), I get python errors:

$ cd ../dyn1
$ tflocal init
Traceback (most recent call last):
  File "/usr/local/bin/tflocal", line 355, in <module>
    main()
  File "/usr/local/bin/tflocal", line 342, in main
    config_file = create_provider_config_file(providers)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/tflocal", line 125, in create_provider_config_file
    tf_config += generate_s3_backend_config()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/tflocal", line 191, in generate_s3_backend_config
    result = result.replace(f"<{key}>", value)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 2 must be str, not bool

You should be able to reproduce this easily using my steps above.

[torenware]

I've gotten this up in a debugger, and the problem occurs on handling key = "skip_credentials_validation", which is being pulled out of the terraform.backend block in terraform.tf from my files.

[whummer]

Thanks for the update @torenware - this was an oversight in the last implementation, we've now added a fix to address this. Can you please upgrade the pip page (to version 0.11) one more time and try again? Thanks for your help.

[torenware]

@whummer The new version works for my file. Looks good.

[torenware]

@whummer this has been fixed now, yes? Should I close this issue?

[sakibstark11]

Hi all
I still seem to be having issues with this
my setup

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  backend "s3" {}
}

and I run terraform init with -backend-config flags and it tries to communicate with actual aws again
I get the following error

│ Error: No valid credential sources found
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded

I have the right perms set for the bucket too. Any help would be greatly appreciated.

[MarcelStranak]

Hello @sakibstark11, kindly report this as a new issue, accompanied by a detailed description of the issue and a reproducible sample. Thank you.

[sakibstark11]

Hello @sakibstark11, kindly report this as a new issue, accompanied by a detailed description of the issue and a reproducible sample. Thank you.

#49

Thanks.