S3 Credentials Issue after upgrading to 1.14.4

[trumpet2012]

We have a lambda function that gets called as part of our cloudformation deployment to run collectstatic. After upgrading django-storages from 1.14.3 to 1.14.4 we started seeing the following errors when running collectstatic:

TypeError: sequence item 0: expected str instance, NoneType found
  File "crhelper/resource_helper.py", line 204, in _wrap_function
    self.PhysicalResourceId = func(self._event, self._context) if func else ''
  File "handlers/manage_custom_resource.py", line 11, in run_manage_command
    manage_handler(event["ResourceProperties"], context)
  File "handlers/manage.py", line 25, in handler
    call_command(*command)
  File "django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
  File "django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "django/contrib/staticfiles/management/commands/collectstatic.py", line 209, in handle
    collected = self.collect()
  File "django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
    handler(path, prefixed_path, storage)
  File "django/contrib/staticfiles/management/commands/collectstatic.py", line 378, in copy_file
    self.storage.save(prefixed_path, source_file)
  File "django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "storages/backends/s3.py", line 564, in _save
    obj.upload_fileobj(content, ExtraArgs=params, Config=self.transfer_config)
  File "boto3/s3/inject.py", line 731, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "boto3/s3/inject.py", line 642, in upload_fileobj
    return future.result()
  File "s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "s3transfer/futures.py", line 266, in result
    raise self._exception
  File "s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "s3transfer/upload.py", line 764, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "botocore/client.py", line 1001, in _make_api_call
    http, parsed_response = self._make_request(
  File "botocore/client.py", line 1027, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "botocore/endpoint.py", line 198, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "botocore/endpoint.py", line 134, in create_request
    self._event_emitter.emit(
  File "botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "botocore/signers.py", line 105, in handler
    return self.sign(operation_name, request)
  File "botocore/signers.py", line 199, in sign
    auth.add_auth(request)
  File "botocore/auth.py", line 432, in add_auth
    self._inject_signature_to_request(request, signature)
  File "botocore/auth.py", line 435, in _inject_signature_to_request
    auth_str = ['AWS4-HMAC-SHA256 Credential=%s' % self.scope(request)]
  File "botocore/auth.py", line 384, in scope
    return '/'.join(scope)

Looking at the error in sentry we are able to see that the None in the list of scopes is coming from self.credentials.access_key.

Reverting back to 1.14.3 fixes the issue.

Other dep versions

django==4.2.14
botocore==1.34.146
boto3==1.34.146
s3transfer==0.10.2

[jschneier]

Thanks for the report. I'm not sure how to reproduce — can you shed any light on your debugging? It seems like to me that 9fca46c is the culprit, could you try reverting just that commit and seeing if that fixes it?

Touching anything with S3 auth is always a nightmare, too many ways to slice it.

[trumpet2012]

I'm working on getting a reproducible CDK stack and django app that I can share to recreate the issue. We have to do some hacky things in CDK to get cloudformation to run our lambda management commands as part of the deployment. I suspect it's going to be something with how that is implemented.