S3 credentials in links missing for private buckets after upgrade to Django 5.1

[GitRon]

Hi @jschneier!

I found an issue with Django 5.1. After the upgrade (using django-storages 1.14.4), all GET-parameter credentials for private S3 buckets are not being added to the links in my templates anymore. This means that I don't have access to the files.

I have no idea and since I have little knowledge on how this amazing package works, I can't really contribute any suggestions. 😔 I just verified that it's really Django 5.1.

Here's my setup, it might help in reconstructing the case.

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class PrivateMediaStorage(S3Boto3Storage):
    location = settings.AWS_PRIVATE_MEDIA_LOCATION
    querystring_expire = 3600  # seconds until the generated link expires
    default_acl = "bucket-owner-full-control"
    file_overwrite = False
    custom_domain = False

Thanks so much for looking into this!
Ronny

PS: It seens unrelated to #1437 since there, Django 4.2 was used.

[christoph-teichmeister]

I've also run in this issue 😞

[kylepollina]

5.1 has also caused issues with AWS_S3_CUSTOM_DOMAIN. With django 5.0 + django rest, when desearializing a model with an image/file field, the key was prepended with whatever value was set for AWS_S3_CUSTOM_DOMAIN, however after the upgrade it seems to default to the hostname.

For example, if I set AWS_S3_CUSTOM_DOMAIN in my settings.py file to be www.test.com

on django==5.0 imagefield deserialization returns http://www.test.com/organizations/0a72d97f-974c-4828-85b1-0276a6c6cf2e/images/m22_ms.png

But on django==5.1 imagefield deserialization returns http://localhost:8000/organizations/0a72d97f-974c-4828-85b1-0276a6c6cf2e/images/m22_ms.png

[kylepollina]

Upon further inspection it looks like the newer version of django is overwriting parts of the storage backend. I was debugging a bit and found that the variable stored here:

https://github.com/django/django/blob/main/django/db/models/fields/files.py#L25

This storage object seems to be the same type, but storage.url used to be pointing to the url function for the s3 storage backend, but after upgrading, it pointed to the url function for the default django filestorage backend. However I couldn't backtrack to see how or where that value is being set. It gets pretty muddled

[rory-ferguson]

Hi @kylepollina, You might of missed the django-storages config change

https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#configuration-settings

This changed for django 4.2

[GitRon]

@rory-ferguson Thanks for the investigation, we didn't use the new STORAGE yet. I'll try it out and report back.

[kylepollina]

@rory-ferguson Thank you! I updated my configurations and this fixed my issue.

[GitRon]

Seems to work here, too. Should this be documented somewhere? 🤔

[kylepollina]

@GitRon it is documented on the bottom of the Django 5.1 release notes. https://docs.djangoproject.com/en/5.1/releases/5.1/

The DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings is removed.

[GitRon]

@kylepollina You are right! Still don't get it, why Django works with the old settings 😆 Well, maybe really not a storages-problem 🤷‍♂️ On the other side, explicit is better than implicit. I'll leave it to the maintainers to decide this.

Thanks for the support everyone ❤️