aws sesv2 list-contacts returns an empty result when filtering with FilteredStatus=OPT_OUT

[delhi09]

Describe the bug

I would like to count OPT_OUT emails in the list management of AWS SES.

I confirmed at least one OPT_OUT email exists using the following command:

aws sesv2 get-contact --contact-list-name=TestContactList [email protected] --region us-east-1
{
    "ContactListName": "TestContactList",
    "EmailAddress": "[email protected]",
    "TopicPreferences": [
        {
            "TopicName": "test-topic",
            "SubscriptionStatus": "OPT_OUT"
        }
    ],
    "UnsubscribeAll": false,
    "CreatedTimestamp": "2024-06-05T11:17:46.771000+09:00",
    "LastUpdatedTimestamp": "2024-06-13T12:52:48.739000+09:00"
}

However, the following command always returns an empty result:

aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --no-paginate --region us-east-1
{
    "Contacts": [],
    "NextToken": "***"
}

When FilteredStatus=OPT_IN, the command successfully returns OPT_IN emails as follows:

aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_IN,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --no-paginate --region us-east-1
{
    "Contacts": [
        {
            "EmailAddress": "[email protected]",
            "TopicPreferences": [
                {
                    "TopicName": "test-topic",
                    "SubscriptionStatus": "OPT_IN"
                }
            ],
            "UnsubscribeAll": false,
            "LastUpdatedTimestamp": "2024-05-23T19:02:54.560000+09:00"
        },
        {
            "EmailAddress": "[email protected]",
            "TopicPreferences": [
                {
                    "TopicName": "test-topic",
                    "SubscriptionStatus": "OPT_IN"
                }
            ],
            "UnsubscribeAll": false,
            "LastUpdatedTimestamp": "2024-05-23T20:01:50.994000+09:00"
        },
...

Expected Behavior

OPT_OUT emails should be returned as follows:

aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --no-paginate --region us-east-1
{
    "Contacts": [
        {
            "EmailAddress": "[email protected]",
            "TopicPreferences": [
                {
                    "TopicName": "test-topic",
                    "SubscriptionStatus": "OPT_OUT"
                }
            ],
            "UnsubscribeAll": false,
            "LastUpdatedTimestamp": "2024-05-23T19:02:54.560000+09:00"
        },
...

Current Behavior

As described in the "Describe the bug" section.

Reproduction Steps

1.Send an email using the SESv2 API with ListManagementOptions:

import boto3

client = boto3.client("sesv2", region_name="us-east-1")

args = {
        "FromEmailAddress": "[email protected]",
        "Destination": {
            "ToAddresses": ["[email protected]"],
        },
        "Content": {
            "Simple": {
                "Subject": {"Data": "test", "Charset": "UTF-8"},
                "Body": {
                    "Text": {"Data": "test", "Charset": charset},
                    "Html": {"Data": "<div>test</test>", "Charset": "UTF-8"},
                },
            },
        },
        "ListManagementOptions": {
            "ContactListName": "TestContactList",
            "TopicName": "[email protected]",
        }
}
client.send_email(**args)

2.Receive the email in a Gmail client and click the "Unsubscribe" link.

3.Confirm "[email protected]" is saved as OPT_OUT using the following command:

aws sesv2 get-contact --contact-list-name=TestContactList [email protected] --region us-east-1
{
    "ContactListName": "TestContactList",
    "EmailAddress": "[email protected]",
    "TopicPreferences": [
        {
            "TopicName": "test-topic",
            "SubscriptionStatus": "OPT_OUT"
        }
    ],
    "UnsubscribeAll": false,
    "CreatedTimestamp": "***",
    "LastUpdatedTimestamp": "***"
}

4. Fetch OPT_OUT emails using the following command. It will probably return an empty result:

aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --no-paginate --region us-east-1
{
    "Contacts": [],
    "NextToken": "***"
}

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.16.7 Python/3.11.8 Darwin/23.4.0 exe/x86_64

Environment details (OS name and version, etc.)

macOS 14.4.1

[RyanFitzSimmonsAK]

Hi @delhi09, thanks for reaching out. Since you received a NextToken, I suspect what's happening here is that pagination is taking place before filtering. You're using the --no-paginate flag, so if your OPT_OUT contact is not in the first page of results, getting no results back makes sense. If you remove that, or continue paginating, what's the response look like?

[delhi09]

Hi @RyanFitzSimmonsAK , thank you for your suggestion. I tried it, but unfortunately, it didn't change the result.

$ aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --region us-east-1
{
    "Contacts": [],
    "NextToken": “xxx”
}

[RyanFitzSimmonsAK]

What if you continue to paginate using the NextToken?

[delhi09]

Hi @RyanFitzSimmonsAK , I apologize for the delay in my response. Thank you for your suggestion.

I tried pagination 10 times as follows. However, the "Contacts" field was empty in all results.

$ aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --region us-east-1
{
    "Contacts": [],
    "NextToken": "xxx"
}
$ aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --next-token xxx --region us-east-1
{
    "Contacts": [],
    "NextToken": "yyy"
}
$ aws sesv2 list-contacts --contact-list-name=TestContactList --filter="FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}" --next-token yyy --region us-east-1
{
    "Contacts": [],
    "NextToken": "zzz"
}

[RyanFitzSimmonsAK]

Are you paginating until there isn't a NextToken in the response?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[delhi09]

Hi @RyanFitzSimmonsAK , thank you for your suggestion. I created the following Python script and verified your suggestion.

import subprocess
import json
import time

request_count = 0

cmd = [
    "aws",
    "sesv2",
    "list-contacts",
    "--contact-list-name=TestContactList",
    "--filter=FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}",
    "--region",
    "us-east-1",
]
output = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result = output.stdout.decode("utf-8")
next_token = json.loads(result)["NextToken"]
request_count += 1

while True:
    time.sleep(5)
    cmd = [
        "aws",
        "sesv2",
        "list-contacts",
        "--contact-list-name=TestContactList",
        "--filter=FilteredStatus=OPT_OUT,TopicFilter={TopicName=test-topic,UseDefaultIfPreferenceUnavailable=true}",
        "--region",
        "us-east-1",
        "--next-token",
        next_token,
    ]
    output = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    result = json.loads(output.stdout.decode("utf-8"))
    next_token = result["NextToken"]
    request_count += 1
    print("request count: ", request_count, "contacts: ", result["Contacts"], "next_token: ", next_token[0:10])
    if "NextToken" not in result or not result["NextToken"]:
        break

print("finished. request count: ", request_count)

I aborted the script because it seemed like it was going to run indefinitely.

Generally, the results showed contacts are empty, but there were a few instances where email addresses were found on some pages. Could you please explain how this is supposed to work?

console log1

request count:  39 contacts:  [] next_token:  AYADeJxWUG
request count:  40 contacts:  [] next_token:  AYADeHByJv
request count:  41 contacts:  [{'EmailAddress': '[email protected]', 'TopicPreferences': [{'TopicName': 'test-topic', 'SubscriptionStatus': 'OPT_OUT'}], 'UnsubscribeAll': False, 'LastUpdatedTimestamp': '2024-05-31T14:22:34.379000+09:00'}] next_token:  AYADeJqWAX
request count:  42 contacts:  [] next_token:  AYADeCqJFg
request count:  43 contacts:  [] next_token:  AYADeDGpKA

console log2

request count:  86 contacts:  [] next_token:  AYADeH3wtH
request count:  87 contacts:  [] next_token:  AYADeEnwYs
request count:  88 contacts:  [{'EmailAddress': '[email protected]', 'TopicPreferences': [{'TopicName': 'test-topic', 'SubscriptionStatus': 'OPT_OUT'}], 'UnsubscribeAll': False, 'LastUpdatedTimestamp': '2024-06-05T10:55:37.937000+09:00'}] next_token:  AYADeOuiB1
request count:  89 contacts:  [] next_token:  AYADeNXxLv
request count:  90 contacts:  [] next_token:  AYADeA9Kbs

[RyanFitzSimmonsAK]

Thanks for your patience. Could you try increasing the --page-size parameter? I suspect that this defaults to a very low value, and it's making these ListContacts operation on very few contacts at a time. Also, roughly how many contacts do you have total? Thanks!

[delhi09]

Hi @RyanFitzSimmonsAK,

Thank you for your suggestion.

I increased the --page-size parameter, as it can be set up to 1000. However, since the total number of contacts in my service is quite large, paging through them even with 1000 entries per page doesn’t seem feasible.
(Unfortunately, I cannot disclose the exact number of contacts due to confidentiality.)

Initially, I assumed that the --filter option for the list-contacts command worked as follows:

1. The total list of contacts would be filtered based on the criteria.
2. The result from step 1 would then be paginated.

However, based on your advice and the results of my tests, I now understand that it works the other way around: the pagination happens first, and only then the results are filtered.

My goal is to retrieve a list of email addresses that have opted out, but given this behavior, it seems difficult to achieve that using the list-contacts command.

Do you know of any more efficient method for this?

If not, I understand the current limitations of the list-contacts command, and I’m happy to close this issue for now.

Thanks again!

[RyanFitzSimmonsAK]

I'm not sure if there's an efficient workaround. I've reached out to the service team about this, and will let you know when I have an update.

Ticket # for internal use : P156959491

[RyanFitzSimmonsAK]

The service team has acknowledged this behavior, and confirmed that pagination taking place before filtering is intended. At this time, there is not a workaround for your specific use case. I've asked them to document this nuance. Thanks again for opening this issue.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.