Cavalcade Log delivery to cloudwatch fails if events span more than 24 hours

[roborourke]

During the Cavalcade healthcheck background task the following error can occur:

The batch of log events in a single PutLogEvents request cannot span more than 24 hours

The logger should process and chunk the delivery of logs into 24 hours batches based on log timestamp.

(as per Rob's comment)

Acceptance criteria

• Reproduce the issue
• bump to v2.0.4

  altis-cloud/composer.json

  Line 41 in dece03b

  "maxbanton/cwh": "^2.0"

• Check if the issue is still presented
• Generate a new ticket with new improvements of the system given that the limits are gone

[roborourke]

Also seeing some rate limit exceeded warnings:

Fluent Bit Exception[Aws\CloudWatchLogs\Exception\CloudWatchLogsException]: Error executing "DescribeLogStreams" on "https://logs.eu-west-1.amazonaws.com"; AWS HTTP error: Client error: `POST https://logs.eu-west-1.amazonaws.com` resulted in a `400 Bad Request` response:
{"__type":"ThrottlingException","message":"Rate exceeded"}
 ThrottlingException (client): Rate exceeded - {"__type":"ThrottlingException","message":"Rate exceeded"} in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195

[roborourke]

This issue can now be resolved by updating the maxbanton/cwh package maxbanton/cwh#96

cc @humanmade/altis-vulcan

[mikelittle]

Is there a way to reliably test this and confirm the fix?

[ferschubert-hm]

https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html

• The maximum batch size is 1,048,576 bytes. This size is calculated as the sum of all event messages in UTF-8, plus 26 bytes for each log event.
• None of the log events in the batch can be more than 2 hours in the future.
• None of the log events in the batch can be more than 14 days in the past. Also, none of the log events can be from earlier than the retention period of the log group.
• The log events in the batch must be in chronological order by their timestamp. The timestamp is the time that the event occurred, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. (In AWS Tools for PowerShell and the AWS SDK for .NET, the timestamp is specified in .NET format: yyyy-mm-ddThh:mm:ss. For example, 2017-09-15T13:45:30.)
  - A batch of log events in a single request cannot span more than 24 hours. Otherwise, the operation fails.
• The maximum number of log events in a batch is 10,000.
  - There is a quota of five requests per second per log stream. Additional requests are throttled. This quota can't be changed.

[rmccue]

AWS has just (Jan 4) removed their rate limiting for PutLogEvents: https://aws.amazon.com/about-aws/whats-new/2023/01/amazon-cloudwatch-logs-log-stream-transaction-quota-sequencetoken-requirement/

With this change we have removed the need for splitting your log ingestion across multiple log streams to prevent log stream throttling.

This should fix the rate exceeded issue I think, and possibly this means we now want to push events much more often?

[ferschubert-hm]

So the AC would be here to bump to v2.0.4

altis-cloud/composer.json

Line 41 in dece03b

"maxbanton/cwh": "^2.0"

[veselala]

Hey team! Please add your planning poker estimate with Zenhub @ferschubert-hm @jerico @kovshenin @mikelittle @sivanovhm @wisyhambolu