You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Over in https://github.com/mapbox/vorpal-sword we've got a worker that is capable of emitting JSON structured logging, which CloudWatch understands how to parse, and make nice-looking queries based on certain fields in log messages.
Currently, the watchbot worker logger adds a string prefix to any stderr/stdout output from the worker process, which breaks the JSON format, and causes CloudWatch not to recognize the log lines as JSON formatted. Example:
[Sat, 30 Oct 2021 02:17:04 GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399] {"name":"@mapbox/vorpal-sword/backend","watchbot_session_id":"45e24f54-86d4-4952-8666-8146042985d6","hostname":"7dc1b301967c","pid":4334,"sns_message_id":"0f29eaca-18f3-5230-b8d0-b33c323c133e","sqs_message_id":"58598cd1-a09e-4d16-be9a-2355b86c1399","problem_id":"7dccbd5c-15f3-4280-908b-5e7c1917acaa","originating_request_id":"JWTRm-jMTDaWPKhc7mZBKOgkB8b4Ad9LV89UNwDJKhzv-iIEO19SUA==","level":20,"timestamp":"2021-10-30T02:17:04.432+00:00","logger":"stderr","process_id":4353,"thread_id":4353,"msg":"Adding service 186 at location 186 as node 188","event":"worker_vorpal_sword_log_line","time":"2021-10-30T02:17:04.448Z","v":0}
If the string prefix of [Sat, 30 Oct 2021 02:17:04 GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399] was not there, then it would be possible to perform cloudwatch log insights queries like:
fields @timestamp, @message
| filter originating_request_id like "JWTRm-jMTDaWPKhc7mZBKOgkB8b4Ad9LV89UNwDJKhzv-iIEO19SUA=="
To trivially find all messages that correlate with an original CloudFront request ID. Of course the fields here are specific to our system.
There are two ways we could address this:
Add a config flag to ecs-watchbot like emitWorkerLogLeader: false to just turn it off. That would put the onus on the worker process to emit the relevant fields (like timestamp, watchbot session ID, etc).
Add a config flag like structuredLogging: true - in this case, watchbot would expect the client to emit JSON formatted messages, and would merge watchbot specific fields into the JSON.
The text was updated successfully, but these errors were encountered:
Over in https://github.com/mapbox/vorpal-sword we've got a worker that is capable of emitting JSON structured logging, which CloudWatch understands how to parse, and make nice-looking queries based on certain fields in log messages.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html
Currently, the
watchbot
worker logger adds a string prefix to anystderr/stdout
output from the worker process, which breaks the JSON format, and causes CloudWatch not to recognize the log lines as JSON formatted. Example:If the string prefix of
[Sat, 30 Oct 2021 02:17:04 GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399]
was not there, then it would be possible to perform cloudwatch log insights queries like:To trivially find all messages that correlate with an original CloudFront request ID. Of course the fields here are specific to our system.
There are two ways we could address this:
emitWorkerLogLeader: false
to just turn it off. That would put the onus on the worker process to emit the relevant fields (like timestamp, watchbot session ID, etc).structuredLogging: true
- in this case, watchbot would expect the client to emit JSON formatted messages, and would merge watchbot specific fields into the JSON.The text was updated successfully, but these errors were encountered: