Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for nested structured logging #363

Open
danpat opened this issue Oct 30, 2021 · 0 comments
Open

Add support for nested structured logging #363

danpat opened this issue Oct 30, 2021 · 0 comments

Comments

@danpat
Copy link
Contributor

danpat commented Oct 30, 2021

Over in https://github.com/mapbox/vorpal-sword we've got a worker that is capable of emitting JSON structured logging, which CloudWatch understands how to parse, and make nice-looking queries based on certain fields in log messages.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html

Currently, the watchbot worker logger adds a string prefix to any stderr/stdout output from the worker process, which breaks the JSON format, and causes CloudWatch not to recognize the log lines as JSON formatted. Example:

[Sat, 30 Oct 2021 02:17:04  GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399]  {"name":"@mapbox/vorpal-sword/backend","watchbot_session_id":"45e24f54-86d4-4952-8666-8146042985d6","hostname":"7dc1b301967c","pid":4334,"sns_message_id":"0f29eaca-18f3-5230-b8d0-b33c323c133e","sqs_message_id":"58598cd1-a09e-4d16-be9a-2355b86c1399","problem_id":"7dccbd5c-15f3-4280-908b-5e7c1917acaa","originating_request_id":"JWTRm-jMTDaWPKhc7mZBKOgkB8b4Ad9LV89UNwDJKhzv-iIEO19SUA==","level":20,"timestamp":"2021-10-30T02:17:04.432+00:00","logger":"stderr","process_id":4353,"thread_id":4353,"msg":"Adding  service 186 at location 186 as node  188","event":"worker_vorpal_sword_log_line","time":"2021-10-30T02:17:04.448Z","v":0}

If the string prefix of [Sat, 30 Oct 2021 02:17:04 GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399] was not there, then it would be possible to perform cloudwatch log insights queries like:

fields @timestamp, @message
| filter originating_request_id like "JWTRm-jMTDaWPKhc7mZBKOgkB8b4Ad9LV89UNwDJKhzv-iIEO19SUA=="

To trivially find all messages that correlate with an original CloudFront request ID. Of course the fields here are specific to our system.

There are two ways we could address this:

  1. Add a config flag to ecs-watchbot like emitWorkerLogLeader: false to just turn it off. That would put the onus on the worker process to emit the relevant fields (like timestamp, watchbot session ID, etc).
  2. Add a config flag like structuredLogging: true - in this case, watchbot would expect the client to emit JSON formatted messages, and would merge watchbot specific fields into the JSON.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant