[WIP] Bazel Build Event Stream support

[aherrmann]

This PR adds support to connect Buck2 to a Bazel Build Event Stream server as found among some Bazel remote build execution platforms. It repurposes the existing RemoteEventSink used to connect to Scribe within Meta's internal Buck2 version and connects it to a Bazel Build Event Service over gRPC instead, and it adds a translation layer between Buck2 build events and Bazel build events.

This is a work in progress, only very few build events are translated so far. To test this code you can

• build Buck2,
• change into the examples/hello_world directory,
• configure a BES service by setting the BES_URI environment-variable, and
• start a build.

$ cargo build --bin=buck2
$ cd examples/hello_world
$ BES_URI=XYZ ../../target/debug/buck2 build :main

For example, with BuildBuddy, if you run bazel run server -- -app.log_level debug locally, set BES_URI=http://localhost:1985.

The Buck2 Trace-ID is used as the BES invocation ID and you can see a Buck2 build on the UI under e.g. http://localhost:8080/invocation/e3123cfa-c350-432b-9784-8b3fe48bed14.

• BuckEvent: provide a Buck Event Publisher proto
• Add buck2_event_publisher_proto to Cargo.toml
• Add Bazel Build Event Protocol/Stream protobuf
• Implement RemoteEventSink to publish to Bazel BES

[nlopezgi]

Hi @aherrmann , thanks for putting this together, we are really looking forward to it. I tested it out and had a couple of questions:

• I see the TLS bit is marked on a TODO as not enabled yet, are you planning on setting this up as part of this CL or a future one?
• Is there any way that I can get some verbose logs from the service publishing BuildEvents? I did not find a way to see anything from this interaction.

Thanks again!

[aherrmann]

I see the TLS bit is marked on a TODO as not enabled yet, are you planning on setting this up as part of this CL or a future one?

It's needed for the target use-case. So, I'm planning to add support for it.

Is there any way that I can get some verbose logs from the service publishing BuildEvents? I did not find a way to see anything from this interaction.

The BES client runs as a thread in the Buck2 daemon. Currently, there is just a bit of printf-debugging in the code. Buck2 has some logging facilities, see --verbose flag and BUCK_LOG env-var. However, this PR so far doesn't make explicit use of these.

[nlopezgi]

Thanks again for your help @aherrmann.
I've been continuing testing this code, I was able to hack together setting TLS certs so that my BES service would connect properly. I've made some progress and was able to see the sysout messages from your code producing the events and sending them to the BES service.
I am still, however, not able to have my BES service process the events successfully. One thing I noticed is that you are not sending any PublishLifecycleEvent calls, and you are setting the check_preceding_lifecycle_events_present on the PublishBuildToolEventStreamRequest. Do you have plans to add the PublishLifecycleEvent calls? Iiuc these are needed for the proper lifecycle of a build's interaction with a BES service. It also seems our implementation of the BES service needs them to happen to properly handle the PublishBuildToolEventStream calls.
Thanks!

[aherrmann]

Do you have plans to add the PublishLifecycleEvent calls?

I'm currently testing and developing against BuildBuddy. To my understanding so far, that implementation seems to largely ignore lifecycle events, at least I haven't spotted a need for them, yet. (I may be wrong, I'm not deeply familiar with the BuildBuddy implementation). So, for now, I'm not planning to add lifecycle events.

[aherrmann]

Incremental update: I've added the necessary events to display targets, actions, command-lines, and command failure messages in the BuildBuddy UI for simple, local builds.

To reproduce

# BuildBuddy
$ bazel run server -- -app.log_level debug -app.enable_target_tracking true

# Buck2
$ cargo build --bin=buck2
$ cd examples/hello_world
$ BES_URI=http://localhost:1985 ../../target/debug/buck2 build :main

[nlopezgi]

Do you have plans to add the PublishLifecycleEvent calls?

I'm currently testing and developing against BuildBuddy. To my understanding so far, that implementation seems to largely ignore lifecycle events, at least I haven't spotted a need for them, yet. (I may be wrong, I'm not deeply familiar with the BuildBuddy implementation). So, for now, I'm not planning to add lifecycle events.

Thanks @aherrmann, I think your current process of develping against BuildBuddy makes sense as an initial approach to validate your POC.

I confirmed with some colleagues and our reading of the protocol is that there should be some lifecycle events in order to produce a legal event stream. Particularly The protocol mandates that BuildEnqueued must be sent before any other event, and this event seems to be something that is documented in the PublishLifecycle event docs as one that is needed.

I'm happy to talk more about this if you feel strongly that lifecycle events are really not needed and ask some of my colleagues with more experience on the protocol to chime in too.

Thanks again for your work on this feature!

[nlopezgi]

Circling back on the state of testing of this PoC with EngFlow UI. I have been able to test all the features on this PR successfully. The key things I had to hardcode to get it to work properly with EngFlow UI are the following:

• TLS cert creation
• Set project in requests to value "default"
• Add lifecycle events called via rpc PublishLifecycleEvent for the following: BuildEnqueued, InvocationAttemptStarted, InvocationAttemptFinished, BuildFinished.
  Thanks again for putting this PoC together for testing @aherrmann . I will keep my eye out if you add more features to test them with EngFlow UI.

[facebook-github-bot]

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[aherrmann]

I've added sufficient TLS and HTTP header support to connect to the BuildBuddy cloud service over https and using an access token.
The configuration is currently performed through env-vars:

• BES_URI: The Build Event Stream service URI.
• BES_RESULT: URL prefix for BES result.
• BES_TLS: Set to 1 to enable TLS. (PEM TBD, see here for an implementation).
• BES_HEADERS: Comma separated list of headers of the form KEY:VALUE, use this to set the access token.

[cjhopman]

you should be able to use unpack_event to get an UnpackedBuckEvent that cuts down a bit on the boilerplate here (compare superconsole's matching on these events:

buck2/app/buck2_client_ctx/src/subscribers/superconsole.rs

Line 504 in 86174c3

match unpack_event(event)? {