buck2_client: download traces via buck2.log_url config key
#770
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
from
7d26fab to
b112eb3
Compare
|
This can't do things like pass authentication credentials just yet, but I'd like to know if this is the right shape. As it is, it's still incredibly useful, so I'd like to know if it looks like roughly the right thing before doing much more. |
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
from
b112eb3 to
bc3001f
Compare
thoughtpolice
changed the title
buck2.log_download_url config keybuck2.log_url config key
|
I'd like to make some small API changes to the GET URI so I'm holding this. But I'd still really like something like this to go in :) |
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
4 times, most recently
from
ba03f55 to
505c4c2
Compare
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
2 times, most recently
from
9206ebd to
7e86204
Compare
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
from
7e86204 to
58a82a3
Compare
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
2 times, most recently
from
0949481 to
947d903
Compare
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
from
947d903 to
bee85c8
Compare
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
from
bee85c8 to
e551169
Compare
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
2 times, most recently
from
3ae8424 to
8f9157c
Compare
When doing a command like `buck2 log replay --trace-id ...` it helps to be
able to download previously created logs from users, automated CI, etc. This
functionality exists within Meta, as logs are available for download through
the "Manifest" system, but it isn't usable in OSS, despite being useful for
diagnostics and getting help.
The only missing thing to really get things working is a download mechanism for
log files, and a way to know where they should come from.
With this patch, if a user configures the `buck2.log_url` key, which is
expected to look something like:
[buck2]
log_url = https://example.com
Then, upon executing a `log` command with a `--trace-id` flag, this server will
be queried with a:
GET /v1/get/{uuid}
request, which is expected to return the raw zst-encoded protobuf file. So,
buck2 will do that and download the trace, and then use it like normal.
The request is done with `curl`, though in principle it could be done within
Buck itself.
This shares as much code as possible with the existing infrastructure and tries
to only insert a small key set of `#[cfg(fbcode_build)]` directives. Notably,
the path Meta uses for their "Manifold" client is still fully available in the
OSS version; `fbcode_build` is only used to gate what the default choice is.
How the server gets access to these files and how they are uploaded is another
question. But for now, this can be done several ways outside of buck2 core, so
this is good enough.
GitHub Issue: facebook#441
Signed-off-by: Austin Seipp <[email protected]>
thoughtpolice
force-pushed
the
aseipp/push-xnmzuqpwomuk
branch
from
8f9157c to
7d9952d
Compare
|
@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When doing a command like
buck2 log replay --trace-id ...it helps to be able to download previously created logs from users, automated CI, etc. This functionality exists within Meta, as logs are available for download through the "Manifest" system, but it isn't usable in OSS, despite being useful for diagnostics and getting help.The only missing thing to really get things working is a download mechanism for log files, and a way to know where they should come from.
With this patch, if a user configures the
buck2.log_urlkey, which is expected to look something like:Then, upon executing a
logcommand with a--trace-idflag, this server will be queried with a:request, which is expected to return the raw zst-encoded protobuf file. So, buck2 will do that and download the trace, and then use it like normal.
The request is done with
curl, though in principle it could be done within Buck itself.This shares as much code as possible with the existing infrastructure and tries to only insert a small key set of
#[cfg(fbcode_build)]directives. Notably, the path Meta uses for their "Manifold" client is still fully available in the OSS version;fbcode_buildis only used to gate what the default choice is.How the server gets access to these files and how they are uploaded is another question. But for now, this can be done several ways outside of buck2 core, so this is good enough.
GitHub Issue: #441