ci improvements, update protoc #13876
Conversation
github-actions
bot
added
the
development-process
|
Thanks @Omega359 is it a draft? I dont see tests in checks |
No, but it should be. I can't seem to make it draft but apparently taiki-e/install-action@nextest is not an allowed action in apache repo's. |
…elease 2.46.19 in the case of this hash.
…elease 2.46.19 in the case of this hash.
|
I've reverted the nextest changes for now as the github action to install the binary of it is not approved and building nextest from source would take longer than the time it saves when running tests. Also noted that some of the non-apache/github actions used in datafusion's ci use versions vs githash of a validated version which is against the apache github action policy. For example: We likely should file another ticket to see change all third-party actions to be githash vs version |
|
an uncached build: https://github.com/apache/datafusion/actions/runs/12449465274 |
|
I can not wait to test / review this but I think I will run out of time today -- hopefully either later today or tomorrow |
|
Thanks for the update - no rush on my side |
| @@ -34,5 +34,6 @@ runs: | |||
| echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV | |||
| echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV | |||
| echo "RUST_BACKTRACE=1" >> $GITHUB_ENV | |||
| echo "CARGO_INCREMENTAL=false" >> $GITHUB_ENV | |||
There was a problem hiding this comment.
It seems like this may be redundant with the line below (or if the line below isn't working. perhaps we could remove it 🤔 )
There was a problem hiding this comment.
Probably. I wasn't sure at one point so I put it there to test.
| @@ -288,17 +318,20 @@ jobs: | |||
| mv *.tbl ../datafusion/sqllogictest/test_files/tpch/data | |||
| - name: Verify that benchmark queries return expected results | |||
| run: | | |||
| # increase stack size to fix stack overflow | |||
| export RUST_MIN_STACK=20971520 | |||
There was a problem hiding this comment.
I think we should have fixed this now with the recursive protection feature -- do we still need to set the min stack size?
There was a problem hiding this comment.
I know I needed it when I tried running this the other day. When was that fixed?
|
|
||
| # ci turns off debug info, etc for dependencies to allow for smaller binaries making caching more effective | ||
| [profile.ci.package."*"] | ||
| debug = false |
There was a problem hiding this comment.
We also found this setting particularly effective for speeding up our CI in influxdb_iox: https://doc.rust-lang.org/cargo/reference/profiles.html#debug
|
|
I am going to continue working on the individual jobs - I think for many the only way to improve their performance is either to build with nightly, somehow make caching effective, run on something that is more powerful than my watch, or break them up into multiple jobs. Currently working on getting the hash collision job to run only on merges to main. |
Which issue does this PR close?
Closes #13846, part of #13845
Rationale for this change
Improve ci run time
What changes are included in this PR?
Are these changes tested?
yes
Are there any user-facing changes?
No