Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate coverage report for fuzz target functions #4966

Open
jouho opened this issue Dec 10, 2024 · 1 comment
Open

generate coverage report for fuzz target functions #4966

jouho opened this issue Dec 10, 2024 · 1 comment

Comments

@jouho
Copy link
Contributor

jouho commented Dec 10, 2024

Security issue notifications

If you discover a potential security issue in s2n we ask that you notify
AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

#4960 introduces coverage report generated by running fuzz tests. This is an aggregate report that merges multiple coverage reports into one. Therefore the report does show high-level insight like "How many % of s2n-tls code is covered/reached by fuzzer" and "Which areas of code are missing fuzzer coverage".

However, when we write individual fuzz tests, we do not intend to fuzz test the entire repo. Instead, each test has its own "target functions", which we intend to fuzz test. We currently are missing explicit coverage of such "target functions".

Solution:

Generate new report/metric on individual fuzz tested functions. Each fuzz test has a comment at the top of the file to indicate which functions are targeted. For example,

/* Target Functions: s2n_negotiate s2n_flush s2n_handshake_write_io
s2n_handshake_read_io s2n_try_delete_session_cache
s2n_read_full_handshake_message */

Requirements / Acceptance Criteria:

  • metric/report should show how many % of the target function is covered by the fuzz test. This can be extracted from llvm's coverage report.
  • the solution needs to be locally testable so that when we write new fuzz tests, we know target functions are effectively fuzzed
  • metric/report should be stored for future access. scheduled fuzz test will be a good place to perform data upload
  • both old & new fuzz tests need to follow the convention to include target functions in a format so they can be easily parsed when identifying list of target functions
@jouho
Copy link
Contributor Author

jouho commented Dec 10, 2024

It seems there's an existing code that extracts the coverage statistics for target functions:

if [[ ! -z "$TARGET_FUNCS" ]];
then
for TARGET in ${TARGET_FUNCS}
do
TARGET_TOTAL+=`sed -n "s/^.*${TARGET} .*% *\([0-9]*\) .*$/\1/p" ${COVERAGE_DIR}/fuzz/${TEST_NAME}_cov.txt`
TARGET_COV+=`sed -n "s/^.*${TARGET} .*% *[0-9]* *\([0-9]*\) .*$/\1/p" ${COVERAGE_DIR}/fuzz/${TEST_NAME}_cov.txt`
done
fi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants