Skip to content

Commit

Permalink
Improvements to benchmark tests (AcademySoftwareFoundation#2030)
Browse files Browse the repository at this point in the history
- Update shaderGenPerformanceTest to use improved MaterialX API workflows.
- Add a readme update to cover Benchmark tests.
  • Loading branch information
ashwinbhat authored Sep 27, 2024
1 parent e6b9650 commit 8fef28d
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 4 deletions.
8 changes: 4 additions & 4 deletions source/MaterialXTest/MaterialXGenShader/GenShaderUtil.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -316,8 +316,7 @@ void testUniqueNames(mx::GenContext& context, const std::string& stage)
void shaderGenPerformanceTest(mx::GenContext& context)
{
mx::DocumentPtr nodeLibrary = mx::createDocument();
mx::FilePath currentPath = mx::FilePath::getCurrentPath();
const mx::FileSearchPath libSearchPath(currentPath);
const mx::FileSearchPath libSearchPath(mx::getDefaultDataSearchPath());

// Load the standard libraries.
loadLibraries({ "libraries" }, libSearchPath, nodeLibrary);
Expand Down Expand Up @@ -351,8 +350,9 @@ void shaderGenPerformanceTest(mx::GenContext& context)
}

// Read mtlx documents
mx::FileSearchPath searchPath = mx::getDefaultDataSearchPath();
mx::FilePathVec testRootPaths;
testRootPaths.push_back("resources/Materials/Examples/StandardSurface");
testRootPaths.push_back(searchPath.find("resources/Materials/Examples/StandardSurface"));

std::vector<mx::DocumentPtr> loadedDocuments;
mx::StringVec documentsPaths;
Expand All @@ -367,7 +367,7 @@ void shaderGenPerformanceTest(mx::GenContext& context)
REQUIRE(loadedDocuments.size() > 0);
REQUIRE(loadedDocuments.size() == documentsPaths.size());

// Shuffle the order of documents and perform document library import validatation and shadergen
// Shuffle the order of documents and perform document library import validation and shadergen
std::mt19937 rng(0);
std::shuffle(loadedDocuments.begin(), loadedDocuments.end(), rng);
for (const auto& doc : loadedDocuments)
Expand Down
10 changes: 10 additions & 0 deletions source/MaterialXTest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,13 @@ When rendering tests are enabled through the `MATERIALX_TEST_RENDER` option, the
#### HTML Render Comparisons
- A `tests_to_html` Python script is provided in the [`python/MaterialXTest`](../../python/MaterialXTest) folder, which can be run to generate an HTML file comparing the rendered results in each shading language.
- Example render comparisons may be found in [commits to the MaterialX repository](https://github.com/AcademySoftwareFoundation/MaterialX/pull/1164), and we encourage developers to post their own results when making changes that have the potential to impact generated shaders.

#### Benchmark Tests

Processing MaterialX documents and generating shaders is an integral part of many pipelines. MaterialX benchmark testing using [Catch2 benchmark framework](https://github.com/catchorg/Catch2/blob/devel/docs/benchmarks.md) and can be enabled using `MATERIALX_BUILD_BENCHMARK_TESTS` CMake macro.

Here is an example of how to run the benchmark test

`MaterialXTest.exe "GenShader: GLSL Performance Test" --benchmark-samples 10`

This will iterate and gather 10 samples of the test case and report low, mean and high timing results.

0 comments on commit 8fef28d

Please sign in to comment.