Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generating Python API docs #2038

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

StefanHabel
Copy link
Contributor

@StefanHabel StefanHabel commented Sep 30, 2024

This PR adds support for generating Python API documentation in HTML format using Sphinx from the MaterialX Python modules that are built in the lib/ directory.

A new CMake build option named MATERIALX_BUILD_PYTHON_DOCS allows developers to turn generating Python API documentation on.

When MATERIALX_BUILD_PYTHON_DOCS is set to ON, MATERIALX_BUILD_PYTHON is set to ON as well, ensuring we have Python modules for which to build the Python API docs.

The core functionality of generating Python API documentation lives in a new directory named documents/PythonAPI/. It is controlled with a new CMakeLists.txt file in that directory, which defines a new target named MaterialXDocsPython, similar to the existing target MaterialXDocs that generates API documentation for the MaterialX C++ API.

Revised docstrings for modules were recently added in the following separate PR:

Revised docstrings for classes, methods, and functions are to be added in subsequent PRs separately.

Documentation in markdown format from the existing DeveloperGuide is integrated into the new Python API documentation by way of symlinking the four main .md files into the documents/PythonAPI/sources/ directory from which Sphinx generates the resulting HTML documentation.

To build the docs from scratch on macOS, I've used the following build script, naming it build.sh in the MaterialX checkout directory:

#!/usr/bin/env tcsh

echo build.sh: Updating Git submodules...
git submodule update --init --recursive

# Create and activate a virtual environment for installing Python dependencies
python3 -m venv /tmp/venv
source /tmp/venv/bin/activate.csh

echo build.sh: Installing dependencies...
python3 -m pip install myst_parser  # https://pypi.org/project/myst-parser/

echo build.sh: Making build directory and changing into it...
mkdir build
cd build

echo build.sh: Configuring...
cmake .. \
    --fresh \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX15.0.sdk \
    -DMATERIALX_BUILD_PYTHON=ON \
    -DMATERIALX_BUILD_VIEWER=ON \
    -DMATERIALX_BUILD_GRAPH_EDITOR=ON \
    -DMATERIALX_BUILD_DOCS=ON \
    -DMATERIALX_BUILD_PYTHON_DOCS=ON \
    -DMATERIALX_BUILD_TESTS=ON \
&& \
echo build.sh: Building... \
&& \
cmake --build . -j 8 \
&& \
echo build.sh: Building target MaterialXDocs... \
&& \
cmake --build . --target MaterialXDocs \
&& \
echo build.sh: Building target MaterialXDocsPython... \
&& \
cmake --build . --target MaterialXDocsPython \
&& \
afplay /System/Library/Sounds/Blow.aiff

# Deactivate the virtual environment
deactivate

The build output currently ends with the following messages:

The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 139 classes
    * 59 attributes
    * 1175 methods
    * 6 exception types

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1019 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 499 methods look like their parameters have not all been named using `py::arg()`.
build succeeded, 4 warnings.

The HTML pages are in ..
[100%] Built target MaterialXDocsPython

Screenshot showing the new Python API documentation main page:

Screenshot 2024-09-29 at 17 56 22

Screenshot showing the modified Viewer.md contents from #2037 in the context of the Python API documentation:

Screenshot 2024-09-30 at 16 04 14

Screenshot showing the (currently) incomplete PyMaterialXCore docs (to be amended in subsequent PRs):

Screenshot 2024-10-06 at 13 11 00

Split from #1567.

Update #342.

StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds a docstring in markdown format to the `PyMaterialXCore` module.

The docstring is defined in a macro named `PyMaterialXCore_DOCSTRING` that
uses the new `PYMATERIALX_DOCSTRING` macro to allow us to place the first
and last lines in lines by themselves, rather than starting the docstring
contents immediately after the `R"docstring(` marker.

The `PyMaterialXCore_DOCSTRING` macro is placed in a separate header file
named `__doc__.md.h` which lives side-by-side with the `PyModule.cpp` file
in which it is included.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <[email protected]>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds a docstring in markdown format to the `PyMaterialXFormat` module.

The docstring is defined in a macro named `PyMaterialXFormat_DOCSTRING`
that uses the new `PYMATERIALX_DOCSTRING` macro that is introduced in AcademySoftwareFoundation#2038
to allow us to place the first and last lines in lines by themselves,
rather than starting the docstring contents immediately after the
`R"docstring(` marker.

The `PyMaterialXFormat_DOCSTRING` macro is placed in a separate header file
named `__doc__.md.h` which lives side-by-side with the `PyModule.cpp` file
in which it is included.

Note that the docstrings for individual classes, methods, and functions are
to be added in separate PRs.

Note that this PR also renames `readFromXmlFileBase()` to `readFromXmlFile()`
allowing us to remove the alias from `python/MaterialX/main.py`.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <[email protected]>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds docstrings in markdown format to the following modules:
- `PyMaterialXGenGlsl`
- `PyMaterialXGenMdl`
- `PyMaterialXGenMsl`
- `PyMaterialXGenOsl`
- `PyMaterialXGenShader`

The docstrings are defined in macros named `PyMaterialXGen*_DOCSTRING` that
use the new `PYMATERIALX_DOCSTRING` macro that is introduced in AcademySoftwareFoundation#2038 to
allow us to place the first and last lines in lines by themselves, rather
than starting the docstring contents immediately after the `R"docstring(`
marker.

The `PyMaterialXGen*_DOCSTRING` macros are placed in separate header files
named `__doc__.md.h` which live side-by-side with their corresponding
`PyModule.cpp` file in which it is included.

Note that the docstrings for individual classes, methods, and functions are
to be added in separate PRs.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <[email protected]>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds docstrings in markdown format to the following modules:
- `PyMaterialXRender`
- `PyMaterialXRenderGlsl`
- `PyMaterialXRenderMsl`
- `PyMaterialXRenderOsl`

The docstrings are defined in macros named `PyMaterialXRender*_DOCSTRING`
that use the new `PYMATERIALX_DOCSTRING` macro that is introduced in AcademySoftwareFoundation#2038
to allow us to place the first and last lines in lines by themselves,
rather than starting the docstring contents immediately after the
`R"docstring(` marker.

The `PyMaterialXRender*_DOCSTRING` macros are placed in separate header
files named `__doc__.md.h` which live side-by-side with their corresponding
`PyModule.cpp` files in which they are included.

Note that the docstrings for individual classes, methods, and functions are
to be added in separate PRs.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <[email protected]>
@StefanHabel
Copy link
Contributor Author

Demo of generated MaterialX Python API Documentation now available here: https://stefanhabel.github.io

Source repo for the live demo available here: https://github.com/StefanHabel/StefanHabel.github.io

This PR adds support for generating Python API documentation in HTML format
using Sphinx from the MaterialX Python that are built in the `lib/`
directory.

A new CMake build option named `MATERIALX_BUILD_PYTHON_DOCS` allows
developers to turn generating Python API documentation on.

When `MATERIALX_BUILD_PYTHON_DOCS` is set to `ON`, `MATERIALX_BUILD_PYTHON`
is set to `ON` as well, ensuring we have Python modules for which to build
the Python API docs.

The core functionality of generating Python API documentation lives in a
new directory named `documents/PythonAPI/`. It is controlled with a new
`CMakeLists.txt` file in that directory, which defines a new target named
`MaterialXDocsPython`, similar to the existing target `MaterialXDocs` that
generates API documentation for the MaterialX C++ API.

To facilitate the curation and addition of docstrings in the implementation
files within `source/PyMaterialX/`, this PR adds a new helper macro named
`PYMATERIALX_DOCSTRING` that allows developers of Python modules to define
docstrings using the following pattern:
```cpp
PYMATERIALX_DOCSTRING(R"docstring(
...markdown text here...
)docstring");
```

Revised docstrings for modules and classes are to be added in subsequent PRs
separately.

Documentation in markdown format from the existing `DeveloperGuide` is
integrated into the new Python API documentation by way of symlinking the
four main `.md` files into the `documents/PythonAPI/sources/` directory
from which Sphinx generates the resulting HTML documentation.

Warnings that are issued when generating the documentation via Sphinx are
to be addressed in a separate PR for the markdown files:
AcademySoftwareFoundation#2037

To build the docs from scratch on macOS, I've used the following build
script, naming it `build.sh` in the `MaterialX` checkout directory:
```bash

echo build.sh: Updating Git submodules...
git submodule update --init --recursive

python3 -m venv /tmp/venv
source /tmp/venv/bin/activate.csh

echo build.sh: Installing dependencies...
python3 -m pip install myst_parser  # https://pypi.org/project/myst-parser/

echo build.sh: Making build directory and changing into it...
mkdir build
cd build

echo build.sh: Configuring...
cmake .. \
    --fresh \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX15.0.sdk \
    -DMATERIALX_BUILD_PYTHON=ON \
    -DMATERIALX_BUILD_VIEWER=ON \
    -DMATERIALX_BUILD_GRAPH_EDITOR=ON \
    -DMATERIALX_BUILD_DOCS=ON \
    -DMATERIALX_BUILD_PYTHON_DOCS=ON \
    -DMATERIALX_BUILD_TESTS=ON \
&& \
echo build.sh: Building... \
&& \
cmake --build . -j 8 \
&& \
echo build.sh: Building target MaterialXDocs... \
&& \
cmake --build . --target MaterialXDocs \
&& \
echo build.sh: Building target MaterialXDocsPython... \
&& \
cmake --build . --target MaterialXDocsPython \
&& \
afplay /System/Library/Sounds/Blow.aiff

deactivate
```
The build output currently ends with the following messages:
```python
The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 139 classes
    * 1175 methods
    * 6 exception types

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1019 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 499 methods look like their parameters have not all been named using `py::arg()`.
build succeeded, 168 warnings.

The HTML pages are in ..
[100%] Built target MaterialXDocsPython
```

Split from AcademySoftwareFoundation#1567.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <[email protected]>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Oct 3, 2024
This PR adds a docstring in markdown format to the `PyMaterialXCore` module.

The docstring is defined in a macro named `PyMaterialXCore_DOCSTRING` that
uses the new `PYMATERIALX_DOCSTRING` macro to allow us to place the first
and last lines in lines by themselves, rather than starting the docstring
contents immediately after the `R"docstring(` marker.

The `PyMaterialXCore_DOCSTRING` macro is placed in a separate header file
named `__doc__.md.h` which lives side-by-side with the `PyModule.cpp` file
in which it is included.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <[email protected]>
We may not need the proposed `PYMATERIALX_DOCSTRING` macro.

Signed-off-by: Stefan Habel <[email protected]>
@StefanHabel
Copy link
Contributor Author

I've rebased the branch of this PR on top of main, and have added another commit to revert the changes in PyMaterialX.h: d76eb5a

This follows a discussion on #2039 in which the proposed new PYMATERIALX_DOCSTRING was first used.

The changes in this PR now stand as +1,174 −0.

@jstone-lucasfilm jstone-lucasfilm changed the title Generating Python API docs. Generating Python API docs Oct 3, 2024
@StefanHabel
Copy link
Contributor Author

I've added a couple of commits to replace the previous Alphabetical Index on a module's documentation page, which did not provide links to module functions, with alphabetical listings of Classes, Exception Types, and Module Functions, each linking to either a separate documentation page or to a section on the same page:

Before After
Screenshot 2024-09-29 at 17 57 32 Screenshot 2024-10-06 at 13 11 00

The only warnings that are now being displayed by Sphinx at the end of building the Python API documentation are those that we add ourselves to highlight missing documentation:

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1019 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 499 methods look like their parameters have not all been named using `py::arg()`.
-build succeeded, 149 warnings.
+build succeeded, 4 warnings.

I've updated the generated HTML pages in the live demo here: https://stefanhabel.github.io/

Example of a class with attribute names and other properties:
https://stefanhabel.github.io/generated/PyMaterialXCore.Backdrop.html#PyMaterialXCore.Backdrop

The summary of the MaterialX Python API is now stated as follows:

The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 139 classes
    * 59 attributes
    * 1175 methods
    * 6 exception types

We can add the missing documentation in subsequent separate PRs dedicated to individual modules and classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants