Woptim/revamp documentation

.readthedocs.yml

@@ -7,7 +7,7 @@ version: 2
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
-  configuration: docs/sphinx/conf.py
+  configuration: docs/conf.py
 
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all

docs/index.rst

@@ -0,0 +1,75 @@
+.. ## Copyright (c) 2019-2022, Lawrence Livermore National Security, LLC and
+.. ## other RADIUSS Project Developers. See the top-level COPYRIGHT file for details.
+.. ##
+.. ## SPDX-License-Identifier: (MIT)
+.. ##
+
+##########
+RADIUSS CI
+##########
+
+Documentation of the CI infrastructure developed for RADIUSS projects.
+
+RADIUSS CI is a sub-project from the RADIUSS initiative focusing on sharing
+resource and documentation regarding Continuous Integration among RADIUSS
+projects.
+
+.. note::
+   LLNL's RADIUSS project (Rapid Application Development via an Institutional
+   Universal Software Stack) aims to broaden usage across LLNL and the open source
+   community of a set of libraries and tools used for HPC scientific application
+   development.
+
+=========================
+Background and Motivation
+=========================
+
+Projects belonging to the RADIUSS scope are targeting the same machines and
+use Spack as a packaging system. We want them to ensure they build with
+similar tool chains.
+
+We designed an automated *CI infrastructure based on GitLab* that we meant to be
+universal enough to be shared among RADIUSS projects. This infrastructure
+involves *using Spack to setup the project dependencies and generate a
+configuration file*. This allows projects to easily *share the full context of
+their builds*. The project is then built and tested as usual and most of *the
+CI infrastructure is shared* to avoid duplication and ease the maintenance.
+
+========
+Overview
+========
+
+We split the design in three steps necessary to adopt RADIUSS CI methodology.
+Those actions will be documented in the `user_guide`_.
+
+1. **Use Spack to configure the project build.**
+   Spack provides a single context to express *toolchains*, *machines
+   setup* and *build sequence*. It is increasingly used to install the
+   dependency tree of large simulation codes.
+2. **Build and test without breaking your habits.**
+   We do not require the adoption of Spack to build your code but we require
+   that your build system accepts the configuration file generated by Spack as
+   an input (``CMakeCache.txt`` for CMake build system). That way, dependencies
+   and options are already set coherently with the spec provided to build the
+   dependency tree.
+3. **Setup the CI using the shared template.**
+   Once you have put in the effort to adopt the first two steps, you should be
+   able to benefit from the shared CI infrastructure. In very complex scenario,
+   you would still be able to use the RADIUSS CI template as a starting point
+   for a custom implementation.
+
+In the `dev_guide`_, we discuss the layout of the RADIUSS CI infrastructure
+and how the different pieces work with one another. Technical choices are also
+explained there.
+
+.. toctree::
+   :hidden:
+   :caption: User Documentation
+
+   sphinx/user_guide/index
+
+.. toctree::
+   :hidden:
+   :caption: Developer Documentation
+
+   sphinx/dev_guide/index

docs/sphinx/dev_guide/how_to.rst

@@ -0,0 +1,87 @@
+.. ##
+.. ## Copyright (c) 2022, Lawrence Livermore National Security, LLC and
+.. ## other RADIUSS Project Developers. See the top-level COPYRIGHT file for details.
+.. ##
+.. ## SPDX-License-Identifier: (MIT)
+.. ##
+
+.. _dev_how_to-label:
+
+******
+How To
+******
+
+=================
+Add a new machine
+=================
+
+Adding a new machine can be done directly in this project so that the
+configuration is shared with all. However it the associated Spack configuration
+must first be added to `Radiuss-Spack-Configs`_.
+
+The sub-pipeline definition
+===========================
+
+To add a new machine, create a corresponding ``<machine>-build-and-test.yml``
+file describing the sub-pipeline. There are two main cases: whether the machine
+uses Slurm or LSF Spectrum as a scheduler.
+
+Machines using Slurm scheduler
+------------------------------
+
+For machines using Slurm scheduler, use ``ruby`` (or ``corona``) as a starting
+point. Then replace all the instances of "RUBY" and "ruby" with the new machine
+name.
+
+Then go to ``customization/custom-variables.yml`` and add the variables:
+
+* ``<MACHINE>_BUILD_AND_TEST_SHARED_ALLOC`` with allocation options sufficient
+  for the shared allocation (salloc) to contain all the jobs.
+* ``<MACHINE>_BUILD_AND_TEST_JOB_ALLOC`` with allocation options for any of the
+  jobs (srun) the machine will take. The job will run under the shared
+  allocation, also, do not prescribe a number of cores here, as they should be
+  defined at Spack and Make/CMake level.
+
+.. note::
+   Use the values we have for ruby and corona as guides, but adapt the
+   partition, number of cpus per task and duration coherently with the machine.
+
+Machines using LSF Spectrum scheduler
+-------------------------------------
+
+For machines using LSF Spectrum scheduler, use ``lassen`` as a starting point.
+Then replace all the instances of "LASSEN" and "lassen" with the new machine
+name.
+
+Then go to ``customization/custom-variables.yml`` and add the variables:
+
+* ``<MACHINE>_BUILD_AND_TEST_JOB_ALLOC`` with allocation options for any of the
+  jobs the machine will take.
+
+.. note::
+   Use the values we have for lassen as guides, but adapt the partition and
+   duration coherently with the machine.
+
+Reference the new sub-pipeline
+==============================
+
+In ``customization/custom-pipelines.yml``, add a new section corresponding to
+the new machine. This is used by ``customization/gitlab-ci.yml`` to control
+which sub-pipelines are effectively generated.
+
+Changelog
+=========
+
+Don’t forget to provide a quick description of your changes in the
+``CHANGELOG.md``.
+
+New tag
+=======
+
+Once the new machine setup is tested and valid, submit a PR. We will review it
+and merge it. We may create a new tag although it is not required for a new
+machine. Indeed, using a new machine is a voluntary change for users: they will
+have to activate it in ``customization/custom-pipelines.yml`` the same way you
+did above (which is a suggested template).
+
+.. _Radiuss-Spack-Configs: https://github.com/LLNL/radiuss-spack-configs

docs/sphinx/dev_guide/index.rst

@@ -0,0 +1,27 @@
+.. ##
+.. ## Copyright (c) 2022, Lawrence Livermore National Security, LLC and
+.. ## other RADIUSS Project Developers. See the top-level COPYRIGHT file for details.
+.. ##
+.. ## SPDX-License-Identifier: (MIT)
+.. ##
+
+.. _dev_guide-label:
+
+###############
+Developer Guide
+###############
+
+There should be two types of contributions to this Radiuss-Shared-CI: adding
+new shared jobs, contributing changes to the CI implementation (and
+documentation).
+
+We provide guidance for both in this section, using an HowTo format where we
+describe the set of actions to perform for several practical use-cases.
+
+This section of the documentation also explains some technical choice.
+
+.. toctree::
+   :maxdepth: 2
+
+   how_to
+   radiuss_ci_explained

docs/sphinx/dev_guide/radiuss_ci_explained.rst

@@ -0,0 +1,81 @@
+.. ##
+.. ## Copyright (c) 2022, Lawrence Livermore National Security, LLC and
+.. ## other RADIUSS Project Developers. See the top-level COPYRIGHT file for details.
+.. ##
+.. ## SPDX-License-Identifier: (MIT)
+.. ##
+
+.. _radiuss_ci_explained-label:
+
+***************************
+RADIUSS Shared CI explained
+***************************
+
+Radiuss-Shared-CI is an infrastructure and documentation repository created to
+help RADIUSS projects adopt the Gitlab CI workflow designed for them.
+
+=================
+Project structure
+=================
+
+Shared CI files
+===============
+
+This project hosts the shared CI configuration, which can be found in
+the YAML files at the root of the project: ``<machine>-build-and-test.yml``.
+
+Each file contains both the configuration and the jobs for one machine. They
+assume that some entries will be provided by including the customization files.
+
+Customization files
+===================
+
+The ``customization`` directory holds all the files allowing to customize the
+pipeline.
+
+The files ``custom-pipelines.yml``, ``custom-variables`` and
+``custom-jobs.yml`` are all included in the configuration in ``gitlab-ci.yml``.
+
+Project must use ``gitlab-ci.yml`` as a base for the ``.gitlab-ci.yml`` at the
+root of their project. This files does not require any change, but can receive
+additional stages if needed by the project.
+
+.. note::
+   We could share ``.gitlab-ci.yml`` directly in Radiuss-Shared-CI. However
+   that would require projects to configure GitLab, through the UI, to use that
+   file. This is not considered a good idea at the moment, and we preffer
+   projects to have the capability to easily add other stages to their CI
+   configuration.
+
+Both ``custom-pipelines.yml`` and ``custom-variablse.yml`` are included
+globally. They will affect all the CI workflow. The file ``custom-jobs.yml`` is
+included in the ``build-and-test`` sub-pipelines and will only affect those
+ones.
+
+Extra jobs
+==========
+
+Extra jobs can be defined by the user appending
+``.gitlab/${CI_MACHINE}-build-and-test-extra.yml``. We provide a working
+minimal template that should always be provided.
+
+.. note::
+   Strictly speaking, the extra jobs file is only needed once the associated
+   sub-pipeline is reference to ``.gitlab-ci.yml`` through
+   ``customization/custom-pipelines.yml``. We make it presence mandatory for
+   the sake of simplicity.
+
+Other files
+=============
+
+The documentation source code is in the ``docs`` directory, while ``cmake``
+aims at receiving BLT submodule to manage the local build of the docs.
+
+==========
+References
+==========
+
+`LC specific documentation for Gitlab <https://gitlab.llnl.gov>`_. In
+particular, the "Getting Started" and "Setup Mirroring" sub-pages.
+
+