Merge branch 'main' into yorickvp/pkgs-fetchurl

.gitignore

@@ -5,4 +5,6 @@
 result
 interpreter
 __pycache__
-*.egg-info
+*.egg-info
+# symlink for devshell
+docs/src/reference

README.md

@@ -13,7 +13,9 @@
   <a href="https://github.com/nix-community/dream2nix/tree/main/examples/packages">Example Packages</a>
 </p>
 
-!!! Warning: dream2nix is unstable software. While simple UX is one of our main focus points, the APIs  are still under development. Do expect changes that will break your setup.
+!!! warning
+
+    dream2nix is unstable software. While simple UX is one of our main focus points, the APIs  are still under development. Do expect changes that will break your setup.
 
 ### legacy dream2nix
 

docs/README.md

@@ -0,0 +1,67 @@
+# About this Documentation
+
+Dream2nix documentation is generated from markdown via 
+[mkdocs](https://www.mkdocs.org/) and [mkdocs-material](https://squidfunk.github.io/mkdocs-material/).
+
+## Build
+
+You can build and server it locally with, i.e.:
+
+``` shellsession
+nix build .#website
+python3 -m http.server -d ./result
+```
+
+# Development shell
+
+Or alternatively run a development environment with:
+
+``` shellsession
+nix develop .#website
+```
+
+Upon entering the devshell, it will change into
+`./docs` and symlink a build of the [options reference](#options-reference)
+into `./docs/src/reference`. 
+
+Normal builds will always use an up-to-date options reference,
+but during development you need to update this symlink yourself
+and remove it after use. 
+
+i.e. from inside the shell in `./docs`:
+
+``` shellsession
+# update
+ln -sfT $(nix build --print-out-paths --no-link ..#optionsReference) ./src/reference
+# remove
+rm ./src/reference
+```
+
+## Options Reference
+
+The reference documentation for [modules](./modules.md) is auto-generated via
+a custom hook in `docs/hooks/render_options.py` and a derivation in
+`.#optionsReference`. 
+
+The derivation includes, for each module, an `options.json` file as generated by nix via `pkgs.nixosOptionsDoc` as well as a `README.md` file, copied from the modules source directory. The existence of such a `README.md` is used as an indicator on whether to include a module in the reference documentation.
+
+The hook runs whenever mkdocs renders one of the `README.md`s. Each of them gets concatenated with header and options reference, after the latter are run through jinja templates in `./docs/theme`.
+
+
+## Notes on Markdown
+
+Mkdocs uses a markdown dialect from [Python-Markdown](https://python-markdown.github.io/) with various, optional extensions listed in `./docs/mkdocs.yml`.
+
+This is different from the CommonMark dialect, as implemented by [markdown-it-py](https://pypi.org/project/markdown-it-py/) and used in NixOS official documentation.
+
+The differences between both don't seem to be too relevant for the markdown features used in our [options reference](#options-reference), but it's good
+to be aware of them when writing longer prose.
+
+[mkdocs-materials reference](https://squidfunk.github.io/mkdocs-material/reference) provides a
+good overview on useful extensions.
+
+## CI
+
+The documentation is published on GitHub pages via a GitHub action, defined in [.github/workflows/pages.yml](https://github.com/nix-community/dream2nix/blob/main/.github/workflows/pages.yml)
+
+

docs/hooks/render_options.py

@@ -0,0 +1,122 @@
+import logging
+import json
+from pathlib import Path
+from collections import OrderedDict
+from typing import Dict, Tuple
+from urllib.request import UnknownHandler
+
+from mkdocs.structure.pages import Page
+from mkdocs.structure.nav import Navigation, Section
+from mkdocs.structure.files import Files
+from mkdocs.config.defaults import MkDocsConfig
+from mkdocs.plugins import event_priority
+
+from pygments import highlight
+from pygments.lexers import get_lexer_by_name
+from pygments.formatters import HtmlFormatter
+
+
+log = logging.getLogger("mkdocs")
+
+
+def is_reference_page(page: Page) -> bool:
+    return page.file.src_path.startswith("reference/")
+
+
+def pygments(code: str, lang: str) -> str:
+    return highlight(code, get_lexer_by_name(lang), HtmlFormatter())
+
+
+def sort_options(item: Tuple[str, Dict], module_name: str) -> int:
+    """
+    Sort the modules. First the one the page is about,
+    then single options, then the rest, alphabetically
+    """
+    name, option = item
+    if name == module_name:
+        return -1
+    elif len(option["children"]) == 0:
+        return 0
+    else:
+        return ord(name[0])
+
+
+def preprocess_options(options, module_name):
+    tree = dict()
+    for name, option in options.items():
+        if name.startswith("_module"):
+            continue
+        cursor = tree
+        parts = name.split(".")
+        for index, part in enumerate(parts):
+            if part not in cursor:
+                if index + 1 == len(parts):
+                    cursor[part] = dict(**option, children={})
+                else:
+                    cursor[part] = dict(children=dict())
+                    cursor = cursor[part]
+            else:
+                cursor = cursor[part]["children"]
+    return OrderedDict(sorted(tree.items(), key=lambda i: sort_options(i, module_name)))
+
+
+def on_page_markdown(
+    markdown: str, page: Page, config: MkDocsConfig, files: Files
+) -> str | None:
+    """Check whether the source path starts with "reference/".
+    If it does:
+    - render a header template, containing values from the source markdown file
+    - render the source markdown file
+    - render an options.json file containing nixos option definitions from the
+      same directory where the source file is found. Then render those options.
+    """
+    if not is_reference_page(page):
+        return markdown
+    src_path = Path(config.docs_dir) / page.file.src_path
+    module_name = src_path.parent.stem
+    env = config.theme.get_env()
+    env.filters["pygments"] = pygments
+
+    header = env.get_template("reference_header.html").render(meta=page.meta)
+
+    options_path = src_path.parent / "options.json"
+    if not options_path.exists():
+        log.error(f"{options_path} does not exist")
+        return None
+    with open(options_path, "r") as f:
+        options = preprocess_options(json.load(f), module_name)
+    reference = env.get_template("reference_options.html").render(options=options)
+
+    return "\n\n".join([header, markdown, reference])
+
+
+@event_priority(-100)
+def on_nav(nav: Navigation, config: MkDocsConfig, files: Files) -> Navigation | None:
+    """Customize the navigation: If a reference section is found,
+    filter for a "state" variable defined in a markdown files front-matter.
+    Leave all items where "state" equals "released" as-is, but put
+    all others in an "experimental" section below that."""
+    try:
+        reference_section = next(filter(lambda i: i.title == "Reference", nav.items))
+        reference_index = nav.items.index(reference_section)
+    except StopIteration:
+        # Return the navigation as-is if we don't find
+        # a reference section
+        return nav
+
+    released = []
+    experimental = []
+    for page in reference_section.children:
+        # to have metadata from the yaml front-matter available
+        page.read_source(config)
+        state = page.meta.get("state")
+        if state == "released":
+            released.append(page)
+        else:
+            experimental.append(page)
+
+    experimental_section = Section("Experimental Modules", experimental)
+    reference_section.children = released + [experimental_section]
+
+    nav.items[reference_index] = reference_section
+    return nav

docs/mkdocs.yml

@@ -0,0 +1,65 @@
+site_name: Dream2Nix
+docs_dir: "src"
+site_url: "https://nix-community.github.io/dream2nix"
+site_dir: !ENV out
+
+repo_url: https://github.com/nix-community/dream2nix
+repo_name: nix-community/dream2nix
+edit_uri: edit/main/docs/src/
+
+plugins:
+  - search
+  - awesome-pages
+
+hooks:
+  - hooks/render_options.py
+
+extra_css:
+  - style.css
+
+markdown_extensions:
+  - toc:
+      permalink: true
+  - tables
+  - admonition
+  - pymdownx.escapeall
+  - pymdownx.highlight:
+      anchor_linenums: true
+  - pymdownx.inlinehilite
+  - pymdownx.snippets
+  - pymdownx.superfences
+  - pymdownx.tasklist
+  - pymdownx.details
+
+validation:
+  omitted_files: warn
+  absolute_links: warn
+  unrecognized_links: warn
+
+theme:
+  custom_dir: theme
+  name: material
+  favicon: favicon.png
+  logo: favicon.png
+  features:
+    - search.suggest
+    - search.highlight
+    - instant
+    - navigation.instant
+    - navigation.instant.prefetch
+    - navigation.instant.progress
+    - navigation.tracking
+    - navigation.path
+    - navigation.top
+    - toc.follow
+    - content.code.copy
+    - content.code.annotate
+
+extra:
+  social:
+    - icon: fontawesome/brands/github
+      link: https://github.com/nix-community/dream2nix
+      name: Dream2nix on Github
+    - icon: fontawesome/solid/comments
+      link: https://matrix.to/#/#dream2nix:nixos.org
+      name: Dream2nix Matrix Channel

docs/src/.pages

@@ -0,0 +1,15 @@
+nav:
+  - "Home": index.md
+  - "Getting Started": gettingstarted.md
+  - Concepts:
+      - "Modules": modules.md
+      - "Overriding Dependencies": overrides.md
+  - Reference: reference
+  - Examples:
+      - "👉 example repository": https://github.com/nix-community/dream2nix/tree/main/examples/repo-with-packages
+      - "👉 example repository using flakes": https://github.com/nix-community/dream2nix/tree/main/examples/repo-with-packages-flake)
+      - "👉 example packages": https://github.com/nix-community/dream2nix/tree/main/examples/packages)
+  - Documentation: docs.md
+  - Notes:
+    - "v1 API design docs": v1-api
+    - development-roundups

docs/src/docs.md

@@ -0,0 +1 @@
+../README.md

website/src/gettingstarted.md → docs/src/gettingstarted.md

@@ -1,5 +1,3 @@
-## Getting started
-
 To package a piece of software with dream2nix, you'd typically start with
 a bare-bones dream2nix flake like this:
 
@@ -60,8 +58,8 @@ And a `default.nix` that looks like this:
 ```
 
 To find out which dream2nix modules to import, browse through the modules
-on the left and the [examples](./examples.md). When getting started, the
-'/packages/languages' collection will be most helpful.
+on the left and the [examples](https://github.com/nix-community/dream2nix/tree/main/examples/packages). When getting started, the
+`./packages/languages` collection will be most helpful.
 
 Once you have imported a module, this module will make ecosystem-dependent
 functions available to create your package definition, such as `mkDerivation`

website/src/modules.md → docs/src/modules.md

@@ -6,9 +6,9 @@ For some more background information, check out the initial exploration of this
 
 @edolstra 's [talk about this topic](https://www.youtube.com/watch?v=dTd499Y31ig) is also worth watching.
 
-# Benefits
+## Benefits
 
-## Deprecate override functions
+### Deprecate override functions
 
 Changing options of packages in nixpkgs can require chaining different override functions like this:
 
@@ -39,7 +39,7 @@ Changing options of packages in nixpkgs can require chaining different override
 
 See htop module definition [here](https://github.com/nix-community/dream2nix/blob/main/examples/packages/basics/htop-with-flags/default.nix).
 
-## Type safety
+### Type safety
 
 The following code in nixpkgs mkDerivation mysteriously skips the patches:
 
@@ -56,7 +56,7 @@ mkDerivation {
 A definition for option `[...].dontPatch' is not of type `boolean' [...]
 ```
 
-## Catch typos
+### Catch typos
 
 The following code in nixpkgs mkDerivation builds **without** openssl_3.
 
@@ -73,7 +73,7 @@ mkDerivation {
 The option `[...].nativBuildInputs' does not exist
 ```
 
-## Environment variables clearly defined
+### Environment variables clearly defined
 
 `dream2nix` requires a clear distinction between known parameters and user-defined variables.
 Defining `SOME_VARIABLE` at the top-level, would raise:
@@ -93,19 +93,19 @@ Instead it has to be defined under `env.`:
 }
 ```
 
-## Documentation / Discoverability
+### Documentation / Discoverability
 
 No more digging the source code to find possible options to override.
 
 Documentation similar to [search.nixos.org](https://search.nixos.org) can be generated for packages declared via `dream2nix`.
 
 Every package built with `dream2nix` has a `.docs` attribute that builds an html documentation describing it's options.
 
-## Package blueprints
+### Package blueprints
 
 With `dream2nix`, packages don't need to be fully declared. Options can be left without defaults, requiring the consumer to complete the definition.
 
-## Flexibility
+### Flexibility
 
 The nixos module system gives maintainers more freedom over how packages are split into modules. Separation of concerns can be implemented more easily.
 For example, the dependency tree of a package set can be factored out into a separate module, allowing for simpler modification.