Add option to preserve comments when parsing templates

[pawamoy]

Add the preserve_comments parameter to Environment.parse to preserve comments in template ASTs.

Create a new Comment node, and always preserve those in ASTs obtained from Environment.parse.

Fixes #1967.

• Add tests that demonstrate the correct behavior of the change. Tests
  should fail without the change.
• Add or update relevant docs, in the docs folder and in code.
• Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• Add .. versionchanged:: entries in any relevant code docs.

[pawamoy]

note: Possible (small?) performance hit. Not sure how to avoid it.

[pawamoy]

Not sure to understand the readthedocs build failure.

[pawamoy]

suggestion: Could maybe add .. versionadded:: 3.2, is that a valid directive?

[davidism]

The docs failure is unrelated and something I'm actively working on, you can ignore it.

What if we just added these nodes to the AST unconditionally? I'm not sure that would be a breaking change, as an existing walker would skip those nodes. I'd rather not have more configuration if possible.

[pawamoy]

What if we just added these nodes to the AST unconditionally? I'm not sure that would be a breaking change, as an existing walker would skip those nodes. I'd rather not have more configuration if possible.

Totally fine by me! I definitely understand not wanting more configuration. I'll run a few code searches on GitHub to see if this would break anything, and then do the change.

[pawamoy]

This would break jinja2schema: https://github.com/aromanovich/jinja2schema.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/data/dev/jinja/.venv/lib/python3.12/site-packages/jinja2schema/core.py", line 49, in infer_from_ast
    rv = visit(ast, {}, config)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/dev/jinja/.venv/lib/python3.12/site-packages/jinja2schema/visitors/util.py", line 20, in visit
    structure = visit_many(node.body, macroses, config)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/dev/jinja/.venv/lib/python3.12/site-packages/jinja2schema/visitors/util.py", line 37, in visit_many
    structure = visit(node, macroses, config, predicted_struct_cls, return_struct_cls)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/dev/jinja/.venv/lib/python3.12/site-packages/jinja2schema/visitors/util.py", line 14, in visit
    structure = visit_stmt(node, macroses, config)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/dev/jinja/.venv/lib/python3.12/site-packages/jinja2schema/visitors/stmt.py", line 53, in visit_stmt
    raise Exception('stmt visitor for {0} is not found'.format(type(ast)))
Exception: stmt visitor for <class 'jinja2.nodes.Comment'> is not found

It hasn't been updated in 7 years though.

[pawamoy]

Other than that, almost all uses of env.parse I find on GitHub are just to pass the AST to meta.find_undeclared_variables.

[pawamoy]

note: This was needed to fix the logic and two failing tests:

FAILED tests/test_inheritance.py::TestInheritance::test_level1_required - jinja2.exceptions.TemplateSyntaxError: Required blocks can only contain comments or whitespace
FAILED tests/test_inheritance.py::TestInheritance::test_invalid_required - jinja2.exceptions.TemplateSyntaxError: Required blocks can only contain comments or whitespace

[pawamoy]

I've pushed a fixup commit that removes the new parameter and changes the code so that comments are always preserved in ASTs. Makes the overall change much smaller 🙂