[RFC] Simulate custom AST elements in Lua

[tarleb]

Quarto has custom AST nodes, and I ventured to add support for this to pandoc proper. I wrote an experimental package, working title castiron, that allows to define custom AST blocks and have them handled (mostly) as if they were native pandoc Block elements.

I'd love to get feedback and hear your thoughts on the whole idea. Is this useful outside of big applications like Quarto? Would it make sense to make functionality like this a part of pandoc Lua at some point?

Please note that the package requires a current nightly build of pandoc for it to work correctly.

[tarleb]

I should probably explain that one of the primary use-cases for custom elements is to attach pure Lua objects to nodes: The pandoc-internal representation can contain some kind of object reference, and the full value can be re-attached once the custom placeholder node is returned to Lua-land.

[massifrg]

I looked at the example in the project README file.

There you define an Example custom Block that is a synonym of a Div that has only one class: "Example".

I guess you are going to use Span (with a class) for a custom Inline.

Div and Span are the best candidates for custom elements, since they produce "transparent" containers of blocks or inlines that are unlikely to disrupt any output format. Moreover, they carry arbitrary data in their Attr.

So it would be:

• "Foo" custom Block = Div + "Foo" class

• "Bar" custom Inline = Span + "Bar" class

I already use that convention to support indexes (Div.index), index terms (Div.index-term) and and references (empty Spans).

So, instead of writing:

local filter = {
  Div = function(div)
    if div.classes == List{'Index'} then
      return filterIndex(div)
    elseif div.classes == List{'IndexTerm'} then
      return filterIndexTerm(div)
    else
       -- real Divs
    end
  end,
  Span = function(span)
    if span.classes == List{'IndexRef'} then
      return filterIndexRef(span)
    else
      -- real Spans
    end
  end,
}

you'd write:

local filter = {
  Index = filterIndex,
  IndexTerm = filterIndexTerm,
  Div = function(div)
    -- ...
  end,
  IndexRef = filterIndexRef,
  Span = function(span)
    -- ...
  end,
}

Those custom elements could also provide default conversions to certain output formats, maybe in terms of RawBlock and RawInline elements. In analogy with __toblock, it could be something like this:

Example:__toformat = {
  html = function(example)
    local l = List{RawInline('html','<example>')}
    l:extend(example.content)
    l:insert(RawInline('html','</example>'))
    return l
  end,
}

IndexRef:__toformat = {
  latex = function(indexref)
    local l = List{RawInline('latex','\\index{')}
    l:extend(indexref.content)
    l:insert(RawInline('latex','}'))
    return l
  end,
  context = function(indexref)
    local l
    if indexref.indexName then
      l = List{RawInline('context','\\register[' .. indexref.indexName .. ']{'
    else
      l = List{RawInline('context','\\index{')}
    end
    l:extend(indexref.content)
    l:insert(RawInline('context','}'))
    return l
  end,
}

Unfortunately, many custom elements would require changes in other parts of a document, or in a different item (file) of the output file (package), as in DOCX and ODT.
Think, for example, of end notes support: in the .docx package, they are in a different file (endnotes.xml, if I remember well), not in the main document.xml file; AFAIK filters are limited to operate inside that document.xml.

[badumont]

It looks great!

Div and Span are the best candidates for custom elements, since they produce "transparent" containers of blocks or inlines that are unlikely to disrupt any output format. Moreover, they carry arbitrary data in their Attr.

The main difference, I think, is that Divs and Spans can carry only one List of Blocks/Inlines. Attributes are plain strings. A custom element could have an arbitrary number of fields containing a List of Blocks or Inlines. I guess that this feature would only make sense used in conjunction with a custom reader that can parse the syntactic construct corresponding to the new object?

[tarleb]

Happy to hear :)

I guess that this feature would only make sense used in conjunction with a custom reader that can parse the syntactic construct corresponding to the new object?

Taking Quarto as an example again, I'd say that some new conventions can be enough. E.g., Quarto Callouts use default Markdown syntax, but it makes sense to treat them as separate Block elements.

I fully agree that the feature is probably most useful when more parts of the transformation chain are customized, and I like the __toformat proposal by @massifrg for that reason.