Proposal: a new general purpose Block extension

[waylan]

I haven't solidified the proposal yet, but I am envisioning something like a fenced code block (except its not for code and would use different delimitators) which might support two different block types:

1. Element Block: The user would define the HTML element (tag) and any attributes. This would be simpler to deal with (read, write and parse) that md_in_html as it avoids lots of HTML tags in your Markdown.
2. Templated Block: The user would define the block 'type' and any attributes. So long as a template exists for the 'type', then the content is inserted into the template after being parsed as Markdown.

Therefore, if the user provided div (for an element block), the content is parsed as Markdown and then wrapped in a div element with any additional attributes being set on the div. However, if the user provides a 'type' of admonition, then the admonition template is used. Presumably the template would expect a set of attributes (including the admonition type), a title and a body and insert those values into the HTML template. Other templates might accept other options.

This would also allow users to use the CSS provided by whichever CSS framework they are using. For example, the MkDocs default theme is using Bootstrap, which provides it own set of alerts. However, MkDocs doesn't use them, but instead also provides CSS for Rst style admonitions because that is what the admonition extension expects. With a new block extension, a Bootstrap alert template could allow Bootstrap's CSS to be used along with all of Bootstrap's alert features (icons, dismissal, etc) removing the need for the MkDocs theme to also include the Rst based CSS.

A few additional things to note:

I would prefer to not add any new extensions to the core library. So I would expect this to be developed in a separate repo. However, I mention it here because it could effect how we proceed with #1174. Also, I would appreciate any feedback on the idea and/or input on a possible syntax proposal.

[waylan]

My initial proposal suggests that these blocks are different from fenced code blocks, but another option would be to use the fenced code block delimitators and also provide for a 'code' type. In our implementation, a user could provide their own 'code' template which might allow them to provide their own custom syntax highlighting solution. If we went this way, the user would simply use this extension instead of the fenced_code extension, avoiding any conflicts.

I am undecided which way to go on this. Do we overload fenced code blocks, or create a completely new and different fenced block? Similar approaches (overloading fenced code blocks) have been taken by other implementations. For example r-markdown uses fenced code blocks to define the r code which is executed with its output being included in the rendered document. I have also seen at least one Markdown-to-reStructuredText bridge which used overloaded fenced code blocks to replicate directives (sorry, I don't recall which one). To my mind, the primary benefit of overloading fenced code blocks is that when a document using the blocks are parsed by other Markdown implementations, the block is simply parsed as a plain code block. In contrast, introducing a completely new syntax could result in some weird output from other implementations. The only reason the existing admonitions extension mostly avoids this is because the content is indented and becomes a code block. But I would prefer to not indent the content of the new block.

[facelessuser]

In general, when I went through the pain of getting fenced blocks to work under lists and such via SuperFences, I just made it so that users could create custom fences to generate whatever kind of block they wanted. With that said, it still doesn't run inline with normal block processors, so it has some limitations as the content is stored under placeholders like code.

I'd love an approach that uses the normal block processors, and if it is possible, I'd love to see how that would work.

1. The way Python Markdown currently normalizes line breaks and such between blocks is frustrating for code blocks as any code blocks with multiple empty lines always get normalized to one.
2. Even in Admontions there is still a degree of complexity involved with appending additional blocks as children and accounting for lists and such. I think this is partly due to how Python Markdown consumes blocks.

Anyways, I'm at least interested to see how this may potentially work.

[waylan]

@facelessuser you make some valid points. I have not looked too closely at how you implemented superfences, and was curious how you worked around the problems with the existing block parser. Sounds like you didn't. It may not make sense to even attempt this until after we refactor how blocks are parsed. However, I'm not really interested in tackling that problem right now. Mostly I am simply trying to get my ideas recorded at this point. I haven't given much thought to how it might be implemented.

[mitya57]

See also Pandoc Markdown's Divs and Spans https://pandoc.org/MANUAL.html#divs-and-spans which have a bit different but similar idea.

[facelessuser]

@facelessuser you make some valid points. I have not looked too closely at how you implemented superfences, and was curious how you worked around the problems with the existing block parser. Sounds like you didn't. It may not make sense to even attempt this until after we refactor how blocks are parsed. However, I'm not really interested in tackling that problem right now. Mostly I am simply trying to get my ideas recorded at this point. I haven't given much thought to how it might be implemented.

Yeah, I accomplish everything through the preprocessor still. It's a bit hacky, but it was the only way to avoid the above issues. Anyways, I think this is a good idea, and generally, if/when block parsers are able to handle blocks better, this will be a good addition on top of it.

[alkisg]

As a user, I think this is a wonderful idea!
I'm currently migrating a mediawiki site to git/markdown. For admonitions, the pandoc mediawiki→markdown converter produced ::: warning, which is the same that markdown-it, docusaurus and some others use. It'll take time to update hundreds of wiki pages to !!! warning and indented content that mkdocs uses. Also, if I ever need to switch away from mkdocs in the future, I'll have to convert the content once more.
The ::: class (and the ::: tag) ideas are so powerful, that will probably make !!! admonitions obsolete if they're implemented! I hope we can see this in production soon! :)

[waylan]

Just stumbled upon MyST, a Commonmark parser which is intended as a reStructuredText replacement. I've seen many before, but this is the first one where I like the directives syntax. It is very similar to what I was trying to accomplish here. See also, their optional colon_fence extension, which looks similar to (but not the same as) Pandoc's syntax.

Also of interest is their discussion of various existing proposals and implementations out in the wild and why they chose what they did.

[facelessuser]

Just stumbled upon MyST, a Commonmark parser which is intended as a reStructuredText replacement. I've seen many before, but this is the first one where I like the directives syntax. It is very similar to what I was trying to accomplish here. See also, their optional colon_fence extension, which looks similar to (but not the same as) Pandoc's syntax.

Unfortunately, us doing code this way, without a huge overhaul of the parser to get block handling where we want, may not be possible. And I definitely wouldn't try to use the ```{directive} format unless the block handling was refactored as the only real way to properly preserve code is using the PreProcessor, and that syntax will conflict with fenced code (at least as things are currently).

But, if you don't need to preserve empty lines and such for code blocks and you just want to handle normal Markdown content, it may be possible to cook something up now. I've been considering possibly tinkering with a way to pull this off now that I've had some time to think about it, at least the logic in regards to the handling of the "fences". Assuming some sort of prototype could be pulled off, we'd at least have something to work with now.

[facelessuser]

So, I actually started to play around with this, and it is in a very early prototype stage. I wanted to make sure I could get it to work in lists and such. I kind of played with the directive syntax some. I'm not sure if anything below, syntax or behavior-wise, will be the final implementation, but this is currently just exploratory.

The thing I like is that you don't have to do indentations for admonitions and such, and it doesn't break my editor's syntax highlighting :). I guess some common admonitions could be created like warning and if people want something different, they could use the generic admonition type. Anyways, this was a simple test:

- 
    ::::{admonition} This is really important!
    ---
    type: warning
    ---

    Don't do that, for these reasons:

    - 
        :::{details} Here is a summary
        This is nested!
        :::

    ::::

    :::{html} div
    ---
    attributes: {id: some-id, class: these are classes}
    ---

    Some other content
    :::

Results

<ul>
<li>
<div class="admonition warning">
<div class="admonition-title">This is really important!</div>
<p>Don't do that, for these reasons</p>
<ul>
<li>
<details>
<summary>Here is a summary</summary>
<p>This is nested!</p>
</details>
</li>
</ul>
</div>
<div class="these are classes" id="some-id">
<p>Some other content</p>
</div>
</li>
</ul>

So far the directives are pretty simple objects. They have on_create events and on_end events. You can store the content as you accumulate it and then process it once you hit on_end, or not store it and let the Markdown parser just parse it as Markdown content. That's really it.

class DirectiveTemplate:
    """Directive template."""

    # Set to something if argument should be split.
    # Arguments will be split and white space stripped.
    ARG_DELIM = ''
    NAME = ''
    STORE = False

    def __init__(self, length):
        """Intitialize."""

        self.store = []
        self.length = length

    def config(self, args, **options):
        """Parse configuration."""

        self.args = [a.strip() for a in args.split(self.ARG_DELIM)] if args and self.ARG_DELIM else [args]
        self.options = options

    def on_create(self, el):
        """On create event."""

    def on_end(self, el):
        """Perform action on end."""

Anyways, I figured I share it and see if people had any thoughts.

[facelessuser]

Yeah, it is pretty easy to just derive to create shortcut for Note admonitions and such:

class Admonition(DirectiveTemplate):
    """Admonition."""

    NAME = 'admonition'

    def on_create(self, parent):
        """Create the element."""

        el = etree.SubElement(parent, 'div')
        t = self.options.get('type', '').lower()
        title = self.args[0] if self.args and self.args[0] else t.title()
        classes = [c for c in self.options.get('classes', '').split(' ') if c]
        if t != 'admonition':
            classes.insert(0, t)
        classes.insert(0, 'admonition')
        ad_title = etree.SubElement(el, 'div', {'class': 'admonition-title'})
        ad_title.text = title
        el.set('class', ' '.join(classes))
        return el


class Note(Admonition):
    """Note."""

    NAME = 'note'

    def config(self, args, **options):
        """Parse configuration."""

        super().config(args, **options)
        self.options['type'] = 'note'

And then this:

:::{admonition} This is really important!
---
type: warning
---

Don't do that!
:::

:::{note}
Just a note
:::

:::{note} With a title
And some words.
:::

Becomes this:

<div class="admonition warning">
<div class="admonition-title">This is really important!</div>
<p>Don't do that!</p>
</div>
<div class="admonition note">
<div class="admonition-title">Note</div>
<p>Just a note</p>
</div>
<div class="admonition note">
<div class="admonition-title">With a title</div>
<p>And some words.</p>
</div>