mmarkdown/mmark

Mmark: a powerful markdown processor in Go geared towards the IETF

title	date	aliases
About	2018-07-22 14:05:51 +0100	/about/

Mmark is a powerful markdown processor written in Go, geared towards writing IETF documents. It is, however, also suited for writing complete books and other technical documentation, like the Learning Go book (mmark source, and I-D text output).

Also see this repository on how to write RFC using Markdown.

It provides an advanced markdown dialect that processes file(s) to produce internet-drafts in XML RFC 7991 format. Mmark can produce xml2rfc (aforementioned RFC 7991), HTML5 output, and manual pages.

Example RFCs in Mmark format can be found in the Github repository.

Mmark uses gomarkdown which is a fork of blackfriday. See its README.md for more documentation.

Syntax

Mmark's syntax and the extra features compared to plain Markdown are detailed in syntax.md.

Mmark adds the following syntax elements to gomarkdown/markdown:

• (Extended) title block.
• Special sections.
• Including other files with the option to specify line ranges, regular expressions and/or prefix each line with a string. By default only files on the same level, or below are allowed to be included (see the -unsafe flag).
• Document divisions.
• Captions for code, tables and quotes
• Asides.
• Figures and Subfigures - bundle (sub)figures into a larger figure.
• Block Level Attributes that allow to specify attributes, classes and IDs for elements.
• Indices to mark an item (and/or a subitem) to be referenced in the document index.
• Citations and adding XML References
• In document cross references, short form of referencing a section in the document.
• Super- and Subscript.
• Callouts in code and text.
• BCP14 (RFC 2119) keyword detection.

Usage

You can download a binary or optionally build mmark your self. You'll need a working Go environment, then check out the code and:

% go get && go build
% ./mmark -version
2.0.0

To output XML2RFC v3 xml just give it a markdown file and:

% ./mmark rfc/3514.md

Making a draft in text form (v3 output)

% ./mmark rfc/3514.md > x.xml
% xml2rfc --v3 --text x.xml

Outputting HTML5 is done with the -html switch.

Files edited under Windows/Mac and using Windows style will be converted into Unix style line ending before parsing. Any output from mmark will use Unix line endings.

Note there are no wrong markdown documents, so mmark will only warn about things that are not right. This may result in invalid XML. Any warning from mmark are send to standard error, to see and check for those you can discard standard output to just leave standard error: ./mmark rfc/3515.md > /dev/null.

Example RFC

The rfc/ directory contains a couple of example RFCs that can be build via v3 tool chain. The build the text files, just run:

cd rfc
make txt

Official RFCs are in rfc/orig (so you can compare the text output from mmark).

Also See

Kramdown-rfc2629 is another tool to process markdown and output XML2RFC XML.

See Syntax.md for a primer on how to use the Markdown syntax to create IETF documents.

mmark's Issues

Add a shortcut for non-breaking spaces

Feature request

Description

The use of non-breaking spaces is useful for better typography and indispensable in some languages, furthermore when writing long texts (documentation, books, or just blog posts).

However, inserting   in all files is not ideal as it is tiresome and can bring some brittleness (it only works as long as the target rendering is HTML).

Could we think of a universal shortcut that would be rendered appropriately?

Some use cases

• This can be useful to make sure that references stay together: See Fig. 3.
• This should be used when writing something with a unit: Processing time: 25 ms.
• In some languages countries, typographic rules make heavy use of the non-breaking space. This is for example the case in France, where :, ;, ? and ! should be preceded with a non-breaking space: Et il déclara : « Que c'est pénible d'écrire correctement ! Faut-il systématiquement saisir six caractères pour une espace ? »

Existing solutions in other software

• AsciiDoctor predefines a {nbsp} variable. It is as verbose as writing   but rules out the brittleness as it can be treated differently depending on the renderer.
• Some other Markdown parsers use an escaped space (\ ) or a tilde (~) as a non-breaking space. For instance: 42\ ms, 1337~ms.

Implement Parts

V2 has dropped Parts, it may make sense to bring that back (not that difficult either).

See https://github.com/miekg/mmark/wiki/Syntax#parts for the syntax.

2.0.32 release in https://github.com/mmarkdown/mmark/releases ?

Thanks, using your new release now. Installed directly. Can you release a new version also?

Update on gomarkdown Parser (parser.ParserOptions => parser.Options)

To help us debug your issue please add these details.

try to build mmark from source and got following error
# github.com/mmarkdown/mmark github.com\mmarkdown\mmark\mmark.go:90:12: undefined: parser.ParserOptions

What went wrong?

failed to build from source after cloning the repo.
check gomarkdown commit

What version of mmark are you using?

2.0.37

What command line did you use?

go get && go build

What platform (Linux/Apple/Windows)?

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

wrong?

To help us debug your issue please add these details.

I use go get to install

What went wrong?

-html

What version of mmark are you using?

master

What command line did you use?

bash

What platform (Linux/Apple/Windows)?

linux

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

mmark -html ~/go/src/github.com/mmarkdown/mmark/rfc/3514.md > x.html

result:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x53d7b3]

goroutine 1 [running]:
github.com/mmarkdown/mmark/render/mhtml.bibliographyItem(0x5cfde0, 0xc000114930, 0xc000114690, 0xc000114901)
/home/king/go/src/github.com/mmarkdown/mmark/render/mhtml/hook.go:102 +0x183
github.com/mmarkdown/mmark/render/mhtml.RenderHook(0x5cfde0, 0xc000114930, 0x5d1d00, 0xc000114690, 0x1, 0x5cfde0, 0xc0000d3718)
/home/king/go/src/github.com/mmarkdown/mmark/render/mhtml/hook.go:37 +0x38c
github.com/gomarkdown/markdown/html.(*Renderer).RenderNode(0xc0000e8b40, 0x5cfde0, 0xc000114930, 0x5d1d00, 0xc000114690, 0x1, 0x0)
/home/king/go/src/github.com/gomarkdown/markdown/html/renderer.go:914 +0x16ce
github.com/gomarkdown/markdown.Render.func1(0x5d1d00, 0xc000114690, 0x4ff401, 0xc0000e8b40)
/home/king/go/src/github.com/gomarkdown/markdown/markdown.go:63 +0x61
github.com/gomarkdown/markdown/ast.NodeVisitorFunc.Visit(0xc00000bd20, 0x5d1d00, 0xc000114690, 0x1, 0xc0000d38e8)
/home/king/go/src/github.com/gomarkdown/markdown/ast/node.go:547 +0x43
github.com/gomarkdown/markdown/ast.Walk(0x5d1d00, 0xc000114690, 0x5d0060, 0xc00000bd20, 0x0)
/home/king/go/src/github.com/gomarkdown/markdown/ast/node.go:519 +0x70
github.com/gomarkdown/markdown/ast.Walk(0x5d1ca0, 0xc0001148c0, 0x5d0060, 0xc00000bd20, 0x0)
/home/king/go/src/github.com/gomarkdown/markdown/ast/node.go:530 +0x1ba
github.com/gomarkdown/markdown/ast.Walk(0x5d1280, 0xc000114000, 0x5d0060, 0xc00000bd20, 0x0)
/home/king/go/src/github.com/gomarkdown/markdown/ast/node.go:530 +0x1ba
github.com/gomarkdown/markdown/ast.Walk(0x5d1220, 0xc000064480, 0x5d0060, 0xc00000bd20, 0xc00000bd20)
/home/king/go/src/github.com/gomarkdown/markdown/ast/node.go:530 +0x1ba
github.com/gomarkdown/markdown/ast.WalkFunc(0x5d1220, 0xc000064480, 0xc00000bd20)
/home/king/go/src/github.com/gomarkdown/markdown/ast/node.go:553 +0x4b
github.com/gomarkdown/markdown.Render(0x5d1220, 0xc000064480, 0x5d0580, 0xc0000e8b40, 0x5a9adc, 0x13, 0x5a861f)
/home/king/go/src/github.com/gomarkdown/markdown/markdown.go:62 +0xd8
main.main()
/home/king/go/src/github.com/mmarkdown/mmark/mmark.go:171 +0x67

Add indexInclude from 7991

Add indexInclude attribute to the TOML block: https://tools.ietf.org/html/rfc7991#section-2.45.4

man output: proper callout support

Currently callouts in manual output are supported, but it lacks the option to specify comments to be detected, this does happen in the other renderers and should be added here as well.

Missing organization in reference

To help us debug your issue please add these details.

What went wrong?

Reference is missing organization in the author block. I am not familiar with writing programs in Go, However, looking at the previous fix, it seems that the test is written to pass if the organization is missing.

What version of mmark are you using?

2.0.39

What command line did you use?

mmark test.md > test.xml

What platform (Linux/Apple/Windows)?

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

Using the same snipped as #61.

%%%

 Title = "Test doc"

 date = 2019-02-19T00:00:00Z

 [seriesInfo] 
 name = "Internet-Draft"
 value = "draft-test-00"
 status = "informational"

 [[author]]
 initials="J."

 %%%

{mainmatter}

# Introduction

Test [@!IANA]

{backmatter}

<reference anchor='IANA' target='https://www.iana.org/assignments/media-types/media-types.xhtml'>
    <front>
        <title abbrev='IANA'>IANA Media Types</title>
        <author>
            <organization>IANA</organization>
        </author>
        <date month='February' year='2019'/>
    </front>
</reference>

From the XML, here's just the reference section. Note the organization in the markdown above does not appear in the XML.

<back>
<references><name>Normative References</name>
<reference anchor="IANA" target="https://www.iana.org/assignments/media-types/media-types.xhtml">
  <front>
    <title>IANA Media Types</title>
    <author>
      <organization></organization>
    </author>
    <date year="2019" month="February"></date>
  </front>
</reference>
</references>

table markdown output with span elements

A markdown table:

 \          | \             | \         
------------|---------------|-----------
`break`	    |`default`	    |`func`	    

is rendered:

          |            |
----------|------------|---------
`break`	  | `default`	 | `func`	

It's off because it does not account for the backticks (or any other span markings)

markdown: def list in quote wrongly rendered

> this is a quote quote with *empp* and more
>
hallo more tetg hallo more tetg hallo more tetg hallo more tetg hallo more tetg hallo more
>     tetg hallo more tetg hallo more tetg
>
>  :  ahllo
>
more1
>
>  :  more
>

is rendered wrong, the > is tripped from the prefix, because we cut the entire prefix which is too much.

Reference blocks with multiple authors only show one

On the latest version of mmark, if a <reference> has multiple authors, only the last one is rendered in the XML. This includes mmark/rfc/3514.md.

support convertedFrom for rfc building

To help us debug your issue please add these details.

Can't build an RFC without warnings:

514.xml(5): Warning: Expected explicit values for year, month, and day, but found None, None and None
3514.xml(3): Warning: Expected a <link> with rel='convertedFrom' providing the datatracker url for the origin draft.

We need a convertedFrom in the toml, maybe 7749 can use it as well.

See examples in the rfc dir

implement index in gomarkdown

This should be relatively easy. Support mmark indices: https://mmark.nl/post/syntax/#indices

This needs upstream support in parser/inline.go (add a new file index.go or something) and a bunch of tests to see if we can parse it.

upstream is https://github.com/gomarkdown/markdown

xml/xml2: table support and caption isn't there yet

Need to double check if tables are rendered correctly, esp. when they have a caption.

Also tests.

support footnotes in xml/xml2 output

In the manual output footnotes are outputted in a "Notes" section, we can do the same in the xml/xml2 renderers, same code (almost) can even be used.

This would be inserted just before the {backmatter} similar to the place of the bibliography, but on the opposite site.

manual page: crash if title is not correctly set

If the little Toml header is not correctly set the manual page output will crash on index out of range.

asides not rendered correctly

they have no content

Old domain still in use in the GitHub project title

Add 'safeInclude' flag/options

Add a 'safeInclude' flag that only allows relative includes and no .. in the paths. This is a simple check.

A more advanced safe include would be to only allow files below the initial file being work on.

xml: Allow html/xml fallthrough

Sometimes you need to fallback to html or xml; currently all blocklevel elements except <reference> are removed from the final output. This may be sub optimal.

includes

Now that gomarkdown/markdown support an IncludeFunc we can implement all the bits we need in mmark.

Spurious [] before cross-references

What went wrong?

With a long-form xref like the XML output is

[]<xref target="RFC2119"></xref>

and the HTML is

[]<a href="#RFC2119"></a>

I expected no [] in the output.

What version of mmark are you using?

2.0.10

What command line did you use?

echo '[](#RFC2119)' >xref.md
~/go/bin/mmark xref.md

What platform (Linux/Apple/Windows)?

• [*] Linux

Missing docName

What went wrong?

xml2rfc --v3 complained with Warning: Expected a 'docName' attribute in the <rfc/> element, but found none., and https://datatracker.ietf.org/submit/ rejected the XML.

Manually adding the docName attribute fixed the issue.

What version of mmark are you using?

2.0.46

What command line did you use?

mmark

What platform (Linux/Apple/Windows)?

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

%%%

    Title = "The ristretto255 Group"
    abbrev = "ristretto255"
    ipr = "trust200902"
    category = "info"
    area = "Internet"
    workgroup = "Crypto Forum Research Group"

    [seriesInfo]
    status = "informational"
    name = "Internet-Draft"
    value = "draft-hdevalence-cfrg-ristretto-01"
    stream = "IETF"

[...]

xml: caption need to be rendered after `<figure>`

<name> needs to be directly after <figure>, right now it comes after the artwork. Simple change and we can look how xml2/renderer.go does it.

Deprecate xml2 parser and renderer

xml2 output for the IETF has already(?) been deprecated and the RFC editor has moved to some form of 7991 XML as the canonical form.

I'll try to get some authoritative source to say so and then delete this xml2 code in Mmark.

Implement example lists

(re-) implement example lists.

Follow the Pandoc implementation, but only support the alfanum implementation (@good) is allowed, but not (@) (as you can't reference those).

Referencing happens with the same syntax (@good)

`<aside>` in rfc 7749 output

Figure out what to do with asides in 7749 output.

markdown output: captionFigure as subfigure is not supported

This is not supported as of yet:

!---
![Array versus slice](fig/array-vs-slice.png)
!---
Figure: An array versus a slice.

allow toml for references

I might be nice to allow toml syntax for reference, so that we unify the preamble (which is in toml) with the references. Downside is that all references come in XML format, so you need to reformat that. We already support the XML from the references natively; so not sure how much functionally this adds.

Add v2 output

It might make sense to add v2 XML output as well to mmark2, as the v3 tool chain for the IETF is still in development. As a new render is relatively easy to add this should be doable. And a lot of stuff could be copied from mmark1 (https://github.com/miekg/mmark)

sort bibliography

Bibliography is stored in a map and when we range over it we get the elements in random order; this should be sorted so a single source markdown file always maps to the same output.

[`link`](http://link.link) fails with Error: Did not expect element t there, at /rfc/middle/t a.xml(3): Error: Invalid document before running preptool

To help us debug your issue please add these details.

What went wrong?

Unable to include a link of a code sample. I commonly do this—and see this—on StackOverflow.

What version of mmark are you using?

2.0.40, and xml2rfc 2.22.0

What command line did you use?

$ mmark a.md > a.xml
$ xml2rfc --v3 a.xml
Parsing file a.xml
a.xml(18): Error: Did not expect element t there, at /rfc/middle/t
a.xml(3): Error: Invalid document before running preptool.
Cannot continue, quitting now

What platform (Linux/Apple/Windows)?

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

$ cat a.md
%%%
title = "Model designer"
abbrev = "ODESIGN"
area = "Internet"
workgroup = "Offscale.io"
keyword = ["platform-independent"]
#date = 2019-12-07T00:00:00Z

[seriesInfo]
name = "RFC"
value = "ODESIGN0.0.1"
status = "informational"

[[author]]
surname="Marks"
fullname="Samuel Marks"
organization = "Sydney Scientific"
  [author.address]
  email = "[email protected]"
%%%

{mainmatter}

You MAY use [`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage) and/or similar built-ins.

{backmatter}

citing an ID doesn't work

For I-Ds you may want to add a draft sequence number, which can be done as such: [@?I-D.blah#06]

There was some special code in mmark1 that hasn't been ported yet.

cite is rendered weirdly

When rendering <cite>s they look a bit off, need to double check if they are ok or what is wrong here.

Expand Syntax.md to describe all the syntax

Cut and paste from blackfriday and/or gomarkdown and flesh out a full document

Support duplicate footnotes

To help us debug your issue please add these details.

Is it possible to support multiple references to a footnote content? Seems that blackfriday has implemented the feature through russross/blackfriday#366.

Sentence one[^1]. Sentence two[^1]. Sentence three[^1].

[^1]: They should all refer to here.

As of Hugo 0.49, whose blackfriday engine outputs one line of footnote while mmark does 3 lines of them with incremental numbers.

complete citations

Citations can have a suffix. This is not implemented. In pandoc this is signaled using a comma as the separation character we now use a space. We should use a comma as well and implement the suffix use in citation rendering.

xml: figure out how to do images

We can include svg, but not(?) other figures... need to double check what rfc7991 says about this.

subfigures

Subfigures in mmark are sorta supported by prefixing a paragraph with F> . This is a bit annoying. The scholary markdown (discontinued) solution seems better:

Do this? http://scholarlymarkdown.com/Scholarly-Markdown-Guide.html#floating-env

xml: images as artwork doesn't work.

for rfc7991 output we can't output images as artwork, as a image element is not an (block)level artwork, this means we must inhibit outputting <t>s, but that is hard. Also using the title for the name attribute looks weird.

A figureblock works, but must be present in the markdown code.

Compilation breaks on: # Foo\n\n ### Bar \n\n

Replication steps (mmark 2.0.40, xml2rfc 2.22.0):

$ mmark index.md > index.xml
$ xml2rfc index.xml -o index.html
Error: Unable to parse the XML document: index.xml
 <string>: Line 24: Opening and ending tag mismatch: middle line 17 and
section
 <string>: Line 24: Opening and ending tag mismatch: middle line 17 and
section
 <string>: Line 28: Opening and ending tag mismatch: rfc line 3 and section
 <string>: Line 30: Extra content at the end of the document

More info:
https://mailarchive.ietf.org/arch/msg/xml2rfc/OPep-8cpM-NZFqQZzX_whWpU9iw

release needed: gomarkdown broke API for parser.

To help us debug your issue please add these details.

The issue #56 was fixed 10 days ago, but this software FTBFS. A stable upstream library or dependency management would be nice, but for now a release would fix this also.

What went wrong?

Current release FTBFS

What version of mmark are you using?

Current Release

What command line did you use?

go build

What platform (Linux/Apple/Windows)?

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

FTBFS

AST transformer

Now that there is a -markdown renderer which output markdown again, we can work on the AST and transform it and then output "new" markdown.

I.e. you can write a super tiny go program that rewrites the AST: for instance to transform all emphasis to strong. (inline are easiest).

But you can also have a more sophisticated one that looks at a code block and if the language matches exec-dot executes dot and converts the whole code block into an dot-generated image.

xml2: anchor needs to be on figure

<figure title="Scenic Routing Option Layout ">
 <artwork anchor="fig-scenic-routing-option-layout"> 

the anchor attribute need to be on the <figure> tag

links in markdown output

This is not rendered correctly in the markdown output.

[I'm an inline-style link](https://www.google.com)

[I'm an inline-style link with title](https://www.google.com "Google's Homepage")

[I'm a reference-style link][Arbitrary case-insensitive reference text]

[I'm a relative reference to a repository file](../blob/master/LICENSE)

[You can use numbers for reference-style link definitions][1]

Or leave it empty and use the [link text itself].

URLs and URLs in angle brackets will automatically get turned into links. 
http://www.example.com or <http://www.example.com> and sometimes 
example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

[arbitrary case-insensitive reference text]: https://www.mozilla.org
[1]: http://slashdot.org
[link text itself]: http://www.reddit.com

markdown: indent all lists with a space

Ident all lists when generating markdown, so

* item1
* item2

will become

 *  item1

 *  item2

this works for kind of listst

mmark math support

I'm trying to use hugo + mmark + katex. Documentation around the web about math support in mmark is sparse and seems to be partly / mostly outdated.

Using mmark as a markdown processor in hugo, $$ E = mc^2 $$ is rendered to <span class='math'>\( E = mc^2\)</span> which is valid TeX, but doesn't seem to be compliant with MathJax as is written in most docs, but rather seems to be MathJS syntax (?).

Trying to run

$("span[class='math']").replaceWith(
  function(){
    var tex = $(this).text();
    return "<span class=\"inline-equation\">" +
           katex.renderToString(tex) +
           "</span>";
});

over it fails because KaTeX cannot deal with the delimiters \( and \).

Is there more documentation on math support in mmark that I have missed and is it possible to configure how the math is rendered?

implement table captions

See gomarkdown/markdown#35

Stuff can be copied from codeblock or quotes, just needs to be done.

Target missing from outputted reference

What went wrong?

References including target attributes on reference blocks do not have a target attribute in the outputted xml. It seems this issue was also reported in a comment on issue #53 but never fixed.

What version of mmark are you using?

2.0.38

What command line did you use?

mmark ./test.md > /tmp/x.xml

What platform (Linux/Apple/Windows)?

• Linux

What is the most minimal markdown snippets that shows the problem?

%%%

 Title = "Test doc"

 date = 2019-02-19T00:00:00Z

 [seriesInfo] 
 name = "Internet-Draft"
 value = "draft-test-00"
 status = "informational"

 [[author]]
 initials="J."

%%%

{mainmatter}

# Introduction

Test [@!IANA]

{backmatter}

<reference anchor='IANA' target='https://www.iana.org/assignments/media-types/media-types.xhtml'>
    <front>
        <title abbrev='IANA'>IANA Media Types</title>
        <author>
            <organization>IANA</organization>
        </author>
        <date month='February' year='2019'/>
    </front>
</reference>

Attribute "start" not supported for list style="numbers"

What went wrong?

mmark logs Attribute "start" not supported for list style="numbers" and makes the 2. element of a list start over at 1..

What version of mmark are you using?

v2.0.46

What command line did you use?

mmark -2

What platform (Linux/Apple/Windows)?

• Linux
• Apple
• Windows

What is the most minimal markdown snippets that shows the problem?

1. Process the internal representation into a field element s as follows:

_some unindented code block here_

2. Return the canonical little-endian encoding of s.