metanorma/coradoc

Coradoc is the Core AsciiDoc Parser used by Metanorma

Metanorma: the standard for standards

Metanorma is dedicated to harmonizing standard documents produced by different standard-setting bodies in a manner that maintains correct semantics while allowing each standard publisher to define appropriate semantic extensions.

Simply put, it allows standards bodies or any other organization to create their own standard or specification document in a best practices manner.

Metanorma is composed of a number of specifications and software implementations. The Metanorma document model is based on the SecureDoc document model.

For more on Metanorma and who uses it, refer to https://www.metanorma.org

Installation on supported platforms

See https://www.metanorma.org/install/

Installing individual components

The Metanorma workflow can be utilized via the metanorma-cli Ruby gem.

gem install metanorma-cli

Usage

Refer to Metanorma man page and Metanorma usage

Threaded execution

Metanorma has threaded execution, to generate output documents from the same Presentation XML input more quickly. Similar to relaton, the METANORMA_PARALLEL environment variable can be used to override the default number of parallel fetches used.

Origin of name

Meta- is a prefix of Greek origin ("μετα") for “with” “after”. In English, it has ended up meaning "about (its own category)"; e.g. meta-discussion (a discussion about discussion). (For the roundabout way it ended up with that meaning, see https://en.wikipedia.org/wiki/Meta#Etymology.)

Norma is Latin for “rule” and “standard”; hence English norm, but also German Norm "standard".

The Metanorma project is for setting a standard for standard documents created by standards-setting organizations (which is a meta thing to do); hence this name.

Metanorma seeks to embrace all standards documents standards, but not possess any: it can give rise to many "standard" standards, but not limit the extension of any of those standards.

The motto of the project is Aequitate verum, "Truth through equity". Dealing with all standards fairly (aequitate), we seek not an abstract virtue (veritas), but a practical reality on the ground (verum), that can be used by stakeholders of multiple standards.

coradoc's Issues

Provide `Coradoc::Document.to_adoc` method to output to AsciiDoc string

Split up UnorderedList and OrderedList to different files

Can we split up UnorderedList and OrderedList to different files?

Originally posted by @ronaldtse in #33 (comment)

Add support for example blocks

Currently the example blocks are not getting parsed correctly, In the example below the example is being parse as a concept because of ====.

==== construction geometry

geometry used as an aid in the construction of a shape that is not an element of
the shape itself

[example]
====
This concept may be illustrated by the case of a 'spine' curve used in the
generation of certain types of swept surfaces. A planar generating curve is
defined, together with a reference point lying in its plane. The reference point
is then swept along the spine curve, the plane of the generating curve being
maintained normal to it. The generating curve sweeps out the desired surface. If
the reference point does not lie on the generating curve the spine curve does
not lie on the generated surface, and it is therefore classified as construction
geometry.
====

Add support to parse document inclusion

From @ronaldtse:

Inclusion of:
- asciidoc file `include::{path}[{options}]`
- image file `image::{path}[{options}]`

Support block tagging

tag::tutorial[]
...
end::tutorial[]

Change default branch to main

From master

Support tables

Create first cut for Coradoc gem

The current parser we use for AsciiDoc (Asciidoctor) does line-by-line regular expression extractions and cannot generate a parse tree for model-based editing.

This task is to develop a formal grammar (which may already be done in the Eclipse AsciiDoc working group) and implement a pure Ruby parser that generates a parse tree.

The parser needs to be extensible: Metanorma adds many specialised blocks and additional language features on top of normal AsciiDoc (Metanorma AsciiDoc), and this parser needs to support that.

Add missing support for section

Todos

• Support for predefined titles
• Support for deviating sections titles
• Attributes with key, value pair, eg: [heading=foreword]
• Attributes with with multi value, eg: [bibliography,heading=normative]
• Support for parsing keep title: eg: [heading=foreword,keeptitle=true]
• Support for blank subclause heading: === {blank}
• Support for inline heading: [%inline-header]\n==== Sieve,
• Support for variant title: [.variant-title,type=sub]
• Support for level attribute in heading: [level=6]

Reference: https://www.metanorma.org/author/topics/sections/

audio, video blocks: use AttributeList for empty attributes?

In audio.rb and video.rb, there is this line:

attrs = @attributes.empty? ? "\[\]" : @attributes.to_adoc

Since we have AttributeList, you can implement AttributeList.to_adoc which returns [] when empty. So this line can go away, and the next line uses attrs.to_adoc instead of attrs.

Originally posted by @ronaldtse in #47 (comment)

Defined options for `video` and `image` blocks

Options need to be defined and validated, e.g. height/width can be auto or a pixel number. When given unknown options we should reject.

Originally posted by @ronaldtse in #33 (comment)

Handle inline formatted text not flanked by whitespaces

`standalone` monospace

``mono``space

*bold* itself

attached**bold**here

_italics_ itself

attached__italics__here

From:

• #33 (comment)
• #33 (comment)
• #33 (comment)

Add support for parsing source

Add support for parsing source blocks that are in the form of

[.source]
some reference

CI: add Rubygems auto-release workflow and test workflow

@CAMOBAP can you please help add the auto-release and testing GHA workflows for this repository? Thanks!

Doc: High level Architecture

Current Architecture

1. Metanorma uses AsciiDoc as input syntax
2. The AsciiDoc syntax Metanorma syntax is an enriched syntax -- not supported by normal AsciiDoc parsers (including the vanilla asciidoctor parser we use)
3. Metanorma AsciiDoc syntax in a file is stored in an .adoc file, and parsed by Metanorma
4. Metanorma uses asciidoctor to parse the .adoc file(s). Metanorma has a "plugin" into asciidoctor to obtain a "pseudo document tree" state of the file as an Asciidoctor object.
5. Metanorma builds an XML file to represent that pseudo document tree object
6. Metanorma writes the XML file into "semantic XML" and "presentational XML" files. The semantic XML file contains mostly data, the presentational XML file contains mostly presentation (e.g. a NOTE is represented in semantic xml as a <note> element, but becomes a <p> in presentational XML)
7. The .xml file is loaded into Metanorma for rendering. Instead of loading the XML file as a Metanorma::Document object, it just runs XPath queries on this XML file using Nokogiri.
8. Metanorma converts the presentational or semantic .xml files into outputs, like HTML, PDF and Word.

The ideal flow:

1. Metanorma loads the AsciiDoc file using the new AsciiDoc parser. This gives us an AsciiDoc parse tree object (a generic AsciiDoc tree)
2. Metanorma "specializes" this AsciiDoc parse tree into a Metanorma::Document document node tree. For example, a generic AsciiDoc section object can be specialized as the Bibliography class object in Metanorma.
3. Metanorma processes the Metanorma::Document such as for link resolution
4. Metanorma calls Document.to_xml to write the XML files
5. The Metanorma Document object is passed to the presentation rendering stage, to generate the outputs

Note:

To support different standard we will need to build an extension mechanism where we can run callbacks on encountering document nodes, so different flavours can build their documents differently. E.g. there would be a Metanorma::Document::Iso vs Metanorma::Document::Ogc

//cc: @ronaldtse

Provide `Coradoc::Document.to_md` method to output to Markdown string

The elements not natively supported by Markdown should be output in HTML.

Support writing to OSCAL Classes

In order to transform AsciiDoc into OSCAL content.

Coradoc-OSCAL will convert the content of the resulting AsciIDoc into OSCAL.

Dependencies:

• #1
• #2
• #13
• #9
• #10

Title class should have content of TextElement

About the class Title, now that I think about it, @content should not be a string but TextElement instead. Or am I wrong and it can be just a string.

Originally posted by @xyz65535 in #33 (comment)

Split Block class into separate files: Example, Sourcecode, etc.

Split Block class into separate files: Example, Sourcecode, etc.

Right now the Block subclasses are in one big file.

Develop Coradoc AsciiDoc grammar

Use Anchor class instead of manually constructed anchors for blocks

Use the Anchor class in classes instead of [[...]] in text.

Originally posted by @ronaldtse in #33 (comment)

Support parsing comments

Add support to parse comments in adoc files.

Handle Definition Lists

Also, we need a DefinitionList class.

Originally posted by @ronaldtse in #33 (comment)

Add support for bibliography, cross-references

From @ronaldtse:

Example:
https://github.com/metanorma/mn-samples-iso/blob/main/sources/international-standard/sections-en/b0-bibliography.adoc

Support for inline elements

Coradoc doesn't support parsing of inline elements, for instance:

• styling (bold, italic)
• hyperlinks
• images

Handle matching pairs of block delimiters of not-default lengths

Let's separate these into different classes. The number of ---- / ___ etc. can change, e.g. 3, 4, 5, 6, 10. But always matching in begin/end.

Originally posted by @ronaldtse in #33 (comment)

Handle image blocks

image: (one colon) is inline image.

image:: (two colons) is block image. We need a new class there, and the two image classes should inherit.

Originally posted by @ronaldtse in #33 (comment)

Parse Markdown into Coradoc

ListItem formatting needs to deal with HardBreaks

For the ListItem class, the formatting needs to deal with HardBreaks too, if the list item contains multiple paragraphs or blocks.

Originally posted by @ronaldtse in #33 (comment)

Implement Ruby classes for all BasicDoc elements