A super fast, highly extensible markdown parser for PHP

What is this?

A set of PHP classes, each representing a Markdown flavor, and a command line tool for converting markdown files to HTML files.

The implementation focus is to be fast (see benchmark) and extensible. Parsing Markdown to HTML is as simple as calling a single method (see Usage) providing a solid implementation that gives most expected results even in non-trivial edge cases.

Extending the Markdown language with new elements is as simple as adding a new method to the class that converts the markdown text to the expected output in HTML. This is possible without dealing with complex and error prone regular expressions. It is also possible to hook into the markdown structure and add elements or read meta information using the internal representation of the Markdown text as an abstract syntax tree (see Extending the language).

Currently the following markdown flavors are supported:

Traditional Markdown according to http://daringfireball.net/projects/markdown/syntax (try it!).
Github flavored Markdown according to https://help.github.com/articles/github-flavored-markdown (try it!).
Markdown Extra according to http://michelf.ca/projects/php-markdown/extra/ (currently not fully supported WIP see #25, try it!)
Any mixed Markdown flavor you like because of its highly extensible structure (See documentation below).

Future plans are to support:

Smarty Pants http://daringfireball.net/projects/smartypants/
... (Feel free to suggest further additions!)

Who is using it?

It powers the API-docs and the definitive guide for the Yii Framework 2.0.

Installation

PHP 5.4 or higher is required to use it. It will also run on facebook's hhvm.

The library uses PHPDoc annotations to determine the markdown elements that should be parsed. So in case you are using PHP opcache, make sure it does not strip comments.

Installation is recommended to be done via composer by running:

composer require cebe/markdown "~1.2.0"

Alternatively you can add the following to the require section in your composer.json manually:

"cebe/markdown": "~1.2.0"

Run composer update cebe/markdown afterwards.

Note: If you have configured PHP with opcache you need to enable the opcache.save_comments option because inline element parsing relies on PHPdoc annotations to find declared elements.

Usage

In your PHP project

To parse your markdown you need only two lines of code. The first one is to choose the markdown flavor as one of the following:

Traditional Markdown: $parser = new \cebe\markdown\Markdown();
Github Flavored Markdown: $parser = new \cebe\markdown\GithubMarkdown();
Markdown Extra: $parser = new \cebe\markdown\MarkdownExtra();

The next step is to call the parse()-method for parsing the text using the full markdown language or calling the parseParagraph()-method to parse only inline elements.

Here are some examples:

// traditional markdown and parse full text
$parser = new \cebe\markdown\Markdown();
echo $parser->parse($markdown);

// use github markdown
$parser = new \cebe\markdown\GithubMarkdown();
echo $parser->parse($markdown);

// use markdown extra
$parser = new \cebe\markdown\MarkdownExtra();
echo $parser->parse($markdown);

// parse only inline elements (useful for one-line descriptions)
$parser = new \cebe\markdown\GithubMarkdown();
echo $parser->parseParagraph($markdown);

You may optionally set one of the following options on the parser object:

For all Markdown Flavors:

$parser->html5 = true to enable HTML5 output instead of HTML4.
$parser->keepListStartNumber = true to enable keeping the numbers of ordered lists as specified in the markdown. The default behavior is to always start from 1 and increment by one regardless of the number in markdown.

For GithubMarkdown:

$parser->enableNewlines = true to convert all newlines to <br/>-tags. By default only newlines with two preceding spaces are converted to <br/>-tags.

It is recommended to use UTF-8 encoding for the input strings. Other encodings may work, but are currently untested.

The command line script

You can use it to render this readme:

bin/markdown README.md > README.html

Using github flavored markdown:

bin/markdown --flavor=gfm README.md > README.html

or convert the original markdown description to html using the unix pipe:

curl http://daringfireball.net/projects/markdown/syntax.text | bin/markdown > md.html

Here is the full Help output you will see when running bin/markdown --help:

PHP Markdown to HTML converter
------------------------------

by Carsten Brandt <[email protected]>

Usage:
    bin/markdown [--flavor=<flavor>] [--full] [file.md]

    --flavor  specifies the markdown flavor to use. If omitted the original markdown by John Gruber [1] will be used.
              Available flavors:

              gfm   - Github flavored markdown [2]
              extra - Markdown Extra [3]

    --full    ouput a full HTML page with head and body. If not given, only the parsed markdown will be output.

    --help    shows this usage information.

    If no file is specified input will be read from STDIN.

Examples:

    Render a file with original markdown:

        bin/markdown README.md > README.html

    Render a file using gihtub flavored markdown:

        bin/markdown --flavor=gfm README.md > README.html

    Convert the original markdown description to html using STDIN:

        curl http://daringfireball.net/projects/markdown/syntax.text | bin/markdown > md.html


[1] http://daringfireball.net/projects/markdown/syntax
[2] https://help.github.com/articles/github-flavored-markdown
[3] http://michelf.ca/projects/php-markdown/extra/

Security Considerations

By design markdown allows HTML to be included within the markdown text. This also means that it may contain Javascript and CSS styles. This allows to be very flexible for creating output that is not limited by the markdown syntax, but it comes with a security risk if you are parsing user input as markdown (see XSS).

In that case you should process the result of the markdown conversion with tools like HTML Purifier that filter out all elements which are not allowed for users to be added.

The list of allowed elements for markdown could be configured as:

[
    'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
    'hr',
    'pre', 'code',
    'blockquote',
    'table', 'tr', 'td', 'th', 'thead', 'tbody',
    'strong', 'em', 'b', 'i', 'u', 's', 'span',
    'a', 'p', 'br', 'nobr',
    'ul', 'ol', 'li',
    'img',
],

The list of allowed attributes would be:

['th.align', 'td.align', 'ol.start', 'code.class']

The above configuration is a general recommendation and may need to be adjusted dependent on your needs.

Extensions

Here are some extensions to this library:

Bogardo/markdown-codepen - shortcode to embed codepens from http://codepen.io/ in markdown.
cebe/markdown-latex - Convert Markdown to LaTeX and PDF
softark/creole - A creole markup parser
hyn/frontmatter - Frontmatter Metadata Support (JSON, TOML, YAML)
... add yours!

Extending the language

Markdown consists of two types of language elements, I'll call them block and inline elements simlar to what you have in HTML with <div> and <span>. Block elements are normally spreads over several lines and are separated by blank lines. The most basic block element is a paragraph (<p>). Inline elements are elements that are added inside of block elements i.e. inside of text.

This markdown parser allows you to extend the markdown language by changing existing elements behavior and also adding new block and inline elements. You do this by extending from the parser class and adding/overriding class methods and properties. For the different element types there are different ways to extend them as you will see in the following sections.

Adding block elements

The markdown is parsed line by line to identify each non-empty line as one of the block element types. To identify a line as the beginning of a block element it calls all protected class methods who's name begins with identify. An identify function returns true if it has identified the block element it is responsible for or false if not. In the following example we will implement support for fenced code blocks which are part of the github flavored markdown.

<?php

class MyMarkdown extends \cebe\markdown\Markdown
{
	protected function identifyFencedCode($line, $lines, $current)
	{
		// if a line starts with at least 3 backticks it is identified as a fenced code block
		if (strncmp($line, '```', 3) === 0) {
			return true;
		}
		return false;
	}

	// ...
}

In the above, $line is a string containing the content of the current line and is equal to $lines[$current]. You may use $lines and $current to check other lines than the current line. In most cases you can ignore these parameters.

Parsing of a block element is done in two steps:

Consuming all the lines belonging to it. In most cases this is iterating over the lines starting from the identified line until a blank line occurs. This step is implemented by a method named consume{blockName}() where {blockName} is the same name as used for the identify function above. The consume method also takes the lines array and the number of the current line. It will return two arguments: an array representing the block element in the abstract syntax tree of the markdown document and the line number to parse next. In the abstract syntax array the first element refers to the name of the element, all other array elements can be freely defined by yourself. In our example we will implement it like this:

 protected function consumeFencedCode($lines, $current)
 {
 	// create block array
 	$block = [
 		'fencedCode',
 		'content' => [],
 	];
 	$line = rtrim($lines[$current]);

 	// detect language and fence length (can be more than 3 backticks)
 	$fence = substr($line, 0, $pos = strrpos($line, '`') + 1);
 	$language = substr($line, $pos);
 	if (!empty($language)) {
 		$block['language'] = $language;
 	}

 	// consume all lines until ```
 	for($i = $current + 1, $count = count($lines); $i < $count; $i++) {
 		if (rtrim($line = $lines[$i]) !== $fence) {
 			$block['content'][] = $line;
 		} else {
 			// stop consuming when code block is over
 			break;
 		}
 	}
 	return [$block, $i];
 }

Rendering the element. After all blocks have been consumed, they are being rendered using the render{elementName}()-method where elementName refers to the name of the element in the abstract syntax tree:
```
 protected function renderFencedCode($block)
 {
 	$class = isset($block['language']) ? ' class="language-' . $block['language'] . '"' : '';
 	return "<pre><code$class>" . htmlspecialchars(implode("\n", $block['content']) . "\n", ENT_NOQUOTES, 'UTF-8') . '</code></pre>';
 }
```
You may also add code highlighting here. In general it would also be possible to render ouput in a different language than HTML for example LaTeX.

Adding inline elements

Adding inline elements is different from block elements as they are parsed using markers in the text. An inline element is identified by a marker that marks the beginning of an inline element (e.g. [ will mark a possible beginning of a link or ` will mark inline code).

Parsing methods for inline elements are also protected and identified by the prefix parse. Additionally a @marker annotation in PHPDoc is needed to register the parse function for one or multiple markers. The method will then be called when a marker is found in the text. As an argument it takes the text starting at the position of the marker. The parser method will return an array containing the element of the abstract sytnax tree and an offset of text it has parsed from the input markdown. All text up to this offset will be removed from the markdown before the next marker will be searched.

As an example, we will add support for the strikethrough feature of github flavored markdown:

<?php

class MyMarkdown extends \cebe\markdown\Markdown
{
	/**
	 * @marker ~~
	 */
	protected function parseStrike($markdown)
	{
		// check whether the marker really represents a strikethrough (i.e. there is a closing ~~)
		if (preg_match('/^~~(.+?)~~/', $markdown, $matches)) {
			return [
			    // return the parsed tag as an element of the abstract syntax tree and call `parseInline()` to allow
			    // other inline markdown elements inside this tag
				['strike', $this->parseInline($matches[1])],
				// return the offset of the parsed text
				strlen($matches[0])
			];
		}
		// in case we did not find a closing ~~ we just return the marker and skip 2 characters
		return [['text', '~~'], 2];
	}

	// rendering is the same as for block elements, we turn the abstract syntax array into a string.
	protected function renderStrike($element)
	{
		return '<del>' . $this->renderAbsy($element[1]) . '</del>';
	}
}

Composing your own Markdown flavor

This markdown library is composed of traits so it is very easy to create your own markdown flavor by adding and/or removing the single feature traits.

Designing your Markdown flavor consists of four steps:

Select a base class
Select language feature traits
Define escapeable characters
Optionally add custom rendering behavior

Select a base class

If you want to extend from a flavor and only add features you can use one of the existing classes (Markdown, GithubMarkdown or MarkdownExtra) as your flavors base class.

If you want to define a subset of the markdown language, i.e. remove some of the features, you have to extend your class from Parser.

Select language feature traits

The following shows the trait selection for traditional Markdown.

class MyMarkdown extends Parser
{
	// include block element parsing using traits
	use block\CodeTrait;
	use block\HeadlineTrait;
	use block\HtmlTrait {
		parseInlineHtml as private;
	}
	use block\ListTrait {
		// Check Ul List before headline
		identifyUl as protected identifyBUl;
		consumeUl as protected consumeBUl;
	}
	use block\QuoteTrait;
	use block\RuleTrait {
		// Check Hr before checking lists
		identifyHr as protected identifyAHr;
		consumeHr as protected consumeAHr;
	}
	// include inline element parsing using traits
	use inline\CodeTrait;
	use inline\EmphStrongTrait;
	use inline\LinkTrait;

	/**
	 * @var boolean whether to format markup according to HTML5 spec.
	 * Defaults to `false` which means that markup is formatted as HTML4.
	 */
	public $html5 = false;

	protected function prepare()
	{
		// reset references
		$this->references = [];
	}

	// ...
}

In general, just adding the trait with use is enough, however in some cases some fine tuning is desired to get most expected parsing results. Elements are detected in alphabetical order of their identification function. This means that if a line starting with - could be a list or a horizontal rule, the preference has to be set by renaming the identification function. This is what is done with renaming identifyHr to identifyAHr and identifyBUl to identifyBUl. The consume function always has to have the same name as the identification function so this has to be renamed too.

There is also a conflict for parsing of the < character. This could either be a link/email enclosed in < and > or an inline HTML tag. In order to resolve this conflict when adding the LinkTrait, we need to hide the parseInlineHtml method of the HtmlTrait.

If you use any trait that uses the $html5 property to adjust its output you also need to define this property.

If you use the link trait it may be useful to implement prepare() as shown above to reset references before parsing to ensure you get a reusable object.

Define escapeable characters

Depending on the language features you have chosen there is a different set of characters that can be escaped using \. The following is the set of escapeable characters for traditional markdown, you can copy it to your class as is.

	/**
	 * @var array these are "escapeable" characters. When using one of these prefixed with a
	 * backslash, the character will be outputted without the backslash and is not interpreted
	 * as markdown.
	 */
	protected $escapeCharacters = [
		'\\', // backslash
		'`', // backtick
		'*', // asterisk
		'_', // underscore
		'{', '}', // curly braces
		'[', ']', // square brackets
		'(', ')', // parentheses
		'#', // hash mark
		'+', // plus sign
		'-', // minus sign (hyphen)
		'.', // dot
		'!', // exclamation mark
		'<', '>',
	];

Add custom rendering behavior

Optionally you may also want to adjust rendering behavior by overriding some methods. You may refer to the consumeParagraph() method of the Markdown and GithubMarkdown classes for some inspiration which define different rules for which elements are allowed to interrupt a paragraph.

Acknowledgements

I'd like to thank @erusev for creating Parsedown which heavily influenced this work and provided the idea of the line based parsing approach.

FAQ

Why another markdown parser?

While reviewing PHP markdown parsers for choosing one to use bundled with the Yii framework 2.0 I found that most of the implementations use regex to replace patterns instead of doing real parsing. This way extending them with new language elements is quite hard as you have to come up with a complex regex, that matches your addition but does not mess with other elements. Such additions are very common as you see on github which supports referencing issues, users and commits in the comments. A real parser should use context aware methods that walk trough the text and parse the tokens as they find them. The only implentation that I have found that uses this approach is Parsedown which also shows that this implementation is much faster than the regex way. Parsedown however is an implementation that focuses on speed and implements its own flavor (mainly github flavored markdown) in one class and at the time of this writing was not easily extensible.

Given the situation above I decided to start my own implementation using the parsing approach from Parsedown and making it extensible creating a class for each markdown flavor that extend each other in the way that also the markdown languages extend each other. This allows you to choose between markdown language flavors and also provides a way to compose your own flavor picking the best things from all. I chose this approach as it is easier to implement and also more intuitive approach compared to using callbacks to inject functionallity into the parser.

Where do I report bugs or rendering issues?

Just open an issue on github, post your markdown code and describe the problem. You may also attach screenshots of the rendered HTML result to describe your problem.

How can I contribute to this library?

Check the CONTRIBUTING.md file for more info.

Am I free to use this?

This library is open source and licensed under the MIT License. This means that you can do whatever you want with it as long as you mention my name and include the license file. Check the license for details.

Contact

Feel free to contact me using email or twitter.

Custom tags/extensions

A nice thing about ReST is its standard for adding custom directives. Would such a thing be useful in cebe/markdown?

Kramdown does this:

{::comment}
This is a comment which is
completely ignored.
{:/comment}

Inline {::nomarkdown}something here{:/}!

{::options auto_ids="false" /}

An alternative:

@tag arg arg
content content content

content content content
/@tag

Docutils (the lib with the ReST parser) has a nice Directive class that does some of the parsing for dumb bunnies like me. For example:

class Exercise(Directive):
    has_content = False
    required_arguments = 1

    def run(self):
        result = '[[[cycoexercise:' + self.arguments[0] + ']]]\n'
        raw_node = nodes.raw('', result, format='html')
        return [raw_node]

Given a tag like this...

.. exercise:: 666

... the Directive class would pick out the 666 and supply it to the subclass in self.arguments. If the tag has content, it shows up in self.content. You can do this...

        self.state.nested_parse(
            self.content,
            self.content_offset,
            content_node
        )

... if the content of the custom tag is to contain ReST.

Any chance of something like this?

Kieran

cebe / markdown Goto Github PK

markdown's Introduction