scalameta/mdoc

Typechecked markdown documentation for Scala

mdoc - typechecked markdown documentation for Scala

mdoc is a markdown documentation tool for Scala inspired by tut. Visit https://scalameta.org/mdoc to learn more about mdoc.

mdoc's Issues

Add fail option to our ammonite integration

False-positive exit code for multipage sites

Currently, the exit code only reflects the latest generated site when running under non --watch mode. This means the exit code can be 0 even if a page failed to build. The exit code should be an aggregate of the generated pages.

Support any language in code fences

Currently, code fences must have the language scala mdoc. However, with custom modifiers it's possible to render any syntax (JSON, markdown, ...) so it would be nice if it was possible to write any language

```sh mdoc
```

or

```mdoc
```

Custom modifiers should declare what language they target.

Make value pretty-printer configurable

Currently, mdoc uses pprint to pretty-print captured binders
https://github.com/olafurpg/mdoc/blob/137b75ca879f21635eedd418eb11b8cd29019b9d/mdoc/src/main/scala/mdoc/internal/markdown/Renderer.scala#L135

It would be nice to allow users to configure a custom pretty-printer. For example, in scalafix I would love the ability to customize rendering of Patch so that it appears like a unified diff without the captured variable name and type.

Code formatting in code fences is not preserved

It seems that the Scalameta tree printer is used instead of the underlying syntax from the parsed source

We don't transform the tree so I wonder what's going on, maybe Input.Slice is to blame? 🤔

Support evaluating code without showing values

I would like to write a code fence like this

```scala mdoc:silent
val program = """
object Math {
  def add(a: Int, b: Int) = a + b
}
"""
val tree = program.parse[Source]
```

and it would be evaluated but render normally without showing the type and runtime values of program and tree

Add file watching to vork

False positive warning for non-markdown links

warning: thesis.md:9:34: warning: Unknown link 'assets/olafur.geirsson-scalafmt-thesis.pdf'.
Read the 55 page glorious report [here](assets/olafur.geirsson-scalafmt-thesis.pdf).
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The pdf exists, but mdoc only takes into account markdown links.

Syntax highlighting

https://github.com/google/code-prettify

Support multiple modifiers in one code fence

For example, the following should be supported

```scala mdoc:reset:silent
val x = 1
```

Add silent option to our ammonite integration

Logo

http://squidfunk.github.io/mkdocs-material/getting-started/#adding-a-logo

Implement proper error reporting pipeline

Currently, errors are aggregated with Either and thrown as exceptions at the end of a source file. It would be nice if errors were reported like in sbt-errors-summary and without using exceptions.

Pass in the classpath from the config to the Ammonite REPL

Add book option to our ammonite integration

MainSettings.toString should show helpful information

Currently it shows mdoc.MainSettings@4ff74f2c

Include file

Something like https://szeiger.github.io/ornate-doc/markdown.html#includecode

Add MainSettings.withArgs(List[String])

It would be nice to pass in List[String] as cli args to MainSettings

ScalaJS support

No pressure, I just wanted an issue I can subscribe to. 😉

Add incrementality at the document level

Add passthrough option to our ammonite integration

Code fences inside quotes don't hide the quotes

The input

> **Pro tip.** If you don't know the name of the tree node you want to match,
> use `tree.structure`
>
> ```scala mdoc
> import scalafix.v1._
> import scala.meta._
> val x = q"complete(true)"
> println(x.structure)
> ```

produces the error

error: /Users/ollie/dev/scalafix/docs/developers/tutorial.md:100:3: error: ; expected but import found
> import scala.meta._
  ^^^^^^
error: Failed to generate /Users/ollie/dev/scalafix/website/target/docs/developers/tutorial.md

I'm not sure how we can support this use-case 🤔

Upgrade flexmark dependency

While trying to upgrade the flexmark dependency I encountered a regression vsch/flexmark-java#271 we are using a fairly old flexmark version so it would be good to upgrade when that issue is fixed.

Admonition

http://squidfunk.github.io/mkdocs-material/extensions/admonition/

Add mdoc:invisible modifiers

It should be exactly like the tut invisible modifier. A workaround is to use the mdoc:passthrough modifier but that works only if the invisible code does not print to stdout.

Support inlining file contents

It would be great if there was a way to inline file contents like this

```scala vork:file:src/main/scala/Main.scala
```

This would then inline the file contents of Main.scala. For bonus points, it would be nice to be able to select line ranges or start/end based on regexp.

Crash modifier looses comments

```scala mdoc:crash
q"final val x = 2" match {
  case q"val x = 2" => // boom!
}
```

results in

q"final val x = 2" match {
  case q"val x = 2" =>
}
// scala.MatchError: final val x = 2 (of class scala.meta.Defn$Val$DefnValImpl)
//  at repl.Session.$anonfun$app$47(/Users/ollie/dev/scalameta-tutorial/docs/trees/guide.md:252)

IllegalArgumentException with fail modifier

Reported on Gitter https://gitter.im/olafurpg/mdoc?at=5bb5cdb5435c2a518e4cc235

$ cat docs/in.md
```scala mdoc:fail
fs2.Stream.eval(println("Do not ever do this"))
```
$ coursier launch com.geirsson:mdoc_2.12.7:0.5.1 co.fs2:fs2-core_2.12:0.10.4
info: Compiling /Users/ollie/dev/metals/target/docs/in.md
mdoc.internal.cli.FileException: /Users/ollie/dev/metals/target/docs/in.md
Caused by: java.lang.IllegalArgumentException: 6 is not a valid line number, allowed [0..1]
	at scala.meta.internal.inputs.InternalInput.lineToOffset(InternalInput.scala:30)
	at scala.meta.internal.inputs.InternalInput.lineToOffset$(InternalInput.scala:26)
	at scala.meta.inputs.Input$Slice.lineToOffset(Input.scala:83)
	at scala.meta.inputs.Position$Range$.apply(Position.scala:58)
	at mdoc.internal.pos.PositionSyntax$XtensionRangePositionMdoc.toMeta(PositionSyntax.scala:50)
	at mdoc.internal.pos.PositionSyntax$XtensionRangePositionMdoc.formatMessage(PositionSyntax.scala:34)
	at mdoc.internal.markdown.Renderer$.$anonfun$renderEvaluatedSection$2(Renderer.scala:134)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at mdoc.internal.markdown.Renderer$.$anonfun$renderEvaluatedSection$1(Renderer.scala:119)
	at mdoc.internal.markdown.Renderer$.$anonfun$renderEvaluatedSection$1$adapted(Renderer.scala:100)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at mdoc.internal.markdown.Renderer$.renderEvaluatedSection(Renderer.scala:100)
	at mdoc.internal.markdown.MdocPostProcessor.$anonfun$processScalaInputs$2(MdocPostProcessor.scala:82)
	at mdoc.internal.markdown.MdocPostProcessor.$anonfun$processScalaInputs$2$adapted(MdocPostProcessor.scala:72)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at mdoc.internal.markdown.MdocPostProcessor.processScalaInputs(MdocPostProcessor.scala:72)
	at mdoc.internal.markdown.MdocPostProcessor.processDocument(MdocPostProcessor.scala:33)
	at com.vladsch.flexmark.internal.PostProcessorManager.postProcess(PostProcessorManager.java:71)
	at com.vladsch.flexmark.internal.PostProcessorManager.processDocument(PostProcessorManager.java:54)
	at com.vladsch.flexmark.parser.Parser.postProcess(Parser.java:389)
```

Add invisible option to our ammonite integration

Compiler warnings do not appear in the output

Given the following code:

```scala mdoc
def unwrap(oi: Option[Int]): Int = oi match {
  case Some(i) => i
}

I would expect the mdoc output to contain the non-exhaustivity warning, or to have a :warnings modifier of some sort to do so.

What I get instead is:

def unwrap(oi: Option[Int]): Int = oi match {
  case Some(i) => i
}

Note that there's an easy workaround, which is to make compiler warnings fatal and use :fail, but this might have other unwanted side effects.

Report error when referencing non-existing section

Vork should report a warning or error when you reference a non-existing section in the current or another page. Sphinx does this by default, read the docs as well.

Config

HOCON seems like a fairly good fit, https://github.com/circe/circe-config

Site variable injection is triggered when it shouldn't

This test here is failing

  check(
    "travis",
    "# Title ![![travis][travis-image]][travis-url]",
    "# Title ![![travis][travis-image]][travis-url]"
  )

with

Missing 'travis' site variable.

False-negative link warning

With the layout

/rules/RemoveUnused.md
/users/faq.md

and linking in faq.md

warning: users/faq.md:166:15: warning: Section '../rules/RemoveUnused.md' does not exist
instructed in [RemoveUnused](../rules/RemoveUnused.md).
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Vork should be released with CrossVersion.full

We transitively depend on scala-compiler via ammonite. I tried out vork in metaconfig where I had scalaVersion := 2.12.2 and it crashed with the following error

[info] Compiling (synthetic)/ammonite/predef/interpBridge.sc
[error] vork.Processor$FileError: /Users/ollie/dev/metaconfig/docs/SUMMARY.md
[info]
[info]
[info]
[info]
[info]
[error] Caused by: java.lang.Exception: Error during predef initialization:
[error] 	at vork.markdown.repl.AmmoniteRepl.$anonfun$new$2(AmmoniteRepl.scala:134)
[error] 	at scala.Option$WithFilter.foreach(Option.scala:257)
[error] 	at vork.markdown.repl.AmmoniteRepl.<init>(AmmoniteRepl.scala:120)
[error] 	at vork.markdown.processors.AmmonitePostProcessor.<init>(AmmonitePostProcessor.scala:12)
[error] 	at vork.markdown.processors.AmmonitePostProcessor$Factory.create(AmmonitePostProcessor.scala:56)
[error] 	at vork.markdown.processors.AmmonitePostProcessor$Factory.create(AmmonitePostProcessor.scala:54)
[error] 	at com.vladsch.flexmark.internal.PostProcessorManager.postProcess(PostProcessorManager.java:71)
[error] 	at com.vladsch.flexmark.internal.PostProcessorManager.processDocument(PostProcessorManager.java:54)
[error] 	at com.vladsch.flexmark.parser.Parser.postProcess(Parser.java:314)
[error] 	at com.vladsch.flexmark.parser.Parser.parse(Parser.java:294)
[error] 	at vork.Processor.handleMarkdown(Processor.scala:33)
[error] 	at vork.Processor.run$1(Processor.scala:53)
[error] 	at vork.Processor.handlePath(Processor.scala:59)
[error] 	at vork.Processor.$anonfun$generateCompleteSite$1(Processor.scala:82)
[error] 	at vork.Processor.$anonfun$generateCompleteSite$1$adapted(Processor.scala:82)
[error] 	at scala.collection.immutable.List.foreach(List.scala:389)
[error] 	at vork.Processor.generateCompleteSite(Processor.scala:82)
[error] 	at vork.Processor.run(Processor.scala:88)
[error] 	at vork.Cli$.main(Cli.scala:16)
[error] 	at vork.Cli.main(Cli.scala)
[error] Caused by: java.lang.AbstractMethodError: Method ammonite/interp/CompilerCompatibility$$anon$2._lockedCount_$eq(I)V is abstract
[error] 	at ammonite.interp.CompilerCompatibility$$anon$2._lockedCount_$eq(CompilerCompatibility.scala)
[error] 	at scala.tools.nsc.typechecker.Namers.$init$(Namers.scala:26)
[error] 	at ammonite.interp.CompilerCompatibility$$anon$2.<init>(CompilerCompatibility.scala:13)
[error] 	at ammonite.interp.CompilerCompatibility$.analyzer(CompilerCompatibility.scala:13)
[error] 	at ammonite.interp.GlobalInitCompat$$anon$2.analyzer$lzycompute(GlobalInitCompat.scala:50)
[error] 	at ammonite.interp.GlobalInitCompat$$anon$2.analyzer(GlobalInitCompat.scala:50)
[error] 	at scala.tools.nsc.Global.computeInternalPhases(Global.scala:629)
[error] 	at scala.tools.nsc.Global.computePhaseDescriptors(Global.scala:675)
[error] 	at scala.tools.nsc.Global.phaseDescriptors$lzycompute(Global.scala:682)
[error] 	at scala.tools.nsc.Global.phaseDescriptors(Global.scala:682)
[error] 	at scala.tools.nsc.Global$Run.<init>(Global.scala:1171)
[error] 	at ammonite.interp.Compiler$$anon$2.<init>(Compiler.scala:252)
[error] 	at ammonite.interp.Compiler$.apply(Compiler.scala:151)
[error] 	at ammonite.interp.CompilerLifecycleManager.init(CompilerLifecycleManager.scala:80)
[error] 	at ammonite.interp.CompilerLifecycleManager.preprocess(CompilerLifecycleManager.scala:49)
[error] 	at ammonite.interp.Interpreter.compileRunBlock$1(Interpreter.scala:466)
[error] 	at ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$15(Interpreter.scala:523)
[error] 	at ammonite.util.Res$Success.flatMap(Res.scala:61)
[error] 	at ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$14(Interpreter.scala:522)
[error] 	at ammonite.util.Res$Success.flatMap(Res.scala:61)
[error] 	at ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$12(Interpreter.scala:519)
[error] 	at scala.Option.getOrElse(Option.scala:121)
[error] 	at ammonite.interp.Interpreter.loop$1(Interpreter.scala:519)
[info] /Users/ollie/dev/metaconfig/vork/target/vork
[error] 	at ammonite.interp.Interpreter.processAllScriptBlocks(Interpreter.scala:555)
[error] 	at ammonite.interp.Interpreter.$anonfun$processModule$6(Interpreter.scala:354)
[error] 	at ammonite.util.Catching.flatMap(Res.scala:114)
[error] 	at ammonite.interp.Interpreter.$anonfun$processModule$5(Interpreter.scala:345)
[error] 	at ammonite.util.Res$Success.flatMap(Res.scala:61)
[error] 	at ammonite.interp.Interpreter.processModule(Interpreter.scala:337)
[error] 	at ammonite.interp.Interpreter.$anonfun$initializePredef$1(Interpreter.scala:109)
[error] 	at ammonite.interp.Interpreter.$anonfun$initializePredef$1$adapted(Interpreter.scala:109)
[error] 	at ammonite.interp.PredefInitialization$.$anonfun$apply$7(PredefInitialization.scala:65)
[error] 	at ammonite.util.Res$.$anonfun$fold$1(Res.scala:31)
[error] 	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:122)
[error] 	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:118)
[error] 	at scala.collection.immutable.List.foldLeft(List.scala:86)
[error] 	at ammonite.util.Res$.fold(Res.scala:29)
[error] 	at ammonite.interp.PredefInitialization$.apply(PredefInitialization.scala:53)
[error] 	at ammonite.interp.Interpreter.initializePredef(Interpreter.scala:111)
[error] 	... 18 more

Upgrading metaconfig to 2.12.4 fixed the problem.

Footnotes are not preserved

input:

Hello [^1]

[^1]: Goodbye

output:

Hello [^1]

[^1]: 

        Goodbye

Close at will if this isn't relevant, since footnotes are extended syntax ✌️

Add evaluated option to our ammonite integration

error: ']' expected but double literal found

$ cat docs/index.md
```scala mdoc
import eu.timepit.refined.W
import eu.timepit.refined.api.Refined
import eu.timepit.refined.numeric.Interval

type Value = Int Refined Interval.Closed[W.`0`.T, W.`7`.T]
```
$  coursier launch com.geirsson:mdoc_2.12.7:0.5.1 eu.timepit:refined_2.12:0.9.2  --
info: Compiling /Users/ollie/dev/mdoc/target/examples/repro/docs/index.md
error: error: ']' expected but double literal found.

The code parses OK with the scalameta parser and scala compiler normally so it appears the problem lies in the instrumented code. The message has no position which indicates something fishy is happening.

Stack traces should contain relative links

Lack of relative links forces us to use --report-relative-links which result in worse error messages.

// scala.MatchError: final val x = 2 (of class scala.meta.Defn$Val$DefnValImpl)
//  at repl.Session.$anonfun$app$47(/Users/ollie/dev/scalameta-tutorial/docs/trees/guide.md:252)

Handle conflicting header IDs in link hygiene

The SemanticDB spec has three "Synthetic" headers https://github.com/scalameta/scalameta/blob/master/semanticdb/semanticdb3/semanticdb3.md#synthetic-1

Currently, mdoc doesn't append the -N suffix for conflicting names.

Console should not clear when generating full site

Currently, the console is always cleared before processing a page under --watch mode. However, the console should only clear after a file-event, not during the initial full-site generation.

Edit this page link

Issues with Future ?

Given the two following blocks:

```scala mdoc
import scala.concurrent._
import scala.concurrent.duration._
import ExecutionContext.Implicits.global

def longRunning(): Int = {
  Thread.sleep(100)
  println("LONG")
  1
}

def immediate(): Int = {
  println("IMMEDIATE")
  2
}

def combine(): Future[Int] = for {
  i <- Future(longRunning())
  j <- Future(immediate())
} yield i + j

And

```scala mdoc
Await.result(combine(), 500.millis)

I would expect the Future to complete.

This is what happens in 1.0.0, but 1.1.0 spits out the following stack trace:

java.lang.NoClassDefFoundError: Could not initialize class repl.Session$App$[error] (run-main-6) java.lang.ExceptionInInitializerError

	at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)
	at scala.util.Success.$anonfun$map$1(Try.scala:251)
	at scala.util.Success.map(Try.scala:209)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[error] java.lang.ExceptionInInitializerError
[error] 	at repl.Session$.app(future_in_comprehensions.md:3)
[error] 	at mdoc.internal.document.DocumentBuilder$$doc$.$anonfun$build$2(DocumentBuilder.scala:76)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] 	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] 	at scala.Console$.withErr(Console.scala:192)
[error] 	at mdoc.internal.document.DocumentBuilder$$doc$.$anonfun$build$1(DocumentBuilder.scala:76)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] 	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] 	at scala.Console$.withOut(Console.scala:163)
[error] 	at mdoc.internal.document.DocumentBuilder$$doc$.build(DocumentBuilder.scala:75)
[error] 	at mdoc.internal.markdown.MarkdownCompiler$.buildDocument(MarkdownCompiler.scala:50)
[error] 	at mdoc.internal.markdown.MdocPostProcessor.processScalaInputs(MdocPostProcessor.scala:80)
[error] 	at mdoc.internal.markdown.MdocPostProcessor.processDocument(MdocPostProcessor.scala:35)
[error] 	at com.vladsch.flexmark.internal.PostProcessorManager.postProcess(PostProcessorManager.java:71)
[error] 	at com.vladsch.flexmark.internal.PostProcessorManager.processDocument(PostProcessorManager.java:54)
[error] 	at com.vladsch.flexmark.parser.Parser.postProcess(Parser.java:389)
[error] 	at com.vladsch.flexmark.parser.Parser.parse(Parser.java:369)
[error] 	at mdoc.internal.markdown.Markdown$.toDocument(Markdown.scala:81)
[error] 	at mdoc.internal.markdown.Markdown$.toMarkdown(Markdown.scala:114)
[error] 	at mdoc.internal.cli.MainOps.handleMarkdown(MainOps.scala:63)
[error] 	at mdoc.internal.cli.MainOps.handleFile(MainOps.scala:90)
[error] 	at mdoc.internal.cli.MainOps.$anonfun$generateCompleteSite$1(MainOps.scala:129)
[error] 	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:122)
[error] 	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:118)
[error] 	at scala.collection.immutable.List.foldLeft(List.scala:85)
[error] 	at mdoc.internal.cli.MainOps.generateCompleteSite(MainOps.scala:127)
[error] 	at mdoc.internal.cli.MainOps.run(MainOps.scala:148)
[error] 	at mdoc.internal.cli.MainOps$.process(MainOps.scala:230)
[error] 	at mdoc.Main$.process(Main.scala:26)
[error] 	at mdoc.Main$.process(Main.scala:21)
[error] 	at mdoc.Main$.main(Main.scala:16)
[error] 	at mdoc.Main.main(Main.scala)
[error] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] 	at java.lang.reflect.Method.invoke(Method.java:497)
[error] Caused by: java.util.concurrent.TimeoutException: Futures timed out after [500 milliseconds]
[error] 	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
[error] 	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
[error] 	at scala.concurrent.Await$.$anonfun$result$1(package.scala:215)
[error] 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
[error] 	at scala.concurrent.Await$.result(package.scala:142)
[error] 	at repl.Session$App$.<init>(future_in_comprehensions.md:48)
[error] 	at repl.Session$App$.<clinit>(future_in_comprehensions.md)
[error] 	at repl.Session$.app(future_in_comprehensions.md:3)
[error] 	at mdoc.internal.document.DocumentBuilder$$doc$.$anonfun$build$2(DocumentBuilder.scala:76)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] 	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] 	at scala.Console$.withErr(Console.scala:192)
[error] 	at mdoc.internal.document.DocumentBuilder$$doc$.$anonfun$build$1(DocumentBuilder.scala:76)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] 	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] 	at scala.Console$.withOut(Console.scala:163)
[error] 	at mdoc.internal.document.DocumentBuilder$$doc$.build(DocumentBuilder.scala:75)
[error] 	at mdoc.internal.markdown.MarkdownCompiler$.buildDocument(MarkdownCompiler.scala:50)
[error] 	at mdoc.internal.markdown.MdocPostProcessor.processScalaInputs(MdocPostProcessor.scala:80)
[error] 	at mdoc.internal.markdown.MdocPostProcessor.processDocument(MdocPostProcessor.scala:35)
[error] 	at com.vladsch.flexmark.internal.PostProcessorManager.postProcess(PostProcessorManager.java:71)
[error] 	at com.vladsch.flexmark.internal.PostProcessorManager.processDocument(PostProcessorManager.java:54)
[error] 	at com.vladsch.flexmark.parser.Parser.postProcess(Parser.java:389)
[error] 	at com.vladsch.flexmark.parser.Parser.parse(Parser.java:369)
[error] 	at mdoc.internal.markdown.Markdown$.toDocument(Markdown.scala:81)
[error] 	at mdoc.internal.markdown.Markdown$.toMarkdown(Markdown.scala:114)
[error] 	at mdoc.internal.cli.MainOps.handleMarkdown(MainOps.scala:63)
[error] 	at mdoc.internal.cli.MainOps.handleFile(MainOps.scala:90)
[error] 	at mdoc.internal.cli.MainOps.$anonfun$generateCompleteSite$1(MainOps.scala:129)
[error] 	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:122)
[error] 	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:118)
[error] 	at scala.collection.immutable.List.foldLeft(List.scala:85)
[error] 	at mdoc.internal.cli.MainOps.generateCompleteSite(MainOps.scala:127)
[error] 	at mdoc.internal.cli.MainOps.run(MainOps.scala:148)
[error] 	at mdoc.internal.cli.MainOps$.process(MainOps.scala:230)
[error] 	at mdoc.Main$.process(Main.scala:26)
[error] 	at mdoc.Main$.process(Main.scala:21)
[error] 	at mdoc.Main$.main(Main.scala:16)
[error] 	at mdoc.Main.main(Main.scala)
[error] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Dele

Leading statement comments are removed

// Either Option class or companion object
val option = SymbolMatcher.normalized("scala.Option")
option.matches(Symbol("scala/Option#")) // Option class
option.matches(Symbol("scala/Option.")) // Option object

// Any println method overload
val assert = SymbolMatcher.normalized("scala.Predef.assert")
assert.matches(Symbol("scala/Predef#assert()."))   // println()
assert.matches(Symbol("scala/Predef#assert(+1).")) // println(String)
assert.matches(Symbol("scala/Predef#assert()."))   // println()

becomes

val option = SymbolMatcher.normalized("scala.Option")
// option: SymbolMatcher = NormalizedSymbolMatcher(Set(scala.Option.))
option.matches(Symbol("scala/Option#")) // Option class
// res4: Boolean = true
option.matches(Symbol("scala/Option.")) // Option object
// res5: Boolean = true


val assert = SymbolMatcher.normalized("scala.Predef.assert")
// assert: SymbolMatcher = NormalizedSymbolMatcher(Set(scala.Predef.assert.))
assert.matches(Symbol("scala/Predef#assert()."))   // println()
// res6: Boolean = true
assert.matches(Symbol("scala/Predef#assert(+1).")) // println(String)
// res7: Boolean = true
assert.matches(Symbol("scala/Predef#assert()."))   // println()
// res8: Boolean = true

I expected the comments "// Either Option class or companion object" and "// Any println method overload" to be preserved.

Support persistent incremental builds

Currently, mdoc supports incremental builds for a long-running session with --watch. However, the full site must be re-generated every time we restart mdoc.

mdoc is already quite fast, but for larger sites with dozens/hundreds of pages it still takes a while to re-process all files. We are after all mostly bottle-necked by the Scala compiler. It would be nice if mdoc had a persistent cache so that it can skip re-compiling unchanged sites on startup.

Challenges

• how do we invalidate cache when custom modifiers and the classpath changes? I think it's best to do nothing smart, let users manually delete --out before re-starting mdoc. Custom modifiers change rarely anyways
• where should the cache be located? I propose to persist the cache in a single file with default location cwd/target/mdoc.cache and can be configured with withCachePath(Path)

The approach I had in mind was that for each file we cache three values

• absolute path of the input file
• md5 hash of file input contents
• md5 hash of file output contents

If all hashes in the cache match with the hashed contents of the input/output file then mdoc skips generating the site.

Add reset option to our ammonite integration

Experiment with CommonMark formatter options to use Pandoc

See vsch/flexmark-java#192

Unable to pass in numbers as values for site variables

When the value for a site variable is a number, the following error occurs.

[info] Running mdoc.Main --in /Users/vlovgr/Projects/blog/posts --out /Users/vlovgr/Projects/blog/hugo/content/post --scalac-options -deprecation -encoding UTF-8 -feature -language:existentials -language:higherKinds -language:implicitConversions -unchecked -Xlint -Yno-adapted-args -Ywarn-dead-code -Ywarn-numeric-widen -Ywarn-value-discard -Xfuture -Xplugin:/Users/vlovgr/.ivy2/cache/org.spire-math/kind-projector_2.12/jars/kind-projector_2.12-0.9.8.jar --site.REFINED_VERSION 0.15 --watch --no-livereload
error: Type mismatch;
  found    : Number (value: 0.15)
  expected : String

Support custom compiler options

Currently, it's not possible to customize compiler flags such as compiler plugins (example: kind-projector). I think the best approach to solve this problem is to support reading the property file mdoc.properties that contains the following keys

• scalacOptions: space separated list of compiler options that we feed to the scala compiler as if they were passed to scalac from the command line
• scalaVersion: to guard that the compiler version matches the mdoc version

For now, users can write the resource generator themselves but in the future we could publish an sbt plugin to generate this resource file.

NullPointerException while reporting compiler diagnostics

It seems the compiler can report diagnostics with a null message

mdoc.internal.cli.FileException: /Users/ollie/dev/scalafix/docs/developers/semantic-type.md
Caused by: java.lang.NullPointerException
	at scalafix.internal.util.PositionSyntax$.formatMessage(PositionSyntax.scala:19)
	at scalafix.internal.util.PositionSyntax$XtensionPositionsScalafix$.formatMessage$extension(PositionSyntax.scala:41)
	at mdoc.internal.io.ConsoleReporter.error(ConsoleReporter.scala:34)
	at mdoc.internal.io.ConsoleReporter.error(ConsoleReporter.scala:30)
	at mdoc.internal.markdown.MarkdownCompiler$.buildDocument(MarkdownCompiler.scala:62)
	at mdoc.internal.markdown.MdocPostProcessor.processScalaInputs(MdocPostProcessor.scala:70)