RFC: TSDoc-flavored-Markdown (TSFM) instead of CommonMark

[microsoft]

RFC: TSDoc-flavored-Markdown (TSFM) instead of CommonMark

[dend]

Something worth calling out here is how this can interact with docs.microsoft.com/DocFX. Now, I know that we are working on a standard here, but fragmentation and a bunch of custom stuff is a bit of a concern. We do have support for Markdown Extensions, so likely that should be a place where we can plug in.

The format you are talking about here is parser-specific - on docs.microsoft.com, we've recently switched to MarkDig, that handles CommonMark parsing much better. It would be preferable to not be inventing our own standard due to the fact that the rest of the documentation stack does not use (and we have no plans to), and guiding people to one set of conventions for TS documentation contributions and another one for the rest of docs seems problematic. Besides, this also adds the added issue of our own parser interpreting the proposed conventions incorrectly.

[sharwell]

@pgonzal The current examples show an input that would be treated differently by the two implementations, but only show one normalized form. Each example should be presented with two normalized forms:

1. The normalized form that causes both parsers to interpret the input in the manner that TSFM interprets the original input
2. The normalized form that causes both parsers to interpret the input in the manner that CommonMark interprets the original input (this is the one that's missing)

[octogonz]

I started prototyping this idea today. If we go this route, it will be a major simplification and should save us a lot of time on the implementation.

[dschnare]

The format you are talking about here is parser-specific - on docs.microsoft.com, we've recently switched to MarkDig, that handles CommonMark parsing much better. It would be preferable to not be inventing our own standard due to the fact that the rest of the documentation stack does not use (and we have no plans to), and guiding people to one set of conventions for TS documentation contributions and another one for the rest of docs seems problematic. Besides, this also adds the added issue of our own parser interpreting the proposed conventions incorrectly.

MarkDig is only a .NET implementation right? Won't that be problematic for JavaScript/TypeScript authors (i.e. they'd need a way to run .NET on their machine or CI server).

[sharwell]

While I agree with encouraging users to use a normalized input form where available, I generally disagree with the premise of this proposal (that CommonMark is problematic and/or suggesting that another Markdown flavor will serve to simplify the space). The following are specific claims which I most disagree with:

I want to say that ">" blockquotes should not be supported at all, since the whitespace handling for these constructs is highly counterintuitive.

This form is widely used (it's even the default behavior when you click the quote formatting button in the GitHub editor), and everyone since the dawn of email knows that > at the beginning of a line means a quote.

However, CommonMark lists also have a lot of counterintuitive rules regarding handling of whitespace.

These rules may cause some confusion for new users, but the general availability of live-preview editors helps users avoid the pitfalls.

No other character (e.g. "*" or "+") can be used to create lists.

This is confusing. Many people (myself included), only use one of these for lists.

… then HTML tags ... should be used …

For any case directly supported by CommonMark without the use of HTML tags, I would oppose a restriction that does not allow a normalized form of the same content to exist without using HTML tags. Markdown provides a set of features which generally allow users to avoid falling back to HTML tags, and a Markdown processor which deviates from this goal feels incomplete. As a user, it would be frustrating to be told Markdown can be used, only to find that it can only sometimes be used.

[octogonz]

@sharwell thanks for your feedback here. As I mentioned in the initial issue description, TSDoc and Markdown have somewhat different requirements, which greatly complicates attempts to embed Markdown inside TSDoc. The two biggest hangups for me are:

1. Predictability: TSDoc cannot assume that a person authoring a code comment will have an easy way to interactively preview how it gets rendered. For example, when they type a * character, it must be completely obvious whether this symbol means boldface, italics, list item, or a literal * character. (By contrast, every Markdown implementation has lots of "gotcha" behaviors that trip people up, but they just fiddle with the interactive preview until it comes out right.)

2. Interoperability: The premise of TSDoc is that different tools must be able to process the same input and agree on its interpretation. In particular, they must exactly agree about questions such as "Does this @ character start a TSDoc tag or not?" (By contrast, CommonMark is more about improving consistency when people move between different projects that use different Markdown engines. In my experience it's pretty rare for the same input file to be production-rendered by two different Markdown engines. Even with CommonMark there are going to be incompatibilities if you attempt that. It's actually quite common for a single Markdown engine to evolve its own syntax in a way that breaks some existing content.)

Keep in mind that TSDoc is not just some English prose for humans to read. TSDoc goes in computer source code, and sometimes it contains tags that affect how a project gets built. Often it gets edited in a Git "merge conflict" editor that doesn't have any nice syntax highlighting.

I want to say that ">" blockquotes should not be supported at all, since the whitespace handling for these constructs is highly counterintuitive.

This form is widely used (it's even the default behavior when you click the quote formatting button in the GitHub editor), and everyone since the dawn of email knows that > at the beginning of a line means a quote.

That seems reasonable. But could you share an example of a realistic TypeScript code comment where someone would need to use >? For our own projects, the content that we write inside code comments (i.e. API reference) seems to be much simpler than what goes in a regular Markdown file (i.e. feature articles and tutorials). So for us at least, there is relatively less demand for advanced text formatting in TSDoc.

As a user, it would be frustrating to be told Markdown can be used, only to find that it can only sometimes be used.

I agree it's frustrating. But maybe it would be less frustrating than a realization that the syntax is not predictable ("I have no idea whether the stuff I'm writing will get rendered correctly by whatever documentation tool runs in this particular repo") or not interoperable ("I marked this API as @beta -- but we accidentally shipped it to production because I didn't understand some esoteric Markdown grammar rule.)

That said, I'm not being dogmatic about this. We modeled TSDoc as an open standard specifically to solicit your input and ideas. :-) A lot of these debates seem to get settled when we switch from philosophy and design, and instead look at specific real-world documentation problems that turn up. For example #128 was fairly enlightening for me personally.

By end of 2018, our API Extractor tool will have fully implemented all the core features of TSDoc (including declaration references) and processed a fairly large corpus of Microsoft APIs. When we write up the spec proposal, I want it to include real-world examples for each design decision.

[sharwell]

That seems reasonable. But could you share an example of a realistic TypeScript code comment where someone would need to use >?

@pgonzal The only case where it's come up for me to date is here:
tunnelvisionlabs/antlr4ts@334007d

Historically, the other thing I've used the quote syntax for is arguably improper, e.g. callouts like this:

⚠️ This method likely does not behave as you expect.

It's the best translation I could think of at the time for what I would prefer to write with C#'s <note> element:

/// <note type="warning">
/// <para>This method likely does not behave as you expect.</para>
/// </note>