Improved & well-documented version of `codeToTokensWithThemes`

[antfu]

Improved & well-documented version of `codeToTokensWithThemes`

[antfu]

The reason we were doing this is because we want to be compatible with the ThemedToken interface (altering color from string to object will break the other tool), while merging all colors into color as a flat string might not be flexible enough.

Can you elaborate more about your usage? If it's a valid concern, we could introduce an additional utility for it maybe.

[fuma-nama]

Since the codeToHast and codeToHtml methods both support multiple(light/dark) themes, codeToThemedTokens should have supported it too, for a better API consistency.

Takes an example of a custom code block in React.js:

export function CodeBlock({ code, lang, className, ...props }: CodeBlockProps) {
  const tokens = useMemo(
    () =>
      highlighter.codeToThemedTokens(code, {
        lang
      }),
    [code]
  )

  return (
    <Pre className={cn('bg-black', className)} {...props}>
      <code className="grid">
        {tokens.map((token, i) => (
          <span data-line key={i}>
            {token.map((s, j) => (
              <span key={j} style={{ color: s.color }}>
                {s.content}
              </span>
            ))}
          </span>
        ))}
      </code>
    </Pre>
  )
}

Obviously if we use codeToTokensWithThemes here, we must preprocess the returned array on our own, to combine to all variants of a node into one. Otherwise, the simplest solution is to render a code block for individual themes, which is not ideal in terms of bundle size.

Although I ended up implementing my own function to flatten nodes, the codeToTokensWithThemes API itself is not well-documented and straightforward enough (feels like a hacky way to me). Would appreciate it if there’s an additional utility to make it more convenience.

[antfu]

I wonder what prevents you from using codeToHast directly for rendering the nodes?

[fuma-nama]

That would be a great solution, I initially missed it because codeToThemedTokens seemed to be more straightforward for me when comes to be rendering and animating it. Hast nodes are not typed and I was unsure of its properties, they are pretty enough for normal code blocks though