`transformerMetaWordHighlight` highlight is not working when word includes symbols

[antfu]

`transformerMetaWordHighlight` highlight is not working when word includes symbols

[antfu]

Still some refactor to do to support multiple token highlight

[fuma-nama]

I am willing to submit a fix

[fuma-nama]

Perhaps we can include other common symbols like " | . - ,. But that is possible to have edge cases where the word is separated into multiple tokens.

Quotes are usually separated into another token, so matching "jsx" will not work with options: "jsx" for example. We have to allow a word to transcend multiple tokens?

[antfu]

Great, here is a reference in twoslash integration:

shikiji/packages/shikiji-twoslash/src/core.ts

Lines 202 to 228 in 4b61e33

	case 'highlight': {
	if (renderer.nodesHighlight) {
	actionsHighlights.push(() => {
	const line = this.lines[node.line]
	let charIndex = 0
	let itemStart = line.children.length
	let itemEnd = 0
	line.children.forEach((token, index) => {
	if (charIndex >= node.character && index < itemStart)
	itemStart = index
	if ((charIndex <= node.character + node.length) && index > itemEnd)
	itemEnd = index
	charIndex += getTokenString(token).length
	})
	if ((charIndex <= node.character + node.length))
	itemEnd = line.children.length
	const targets = line.children.slice(itemStart, itemEnd)
	const length = targets.length
	const highlighted = renderer.nodesHighlight?.call(this, node, targets) || targets
	line.children.splice(itemStart, length, ...highlighted)
	})
	}
	break
	}

[brc-dd]

I think vuejs/vitepress#3503 is also related to this. There, the text that's being matched against the regex is not the whole line instead individual tokens 👀 The comments like // [!code error] don't work if they are not entirely in the same text node.

[fuma-nama]

I think vuejs/vitepress#3503 is also related to this. There, the text that's being matched against the regex is not the whole line instead individual tokens 👀 The comments like // [!code error] don't work if they are not entirely in the same text node.

That is caused by the limitation of design rather than the transformer itself, adding comments after attributes is not valid in Vue. Therefore, you'll see a weird highlighting here:

<template>
  <div
    class="container max-w-250 mx-a my-10 min-h-250 vp-doc p4" // something
    of-hidden
  >
    <ShikijiMiniPlayground />
  </div>
</template>

Writing this in real code even gives you syntax errors, which is not supposed to work.

[antfu]

@brc-dd Feel free to open an issue on https://github.com/shikijs/shiki and we could discuss what would be the solution for that :)