code-hike/lighter

The syntax highlighter used by Code Hike.

lighter's Issues

Use CSS classes instead of colors

To support darkmode I was hoping to switch to using class names instead of just colors. My guess was I can do this easiest by using annotations and then converting those to class names?

Export list of theme names and valid aliases

Add highlightSync

that fails if the theme and grammars are not loaded

(will need an async loadGrammars function too)

fenchJSON must use external network

I looked at the source code and found that the selection and configuration of the language must be from the https://lighter.codehike.org/api/ api, but due to the work environment, we cannot access this website, resulting in failure to start code-hike

Throw typed error when grammar is unknown

Allow the user to control the process of loading grammars & themes

The current behaviour is a good default as it mostly just works across all environments. However there are cases where I want to be in control of how/when the grammars and themes are loaded. In general: as a user I want to tightly control the IO operations of the library.

Consider the following use cases:

• I'm running lighter in an environment without network access, I instead want to choose to read the grammar/theme from the filesystem or other async IO
• I'm running lighter without access to a filesystem or any other kind of IO, instead I want to read the grammar/theme from memory
• I'm running lighter in an environment where I want to customise the fetch call. eg. in Next.js I want to pass next: { revalidate: ... }

Use ansi codes to highlight `terminal` code

Example code from Code Hike

```terminal
�[2m Test Files �[22m �[1m�[32m6 passed�[39m�[22m�[90m (6)�[39m
�[2m      Tests �[22m �[1m�[32m52 passed�[39m�[22m�[90m (52)�[39m
�[2m   Start at �[22m 19:43:40
�[2m   Duration �[22m 3.86s�[2m (transform 2.49s, setup 1ms, collect 6.72s, tests 2.19s)�[22m

�[?25h�[2K�[1GDone in 4.32s.
```

Parser example: https://github.com/travis-ci/travis-web/blob/76af32013bc3ab1e5f540d69da3a97c3fec1e7e9/assets/scripts/vendor/ansiparse.js

Add

Possible to see if a language is available?

This library looks really great and I am hoping to start using it. One thing I need to do, though, is see if a language is available (right now we are just parsing the extension by split(".").last() and trying to use that for lang). That will cause Unknown language errors though for filenames like "env.development" though.

Is it possible to check to see if a language is available in a relatively fast way so I can just default to "text" if not?

Preload common grammars

js, html, css, py

Edge support?

I think right now this wont support edge based on how it is loading the WASM (and due to dependence on FS although maybe the browser version can be used?). I wondered if somehow using https://vercel.com/docs/concepts/functions/edge-functions/wasm this could be shipped to an edge function, though.

Update vscode-textmate and onigurama

Add CommonJS output

Support static builds in Next.js app directory

I'm not sure why but whenever I use this library it forces the page to be dynamically rendered. I think in theory this isn't required right? Any fetches can happen at build time in app directory, and we should have access to the filesystem for reading any grammars/themes distributed in node_modules.

Fix terminal out of memory

See blake-mealey/ansi-sequence-parser#1

Escape annotations

There should be a way to escape comment annotations.

See code-hike/codehike#367 for an unexpected annotation with no way to escape.

Also, for Code Hike docs, I want to show the comments without converting them to annotations:

Add `text` grammar

build error on vite

This is not this project itself issue, but I will find a way to solve the problem and post it here

/Users/himself65/Library/Caches/fnm_multishells/85522_1705359274939/bin/pnpm run build

> [email protected] build /Users/himself65/Code/waku/packages/website
> waku build --with-ssr

vite v5.0.10 building SSR bundle for production...
src/hooks/index.ts (1:0) Error when using sourcemap for reporting an error: Can't resolve original location of error.
✓ 348 modules transformed.
Unexpected eof (Note that you need plugins to import files that are not JavaScript)
file: /Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@code-hike/lighter/dist/onig.wasm?module:1:5
1: asm�`���`��`����`�����`�`������`���`��`�������`��������`�`   �������������������``...
        ^
2: 
3:      
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error [RollupError]: Unexpected eof
    at error (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:337:30)
    ... 6 lines matching cause stack trace ...
    at async ModuleLoader.addModuleSource (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17760:13) {
  cause: Error [RollupError]: Unexpected eof
      at error (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:337:30)
      at nodeConverters (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:2084:9)
      at convertNode (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:969:12)
      at convertProgram (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:960:48)
      at parseAstAsync (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:2150:20)
      at async Module.tryParseAsync (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:13514:21)
      at async Module.setSource (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:13095:35)
      at async ModuleLoader.addModuleSource (file:///Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17760:13) {
    code: 'PARSE_ERROR',
    pos: 5
  },
  code: 'PARSE_ERROR',
  id: '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@code-hike/lighter/dist/onig.wasm?module',
  pos: 5,
  loc: {
    column: 5,
    file: '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@code-hike/lighter/dist/onig.wasm?module',
    line: 1
  },
  frame: '1: \x00asm\x01\x00\x00\x00\x01�\x01\x16`\x02\x7F\x7F\x01\x7F`\x01\x7F\x01\x7F`\x03\x7F\x7F\x7F\x01\x7F`\x04\x7F\x7F\x7F\x7F\x01\x7F`\x01\x7F\x00`\x05\x7F\x7F\x7F\x7F\x7F\x01\x7F`\x03\x7F\x7F\x7F\x00`\x02\x7F\x7F\x00`\x06\x7F\x7F\x7F\x7F\x7F\x7F\x01\x7F`\x07\x7F\x7F\x7F\x7F\x7F\x7F\x7F\x01\x7F`\x00\x01\x7F`\t\x7F\x7F\x7F\x7F\x7F\x7F\x7F\x7F\x7F\x01\x7F`\b\x7F\x7F\x7F\x7F\x7F\x7F\x7F\x7F\x01\x7F`\x00\x00`\x04...\n' +
    '        ^\n' +
    '2: \x01\x00\x01\x00\x03\x03\x02\x00\x00\x02\x06\x02\x0E\x02\x01\x0F\x01\x01\x01\n' +
    '3: \x13\x02\x05\t\x06\x01\x0E\x14\x15\x10\x07\x03\x02\x01\x00\x01\x04',
  watchFiles: [
    '/Users/himself65/Code/waku/packages/website/src/entries.tsx',
    '/Users/himself65/Code/waku/packages/waku/package.json',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next-mdx-remote/rsc.js',
    '/Users/himself65/Code/waku/packages/website/package.json',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next-mdx-remote/dist/rsc.js',
    '/Users/himself65/Code/waku/packages/website/src/templates/root-layout.tsx',
    '/Users/himself65/Code/waku/packages/waku/dist/main.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/react.shared-subset.js',
    '/Users/himself65/Code/waku/packages/website/src/templates/blog-article-page.tsx',
    '/Users/himself65/Code/waku/packages/website/src/templates/home-page.tsx',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/cjs/react.shared-subset.production.min.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/cjs/react.shared-subset.development.js',
    '/Users/himself65/Code/waku/packages/waku/dist/router/server.js',
    '/Users/himself65/Code/waku/packages/waku/dist/router/client.js',
    '/Users/himself65/Code/waku/packages/waku/dist/server.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/jsx-runtime.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/cjs/react-jsx-runtime.production.min.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/cjs/react-jsx-runtime.development.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next-mdx-remote/dist/jsx-runtime.cjs',
    '/Users/himself65/Code/waku/packages/waku/dist/client.js',
    '/Users/himself65/Code/waku/packages/waku/dist/router/common.js',
    '/Users/himself65/Code/waku/packages/waku/dist/lib/utils/path.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/jsx-dev-runtime.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]_4u4q4hpyfvx2kxhzws2c33xndq/node_modules/react-server-dom-webpack/client.edge.js',
    '/Users/himself65/Code/waku/packages/website/src/components/analytics.tsx',
    '/Users/himself65/Code/waku/packages/website/src/components/page.tsx',
    '/Users/himself65/Code/waku/packages/website/src/components/meta.tsx',
    '/Users/himself65/Code/waku/packages/website/src/styles.css',
    '/Users/himself65/Code/waku/packages/website/src/components/providers.tsx',
    '/Users/himself65/Code/waku/packages/website/src/components/mdx.tsx',
    '/Users/himself65/Code/waku/packages/website/src/components/start.tsx',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/cjs/react-jsx-dev-runtime.development.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/react/cjs/react-jsx-dev-runtime.production.min.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]_4u4q4hpyfvx2kxhzws2c33xndq/node_modules/react-server-dom-webpack/cjs/react-server-dom-webpack-client.edge.development.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]_4u4q4hpyfvx2kxhzws2c33xndq/node_modules/react-server-dom-webpack/cjs/react-server-dom-webpack-client.edge.production.min.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/react-dom.shared-subset.js',
    '/Users/himself65/Code/waku/packages/website/src/atoms/index.ts',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/jotai/esm/index.mjs',
    '/Users/himself65/Code/waku/packages/website/src/components/menu.tsx',
    '/Users/himself65/Code/waku/packages/website/src/components/credits.tsx',
    '/Users/himself65/Code/waku/packages/website/src/utils/index.ts',
    '/Users/himself65/Code/waku/packages/website/src/components/scroll.tsx',
    '/Users/himself65/Code/waku/packages/website/src/components/fade.tsx',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/clsx/dist/clsx.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@mdx-js/mdx/index.js',
    '/Users/himself65/Code/waku/packages/website/src/components/code.tsx',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom.shared-subset.development.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom.shared-subset.production.min.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/jotai/esm/vanilla.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/jotai/esm/react.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@mdx-js/mdx/lib/compile.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@mdx-js/mdx/lib/run.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@mdx-js/mdx/lib/core.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@mdx-js/mdx/lib/node-types.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/vfile/index.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/@[email protected]/node_modules/@mdx-js/mdx/lib/evaluate.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected]/node_modules/bright/dist/index.mjs',
    '/Users/himself65/Code/waku/packages/website/src/theme.json',
    '/Users/himself65/Code/waku/packages/website/src/hooks/index.ts',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/unified/index.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/vfile-matter/index.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/render/dom/motion.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/render/dom/motion-minimal.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected]/node_modules/vfile/lib/index.js',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/components/LazyMotion/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/components/LayoutGroup/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/components/MotionConfig/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/components/Reorder/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/components/AnimatePresence/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/render/dom/features-animation.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/render/dom/features-max.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-motion-template.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-motion-value.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/utils/resolve-motion-value.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-transform.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-spring.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-scroll.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-velocity.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/scroll/use-element-scroll.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/scroll/use-viewport-scroll.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-time.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/utils/use-motion-value-event.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/utils/reduced-motion/use-reduced-motion.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/utils/reduced-motion/use-reduced-motion-config.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/value/use-will-change/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/animation/hooks/use-animate.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/animation/hooks/use-animation.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/utils/use-animation-frame.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/utils/use-cycle.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/animation/interfaces/visual-element.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/components/AnimatePresence/use-presence.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/utils/use-in-view.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/gestures/drag/use-drag-controls.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/events/use-dom-event.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/animation/hooks/animation-controls.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/motion/utils/valid-prop.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/motion/utils/is-motion-component.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/motion/index.mjs',
    '/Users/himself65/Code/waku/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/framer-motion/dist/es/motion/utils/unwrap-motion-component.mjs',
    ... 240 more items
  ]
}

Node.js v20.10.0
 ELIFECYCLE  Command failed with exit code 1.

Process finished with exit code 1

dai-shi/waku#394

Error with non extensible themes

If the theme object is non extensible (object.freeze(), object.seal()), highlight fails with:

browser.esm.mjs?43bf:284 Uncaught (in promise) TypeError: Cannot add property 85, object is not extensible
    at Array.unshift (<anonymous>)
    at toFinalTheme

More theme colors

• diff colors: https://code.visualstudio.com/api/references/theme-color#diff-editor-colors
• tab colors: https://github.com/code-hike/codehike/blob/71d5ea2d5a354a65d99e22a4d6fe35eb6b29526d/packages/mdx/src/mini-editor/editor-frame.tsx#L175

Add getThemeColors

should be possible to get the colors of a theme synchronously (at least background and foreground) to show a decent fallback while the code is highlighted.

Export types

At least Theme and LanguageAlias

Avoid loading the same theme twice

Map theme name to promise instead of theme.

Handle missing alias in grammars endpoint

https://lighter.codehike.org/api/grammars?lang=j&v=0.0.3

Support Gatsby Partial Hydration

When I try to use this project inside of Gatsby with partial hydration it doesn't work.

../node_modules/.pnpm/@[email protected]/node_modules/@code-hike/lighter/dist/browser.esm.mjs
Module not found: Can't resolve 'https' in '/Users/tom.sherman/code/perfect-syntax/node_modules/.pnpm/@[email protected]/node_modules/@code-hike/lighter/dist'

I'm raising it here because it's weird to me that the browser entry contains a reference to a node module, feels like a bug with lighter and not gatsby

First call to markdown endpoint takes too long

After a new version, a call to https://lighter.codehike.org/api/grammars?lang=markdown&v=0.2.3 will take more than 30sec (vercel's edge limit).

Add browser export

without the node imports (replace file-system.js)

Annotations for languages without comment syntax

For example, how do you use annotations with json or text.

`from-css` themes don't work with `scopes: true`

highlight("console.log(1)", "js", "github-from-css", {
  scopes: true
})

returns

"style": {
  "color": "#000000"
},

instead of the CSS var names

JSX comment syntax and annotations

This is a comment annotation in MDX:

# Hello
{/* focus */}
## World

The problem is that the {} aren't part of the comment according to the mdx grammar. So after extracting the annotation, the code becomes:

# Hello
{}
## World