babashka/nbb

Scripting in Clojure on Node.js using SCI

nbb's Issues

Investigate refer conflict with clojure.core var

(require '["shelljs" :as sh :refer [cat]])

(prn (str (cat "shelljs.cljs")))

We should support :refer-clojure :exclude [cat] in this case or else give an error.

Support clojure.test

This will be handy for writing browser tests with puppeteer or https://github.com/microsoft/playwright

Add tools.cli as optional module

Moreover, it would be nice if this entire project could run with nbb:

https://github.com/wildwestrom/uniorg-util

/cc @wildwestrom

Add test suite

I've implemented a couple of async tests, but it's a pain in the neck to write tests that way.

Instead I'm going to write the tests in babashka and will just invoke the main JS on every test, testing the output of the process.

Consider wrapper for setting NODE_PATH

Problem

Scripts can require libraries that are local to the script, not local to the nbb npm library itself.
To accomodate this I've used createRequire with the path of the script for requiring libs, but this doesn't work for built-in libs. E.g. the built-in reagent module which is lazy loaded on demand, only loads react from the nbb install itself, not from a local library.
But node has another workaround for this, which is called NODE_PATH which allows you to set another node_modules directory to be included and this would solve the above problem.

However, I don't like writing a .js script that wraps another instance of node. This will take another 50ms or so of startup time which is not optimal.

This could be solved using a simple bash / .cmd script which wraps node but these scripts would have to do the arg parsing to decide what the node_modules relative to the script is, which is very annoying in bash.

Possible solution

So I was thinking, maybe we could write a tiny nbb wrapper in golang, Rust or Zig (even smaller) which does the arg parsing for us and then calls the nbb runner with just the args parsed as JSON + sets the correct NODE_PATH env variable.
Then the 'real' nbb doesn't have to concern itself with arg parsing and will just receive the already JSON-parsed args in an entry point function.
If we write the wrapper in GraalVM we might be able to leverage gitlibs as well and resolve deps in the CLI rather than in node (see #20) and simply pass the entire classpath to the node part, which at that point, is just a list of plain directories. This saves us a bunch of JS coding while still being able to tap into the JS ecosystem.

Alternatives

1. Hack

Hack: https://stackoverflow.com/a/60537950/6264

This hack seems to work, e.g. when you have a /tmp/foo/node_modules with shelljs and the below script lives in /tmp/start.cljs:

module.paths.unshift('/tmp/foo/node_modules');
process.env.NODE_PATH = process.env.NODE_PATH + ':/tmp/foo/node_modules';
module.constructor._initPaths();

require("shelljs");

If we can rely on this "hack" to work in the future, then this may be preferred as it's a much simpler workaround.
The question is if loading esm modules will be affected by this hack. If not, then we still would need the wrapper.

2. Require nbb users to use babashka and use this wrapper script:

#!/usr/bin/env bb

(require '[babashka.fs :as fs]
         '[babashka.process :as p])

(def script (first *command-line-args*))

(-> (p/process (cond-> ["nbb"]
                 script (conj script))
               (cond-> {:inherit true}
                 script
                 (assoc-in [:extra-env "NODE_PATH"]
                           ;; TODO: preserve previous NODE_PATH if populated
                           (str (fs/path (fs/parent script) "node_modules")))))
    p/check)

nil

3. Built-in support into babashka

ink: invalid hook call + reagent-cursor cannot be resolved

ink with its useInput-hook do not work for me. Since I was not sure (being a beginner in reagent) - i set up the same code without nbb and got it to work.

With nbb I get

  ERROR  Invalid hook call. Hooks can only be called inside of the body of a function component. This could happen for one of the following reasons:
        1. You might have mismatching versions of React and the renderer (such as React DOM)
        2. You might be breaking the Rules of Hooks
        3. You might have more than one copy of React in the same app
        See https://fb.me/react-invalid-hook-call for tips about how to debug and fix this problem.

I've put both code-bases into repos:

• nbb: https://github.com/MrGung/list-selector
• non-nbb: https://github.com/MrGung/clj-list-selector

---

In the nbb-code it is also not possible to resolve r/cursor (which does work in the non-nbb variant)...

Get core.async working in nbb

See #20 (comment). Of particular interest is <p! which pulls from a js Promise.

Mike Fikes' Andare might be a useful reference.

Print a ”standard” message when the nREPL server is started

Is your feature request related to a problem? Please describe.

In adding nbb Jack-in to Calva the current message format when nbb has started its nREPL server tripped me a bit. It looks like so:

nRepl server started on port %d . nrepl-cljs-sci version %s 1337 TODO

So clearly it is just temporary and because the nREPL server is totally new.

Describe the solution you'd like

When babashka starts it prints a message like so:

Started nREPL server at 127.0.0.1:1667

Which Calva is prepared for and from where it picks up the host and port.

When the ”regular” Clojure nREPL server is started it prints:

nREPL server started on port 54836 on host localhost - nrepl://localhost:54836

Which Calva also is prepared for.

I know this is not really an API, but Calva certainly uses it as if it were. It would be nice if both nbb and babashka followed the nREPL pattern.

But since Calva already is adapted to babashka's format, I would be fine with nbb using that as well. I just wish that nbb will not invent some new format here.

Support library property namespaces

https://clojurescript.org/news/2021-04-06-release#_library_property_namespaces

Allow symbols to refer to JS libs

e.g. (:require [console :refer [log]])

Provide cljs.pprint

Better error handling

It seems in the sqlite3 honeysql example, a lot of times errors aren't reported but the script just outputs nothing.

Basic stdin/stdout REPL

We should support a basic stdin/stdout REPL.

Very rudimentary one in user space:

$ nbb -e '(ns user (:require ["readline" :as readline])) (defn repl [] (let [rl (.createInterface readline #js {:input js/process.stdin :output js/process.stdout})] (.question rl (str *ns* "> ") (fn [answer] (prn (try (load-string answer) (catch :default e e))) (.close rl) (repl))))) (repl)'

Also see https://github.com/anmonteiro/lumo/blob/master/src/js/repl.js

Support server side rendering with reagent.dom.server

A pattern I am using in cljs web servers is to use Reagent's render-to-static-markup and inject the result into my template to be returned:

(:require [reagent.dom.server :refer [render-to-static-markup] :rename {render-to-static-markup r}])

;...

(aset template "innerHTML" (r [:div "My response]))

(where template here is an instance of node-html-parser on the server side)

This currently throws and error in nbb:

$ npx nbb myserver.cljs

> @ serve /home/chrism/dev/sitefox/examples/nbb
> nbb --classpath ../../src/ webserver.clj

----- Error --------------------------------------
Message:  No namespace: reagent.dom.server

Would be great to have access to reagent.dom.server!

Evaluations can be promises and those should not be resolved

e.g.

(Promise.resolve 1)

should return that promise but when you evaluate it, the result via the API is a promise with the result and then the returned promise is in turn resolved. This is JS behavior: nested promises are flattened. I.e. Promise<Promise<T>> cannot exist.

microsoft/TypeScript#27711

As such we should wrap evaluation results.

Investigate :node-library target

From the shadows:

thheller as I said before from the build perspective this is all trivial. there just isn't a target that does this just yet for node. it could easily be built but for that I need more details about how you actually plan on doing any of this

12:10 PM
my suggestion is to build all of this completely on top of :node-library and then deal with the code splitting stuff later
12:11 PM
it is an optimization after all so you don't need it from the start. if you just keep namespaces separate it'll be trivial to split out later
12:12 PM
once I can actually see what you are doing I can make a suggestion and maybe a custom target to do what you need

Also look at https://github.com/thheller/shadow-cljs/blob/master/packages/shadow-cljs/cli/runner.js

In your package.json you'll specify a bin. that should be the runner. as seen here https://github.com/thheller/shadow-cljs/blob/master/packages/shadow-cljs/package.json#L21

Windows CI

v0.0.70 has a fix for Windows, but we should test this in CI

Support or at least ignore cljs analyzer settings

In my specific case, I have a cljs program that does (set! *warn-on-infer* false) for a small section of the code that interacts tightly with node. nbb gives a "Could not resolve symbol: *warn-on-infer*" error and stops. If I comment out that setting it works with nbb. But on the flip-side normal cljs won't compile. Complicating this is that there is not nbb specific reader macro (or is there?) so I can't make the code support both. This happens during analysis so it can't be wrapped in a try/catch block.

Supporting the warnings probably is overkill for nbb, but perhaps those variables could at least be predefined so that those settings are a no-op. Or alternatively, perhaps undefined *warn-on-... variables could trigger a non-fatal warning instead of a stop-the-world error.

error in ubuntu/WSL2

$ nbb -e '(+ 1 2 3)'
/usr/local/lib/node_modules/nbb/out/nbb_main.js:3
import * as import$fs from "fs";
^

SyntaxError: Unexpected token *
at Module._compile (internal/modules/cjs/loader.js:723:23)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)

Implement :refer-clojure :exclude in ns form

(js/import "module") doesn't work when using global ver of nbb

Just found out that (js/import "module") won't work with global nbb, but works if I've installed local nbb but running using global nbb.

For example this will work:

yarn global add nbb
yarn add nbb term-size
# uses (js/import "term-size") in script.cljs
nbb script.cljs

This will not work:

yarn global add nbb
yarn add term-size
# uses (js/import "term-size") in script.cljs
nbb script.cljs
# Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'term-size' imported from /Users/macair/.volta/tools/image/packages/nbb/lib/node_modules/nbb/out/nbb_core.js

Implement *command-line-args*

See https://github.com/arichiardi/lumo/blob/6b96ff2463a36ca21d23744b3df72e6ea07759c2/src/js/cli.js

Support `:end-run-tests` multi-method for cljs.test

That's a good point. Normally clojure.test doesn't do anything with the exit code and you should inspect the test results to do that.

In the cljs.test docs about run-tests:

Runs all tests in the given namespaces; prints results.
Defaults to current namespace if none given. Does not return a meaningful
value due to the possiblity of asynchronous execution. To detect test
completion add a :end-run-tests method case to the cljs.test/report
multimethod.

So I think we should support this as the next step. I'll make an issue.

Originally posted by @borkdude in #41 (comment)

/cc @ErikSchierboom

(println ...) in nrepl should send output to client

js literal with expression doesn't eval expression

$ nbb -e '(def f (fn [] #js {:a (+ 1 2 3)})) (f)'
#js {:a (+ 1 2 3)}

$ plk -e '(def f (fn [] #js {:a (+ 1 2 3)})) (f)'
#js {:a 6}

Errors report "success" with exit code `0`.

version

$ which node; node --version
/usr/bin/node
v13.14.0
$ npm list -g | grep nbb
├── [email protected]

platform

Ubuntu 20.04 LTS with packaged Node.js v13.14.0.

problem

Scripting errors exit with status code 0 meaning "success."

repro

$ nbb -e '(print (* 6 7))'; echo "exit code: $?"
42
exit code: 0
$ nbb -e '(printt (* 6 7))'; echo "exit code: $?"
----- Error --------------------------------------
Message:  Could not resolve symbol: printt
Location: 1:2
Phase:    analysis
exit code: 0
$ 

expected behavior

Return anything but 0 as the exit code.

When errors return non-zero status code, nbb could be used as part of a larger script that checks its exit code.

Support requiring classes without string name

E.g. (require '[goog.object :as gobj]) should be possible by mapping goog.object under :classes

reagent / ink loading order problem

In the following example ink and react are loaded before reagent. In the reagent fork, the module is undefined, perhaps because it was already loaded?

The error: Cannot read property 'Component' of undefined.

(ns reagent
  (:require ["ink" :refer [render Text]]
            ["react" :as react]
            [reagent.core :as r]))

(defn example []
  (let [[count set-count] (react/useState 0)]
    (react/useEffect (fn []
                       (let [timer (js/setInterval #(set-count (inc count)) 500)]
                         (fn []
                           (js/clearInterval timer)))))
    [:> Text {:color "green"} "Hello, world! " count]))

(defn root []
  [:f> example])

(render (r/as-element [root]))

Resolve deps via file argument path

It seems for the import stuff to work properly, I need access to:

import.meta.resolve also accepts a second argument which is the parent module from which to resolve from:
await import.meta.resolve('./dep', import.meta.url);

source: https://nodejs.org/api/esm.html#esm_import_meta_resolve_specifier_parent

Make library aliases work with fully qualified symbols

E.g.:

(ns foo.script
  (:require ["fs" :as fs]))
(println (count (str (fs/readFileSync "script.cljs"))))

Does normal CLJS create a "real" namespace for fs or is fs just the fs object from the library itself? And is fs/readFileSync then translated into an object lookup in this library, or is there a "real" namespace with a readFileSync function created behind the scenes?
It seems like it's doing something like the latter (with lumo):
cljs.user=> (str (fn [] (fs/readFileSync "script.cljs")))
"function (){\nreturn cljs.user.node$module$fs.readFileSync.call(null,"script.cljs");\n}"

Interesting bits from CLJS

• https://github.com/clojure/clojurescript/blob/c0045c5718014f4cc40db55742140d2eb561807c/src/main/clojure/cljs/analyzer.cljc#L795
• https://github.com/clojure/clojurescript/blob/c0045c5718014f4cc40db55742140d2eb561807c/src/main/clojure/cljs/analyzer.cljc#L1129-L1132

Provide JS API so you can evaluate CLJS files from JS

E.g.:

import nbb from 'nbb'
const script = fs.readFileSync('./test.cljs', 'utf8')
nbb(script, 'args');

Rationale? Why nbb if there's already other similar tools?

For example, there's planck, a standalone CLJS REPL, that compiles to C.

Support :rename in ns form

(ns foo.bar (:require ["shelljs" :refer [cat] :rename {cat scat}]))

Instantiating a js class throws an error

Steps to reproduce:

npm i keyv

(ns t
  (:require ["keyv" :as Keyv]))
(js/console.log Keyv)
(Keyv. "sqlite://./database.sqlite")

Result:

$ npx nbb test.cljs
[class Keyv extends EventEmitter]
#error {:message "Bind must be called on a function", :data {:type :sci/error, :line 4, :column 2, :message "Bind must be called on a function", :sci.impl/callstack #object[cljs.core.Delay {:status :pending, :val nil}], :file nil, :locals {}}, :cause #object[TypeError TypeError: Bind must be called on a function]}

(js/process.argv.slice 2) works in CLJS but not in nbb

However (.slice js/process.argv.slice 2) does work.

I also noticed that (js/Promise.resolve 1) doesn't work but (.resolve js/Promise 1) does.

Support more babashka command line args (--main, --help/help, version)

It would be nice if nbb supported a few of the same basic babashka command line args like --main, --help, help, version, --, etc.

nREPL server module

See https://github.com/viesti/nrepl-cljs-sci

Consider adding lazy loaded module js-interop

https://github.com/applied-science/js-interop

Socket server REPL

Related to #63

Requiring *file* from nbb.core causes cljs requires to fail

Replicate:

a.cljs:

(ns a
  (:require
    ; [nbb.core :refer [*file*]]
    [b :refer [hi]]))
(hi)

b.cljs:

(ns b)

(defn hi []
  (print "hi"))

With the nbb.core require commented out:

$ npx nbb a.cljs 
hi

With the nbb.core require uncommented:

$ npx nbb a.cljs 
#error {:message "Could not resolve symbol: hi", :data {:type :sci/error, :line 6, :column 8, :file "/home/chrism/dev/c64core/no.cljs", :phase "analysis"}}

Which style of requires should be support: cjs or esm style?

@thheller: if you have an opinion on this, hearing it would be most interesting. I'm aware of the foo$default vs "foo" :default foo syntax, that's not what this issue is about, when I write :default below, just assume I mean the dollar syntax.

/cc @filipesilva

Set up CI for tests

This is the order I use locally, but we should do this in CI instead:

• bb run-tests
• bb release
• bb run-integration-tests
• npm publish

Perhaps only when pushing a new tag we should deploy.

Expose globals __filename and __dirname for the cljs file being executed

Node users are used to having these around. https://nodejs.org/api/modules.html#modules_filename

In wisp and cljs-lumo these are available via js/__filename.

Print nice stacktrace like bb in case of error instead of garbled advanced compiled JS

support with-out-str

Depends on babashka/sci#365

load libraries from .jar files and directories

See - https://github.com/anmonteiro/lumo/blob/6709c9f1b7b342c8108e6ed6e034e19dc786f00b/src/cljs/bundled/lumo/classpath.cljs

• https://github.com/anmonteiro/lumo/blob/6709c9f1b7b342c8108e6ed6e034e19dc786f00b/src/cljs/bundled/lumo/cljs_deps.cljs

Support -e

nbb -e '(+ 1 2 3)'

This already works by doing nbb /dev/stdin <<< '(prn (+ 1 2 3))' lol :)

Investigate loading of zx

(require '[promesa.core :as p]
         '["url" :as url])

;; (prn (url/pathToFileURL (.resolve js/require "zx/index.mjs")))
(js/import (doto (str (url/pathToFileURL (.resolve js/require "zx/index.mjs")))
             prn))

$ nbb zx.cljs
"file:///private/tmp/zx/node_modules/zx/index.mjs"
Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only file and data URLs are supported by the default ESM loader

Interop converts Clojure value to JS value automatically while it should not

(.then (.resolve js/Promise {:a 1}) (clojure.core/fn [{:keys [:a] :as m}] (prn a m))) prints nil #js {:a 1}.

Try pre-compiled nbb sources in native image + graaljs / node

This would mostly be a fun issue and not yet something which I consider as a serious distribution of nbb: a native image which contains node + the sources of NBB loaded at build time and then the rest of the evaluation would happen at runtime.