detChoice problems in browser

[earthstar-project]

detChoice problems in browser

[cinnamon-bun]

Thanks for the report!

That's the right usage (assuming author.address is a string).

earthstar/src/util/detRandom.ts

Lines 43 to 45 in e0eb930

	export let detChoice = <T>(s: string | Buffer, array: T[]): T =>
	// deterministically choose a random-ish item from the array
	array[Math.floor(detRandom(s) * array.length)];

earthstar/src/util/detRandom.ts

Lines 24 to 33 in e0eb930

	export let detRandom = (s: string | Buffer): number => {
	// return a random-ish float between 0 and 1, deterministically derived from a hash of the string
	let hashBuffer = LowLevelCrypto.sha256(s);
	// take first 4 bytes of hash and convert to unsigned 32 bit integer (big-endian)
	// https://github.com/nodejs/node/blob/44161274821a2e81e7a5706c06cf8aa8bd2aa972/lib/internal/buffer.js#L291-L302
	let randInt = hashBuffer.slice(4).readUInt32BE();
	// divide by max possible value
	return randInt / 2 ** 32;
	};

...I haven't tested this in the browser yet. Since the error is coming from node_modules/buffer I'm guessing it's related to the browserification of node Buffers into browser ArrayBuffers. Maybe readUInt32BE isn't supported in the browser polyfill. I think sha256 is working so it's not that (if signing documents works, sha256 works).

How are you bundling this, is it browserify or webpack?

Can you get a more detailed traceback?

Speed

It should be very fast to run once, the problem is running it for every rendered React component over and over.

Faster hash function

We should switch from sha256 to md5 or something even faster. This isn't security critical.

Wait for sodium to load WASM

Investigate this note about waiting for sodium to load its WASM module. Is the slowness only temporary until WASM is loaded, or does it get stuck with the slow version if we use it without waiting?

Memoize

Memoizing / caching the function should fix it, but it will still be slow on the first render.

Something like:

let colorCache = {}  // mapping from author address to color string

let authorToColor(author: string): string => {
    if (colorCache[author]) { return colorCache[author]; }
    let color = detRandom(author, ['red', 'green', 'blue']);
    colorCache[author] = color;
    // todo: if cache is getting too big, randomly delete an item here
    return color;
}

[cinnamon-bun]

This reminds me about #2 Make Tests Run In Browsers ;)

If you have experience with that, I'd appreciate help.

[cinnamon-bun]

I verified that detChoice works in the browser when bundled using Browserify.

I'm guessing this error happened in earthstar-lobby which uses Webpack (via react-scripts). I think this is the webpack config, but it's kind of intimidating.

Theory

Maybe Webpack doesn't bundle earthstar correctly? This could cause a problem in 3 places

• A. Need to skip sqlite
• B. Node's Buffer-related code might not be translated to browser code properly
• C. Node's crypto code might not be translated to browser code properly

I'm not sure how everything could have worked so far but broken when detChoice appeared. I suspect it's this line:

earthstar/src/util/detRandom.ts

Line 30 in e0eb930

let randInt = hashBuffer.slice(4).readUInt32BE();

Webpack uses a pretty old polyfill, node-libs-browser, and I am guessing it doesn't understand readUInt32BE which might be new-ish.

If this theory is true, all of the earthstar code should work except any of the det functions (detChoice, detRandom, etc), which should all fail.

Other theory

Back over here in earthstar, Browserify is instructed to swap out some files when bundling. It uses this section of package.json:

  // we need to skip all the sqlite-related code when bundling for the browser
  "browser": {
    "./build/index.js": "./build/index.browser.js",   // swap out index for a version that doesn't import sqlite
    "./build/storeSqlite.js": false,   // skip the other file that imports sqlite
    "tap": "tape"  // use a browser-compatible test module
  },

I can't figure out if Webpack understands this browser field. I think it expects it to point to a single alternate index file, like this:

    "browser": "./build/index.browser.js",

???

[cinnamon-bun]

@sgwilym

I published earthstar 5.2.4 -- can you check if this fixes the problem?

4f9c89d

I just copy-pasted node's code for buffer.readUInt32BE right into Earthstar, to escape from the clutches of a browser polyfill.

If it doesn't work can you point me to a commit in earthstar-lobby that triggers the error?

[cinnamon-bun]

Closing this for now since it seems to not be an issue lately. Please re-open if it can be reproduced.