Blob support

[dumbmatter]

Blob support

[vdechef]

I have an app that stores data as Blob in browser's indexeddb. For my unit tests, I use Jest and FakeIndexedDB.
I had this kind of problem: Error [DataCloneError]: The data being stored could not be cloned by the internal structured cloning algorithm

After a lot of trial and errors, I figured out that my problem was caused by jsdom (Jest internal browser) not implementing URL.createObjectURL(). Using code from https://github.com/bjornstar/blob-polyfill, I was able to mock Blob and createObjectURL and now I have my unit tests working nicely.

Here is my workaround. The code does not handle every corner case as it is only meant for unit testing, but I hope it may help someone.

---

jest.setup.js:

import { MockBlob, mockCreateObjectURL } from "./myMocks.js"

// mock indexeddb, because jest browser does not handle it.
// Blobs in indexeddb are created using URL.createObjectURL(), but jest does not handle this function. Thus we also need to mock it.
// createObjectURL is synchronous, whereas convering Blob to array is asynchronous, thus we need to mock Blob.
global.Blob = MockBlob
global.URL.createObjectURL = mockCreateObjectURL

myMocks.js:

/**
 * Code is copied from https://github.com/bjornstar/blob-polyfill
 * Minor changes where done, because we don't need the whole polyfill, and because we want to force the FakeBlobBuilder
 */

function stringEncode (string) {
    var pos = 0;
    var len = string.length;
    var Arr = global.Uint8Array || Array; // Use byte array when possible

    var at = 0; // output position
    var tlen = Math.max(32, len + (len >> 1) + 7); // 1.5x size
    var target = new Arr((tlen >> 3) << 3); // ... but at 8 byte offset

    while (pos < len) {
        var value = string.charCodeAt(pos++);
        if (value >= 0xd800 && value <= 0xdbff) {
            // high surrogate
            if (pos < len) {
                var extra = string.charCodeAt(pos);
                if ((extra & 0xfc00) === 0xdc00) {
                    ++pos;
                    value = ((value & 0x3ff) << 10) + (extra & 0x3ff) + 0x10000;
                }
            }
            if (value >= 0xd800 && value <= 0xdbff) {
                continue; // drop lone surrogate
            }
        }

        // expand the buffer if we couldn't write 4 bytes
        if (at + 4 > target.length) {
            tlen += 8; // minimum extra
            tlen *= (1.0 + (pos / string.length) * 2); // take 2x the remaining
            tlen = (tlen >> 3) << 3; // 8 byte offset

            var update = new Uint8Array(tlen);
            update.set(target);
            target = update;
        }

        if ((value & 0xffffff80) === 0) { // 1-byte
            target[at++] = value; // ASCII
            continue;
        } else if ((value & 0xfffff800) === 0) { // 2-byte
            target[at++] = ((value >> 6) & 0x1f) | 0xc0;
        } else if ((value & 0xffff0000) === 0) { // 3-byte
            target[at++] = ((value >> 12) & 0x0f) | 0xe0;
            target[at++] = ((value >> 6) & 0x3f) | 0x80;
        } else if ((value & 0xffe00000) === 0) { // 4-byte
            target[at++] = ((value >> 18) & 0x07) | 0xf0;
            target[at++] = ((value >> 12) & 0x3f) | 0x80;
            target[at++] = ((value >> 6) & 0x3f) | 0x80;
        } else {
            // FIXME: do we care
            continue;
        }

        target[at++] = (value & 0x3f) | 0x80;
    }

    return target.slice(0, at);
}

/********************************************************/
/*               String Decoder fallback                */
/********************************************************/
function stringDecode (buf) {
    var end = buf.length;
    var res = [];

    var i = 0;
    while (i < end) {
        var firstByte = buf[i];
        var codePoint = null;
        var bytesPerSequence = (firstByte > 0xEF) ? 4
            : (firstByte > 0xDF) ? 3
                : (firstByte > 0xBF) ? 2
                    : 1;

        if (i + bytesPerSequence <= end) {
            var secondByte, thirdByte, fourthByte, tempCodePoint;

            switch (bytesPerSequence) {
            case 1:
                if (firstByte < 0x80) {
                    codePoint = firstByte;
                }
                break;
            case 2:
                secondByte = buf[i + 1];
                if ((secondByte & 0xC0) === 0x80) {
                    tempCodePoint = (firstByte & 0x1F) << 0x6 | (secondByte & 0x3F);
                    if (tempCodePoint > 0x7F) {
                        codePoint = tempCodePoint;
                    }
                }
                break;
            case 3:
                secondByte = buf[i + 1];
                thirdByte = buf[i + 2];
                if ((secondByte & 0xC0) === 0x80 && (thirdByte & 0xC0) === 0x80) {
                    tempCodePoint = (firstByte & 0xF) << 0xC | (secondByte & 0x3F) << 0x6 | (thirdByte & 0x3F);
                    if (tempCodePoint > 0x7FF && (tempCodePoint < 0xD800 || tempCodePoint > 0xDFFF)) {
                        codePoint = tempCodePoint;
                    }
                }
                break;
            case 4:
                secondByte = buf[i + 1];
                thirdByte = buf[i + 2];
                fourthByte = buf[i + 3];
                if ((secondByte & 0xC0) === 0x80 && (thirdByte & 0xC0) === 0x80 && (fourthByte & 0xC0) === 0x80) {
                    tempCodePoint = (firstByte & 0xF) << 0x12 | (secondByte & 0x3F) << 0xC | (thirdByte & 0x3F) << 0x6 | (fourthByte & 0x3F);
                    if (tempCodePoint > 0xFFFF && tempCodePoint < 0x110000) {
                        codePoint = tempCodePoint;
                    }
                }
            }
        }

        if (codePoint === null) {
            // we did not generate a valid codePoint so insert a
            // replacement char (U+FFFD) and advance only 1 byte
            codePoint = 0xFFFD;
            bytesPerSequence = 1;
        } else if (codePoint > 0xFFFF) {
            // encode to utf16 (surrogate pair dance)
            codePoint -= 0x10000;
            res.push(codePoint >>> 10 & 0x3FF | 0xD800);
            codePoint = 0xDC00 | codePoint & 0x3FF;
        }

        res.push(codePoint);
        i += bytesPerSequence;
    }

    var len = res.length;
    var str = "";
    var j = 0;

    while (j < len) {
        str += String.fromCharCode.apply(String, res.slice(j, j += 0x1000));
    }

    return str;
}

// string -> buffer
var textEncode = typeof TextEncoder === "function"
    ? TextEncoder.prototype.encode.bind(new TextEncoder())
    : stringEncode;

// buffer -> string
var textDecode = typeof TextDecoder === "function"
    ? TextDecoder.prototype.decode.bind(new TextDecoder())
    : stringDecode;

var viewClasses = [
    "[object Int8Array]",
    "[object Uint8Array]",
    "[object Uint8ClampedArray]",
    "[object Int16Array]",
    "[object Uint16Array]",
    "[object Int32Array]",
    "[object Uint32Array]",
    "[object Float32Array]",
    "[object Float64Array]"
];

var isArrayBufferView = ArrayBuffer.isView || function (obj) {
    return obj && viewClasses.indexOf(Object.prototype.toString.call(obj)) > -1;
};

function isDataView (obj) {
    return obj && Object.prototype.isPrototypeOf.call(DataView, obj);
}

function bufferClone (buf) {
    var view = new Array(buf.byteLength);
    var array = new Uint8Array(buf);
    var i = view.length;
    while (i--) {
        view[i] = array[i];
    }
    return view;
}

function concatTypedarrays (chunks) {
    var size = 0;
    var j = chunks.length;
    while (j--) { size += chunks[j].length; }
    var b = new Uint8Array(size);
    var offset = 0;
    for (var i = 0; i < chunks.length; i++) {
        var chunk = chunks[i];
        b.set(chunk, offset);
        offset += chunk.byteLength || chunk.length;
    }

    return b;
}

function MockBlob(chunks, opts) {
    chunks = chunks || [];
    opts = opts == null ? {} : opts;
    for (var i = 0, len = chunks.length; i < len; i++) {
        var chunk = chunks[i];
        if (chunk instanceof MockBlob) {
            chunks[i] = chunk._buffer;
        } else if (typeof chunk === "string") {
            chunks[i] = textEncode(chunk);
        } else if (Object.prototype.isPrototypeOf.call(ArrayBuffer, chunk) || isArrayBufferView(chunk)) {
            chunks[i] = bufferClone(chunk);
        } else if (isDataView(chunk)) {
            chunks[i] = bufferClone(chunk.buffer);
        } else {
            chunks[i] = textEncode(String(chunk));
        }
    }

    this._buffer = global.Uint8Array
        ? concatTypedarrays(chunks)
        : [].concat.apply([], chunks);
    this.size = this._buffer.length;

    this.type = opts.type || "";
    if (/[^\u0020-\u007E]/.test(this.type)) {
        this.type = "";
    } else {
        this.type = this.type.toLowerCase();
    }
}

MockBlob.prototype.arrayBuffer = function () {
    return Promise.resolve(this._buffer.buffer || this._buffer);
};

MockBlob.prototype.text = function () {
    return Promise.resolve(textDecode(this._buffer));
};

MockBlob.prototype.slice = function (start, end, type) {
    var slice = this._buffer.slice(start || 0, end || this._buffer.length);
    return new MockBlob([slice], {type: type});
};

MockBlob.prototype.toString = function () {
    return "[object Blob]";
};


function array2base64 (input) {
    var byteToCharMap = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";

    var output = [];

    for (var i = 0; i < input.length; i += 3) {
        var byte1 = input[i];
        var haveByte2 = i + 1 < input.length;
        var byte2 = haveByte2 ? input[i + 1] : 0;
        var haveByte3 = i + 2 < input.length;
        var byte3 = haveByte3 ? input[i + 2] : 0;

        var outByte1 = byte1 >> 2;
        var outByte2 = ((byte1 & 0x03) << 4) | (byte2 >> 4);
        var outByte3 = ((byte2 & 0x0F) << 2) | (byte3 >> 6);
        var outByte4 = byte3 & 0x3F;

        if (!haveByte3) {
            outByte4 = 64;

            if (!haveByte2) {
                outByte3 = 64;
            }
        }

        output.push(
            byteToCharMap[outByte1], byteToCharMap[outByte2],
            byteToCharMap[outByte3], byteToCharMap[outByte4]
        );
    }

    return output.join("");
}

function mockCreateObjectURL(blob) {
    if (blob instanceof MockBlob) {
        return "data:" + blob.type + ";base64," + array2base64(blob._buffer)
    }
    else {
        console.error("MOCK ERROR ! Bad Blob type:", typeof blob)
    }
};

exports.MockBlob = MockBlob
exports.mockCreateObjectURL = mockCreateObjectURL

[vdechef]

The code is pure JS, but you use typescript. Typescript is more precise when analyzing the objetcs types.
The mock I provided is missing the property stream, but you can easily add it in the code above :

MockBlob.prototype.stream = function () {
    return null; // if you need a real value, replace null by this value
};

[dumbmatter]

That's nice to hear! It is just a happy side effect, probably due to native structuredClone in recent versions of Node.

[tedsecretsource]

@dumbmatter Thank you so much for this detailed reply. As I'm not ready to eject from React and there doesn't seem to be any way to "undelete" structuredClone, I think the only solution I'm left with is putting this logic into a custom hook that I can mock until Jest 28 is included with CRA. I have very mixed feelings about this approach, though, but I'll save that rant for another day. Thanks again and nice work on fake-indexeddb!

[dumbmatter]

I don't have any plans, but would accept a PR if someone is able to improve the current situation :)

Maybe https://github.com/sidequestlegend/node-blob would help?

[nemanjam]

I get this error

[kepta]

Blob is now supported by nodejs 18. A simple fix for me was to use node 18 and set the following global variable:

globalThis.Blob = require('buffer').Blob;

[lucas42]

When using nodejs18, it looks like Blobs are now supported with fakeIndexedDB versions 4.0.0-beta.2 and above (including 4.0.0).

They now work for me without any of the workarounds listed above. 🎉
For anyone watching this thread, I'd recommending upgrading and see if the latest versions solve your Blob issues too.

Thanks @dumbmatter for the fix (whether it was deliberate, or a happy side-effect of changes to the cloning logic)!

[tedsecretsource]

Even with node 18..11.10 and fake-indexeddb 4.0.1 this is still failing for me and I would be very grateful for any pointers others can provide. I'm getting the error "DataCloneError: The data being stored could not be cloned by the internal structured cloning algorithm" when trying to put an object with a Blob member.

Versions

React: 17.0.2
Jest: 29.0.3
fake-indexeddb: 4.0.1
ts-jest: 28.0.8
idb: 7.1.1

I am using idb/with-async-ittr. My Jest test file looks mostly like this (removing anything not directly related to this issue):

import "fake-indexeddb/auto"
import { SoundRecorderDB } from '../../SoundRecorderTypes'
import { openDB } from 'idb/with-async-ittr'

describe('With an empty list of recordings', () => {
  beforeEach( async () => {
    const db = await openDB<SoundRecorderDB>('test-db', 1, {
      upgrade(db) {
        db.createObjectStore('recordings', { keyPath: 'id', autoIncrement: true });
      }
    })
    await act(async () => {
      await render(<Recorder db={db} />);
    })
  });

  it('user can start a recording pressing the button', async () => {
    const button = screen.getByRole("button", { name: 'Record' })
    expect(button).toHaveClass('record-play')
    await user.click(button)
    expect(button).toHaveTextContent(/stop/i);
    // saves the recording to indexeddb
    // this is the line that produced the error because the component calls db.put()
    await user.click(button)
  });
)}

The data schema looks like this:

import { DBSchema } from 'idb'

export interface SoundRecorderDB extends DBSchema {
    recordings: {
        key: SoundRecorderDB['recordings']['value']['id']
        value: {
            id?: number
            data?: Blob // this bad boy is what's causing all the hullabaloo
            name: string
            length: number
            audioURL: string
        }
        indexes: {
            name: string
        }
    }
}

I'd be happy to mock the put call but to be completely honest, I can't quite figure out how… I have not tried the solution above (polyfilling the blob object). Any help would be greatly appreciated.

[dumbmatter]

@tedsecretsource can you give a minimal reproduction that doesn't depend on any other code? Ideally a repo I can just run. Cause it should work with Node 17 or higher, since structuedClone supports blobs:

Welcome to Node.js v18.14.0.
Type ".help" for more information.
> const x = new Blob()
Blob { size: 0, type: '' }
> structuredClone(x)
Blob { size: 0, type: '' }

Like maybe your code is not using a "real" blob and instead it's some other type of object that can't be cloned? idk

[tedsecretsource]

@dumbmatter First of all, thank you so much for taking the time to look into this and for anyone else who is reading, when a maintainer takes the time to respond to your queries, it's the least you can do to do what they ask!

I've set up a sample repo to the best of my ability. I'm not great working with promises, especially under Jest, so that could well be the whole problem. This repo is a very basic React application that demonstrates the issue when you run yarn test you'll see Error: Uncaught [DataCloneError: The data being stored could not be cloned by the internal structured cloning algorithm.] in the console.

As I said, please go easy on me here. Promises are not my forte! And thank you so, so much for bothering to look at this!

[dumbmatter]

It's a little complicated :)

The built-in Node structuredClone works with blobs, but the structuredClone polyfill I'm using for old versions of Node does not work with blobs. So if you are using a recent version of Node (17 or higher), it should work.

But under some configurations, Jest attempts to delete Node's global variables to make their tests more similar to the browser environment, so that you don't accidentally rely on Node functionality in your code that will then fail when you run it in the browser.

The problem is you're using Jest 27, which is old enough that it doesn't realize structuredClone should be available in the browser too jestjs/jest#12628 which was only fixed in Jest 28. So it deletes structuredClone, and then fake-indexeddb uses its polyfill, which fails for blobs.

You could either upgrade Jest (maybe not easy because you're only indirectly using it through create-react-app) or somehow configure Jest to stop deleting structuredClone (IDK how to do that off the top of my head, but I think there is some way) @tedsecretsource