stardazed/sd-streams

Web Streams Standard implementation and polyfill

Stardazed

A library to enable quick development of 3D games in the browser.

Project status

In Development (Pre-Alpha)

Features and APIs still very much in flux, but functional and usable for actual development (I'm using it for prototypes and Game Jams).

⚠️ Important: Currently the project is being reworked internally and is not in a usable state.

Project Goal

"A web-native, modular and comprehensive 3D game creation platform"

There is still quite a ways to go for this to become a reality, but I've already used the library succesfully for small projects. For the foreseeable future the library will require medium to high technical expertise to use.

Stardazed is an educational project for myself and the design and feature set currently directly corresponds to my interests. For example, I'm focusing on desktop WebGL 1 & 2, mobile support is currently not a priority.

Sub goals

• Learn about all aspects of game (engine) programming by implementing them.
• Learn about build systems, large project design and modularity in the context of the web platform.
• Memory & GC efficiency: a lot of the data is kept in large linear typed arrays, not in millions of tiny objects
• Scalability: use workers, atomics and shared buffers to allow for kinda multi-threaded rendering and game logic handling
• Solid but performant code using TypeScript, linting and (runtime) function contracts that can be omitted in release builds
• Flexible and modular shader programming (composable shaders and features like PBR, GI, etc.)
• Network-oriented project design with streaming, partial assets, fallback shaders, etc.
• Minimal external dependencies and small code footprint

License

MIT

© 2015-Present by @zenmumbler

sd-streams's Issues

No types for Compression Streams

It seems like the types for DecompressionStream and CompressionStream are not included in lib yet - the other streams types are. It would make sense to include their type definitions with the polyfill package

CompressionStreams polyfill not working in Firefox

The polyfill seems not to work with File and/or fetch() in Firefox at the moment.
When compressing a file and converting to a blob with fetch() its sizie is always 23B and content is just [Object Object] or [Object ReadableStream].

Maybe this is related to issue #8 but I am pretty sure this worked in the past (2 months or so ago).

I prepared a minimal example: https://stackblitz.com/edit/angular-ivy-phtsgd
(Note: as this displays the compressed content, maybe just upload small files.)

Thanks!

Could `stream.tee()` copy the buffer for at least one of the output?

Hey there,

Thanks for the work on this, it's a great library.

The way we're using ReadableStreams, when we read from them and send back a buffer to the "host" (in our case: Rust + V8), we externalize and neuter the ArrayBuffer. This essentially renders the ArrayBuffer unusable. So if we tee() the stream, once a chunk has been pushed to our servers, the backing ArrayBuffer can't be used for the second chunk.

I used value2.slice(0) here: https://github.com/stardazed/sd-streams/blob/master/packages/streams/src/readable-stream.ts#L115 as a local fix.

Looks like it's part of the spec as a parameter (cloneForBranch2) on ReadableStreamTee. It doesn't look like it should be a parameter on tee() though. I see you've got the assertion commented out:

sd-streams/packages/streams/src/readable-stream.ts

Line 81 in ee1c581

// Assert: Type(cloneForBranch2) is Boolean.

Any thoughts on a workaround or how to implement the cloneForBranch2 param?

This is a great polyfill. Is it still active?

pipeThrough() not working in Safari

When using the CompressionStreams polyfill with Safari like this:

file.stream().pipeThrough(new CompressionStream('gzip'));

the invocation of pipeThrough() fails with readable should be ReadableStream.

The error is thrown after isReadableStream check failed [1]. After debugging I'm sure, that it is failing, because the symbol readableStreamController_ is not defined in the tested value [2].

Note that Safari is also polyfilling the basic streams implemenation, bacause there is no support for BYOB/bytsoures.
Not using the polyfill and instead directly importing CompressionStream from the lib works.

This issue occurs at least on Safari 16.3 and 15.5 and TP.

I made a minimal reproducer in StackBlitz (check console output): https://stackblitz.com/edit/js-gaksqm

1:

sd-streams/packages/streams/src/readable-stream.ts

Line 110 in 009c982

throw new TypeError("readable must be a ReadableStream");

2:

sd-streams/packages/streams/src/readable-internals.ts

Line 201 in c9db3d0

return readableStreamController_ in value;

This will be better

export declare global {
  declare export class DecompressionStream {
    constructor(format: keyof Formats);

    readonly readable: ReadableStream<BufferSource>;
    readonly writable: WritableStream<Uint8Array>;
  }

  declare export class CompressionStream {
    constructor(format: keyof Formats);

    readonly readable: ReadableStream<BufferSource>;
    readonly writable: WritableStream<Uint8Array>;
  }
}

interface Formats {
  deflate: never,
  "deflate-raw": never,
  gzip: never,
}

Missing type definitions in streams-polyfill?

Thanks for a useful project!

I'm trying to use streams-polyfill in Typescript 3.8 node project, and tsc isn't finding types for streaming classes (I'm referencing WritableStream, TextDecoderStream, and TransformStream).

I've tried both adding the import you suggest

import "@stardazed/streams-polyfill;

as well as explicitly importing the types

import { WritableStream, TextDecoderStream, and TransformStream } from "@stardazed/streams-polyfill;

I understand from your readme that Typescript > 3.2 is supposed to supply these. Is there some tsconfig magic I need to do to have that work?

stream-polyfill should use (babel?) transformation in rollup build to support legacy browsers

The output dist code in stream-polyfill assumes the browser know how to handle the "=>" expression.
I suggest to automate the conversion by using babel rollup plugin, together with babel/preset-env plugin.

Missing Blob/File stream methods

This polyfill works great for streaming fetches, but it doesn't patch the ReadableStream implementation of Blob (and by extension, File).

The following returns a function in Chrome but not in Firefox when using the polyfill:
x=new Blob().stream().pipeThrough

Add missing "type": "module" to @stardazed/streams-compression

I'm trying to import the module in vite, but it's complaining about an Unexpected token 'export' due to the package not being marked as a module.

Possible scaling issue with QueueContainer<V>

If QueueContainer<V>.[queue_] is compiled to a JS array, then it's likely to become slow when the queue is very large.

See https://bugs.chromium.org/p/chromium/issues/detail?id=681493 for background information about when the same issue happened in Chrome.

"deflate-raw" option

Thanks for the good project! I find that it does not expose the "deflate-raw" format in the current polyfiller even though the related functionality has already been implemented. I can get it work simply by modify the code and change the argument passed to the constructor of the Inflater and Deflater classes.

As "deflate-raw" has been accepted as a part of the standard, will this project adds support for it?

TypeError: writable must be a WritableStream

I think I'm misunderstanding some usage of this polyfill. I've added the following before my scripts:

    <script src="//unpkg.com/@stardazed/streams-polyfill/dist/sd-streams-polyfill.min.js"></script>

In my script, I have a simple Fetch:

fetch(url).then((res) => {
  if (!res.ok) {
    throw res;
  }

  return res.body.pipeThrough(new TextDecoderStream());
});

This causes the error:

Uncaught (in promise) TypeError: writable must be a WritableStream
at SDReadableStream.pipeThrough (sd-streams-polyfill.min.js:7)

Any thoughts on what I'm missing? Thanks.