elm-explorations/webgl

Functional rendering with WebGL in Elm

webgl's Issues

Memory leak when creating a mesh in update/view

Just pointing to the same issue in the old repo that is still relevant
elm-community/webgl#46

WebGL content can fail to show up immediately on Edge

This was first reported at ianmackenzie/elm-3d-scene#87 but seems to be a general WebGL issue. On some versions of Microsoft Edge (tested on version 44.19041.1.0 on Windows 10), replacing a general HTML element with a WebGL canvas some time after the Elm app has started results in the WebGL scene not actually being displayed until a redraw is triggered (window resize, user drags the mouse over the canvas element, etc.).

SSCCE is at https://ellie-app.com/9tMP6X3FjVTa1. The expected behavior is that the page will display "Loading..." for 1 second and then switch to a blue square. On Edge, this can fail and the text disappears but the blue square does not appear until the page is resized or a redraw is otherwise forced. Replacing the Html.text "Loading..." with a dummy WebGL scene like

WebGL.toHtml
    [ Html.Attributes.width 300
    , Html.Attributes.height 300
    ]
    []

fixes the issue, so it seems to be a problem when the virtual DOM replaces a non-WebGL element with a WebGL one.

GLSL with apostrophes breaks JS output

Steps to replicate

A fragment shader like this:

fragmentShader : Shader {} Uniforms { vcoord : Vec2 }
fragmentShader =
    [glsl|
        uniform sampler2D texture;
        varying vec2 vcoord;

        // This ain't gonna work now

        void main () {
          gl_FragColor = texture2D(texture, vcoord);
        }
    |]

Expected

• Compilation succeeds

Actual

(The build uses Webpack, hence these messages)

ERROR in ./src/elm/Main.elm 12271:97
Module parse failed: Unexpected token (12271:97)
You may need an appropriate loader to handle this file type.
| };
| var author$project$GL$fragmentShader = {
>       src: '\n        uniform sampler2D texture;\n        varying vec2 vcoord;\n\n        // This ain't gonna compile now\n\n        void main () {\n          gl_FragColor = texture2D(texture, vcoord);\n        }\n    ',
|       attributes: {},
|       uniforms: {texture: 'texture'}
 @ ./src/index.js 4:0-37 132:8-11

Note the src of the author$project$GL$fragmentShader object is delimited by ', but the inline comment's ' is not escaped here thus breaking the JS (hence Webpack then failing).

Use gl.drawArrays for non-indexed meshes

Currently, the code in WebGL.js uses gl.drawElements to draw all entities, constructing a 'dummy' index buffer for non-indexed meshes (which are in fact all current meshes except for those constructed using WebGL.indexedTriangles).

It would be better to use gl.drawArrays instead for non-indexed meshes; this would have a few advantages:

• less memory required on the GPU
• less work on the CPU constructing the dummy index buffer
• most importantly, it would remove the current restriction where (if I recall correctly) only up to 65536 vertices can be supported in a single mesh, since WebGL uses 16-bit vertex indices; there is no such restriction on the number of points/lines/triangles to draw using gl.drawArrays, since they don't have explicit indices

Cache attribute locations for a program

It seems that gl.getProgramParameter, gl.getActiveAttrib, gl.getAttribLocation can be cached for each program and not called for each draw.

This could improve the performance a little bit.

Allow efficiently clearing depth and stencil buffers

Right now to clear the stencil or depth buffer it is necessary to do a hack like drawing a full-screen quad. It would be useful (and presumably more efficient) to have something like some special Entity values WebGL.clearDepthBuffer and WebGL.clearStencilBuffer that would end up simply calling gl.clearDepth() or gl.clearStencil() when drawn.

Enable OES_element_index_uint

The OES_element_index_uint extension is listed as a "universally supported extension", and enabling it should allow elm-explorations/webgl to support indexed meshes with a basically unlimited number of vertices (theoretically billions instead of the current 65536 vertex limit).

This will also require updating the JS code to use Uint32Array instead of Uint16Array.

Mesh instancing support

Support for instancing, aka being able to draw the same mesh many times with different uniforms, would be very useful for my project.

Here's an example of my current work around to be able to efficiently draw many instances of a single mesh (I've excluded some details but this is the general idea).

type alias ChipTileVertex =
    { position : Vec3
    , instanceIndex : Float
    }

type alias ChipTileUniforms =
    { perspective : Mat4
    , camera : Mat4
    , time : Second
    , instanceOffset0 : Vec3
    , instanceOffset1 : Vec3
    , instanceOffset2 : Vec3
    , instanceOffset3 : Vec3
    , instanceOffset4 : Vec3
    , instanceOffset5 : Vec3
    , instanceOffset6 : Vec3
    , instanceOffset7 : Vec3
    , instanceOffset8 : Vec3
    , instanceOffset9 : Vec3
    , instanceOffset10 : Vec3
    , instanceOffset11 : Vec3
    , instanceOffset12 : Vec3
    , instanceOffset13 : Vec3
    , instanceOffset14 : Vec3
    , instanceOffset15 : Vec3
    , instanceOffset16 : Vec3
    , instanceOffset17 : Vec3
    , instanceOffset18 : Vec3
    , instanceOffset19 : Vec3
    }

chipTileUniforms : InGameModel -> Size Int -> Array Point2d -> ChipTileUniforms
chipTileUniforms inGameModel windowSize particles =
    let
        getOffset index =
            Array.get index particles
                |> Maybe.map
                    (\position ->
                        Vec3.vec3 (Point2d.xCoordinate position) (Point2d.yCoordinate position) 0
                    )
                |> Maybe.withDefault (Vec3.vec3 -999999 -999999 0)
    in
    { perspective = perspective windowSize
    , camera = cameraMatrix inGameModel
    , time = inGameModel.inGameTime
    , instanceOffset0 = getOffset 0
    , instanceOffset1 = getOffset 1
    , instanceOffset2 = getOffset 2
    , instanceOffset3 = getOffset 3
    , instanceOffset4 = getOffset 4
    , instanceOffset5 = getOffset 5
    , instanceOffset6 = getOffset 6
    , instanceOffset7 = getOffset 7
    , instanceOffset8 = getOffset 8
    , instanceOffset9 = getOffset 9
    , instanceOffset10 = getOffset 10
    , instanceOffset11 = getOffset 11
    , instanceOffset12 = getOffset 12
    , instanceOffset13 = getOffset 13
    , instanceOffset14 = getOffset 14
    , instanceOffset15 = getOffset 15
    , instanceOffset16 = getOffset 16
    , instanceOffset17 = getOffset 17
    , instanceOffset18 = getOffset 18
    , instanceOffset19 = getOffset 19
    }

chipTileVS : Shader ChipTileVertex ChipTileUniforms { vcolor : Vec4 }
chipTileVS =
    [glsl|
        attribute vec3 position;
        attribute float instanceIndex;
        uniform mat4 perspective;
        uniform mat4 camera;
        uniform vec3 instanceOffset0;
        uniform vec3 instanceOffset1;
        uniform vec3 instanceOffset2;
        uniform vec3 instanceOffset3;
        uniform vec3 instanceOffset4;
        uniform vec3 instanceOffset5;
        uniform vec3 instanceOffset6;
        uniform vec3 instanceOffset7;
        uniform vec3 instanceOffset8;
        uniform vec3 instanceOffset9;
        uniform vec3 instanceOffset10;
        uniform vec3 instanceOffset11;
        uniform vec3 instanceOffset12;
        uniform vec3 instanceOffset13;
        uniform vec3 instanceOffset14;
        uniform vec3 instanceOffset15;
        uniform vec3 instanceOffset16;
        uniform vec3 instanceOffset17;
        uniform vec3 instanceOffset18;
        uniform vec3 instanceOffset19;
        uniform float time;
        varying vec4 vcolor;

        void main () {
            mediump int index = int(instanceIndex);
            mediump vec3 offset =
                (instanceOffset0 * float(index == 0)) +
                (instanceOffset1 * float(index == 1)) +
                (instanceOffset2 * float(index == 2)) +
                (instanceOffset3 * float(index == 3)) +
                (instanceOffset4 * float(index == 4)) +
                (instanceOffset5 * float(index == 5)) +
                (instanceOffset6 * float(index == 6)) +
                (instanceOffset7 * float(index == 7)) +
                (instanceOffset8 * float(index == 8)) +
                (instanceOffset9 * float(index == 9)) +
                (instanceOffset10 * float(index == 10)) +
                (instanceOffset11 * float(index == 11)) +
                (instanceOffset12 * float(index == 12)) +
                (instanceOffset13 * float(index == 13)) +
                (instanceOffset14 * float(index == 14)) +
                (instanceOffset15 * float(index == 15)) +
                (instanceOffset16 * float(index == 16)) +
                (instanceOffset17 * float(index == 17)) +
                (instanceOffset18 * float(index == 18)) +
                (instanceOffset19 * float(index == 19));

            gl_Position = perspective * camera * vec4(position + vec3(offset.xy, 0.0), 1.0);
            vcolor = vec4(0.0, 0.5, 0.0, 0.5);
        }
    |]

It's very repetative code and if I want a new shader that also draws multiple instances, I need to copy paste this code.

Also while it's faster than making one draw call per instance, having to do the following in the vertex shader is probably expensive.

        mediump vec3 offset =
                (instanceOffset0 * float(index == 0)) +
                (instanceOffset1 * float(index == 1)) +
                (instanceOffset2 * float(index == 2)) +
                (instanceOffset3 * float(index == 3)) +
                (instanceOffset4 * float(index == 4)) +
                (instanceOffset5 * float(index == 5)) +
                (instanceOffset6 * float(index == 6)) +
                (instanceOffset7 * float(index == 7)) +
                (instanceOffset8 * float(index == 8)) +
                (instanceOffset9 * float(index == 9)) +
                (instanceOffset10 * float(index == 10)) +
                (instanceOffset11 * float(index == 11)) +
                (instanceOffset12 * float(index == 12)) +
                (instanceOffset13 * float(index == 13)) +
                (instanceOffset14 * float(index == 14)) +
                (instanceOffset15 * float(index == 15)) +
                (instanceOffset16 * float(index == 16)) +
                (instanceOffset17 * float(index == 17)) +
                (instanceOffset18 * float(index == 18)) +
                (instanceOffset19 * float(index == 19));

I admit, I haven't entirely thought out how the elm-webgl API should be modified in order to support instancing. My initial idea is to change the uniform parameter in entity to a list. I'm happy to discuss how this might be organized.

Workaround for uniform array type inference failure

I have the following Elm type for uniforms:

type alias Uniforms = { cellColors : List Vec3 }

In a vertex shader I have the following declaration for an array of vec3 uniform:

uniform vec3 cellColors[9];

The GLSL parser, however, is inferring/parsing the type of the cellColors variable to be Vec3, and not Array Vec3.

It seems as if this is a bug in the underlying GLSL parsing library: noteed/language-glsl#19

Is there a way to work around this bug?

Parallelise shader compilation and programs linking

The first frame may take a while to render because the WebGL scene may be composed out of different combinations of shaders.

WebGL best practices recommend that the shader compilation and program linking should be run in parallel.

This would probably require the following steps:

1. Loop through all the entities and initiate the shader compilation and figure out all the programs that need to be linked
2. Loop through the programs to start linking these
3. Loop through the programs again to check the linking status
4. Loop through all the entities to draw them

Mesh generation in view clarification

In your documentation, you said

Do not generate meshes in view, read more about this here

I didn't see any way of the alternative. All your example call WebGL.toHtml that end up generating a mesh on the fly.

Did a miss something? Can you clarify what you mean by that? I supposed it's better to generate the mesh on a init function and store them the application state somehow?

Thx!

Dynamic textures support

We have several projects using Elm+WebGL for now and we produce generative graphics with that and we're very happy with using it. Thanks to Elm, we have type safety, and our code defines algorithms in a very clear way. Thanks to WebGL package and its simple and powerful API, doing shaders and passing complex structures to them is easy.

However, for more and more times we encounter the issue that it would be interesting to work with dynamic textures—for example, one would create a kaleidoscopic effect using the part of the viewport that is already rendered. Sure, we can use several canvases for that, but sometimes one layer is needs to be reflected and another is not, or the effect result itself is transforming inside the complex scene.

So, I would kindly ask to think once again on adding this functionality to the API. I did the Pull Request for the Elm 0.18 version, but now in 0.19 it's harder to have your own forks of Elm Packages which use JavaScript, except patching them on build.

Hot-reloading corrupts shaders

Sometimes when hot-reloading the webpage, artifacts can get introduced into the scene such as where the red arrow is pointing

Hot-reloading sometimes also causes entities to not render.

In terms of frequency, the former seems to rarely happen (I delayed posting this issue because I couldn't reproduce it for a couple days) and the latter happens maybe 1 in 5 hot-reloads.

The mesh corruption artifacts don't seem to go away until I do a page refresh while the entities disappearing usually is fixed by doing one or two more hot-reloads.

This is occurring on Chrome both on my Mac and Windows computers. It might have also occurred on Firefox though I might be misremembering as I haven't tested that in a while.

Edit: Now I'm getting the mesh corruption repeatedly. I'm not sure what conditions cause it to happen more frequently.

Redundant WebGL calls

I did some GPU profiling with Spector.js. It's reporting that there are some redundant calls made before and after each drawElements call.

In this screeshot for example, all the calls with orange text are apparently redundant.

Here is a capture I took of my game if you want to see more details.
capture 17_59_02.json.zip

preserveDrawingBuffer

Is there any way to use this package with the option of preserveDrawingBuffer True?

Inconsistent default behavior for WebGL.entity between Firefox and Chrome

The following code (Ellie link) will show a green rectangle on Firefox and a black rectangle on Chrome and Safari. This behavior is consistent both on Windows and Mac OS.

If you change WebGL.entity to WebGL.entityWith [] then all browsers will be consistent in showing a green rectangle.

module Main exposing (main)

import Browser.Events
import Browser
import Html exposing (Html, button, div, text)
import Html.Events exposing (onClick)
import Math.Matrix4 exposing (Mat4)
import Math.Vector2 as Vec2 exposing (Vec2)
import Math.Vector4 as Vec4 exposing (Vec4)
import WebGL exposing (Mesh, Shader)

main : Platform.Program () () ()
main =
    Browser.element
        { init = \_ -> ((), Cmd.none)
        , update = \_ _ -> ((), Cmd.none)
        , view = view
        , subscriptions =
            \_ -> Browser.Events.onAnimationFrame (\_ -> ())
        }


view model =
    WebGL.toHtmlWith
        [ WebGL.stencil 0
        ]
        []
        [ WebGL.entity
            vertexShader
            fragmentShader
            viewportSquare
            { color = Vec4.vec4 0 1 0 1
            , matrix = Math.Matrix4.identity
            }
        ]


viewportSquare : Mesh { position : Vec2 }
viewportSquare =
    WebGL.triangleFan
        [ { position = Vec2.vec2 -1 -1 }
        , { position = Vec2.vec2 1 -1 }
        , { position = Vec2.vec2 1 1 }
        , { position = Vec2.vec2 -1 1 }
        ]


vertexShader : Shader { position : Vec2 } { a | matrix : Mat4 } {}
vertexShader =
    [glsl|

attribute vec2 position;
uniform mat4 matrix;

void main () {
  gl_Position = matrix * vec4(position, 0.0, 1.0);
}

|]


fragmentShader : Shader {} { a | color : Vec4 } {}
fragmentShader =
    [glsl|
precision mediump float;
uniform vec4 color;

void main () {
    gl_FragColor = color;
}
    |]

Add support for OES_standard_derivatives extension?

Following up from elm-community/webgl#48, https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/WebGL_best_practices lists OES_standard_derivatives as a "universally supported WebGL 1 extension". So it should be fairly safe to add

gl.getExtension('OES_standard_derivatives');

to the JS context initialization code and prepend

#extension GL_OES_standard_derivatives : enable

to each generated fragment shader. This would then open up a bunch of cool possibilities for things like edge highlighting.

Version 1.1.0 Runtime error "Uncaught Error: Trying to use (==)"

Uncaught Error: Trying to use (==) on functions. There is no way to know if functions are "the same" in the Elm sense. Read more about this at https://package.elm-lang.org/packages/elm/core/latest/Basics#== which describes why it is this way and what the better version will look like.

Ellie

module Main exposing (main)

import Browser
import Html exposing (Html)
import Json.Decode exposing (Value)
import Task exposing (Task)
import WebGL.Texture as WebGL


init _ =
    ( ()
    --here is problem
    , Task.map2 (==) (WebGL.load image) (WebGL.load image)
        |> Task.attempt identity
    )


main : Program Value () (Result WebGL.Error Bool)
main =
    Browser.element
        { init = init
        , view = \_ -> Html.text "Success"
        , subscriptions = \_ -> Sub.none
        , update = \_ m -> ( m, Cmd.none )
        }


image =
    "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mPUDLS6DAAC6AGJTI9bHgAAAABJRU5ErkJggg=="

Don't bind attribute buffers for the same program if they don't change

If the same mesh is rendered many times in a row using the same shaders, the current behaviour would still call gl.bindBuffer, gl.enableVertexAttribArray and gl.vertexAttribPointer for the same mesh. These extra calls could be potentially avoided.

This should improve the performance of 2D games with many sprites, that are rendered using the same quad mesh, the same shaders, but different uniforms. This should also improve the rendering of scenes like this http://unsoundscapes.com/elm-physics/examples/boxes/

Dynamic mesh support

Hi, I'm interested in rendering circuit boards similar to this
https://3c1703fe8d.site.internapcdn.net/newman/gfx/news/hires/2009/runningelect.jpg

Ideally I'd generate one large mesh for the entire circuit board. The layout of the circuit board isn't known at compile time though and as I understand it, all meshes need to be known at compile time, as generating them in the view causes a memory leak.

It seems like currently the best I can do then is have a rectangle and circle mesh at compile time and use a bunch of them to render the entire circuit board.

As a test to see if it would be feasible to do this I ran a quick benchmark by drawing the same circle mesh with varying transformation matrices. Around 600 circles per frame was the limit before the frame rate would start to drop. This seems too low to achieve what I'm aiming for.

So if possible, it would be great to support creating new meshes after compile time.

Edit: I realized that meshes can still be created during runtime so long as they are stored in the model and not modified. This works well enough for my purposes though it would still be nice to have some way to free/modify existing meshes.

Allow creation of Textures from existing image data

Apologies if this is already supported or due to be released, I couldn't find any references, but I think it's sort of related to the open PR about creating textures from bytes.

Textures can be loaded from a url, but I'd like to be able to read image data directly from an image I've already loaded, or a 2D canvas that is already on the page.

Texture colors altered on some browsers

I've noticed that on two occasions (once in someone's Firefox browser and I don't recall which browser the other person was using), textures would have some kind of color correction applied. This would cause problems for this app I'm working on https://town-collab.app where certain texture colors (such as FF00FF) get replaced with user chosen colors at runtime. The color correction causes the game to look like this

The fix was to add gl.pixelStorei(gl.UNPACK_COLORSPACE_CONVERSION_WEBGL, false);. Ideally the webgl package should do this automatically though?

Perhaps also gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false); should be used. I haven't encountered any problems with premultiplication but I suspect some browsers are going to randomly have different default values here too.

Edit: For reference, I got the two lines of code from here https://stackoverflow.com/questions/31500597/fragment-shader-wrong-color-value-from-texture

Allow Mat4 to be used as attribute to vertex shader

Just ran into an issue where I was getting this error:

Uncaught Error: No info available for: 35676
    at _WebGL_doBindAttribute (Main.elm:6110)
    ...

After a little digging I discovered it's because there's no support for mat4 attributes.

It seems like this should be possible to do, and pretty straightforward: https://stackoverflow.com/questions/38853096/webgl-how-to-bind-values-to-a-mat4-attribute

My use case is passing in a separate transform for each Entity as an attribute, in order to do the transformation on the GPU. I'm willing to make the change, just wanted to get some opinions first.

Add premultiply option for texture loading

In order to alpha blend textures correctly, their color channel needs to be premultiplied by the alpha channel. I believe WebGL supports doing this when loading textures but that feature isn't exposed in this package.

For that reason I think premultiply : Bool should be added as an additional field available when calling WebGL.Texture.loadWith. For WebGL.Texture.load the default value will be False.

How to build a Shader without the [glsl|...|] syntax ?

The documentation states

Normally you specify a shader with a [glsl| |] block.

Is there any other way to build a Shader without the [glsl| ...|] syntactical construct ? I would like to generate shader code in Elm, and therefore I don't have any shader code at compile time to put in there.