Giter Site home page Giter Site logo

brendan-duncan / wgsl_reflect Goto Github PK

View Code? Open in Web Editor NEW
180.0 9.0 19.0 651 KB

A WebGPU Shading Language parser and reflection library for Javascript.

License: MIT License

HTML 1.24% CSS 0.12% JavaScript 70.54% TypeScript 28.09%
webgpu wgsl javascript

wgsl_reflect's Introduction

WebGPU Shading Language Reflection Library

A WebGPU Shading Language parser and reflection library for Typescript and Javascript.

wgsl_reflect can parse a WGSL shader and analyze its contents, providing information about the shader. It can determine the bind group layout of the shader, resource bindings, uniform buffers, the members of a uniform buffer, their names, types, sizes, offsets into the buffer.

Usage

From NPM

npm install wgsl_reflect

The wgsl_reflect.module.js file is a self-contained roll-up of the library that can be included in your project and imported with:

import { WgslReflect } from "wgsl_reflect/wgsl_reflect.module.js";
const reflect = new WgslReflect(shader_code);

Example

WGSL Reflect Example

Documentation

class WgslReflect {
  /// All top-level uniform vars in the shader.
  uniforms: Array<VariableInfo>;
  /// All top-level storage vars in the shader, including storage buffers and textures.
  storage: Array<VariableInfo>;
  /// All top-level texture vars in the shader;
  textures: Array<VariableInfo>;
  // All top-level sampler vars in the shader.
  samplers: Array<VariableInfo>;
  /// All top-level type aliases in the shader.
  aliases: Array<AliasInfo>;
  /// All top-level overrides in the shader.
  overrides: Array<OverrideInfo> = [];
  /// All top-level structs in the shader.
  structs: Array<StructInfo>;
  /// All entry functions in the shader: vertex, fragment, and/or compute.
  entry: EntryFunctions;
  /// All functions in the shader, including entry functions.
  functions: Array<FunctionInfo>;

  // Find a resource by its group and binding.
  findResource(group: number, binding: number): VariableInfo | null;

  // Get the bind groups used by the shader, bindGroups[group][binding].
  getBindGroups(): Array<Array<VariableInfo>>;
}

enum ResourceType {
  Uniform,
  Storage,
  Texture,
  Sampler,
  StorageTexture
}

class VariableInfo {
  name: string; // The name of the variable
  type: TypeInfo; // The type of the variable
  group: number; // The binding group of the variable
  binding: number; // The binding index of the variable
  resourceType: ResourceType; // The resource type of the variable
  access: string; // "", read, write, read_write

  get isArray(): boolean; // True if it's an array type
  get isStruct(): boolean;  // True if it's a struct type
  get isTemplate(): boolean; // True if it's a template type
  get size(): number; // Size of the data, in bytes
  get align(): number; // The alignment size if it's a struct, otherwise 0
  get members(): Array<MemberInfo> | null; // The list of members if it's a struct, otherwise null
  get format(): TypeInfo | null; // The format if it's a template or array, otherwise null
  get count(): number; // The array size if it's an array, otherwise 0
  get stride(): number; // The array stride if it's an array, otherwise 0
}

class TypeInfo {
  name: string;
  size: number; // Size of the data, in bytes

  get isArray(): boolean;
  get isStruct(): boolean;
  get isTemplate(): boolean;
}

class StructInfo extends TypeInfo {
  members: Array<MemberInfo>;
  align: number;
  startLine: number;
  endLine: number;
  inUse: boolean; // true if the struct is used by a uniform, storage, or directly or indirectly by an entry function.
}

class ArrayInfo extends TypeInfo {
  format: TypeInfo;
  count: number;
  stride: number;
}

class TemplateInfo extends TypeInfo {
  format: TypeInfo;
  access: string; // "", read, write, read_write
}

class MemberInfo {
  name: string;
  type: TypeInfo;
  offset: number;
  size: number;

  get isArray(): boolean;
  get isStruct(): boolean;
  get isTemplate(): boolean;
  get align(): number;
  get members(): Array<MemberInfo> | null;
  get format(): TypeInfo | null;
  get count(): number;
  get stride(): number;
}

class AliasInfo {
  name: string;
  type: TypeInfo;
}

class EntryFunctions {
  vertex: Array<FunctionInfo>;
  fragment: Array<FunctionInfo>;
  compute: Array<FunctionInfo>;
}

class FunctionInfo {
  name: string;
  stage: string | null;
  inputs: Array<InputInfo>;
  outputs: Array<OutputInfo>;
  resources: Array<VariableInfo>;
  startLine: number;
  endLine: number;
  inUse: boolean;  // true if called directly or indirectly by an entry function.
  calls: Set<FunctionInfo>; // All custom functions called directly by this function.
}

class InputInfo {
  name: string;
  type: TypeInfo | null;
  locationType: string;
  location: number | string;
  interpolation: string | null;
}

class OutputInfo {
  name: string;
  type: TypeInfo | null;
  locationType: string;
  location: number | string;
}

class OverrideInfo {
  name: string;
  type: TypeInfo | null;
  id: number;
}

Examples

Calculate the bind group information in the shader:

import { WgslReflect } from "./wgsl_reflect.module.js";

const shader = `
struct ViewUniforms {
    viewProjection: mat4x4<f32>
}

struct ModelUniforms {
    model: mat4x4<f32>,
    color: vec4<f32>,
    intensity: f32
}

@binding(0) @group(0) var<uniform> viewUniforms: ViewUniforms;
@binding(1) @group(0) var<uniform> modelUniforms: ModelUniforms;
@binding(2) @group(0) var u_sampler: sampler;
@binding(3) @group(0) var u_texture: texture_2d<f32>;

struct VertexInput {
    @location(0) a_position: vec3<f32>,
    @location(1) a_normal: vec3<f32>,
    @location(2) a_color: vec4<f32>,
    @location(3) a_uv: vec2<f32>
}

struct VertexOutput {
    @builtin(position) Position: vec4<f32>,
    @location(0) v_position: vec4<f32>,
    @location(1) v_normal: vec3<f32>,
    @location(2) v_color: vec4<f32>,
    @location(3) v_uv: vec2<f32>
}

@vertex
fn main(input: VertexInput) -> VertexOutput {
    var output: VertexOutput;
    output.Position = viewUniforms.viewProjection * modelUniforms.model * vec4<f32>(input.a_position, 1.0);
    output.v_position = output.Position;
    output.v_normal = input.a_normal;
    output.v_color = input.a_color * modelUniforms.color * modelUniforms.intensity;
    output.v_uv = input.a_uv;
    return output;
}`;

const reflect = new WgslReflect(shader);

console.log(reflect.functions.length); // 1
console.log(reflect.structs.length); // 4
console.log(reflect.uniforms.length); // 2

// Shader entry points
console.log(reflect.entry.vertex.length); // 1, there is 1 vertex entry function.
console.log(reflect.entry.fragment.length); // 0, there are no fragment entry functions.
console.log(reflect.entry.compute.length); // 0, there are no compute entry functions.

console.log(reflect.entry.vertex[0].name); // "main", the name of the vertex entry function.

console.log(reflect.entry.vertex[0].resources.length); // 2, main uses modelUniforms and viewUniforms resource bindings.
console.log(reflect.entry.vertex[0].resources[0].name); // viewUniforms
console.log(reflect.entry.vertex[0].resources[1].name); // modelUniforms

// Vertex shader inputs
console.log(reflect.entry.vertex[0].inputs.length); // 4, inputs to "main"
console.log(reflect.entry.vertex[0].inputs[0].name); // "a_position"
console.log(reflect.entry.vertex[0].inputs[0].location); // 0
console.log(reflect.entry.vertex[0].inputs[0].locationType); // "location" (can be "builtin")
console.log(reflect.entry.vertex[0].inputs[0].type.name); // "vec3"
console.log(reflect.entry.vertex[0].inputs[0].type.format.name); // "f32"

// Gather the bind groups used by the shader.
const groups = reflect.getBindGroups();
console.log(groups.length); // 1
console.log(groups[0].length); // 4, bindings in group(0)

console.log(groups[0][1].resourceType); // ResourceType.Uniform, the type of resource at group(0) binding(1)
console.log(groups[0][1].size); // 96, the size of the uniform buffer.
console.log(groups[0][1].members.length); // 3, members in ModelUniforms.
console.log(groups[0][1].members[0].name); // "model", the name of the first member in the uniform buffer.
console.log(groups[0][1].members[0].offset); // 0, the offset of 'model' in the uniform buffer.
console.log(groups[0][1].members[0].size); // 64, the size of 'model'.
console.log(groups[0][1].members[0].type.name); // "mat4x4", the type of 'model'.
console.log(groups[0][1].members[0].type.format.name); // "f32", the format of the mat4x4.

console.log(groups[0][2].resourceType); // ResourceType.Sampler

console.log(groups[0][3].resourceType); // ResourceType.Texture
console.log(groups[0][3].type.name); // "texture_2d"
console.log(groups[0][3].type.format.name); // "f32"

Calculate the member information for a uniform buffer block:

import { WgslReflect } from "./wgsl_reflect.module.js";

// WgslReflect can calculate the size and offset for members of a uniform buffer block.

const shader = `
struct A {                                     //             align(8)  size(32)
    u: f32,                                    // offset(0)   align(4)  size(4)
    v: f32,                                    // offset(4)   align(4)  size(4)
    w: vec2<f32>,                              // offset(8)   align(8)  size(8)
    @size(16) x: f32                          // offset(16)  align(4)  size(16)
}

struct B {                                     //             align(16) size(208)
    a: vec2<f32>,                              // offset(0)   align(8)  size(8)
    // -- implicit member alignment padding -- // offset(8)             size(8)
    b: vec3<f32>,                              // offset(16)  align(16) size(12)
    c: f32,                                    // offset(28)  align(4)  size(4)
    d: f32,                                    // offset(32)  align(4)  size(4)
    // -- implicit member alignment padding -- // offset(36)            size(12)
    @align(16) e: A,                           // offset(48)  align(16) size(32)
    f: vec3<f32>,                              // offset(80)  align(16) size(12)
    // -- implicit member alignment padding -- // offset(92)            size(4)
    g: @stride(32) array<A, 3>,                // offset(96)  align(8)  size(96)
    h: i32,                                    // offset(192) align(4)  size(4)
    // -- implicit struct size padding --      // offset(196)           size(12)
}

@group(0) @binding(0)
var<uniform> uniform_buffer: B;`;

const reflect = new WgslReflect(shader);

const u = reflect.uniforms[0];
console.log(u.size); // 208, the size of the uniform buffer in bytes
console.log(u.group); // 0
console.log(u.binding); // 0
console.log(u.members.length); // 8, members in B
console.log(u.members[0].name); // "a"
console.log(u.members[0].offset); // 0, the offset of 'a' in the buffer
console.log(u.members[0].size); // 8, the size of 'a' in bytes
console.log(u.members[0].type.name); // "vec2", the type of 'a'
console.log(u.members[0].type.format.name); // "f32", the format of the vec2.

console.log(u.members[4].name); // "e"
console.log(u.members[4].offset); // 48, the offset of 'e' in the buffer
console.log(u.members[4].size); // 32, the size of 'e' in the buffer

wgsl_reflect's People

Contributors

brendan-duncan avatar greggman avatar mflament avatar mighdoll avatar wardenfeng avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wgsl_reflect's Issues

Support for inferred typed arrays

the current version doesn't like this style of array declaration

  @vertex fn vs(@builtin(vertex_index) vertexIndex : u32) -> @builtin(position) vec4f {
    let pos = array(
      vec2f( 0.0,  0.5),  // top center
      vec2f(-0.5, -0.5),  // bottom left
      vec2f( 0.5, -0.5)   // bottom right
    );

    return vec4f(pos[vertexIndex], 0.0, 1.0);
  }

array works as a constructor if all the arguments are of the same type so in the case above, the type of pos is inferred as array<vec2<f32>, 3>. I'm not sure wgsl_reflect has to figure that out but it probably needs to parse the expression.

Support for arrays of arrays and other stuff

I think there is some inconsistency in how nested types are handled.

For example I have this test


const reflect = new WgslReflect(`
    struct InnerUniforms {
        bar: u32,
    };

    struct VSUniforms {
        foo: u32,
        moo: InnerUniforms,
    };
@group(0) @binding(0) var<uniform> foo0: vec3f;
@group(0) @binding(1) var<uniform> foo1: array<vec3f, 5>;
@group(0) @binding(2) var<uniform> foo2: array<array<vec3f, 5>, 6>;
@group(0) @binding(3) var<uniform> foo3: array<array<array<vec3f, 5>, 6>, 7>;

@group(0) @binding(4) var<uniform> foo4: VSUniforms;
@group(0) @binding(5) var<uniform> foo5: array<VSUniforms, 5>;
@group(0) @binding(6) var<uniform> foo6: array<array<VSUniforms, 5>, 6>;
@group(0) @binding(7) var<uniform> foo7: array<array<array<VSUniforms, 5>, 6>, 7>;
`);

If I then print out stuff like this

reflect.uniforms.map(uniform => {
    const info = reflect.getUniformBufferInfo(uniform);
    console.log('---------:', uniform.name);
    console.log(JSON.stringify(info, null, 2));
});

I get this

---------: foo0
{
  "align": 16,
  "size": 12,
  "name": "foo0",
  "type": {
    "name": "vec3f",
    "attributes": null
  },
  "isArray": false,
  "isStruct": false,
  "arrayStride": 12,
  "arrayCount": 0,
  "group": 0,
  "binding": 0
}
---------: foo1
{
  "align": 16,
  "size": 80,
  "name": "foo1",
  "type": {
    "name": "array",
    "attributes": null,
    "format": {
      "name": "vec3f"
    },
    "count": 5
  },
  "isArray": true,
  "isStruct": false,
  "arrayStride": 12,
  "arrayCount": 5,
  "group": 0,
  "binding": 1
}
---------: foo2
{
  "align": 16,
  "size": 480,
  "name": "foo2",
  "type": {
    "name": "array",
    "attributes": null,
    "format": {
      "name": "array",
      "attributes": null,
      "format": {
        "name": "vec3f"
      },
      "count": 5
    },
    "count": 6
  },
  "isArray": true,
  "isStruct": false,
  "arrayStride": 80,
  "arrayCount": 6,
  "group": 0,
  "binding": 2
}
---------: foo3
{
  "align": 16,
  "size": 3360,
  "name": "foo3",
  "type": {
    "name": "array",
    "attributes": null,
    "format": {
      "name": "array",
      "attributes": null,
      "format": {
        "name": "array",
        "attributes": null,
        "format": {
          "name": "vec3f"
        },
        "count": 5
      },
      "count": 6
    },
    "count": 7
  },
  "isArray": true,
  "isStruct": false,
  "arrayStride": 480,
  "arrayCount": 7,
  "group": 0,
  "binding": 3
}
---------: foo4
{
  "align": 4,
  "size": 8,
  "name": "foo4",
  "type": {
    "name": "VSUniforms",
    "attributes": null
  },
  "members": [
    {
      "node": {
        "name": "foo",
        "type": {
          "name": "u32",
          "attributes": null
        },
        "attributes": null
      },
      "name": "foo",
      "offset": 0,
      "size": 4,
      "type": {
        "name": "u32",
        "attributes": null
      },
      "isArray": false,
      "arrayCount": 0,
      "arrayStride": 4,
      "isStruct": false
    },
    {
      "node": {
        "name": "moo",
        "type": {
          "name": "InnerUniforms",
          "attributes": null
        },
        "attributes": null
      },
      "name": "moo",
      "offset": 4,
      "size": 4,
      "type": {
        "name": "InnerUniforms",
        "attributes": null
      },
      "isArray": false,
      "arrayCount": 0,
      "isStruct": true,
      "members": [
        {
          "node": {
            "name": "bar",
            "type": {
              "name": "u32",
              "attributes": null
            },
            "attributes": null
          },
          "name": "bar",
          "offset": 0,
          "size": 4,
          "type": {
            "name": "u32",
            "attributes": null
          },
          "isArray": false,
          "arrayCount": 0,
          "arrayStride": 4,
          "isStruct": false
        }
      ]
    }
  ],
  "isArray": false,
  "isStruct": true,
  "arrayCount": 0,
  "group": 0,
  "binding": 4
}
---------: foo5
{
  "align": 4,
  "size": 40,
  "name": "foo5",
  "type": {
    "name": "array",
    "attributes": null,
    "format": {
      "name": "VSUniforms"
    },
    "count": 5
  },
  "isArray": true,
  "isStruct": true,
  "members": [
    {
      "node": {
        "name": "foo",
        "type": {
          "name": "u32",
          "attributes": null
        },
        "attributes": null
      },
      "name": "foo",
      "offset": 0,
      "size": 4,
      "type": {
        "name": "u32",
        "attributes": null
      },
      "isArray": false,
      "arrayCount": 0,
      "arrayStride": 4,
      "isStruct": false
    },
    {
      "node": {
        "name": "moo",
        "type": {
          "name": "InnerUniforms",
          "attributes": null
        },
        "attributes": null
      },
      "name": "moo",
      "offset": 4,
      "size": 4,
      "type": {
        "name": "InnerUniforms",
        "attributes": null
      },
      "isArray": false,
      "arrayCount": 0,
      "isStruct": true,
      "members": [
        {
          "node": {
            "name": "bar",
            "type": {
              "name": "u32",
              "attributes": null
            },
            "attributes": null
          },
          "name": "bar",
          "offset": 0,
          "size": 4,
          "type": {
            "name": "u32",
            "attributes": null
          },
          "isArray": false,
          "arrayCount": 0,
          "arrayStride": 4,
          "isStruct": false
        }
      ]
    }
  ],
  "arrayStride": 8,
  "arrayCount": 5,
  "group": 0,
  "binding": 5
}
---------: foo6
{
  "align": 4,
  "size": 240,
  "name": "foo6",
  "type": {
    "name": "array",
    "attributes": null,
    "format": {
      "name": "array",
      "attributes": null,
      "format": {
        "name": "VSUniforms"
      },
      "count": 5
    },
    "count": 6
  },
  "isArray": true,
  "isStruct": false,
  "arrayStride": 40,
  "arrayCount": 6,
  "group": 0,
  "binding": 6
}
---------: foo7
{
  "align": 4,
  "size": 1680,
  "name": "foo7",
  "type": {
    "name": "array",
    "attributes": null,
    "format": {
      "name": "array",
      "attributes": null,
      "format": {
        "name": "array",
        "attributes": null,
        "format": {
          "name": "VSUniforms"
        },
        "count": 5
      },
      "count": 6
    },
    "count": 7
  },
  "isArray": true,
  "isStruct": false,
  "arrayStride": 240,
  "arrayCount": 7,
  "group": 0,
  "binding": 7
}
--end--

You can see foo4 has members ['foo', 'bar']

foo5 also does but lists the top level as an array

foo6 and foo7 are missing all "member" info.

Similar issues exist for foo2 and foo3 vs foo0 and foo1.

How should this be handled? Maybe some other info for array should then nest to a type definition rather than how they are currently combined at the top level? In other words, right now the top level has things like "arrayCount" and "arrayStride" and "isArray" and "isStruct" but as these are nested deeper this seems to break?

shader that doesn't compile

Thanks for this library!

If you're interested, this shader works in webgpu but doesn't work here

    struct VertexDesc {
        offset: u32,
        stride: u32,
        size: u32,
        padding: u32,
    };

    struct LineInfo {
        triDiv: u32,
        triMul: u32,
        midMod: u32,
        midDiv: u32,
        oddMod: u32,
        triMod: u32,
        pad0: u32,
        pad1: u32,
    };

    struct VSUniforms {
        worldViewProjection: mat4x4<f32>,
        position: VertexDesc,
        lineInfo: LineInfo,
        color: vec4<f32>,
        lightDirection: vec3<f32>,
    };

    @group(0) @binding(0) var<uniform> vsUniforms: VSUniforms;
    @group(0) @binding(1) var<storage> vertData: array<f32>;

    fn getVert(desc: VertexDesc, index: u32) -> vec4<f32> {
        var v = vec4<f32>(0, 0, 0, 1);
        let offset = desc.offset + index * desc.stride;
        for (var i: u32 = 0u; i < desc.size; i += 1u) {
            v[i] = vertData[offset + i];
        }
        return v;
    }

    struct MyVSOutput {
        @builtin(position) position: vec4<f32>,
    };

    @vertex
    fn myVSMain(@builtin(vertex_index) vertex_index: u32) -> MyVSOutput {
        var vsOut: MyVSOutput;
        var i = (vertex_index / vsUniforms.lineInfo.triDiv) * vsUniforms.lineInfo.triMul +
                ((vertex_index % vsUniforms.lineInfo.midMod) / vsUniforms.lineInfo.midDiv +
                (vertex_index % vsUniforms.lineInfo.oddMod)) % vsUniforms.lineInfo.triMod;
        let position = getVert(vsUniforms.position, i);
        vsOut.position = vsUniforms.worldViewProjection * position;
        return vsOut;
    }

    @fragment
    fn myFSMain(v: MyVSOutput) -> @location(0) vec4<f32> {
        return vsUniforms.color + vec4(vsUniforms.lightDirection, 0) * 0.0;
    }

gets

wgsl_reflect.module.js:691 Token_lexeme: "+"_line: 33_type: {name: 'plus', type: 'token', rule: '+', toString: ฦ’}[[Prototype]]: Object "Expected '='."
_error @ wgsl_reflect.module.js:691
wgsl_reflect.module.js:719 Uncaught Objectmessage: "Expected '='."toString: ฦ’ ()token: Tokenย {_type: {โ€ฆ}, _lexeme: '+', _line: 33}[[Prototype]]: Object

Resources not included in reflection if used directly as array index

Hi again,

I hope I don't bother you too much with those issues, I just love your library and rely on it every day.

I found a little bug when using a binding as array index, the binding is not added to the resources array in the program entry.

Here is an example:

I have two bindings, a uniform and a storage buffer:

@group(0) @binding(2) var<uniform>  batchIndex: u32;
@group(0) @binding(3) var<storage, read> batchOffsets: array<u32>;

When accessing batchOffsets directly with batchIndex like below batchIndex is not added to entry.vertex.0.resources

@vertex
fn vertex_main(
  vertex: VertexInput
) -> VertexOutput {
    let batchOffset = batchOffsets[batchIndex]; // will NOT work
    // ...
}

A simple workaround is simply to create a temporary variable:

@vertex
fn vertex_main(
  vertex: VertexInput
) -> VertexOutput {
    let index = batchIndex;
    let batchOffset = batchOffsets[index]; // will work
    // ...
}

Additonally, the issue is the same if the batchIndex uniform is a struct rather than a value:

struct BatchIndex {
   index: u32,
}
@group(0) @binding(2) var<uniform> batchIndex: BatchIndex;
@group(0) @binding(3) var<storage, read> batchOffsets: array<u32>;

@vertex
fn vertex_main(
  vertex: VertexInput
) -> VertexOutput {
    let batchOffset = batchOffsets[batchIndex.index]; // will NOT work
    // ...
}

Performance issues on medium-sized shaders (> 500+ line of code)

Hi again!

I wanted to report an issue that I am having with the library:

it's quite slow specially for shader > 500 lines of code (not huge but just a bit longer).

For instance, my PBR / physical shader is about 1000 lines of WGSL code and the reflection takes 200ms on a beefy desktop machine.
Note that it then takes 20ms to compile the WGSL shader itself (for comparison).

Simply put: is there anything we could do to speed up the reflection?

The problem of the web and non-binary shaders is that we must recompile and reflect our shaders are runtime on the fly
since compiling all shader variants right from the start of the app would dramatically slow down the start time.

But waiting 200ms for a simple PBR shader variant to be reflected is a tough pile to swallow, specially if it happens often :(

Do you have any tips to mitigate this?
Would the reflection be faster if function bodies would be empty for instance?

for ++ fails to compile

Hi,

Great library however there is an issue with for loops.
The i++ generates an error and prevents the reflection to happen.

fn foo() {
   for (var i = 0; i < 4; i++) {
   }
}

Needs support for `alias`

example

        alias material_index = u32;
        alias color = vec3f;

        struct material {
            index: material_type,
            diffuse: color,
        }

        @group(0) @binding(1) var<storage> materials: array<material>;

reflect returns nothing for this

WgslReflect {structs: Array(0), uniforms: Array(0), storage: Array(0), textures: Array(0), samplers: Array(0), โ€ฆ}
aliases: []
ast: []
entry: EntryFunctions
  compute: []
  fragment: []
  vertex: []
samplers: []
storage: []
structs: []
textures: []
uniforms: []

Fantastic project!

Thanks for sharing this project.

Maybe I be so free to ask why it wasn't written in Typescript?

WgslReflect.getBindGroups() does not inlcude storage buffers

WgslReflect.getBindGroups() does not list storage buffers

Example Shader:

      @group(0) @binding(0) var<storage, read> x : array<f32>;
      @group(0) @binding(1) var<storage, read> y : array<f32>;
      @group(0) @binding(2) var<storage, write> z : array<f32>;
      
      @stage(compute) @workgroup_size(64)
      fn main(@builtin(global_invocation_id) global_id : vec3<u32>) {
        // Guard against out-of-bounds work group sizes
        var idx = global_id.x;
        z[idx] = x[idx] + y[idx];
      }

Calling .getBindGroups() on shader reflection return empty array.

Would be interested to submit a pull request if you accept contributions.

why there is no sampler info in reflection information

on the example page i tried a shader with sampler but there is no sampler info in the reflection information.
here is the shader:

struct VertexOutput {
@Builtin(position) Position: vec4,
@location(0) attr_var_UV: vec2
}

@binding(1) @group(1) var color: texture_2d;
@binding(0) @group(1) var tex_sprite: sampler;

@Fragment
fn fragment(fragData: VertexOutput) -> @location(0) vec4f {
var _33: vec4 = textureSample(color, tex_sprite, fragData.attr_var_UV);
var _34: f32 = _33.x;
var _41: f32 = _33.y;
var _48: f32 = _33.z;
var _57: vec3 = vec3(pow((_34 + 0.054999999701976776123046875) * 0.947867333889007568359375, 2.400000095367431640625), _41, _48) * _33.w;
return vec4(_57.x, _57.y, _57.z, _33.w);
}

and the reflection info

TIME: 1.2000000476837158ms
Entry Points
fragment: fragment
Inputs
Position: vec4 location: position
attr_var_UV: vec2 location: 0
Uniforms
color: texture_2d
Buffer Size: 0
Bind Group: 1
Bind Index: 1
Members:

I wonder if there should be the sampler info or not?

Add support for const

Example:

          @vertex fn vs(
            @builtin(vertex_index) VertexIndex : u32
          ) -> @builtin(position) vec4<f32> {
            var pos = array<vec2<f32>, 3>(
              vec2(0.0, 0.5),
              vec2(-0.5, -0.5),
              vec2(0.5, -0.5)
            );

            return vec4(pos[VertexIndex], 0.0, 1.0);
          }

          const FOO = radians(90);
          const BAR = sin(FOO);
          const NUM_COLORS = u32(BAR + 3); // result 4
          @group(0) @binding(0) var<uniform> uni: array<vec4f, NUM_COLORS>;

          @fragment fn fs() -> @location(0) vec4<f32> {
            return uni[NUM_COLORS - 1];
          }

works: https://jsgist.org/?src=bcebe5c22fc9ed9ecc18f46fdf986ffc

I get that this is non-trivial as you actually have to do the math and that includes all of the math functions :(

Documentation bug?

console.log(groups[0][1].size); // 108, the size of the uniform buffer.

console.log(groups[0][1].size); // 108, the size of the uniform buffer.

The code above doesn't seem correct. Based on your readme, I wrote typescript based unit tests to verify the code and I'm getting 96, not 108. Seems like a mat4x4+vec4+f32 = 64+16+4+12(extra padding) = 96.

wgsl_reflect_unit_tests.txt

Bug: storage buffer resource missing from entry when writing only (no reading)

Hello,

Amazing work on this library!

I just discovered a little bug when using compute shaders with it:

When defining a read_write storage buffer and only writing to it in my compute shader, the buffer will not appear in the entry's resources of the program. However if I add read operation then the buffer does appear.

In the code below, only particlesA appears in the resources array of the compute program.

struct Particle {
  pos : vec2<f32>,
  vel : vec2<f32>,
}

struct Particles {
  particles : array<Particle>,
}

@binding(0) @group(0) var<storage, read> particlesA: Particles;
@binding(1) @group(0) var<storage, read_write> particlesB : Particles;

@compute @workgroup_size(64)
fn main(@builtin(global_invocation_id) GlobalInvocationID : vec3<u32>) {
  var index = GlobalInvocationID.x;
  var vPos = particlesA.particles[index].pos;
  particlesB.particles[index].pos = vPos;
}

Adding a blank read operation (var _ = particlesB.particles[index];) will add the particlesB in the resources array:

struct Particle {
  pos : vec2<f32>,
  vel : vec2<f32>,
}

struct Particles {
  particles : array<Particle>,
}

@bind var<storage, read> particlesA: Particles;
@bind var<storage, read_write> particlesB : Particles;

@compute @workgroup_size(64)
fn main(@builtin(global_invocation_id) GlobalInvocationID : vec3<u32>) {
  var index = GlobalInvocationID.x;
  var vPos = particlesA.particles[index].pos;

  var _ = particlesB.particles[index]; // <-- to enable particlesB in the resources of the shader

  particlesB.particles[index].pos = vPos;
}  

Support for var usage by entry

AFAICT, right now, entry is EntryFunctions as defined below.

class FunctionInfo {
    constructor(name, stage = null) {
        this.stage = null;
        this.inputs = [];
        this.outputs = [];
        this.name = name;
        this.stage = stage;
    }
}
class EntryFunctions {
    constructor() {
        this.vertex = [];
        this.fragment = [];
        this.compute = [];
    }
}

It would be useful if each entry point listed which bindings it uses

For example

@group(0) @binding(0) var<uniform> u1: vec4f;
@group(0) @binding(1) var<uniform> u2: vec4f;
@group(0) @binding(1) var<uniform> u3: vec4f;

@vertex fn vs1() -> @builtin(position) vec4f {
  return u1;
}

@vertex fn vs2() -> @builtin(position) vec4f {
  return u2;
}

@vertex fn vs3() -> @builtin(position) vec4f {
  return u1 + u2;
}

@vertex fn vs4() -> @builtin(position) vec4f {
  _ = u1;
  _ = u2;
  return vec4f(0);
}

@vertex fn vs5() -> @builtin(position) vec4f {
  return u1 + u3;
}

It would be nice if some where it told me

  • vs1 uses u1
  • vs2 uses u2
  • vs3 uses u1 and u2
  • vs4 uses u1 and u2
  • vs5 uses u1 and u3

etc... where "uses" is the storage, uniform, sampler, texture info. Then I could generate GPUBindGroupLayoutDescription from the info.

As it is now I don't see that info. I can see this is not entirely trivial since the usage might not appear until multiple function calls and aliases etc make it more complicated. Right now I can guess that all uniforms, samplers, textures, storage are used but that wouldn't work for more complicated cases

Get unused members

Hello,

I was getting ready to write something similar and came across your repo. I took a quick look and it doesn't seem to have a few features I need. Sorry if I missed them.

  • List of structs that are not referenced
  • Starting and ending line number of each member (structs and functions)
  • List of custom functions referenced by a function

My use case:

I build up shaders by including snippets. This may cause the shader code to be unnecessarily large because it includes code that is not being used. I would like to remove all the unused elements after the includes have been processed.

Example:

#include common_structs
#include vec2_utils
...

@vertex
fn vs...

After expansion of the includes, I would run reflect and identify any structs and functions that are unused by the vertex shader. I would then remove them so the final shader code is smaller.

Thanks,
Wil

Add strict mode

Currently invalid syntax will silently fail. I'd like an error returned or an exception thrown instead of the results up until the invalid syntax.

Example:

import wgsl_reflect from "@feng3d/wgsl_reflect";
const { WgslReflect } = wgsl_reflect;

const reflect = new WgslReflect(`
const valid_not_skipped: u32 = 0u;
some invalid syntax;
const valid_but_skipped: u32 = 0u;
`);
console.log(reflect);
/*
_WgslReflect2 {
  structs: [],
  overrides: [],
  uniforms: [],
  storages: [],
  textures: [],
  samplers: [],
  functions: [],
  aliases: [],
  ast: [
    Const {
      name: 'valid_not_skipped',
      type: [Type],
      storage: '',
      access: '',
      value: [LiteralExpr],
      attributes: null
    }
  ],
  entry: EntryFunctions { vertex: [], fragment: [], compute: [] }
}
*/

Failed to handle value constructor in module scope

tint will generate some code like this:

const v2 = vec2f(0.0f, 0.0f);
const v3 = vec3f(0.0f, 0.0f, 0.0f);
const v4 = vec4f(0.0f, 0.0f, 0.0f, 0.0f);

const M3 = mat3x3f(v3, v3, v3);
const M4 = mat4x4f(v4, v4, v4, v4);

fn fv2(outVar : ptr<function, vec2f>) {
  *(outputValue) = v2;
  return;
}

fn fv3(outVar : ptr<function, vec3f>) {
  *(outputValue) = v3;
  return;
}

fn fv4(outVar : ptr<function, vec4f>) {
  *(outVar ) = v4;
  return;
}

fn fM3(outVar : ptr<function, mat3x3<f32>>) {
  *(outVar) = M3;
  return;
}

fn fM4(outVar : ptr<function, mat4x4<f32>>) {
  *(outVar) = M4;
  return;
}

Error: Non const function: vec3f
Error: Non const function: vec2f
Error: Non const function: vec4f
Error: Non const function: mat3x3f
Error: Non const function: mat4x4f

missing members?

If I take the shader in this test

const shader = `alias material_index = u32;

Which is this

alias material_index = u32;
alias color = vec3f;
struct material {
    index: material_type,
    diffuse: color,
}
@group(0) @binding(1) var<storage> materials: array<material>;

Then I make a WgslReflect

 const reflect = new WgslReflect(code);

and I look at the result it looks good.

Screenshot 2023-04-17 at 16 29 14

But then I call

 const info = reflect.getStructInfo(relect.structs[0);

and that's missing a member

Screenshot 2023-04-17 at 16 30 31

No reflection data for raw arrays

If I have a shader that has

@binding(0) @group(0) var<uniform> viewUniforms: array<vec4<f32>, 7>;

It appears the no reflection data is returned. Sorry, I didn't try to fix it myself yet.

Token error in minified shaders

Hi! First of all, this is a great tool, it makes extracting vertex attributes and shader uniforms so much easier. Thank you for that!

I've run into an issue I'm not sure if possible to solve, so I'd like your opinion on this one.
I have a tool to minify GLSL and WGSL shaders, but when used in combination with wgsl_reflect, it may cause one of the errors shown in the screenshots. The cause seems to be the same: -1.

For example, this is a valid, minified shader produced by vite-plugin-glsl:

@group(0)@binding(0)var<uniform>resolution: vec2f;fn GetClipSpace(position: vec2f)->vec2f{let clipSpace=position/resolution*2-1;return clipSpace*vec2f(1,-1);}

but the error is caused for 2-1 being attached together. The only thing required to make it work was to separate them with a space:

@group(0)@binding(0)var<uniform>resolution: vec2f;fn GetClipSpace(position: vec2f)->vec2f{let clipSpace=position/resolution*2 - 1;return clipSpace*vec2f(1,-1);}

I have several other shaders and the fix was always the same. Strangely enough, other operations (*,/ and +) cause no errors when two operands are attached together. Is it possible to do something with this or is it some limitation of the parsing algorithm? As a workaround for me, it's possible to disable minification all together, but of course, I'd prefere not to. ๐Ÿ˜„

Thanks!

Screenshot 2024-08-28 232101

Screenshot 2024-08-28 231328

Error with new WGSL "enable" and "requires" directives

Hello again,

WGSL introduced some new directives, however when adding those to my wgsl code the library will no longer detect any module entry.
Adding support for requires would be really awesome to use some of the latest language features.

Unlock language features with requires:
requires readonly_and_readwrite_storage_textures;
https://developer.chrome.com/blog/new-in-webgpu-124

if (!navigator.gpu.wgslLanguageFeatures.has("readonly_and_readwrite_storage_textures")) {
  throw new Error("Read-only and read-write storage textures are not available");
}

const adapter = await navigator.gpu.requestAdapter();
const device = await adapter.requestDevice();

const bindGroupLayout = device.createBindGroupLayout({
  entries: [{
    binding: 0,
    visibility: GPUShaderStage.COMPUTE,
    storageTexture: {
      access: "read-write", // <-- New!
      format: "r32uint",
    },
  }],
});

const shaderModule = device.createShaderModule({ code: `
  requires readonly_and_readwrite_storage_textures;

  @group(0) @binding(0) var tex : texture_storage_2d<r32uint, read_write>;

  @compute @workgroup_size(1, 1)
  fn main(@builtin(local_invocation_id) local_id: vec3u) {
    var data = textureLoad(tex, vec2i(local_id.xy));
    data.x *= 2;
    textureStore(tex, vec2i(local_id.xy), data);
  }`
});

Experimental flag with enable:
enable chromium_experimental_subgroups;
https://developer.chrome.com/blog/new-in-webgpu-125#subgroups_feature_in_development

const adapter = await navigator.gpu.requestAdapter();
if (!adapter.features.has("chromium-experimental-subgroups")) {
  throw new Error("Experimental subgroups support is not available");
}
// Explicitly request experimental subgroups support.
const device = await adapter.requestDevice({
  requiredFeatures: ["chromium-experimental-subgroups"],
});

const shaderModule = device.createShaderModule({ code: `
  enable chromium_experimental_subgroups;

  @compute @workgroup_size(64) fn main(
      @builtin(global_invocation_id) global_id : vec3u,
      @builtin(subgroup_size) sg_size : u32,
      @builtin(subgroup_invocation_id) sg_id : u32) {
    // TODO: Use subgroupBallot() and subgroupBroadcast().
  }`,
});

Can't get storage buffer's access mode

For example:

WGSL:

struct ReadonlyStorageBufferBlockName {
  /* @offset(0) */
  a : f32,
}

struct ReadWriteStorageBufferBlockName {
  /* @offset(0) */
  b : f32,
}

@group(3) @binding(1) var<storage, read> readonlyStorageBuffer : ReadonlyStorageBufferBlockName;         // <= read-only
@group(3) @binding(2) var<storage, read_write> readWriteStorageBuffer : ReadWriteStorageBufferBlockName; // <= read-write
@compute @workgroup_size(1,1,1)
fn main() {}

JS:

const reflect = new WgslReflect(wgsl);
const groups = reflect.getBindGroups()

image

this infomation is useful for:

enum GPUBufferBindingType {
    "uniform",
    "storage",
    "read-only-storage",
};

Consider providing types

I'm noticing that there is no top level .d.ts file, would you consider providing one? This shouldn't be too hard given that the project is seemingly written in typescript and would help integrations into other projects which also are written in typescript.

Failed to handle `continuing` statements

for example:

var a: i32 = 2;
var i: i32 = 0;
loop {
  if i >= 4 { break; }

  let step: i32 = 1;

  if i % 2 == 0 { continue; }

  a = a * 2;

  continuing {   // <2>
    i = i + step;
  }
}

Support for unsized arrays of arrays

This fails to parse

@group(0) @binding(0) var<storage> u: array<array<vec4f, 20>>;

This works (added a space between the two > at the end)

@group(0) @binding(0) var<storage> u: array<array<vec4f, 20> >;

I'm happy to add the space in my own code, I just thought I'd pass on the issue

ะก/C++ version

Are there any plans to create a C/C++ version of the library?
That way we can parse WGSL for native WebGPU(Dawn/WGPU), as well as for Wasm

Can't process "]]"

Code:

struct Uniforms {
    positions: array<vec3f, 2>,
    ids: array<i32, 2>
}

@binding(0) @group(0) var<uniform> uniforms: Uniforms ;

@compute @workgroup_size(1i, 1i, 1i)
fn main() {
    uniforms.positions[uniforms.ids[0]];
}

Result:
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.