virtual-cast/babylon-vrm-loader

glTF VRM extension Loader for babylon.js

babylon-vrm-loader

VRM porting to babylon.js.

This loader is used as VirtualCast web VRM/VCI/glb viewer.

Supported version table

babylon.js version	babylon-vrm-loader version
~4.1.0	<1.5.0
~4.2.0	^1.5.0
^5.19.0	^2.0.0
^6.0.0	will be ^3.0.0

Features

• Supports .vrm v0.x file loading
  • with extensions.VRM glTF Extension
  • TODO VRM v1.0 file loading
• Supports .vci file loading
• Supports MToonMaterial
• Get bone(TransformNode) from Unity Humanoid bone mapping name
• BlendShape morphing
• SpringBone
• Supports VCI features(partial support)
  • VCAST_vci_material_unity
  • TODO: VCAST_vci_meta
  • TODO: VCAST_vci_embedded_script
  • TODO: VCAST_vci_audios
  • TODO: VCAST_vci_colliders
  • TODO: VCAST_vci_rigidbody
  • TODO: VCAST_vci_joints
  • TODO: VCAST_vci_item

Usage

on browser

example is here.

on Babylon.js Playgound

example is here.

with npm/yarn

$ npm install --save @babylonjs/core @babylonjs/loaders babylon-vrm-loader
# or
$ yarn add @babylonjs/core @babylonjs/loaders babylon-vrm-loader

import * as BABYLON from '@babylonjs/core'

// has side-effect
// ref. https://webpack.js.org/guides/tree-shaking#mark-the-file-as-side-effect-free
import 'babylon-vrm-loader'

// vrmFile is File object retrieved by <input type="file">.
const scene = await BABYLON.SceneLoader.LoadAsync('file:', vrmFile, engine);
const vrmManager = scene.metadata.vrmManagers[0];

// Update secondary animation
scene.onBeforeRenderObservable.add(() => {
    vrmManager.update(scene.getEngine().getDeltaTime());
});

// Model Transformation
vrmManager.rootMesh.translate(new BABYLON.Vector3(1, 0, 0), 1);

// Work with HumanoidBone
vrmManager.humanoidBone.leftUpperArm.addRotation(0, 1, 0);

// Work with BlendShape(MorphTarget)
vrmManager.morphing('Joy', 1.0);

Contributing

See CONTRIBUTING.md.

Build

$ yarn build

Debugging MToonMaterial

$ yarn debug

You can see inspector on http://localhost:8080/

Related Links

• BabylonJS/Babylon.js: Babylon.js: a complete JavaScript framework for building 3D games with HTML 5 and WebGL
• vrm-c/UniVRM: Unity package that can import and export VRM format
• virtual-cast/babylon-mtoon-material: Unity MToon Shader WebGL porting to babylon.js.

Licenses

see LICENSE.

This project uses babylon.js with Apache License, Version 2.0.

babylon-vrm-loader's Issues

colliderGroups が定義されていない VRM がある

undefined is not an object (evaluating 'this.ext.colliderGroups.forEach')

Why the morphTarget is an array ?

I want to replace three-vrm with babylon-vrm-loader, but I have confusion about the morph target:
In three-vrm, change blendeshape using as following (demo):

// set value
currentVrm.blendShapeProxy.setValue( THREE.VRMSchema.BlendShapePresetName.A, 0.5 + 0.5 * s );
currentVrm.blendShapeProxy.setValue( THREE.VRMSchema.BlendShapePresetName.BlinkL, 0.5 - 0.5 * s );
// get value
currentVrm.blendShapeProxy.getValue( THREE.VRMSchema.BlendShapePresetName.A);
currentVrm.blendShapeProxy.getValue( THREE.VRMSchema.BlendShapePresetName.BlinkL);

In babylon-vrm-loader, I think using morphing method can set value.

currentVrm.morphing('A', 1)
currentVrm.morphing('Blink_L', 1)

However, I don't know how to get the morphTarget value because the target is an array. And the array has the same object as this:

three-vrm get value method source code

Merge babylon-mtoon-material repository into here

monorepo

Whether there are plans to support physical engines

Such as title.

Prefer to use with SceneLoader.LoadAssetContainer

現在 glTFLoader の読み込み完了後に scene.metadata.vrmManagers にマネージャを代入しているが、使い勝手が悪い。

SceneLoader.LoadAssetContainer を利用してアセットコンテナオブジェクトをコンストラクタに渡し、ユーザランドで VRMManager の管理を出来るようにした方が良い。

interface VRMContainer {
  public constructor(public readonly assetContainer: AssetContainer): void;
  public get humanoidBone(): HumanoidBone;
  public get blendShape(): BlendShape;
  public get firstPerson(): FirstPerson;
  public get meta(): Meta;
  public get inScene(): boolean;
  public addToScene(): void;
  public removeFromScene(): void;
  public update(deltaTime: number): Promise<void>;
  public dispose();
}

ref #16

https://doc.babylonjs.com/how_to/load_from_any_file_type#sceneloaderloadassetcontainer

AssetsManager を利用しても問題なく使えるか確認が必要 https://doc.babylonjs.com/how_to/how_to_use_assetsmanager

現状 gltf json を読み込む場所がないので、 vrm_extension で scene.metadata[loadName].vrm などにエクスポートする必要がありそう

Use this with React.js

Can this be used with React.js ?

VRM without vertex Normal attribute will unexpected behaviour.

https://twitter.com/iCyP/status/1163691288849338368 のVroidHubよりDLできるモデルについての問題です。
法線属性をgltfのjsonから削除したVRMを読み込ませると、TSOでは新ﾋﾞｭｰﾜモデルの大部分が表示されず、サンプルページ（ https://codepen.io/akai_inu/pen/zQXyxL?editors=1010 ）ではロードが終了しません。
期待される動作は、uniVRM0.53およびTSO旧ビューワにおける表示になります。

Use Bone instead of TransformNode

TransformNode.getAbsolutePosition() が異常な値を返す?ようなので、 Humanoid Bone を取得・操作したい場合はちゃんと Bone オブジェクトを利用するようにする

• ボーンを操作するには、操作対象の Mesh が必要である
• glTF からボーンを読み込むと、 Mesh ごとに Skeleton が生成される
• Mesh についてる Skeleton から Hips 等に該当するボーンがある奴を取り出す
• 今使っている TransformNode が紐づいている Bone._linkedTransformNode を探索して nodeIndex と Bone を紐づける
• Hips ボーンと Hips ボーンを持っている N 個のメッシュを紐づけて保持
• ボーンを動かすときは TransformNode ではなく N 個のメッシュに対して Bone をそれぞれ適用する

v3 Roadmap

• #103
• #101
• #75
• #28
• #25
• #23

NOTICE: Supports VRM 1.0 #64 is not part of v3, it will be processed another time.

Supports Babylon.js 6.x

• https://github.com/BabylonJS/Babylon.js/releases/tag/6.0.0

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 12.0.12 to 12.6.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ ci/circleci: build: Your tests failed on CircleCI (Details).
• ❌ build_and_test: * build - Failed

• lint - Blocked

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Add WebXR sample with First Person

ref. 一人称表示 - VRM

Error: Self is not defined

I'm loading this module just as shown in the example:

import * as BABYLON from "@babylonjs/core";
import 'babylon-vrm-loader'

in NextJS and Typescript.

The app crashes instantly:

Error [ReferenceError]: self is not defined
    at Object.<anonymous> (C:\Users\Benjy\Documents\Github\project\s\node_modules\babylon-vrm-loader\dist\index.module.js:1:4903)

Not sure what's going on but would love a fix...

[v2] vrmManagers to class

BREAKING CHANGE

scene.metadata.vrmManagers: VRMManager[] to scene.metadata.vrmManagers: VRMManagers

interface VRMManagers {
  getByName(name: string): Nullable<VRMManager>;
  getByIndex(index: number): Nullable<VRMManager>;
  updateAll(): Promise<void>;
}

Instead, SceneLoader.LoadAssetContainer based loading is preferred?

https://doc.babylonjs.com/how_to/load_from_any_file_type#sceneloaderloadassetcontainer

define VRMFinder in scene

/**
 * @link https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/core/src/Rendering/outlineRenderer.ts
 */

declare module "@babylonjs/core/scene" {
    export interface Scene {
        _vrmFinder: VRMFinder;
        getVRMFinder(): VRMFinder;
    }
}

Scene.prototype.getVRMFinder = function (): VRMFinder {
    if (!this._vrmFinder) {
        this._vrmFinder = new VRMFinder(this);
    }
    return this._vrmFinder;
};

export class VRMFinder implements ISceneComponent {
    public name = 'VRMFinder';

    public constructor(public readonly scene: Scene) {
        this.scene._addComponent(this);
    }

    public getByTitle(title: string): Nullable<VRMManager> {
        // ...
    }

    public getAll(): VRMManager[] {
        // ...
    }

    // ...
}

恐らくこんな感じのインターフェースを想定。

// use case
const vrm = this.scene.getByTitle('Alicia Solid') ?? throw new Error(`VRM not found`);
vrm.morphing('a', 1.0);

Support babylon-vrm-loader babylon.js 5.x

https://github.com/BabylonJS/Babylon.js/blob/master/dist/preview%20release/what's%20new.md#500

Delete `VRMManager.findMesh`

Use findMeshes instead.

See #24

Delete `VRMManager.getBone`

Use VRMManager.humanoidBone instead.

see #22

BlendShape(with MorphTarget) is broken

Update: It will be fixed at babylon.js v4.1.0!

Summary

some vrm BlendShape(uses MorphTarget) do not morph correctly.

Specs

https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#morph-targets

A Morph Target is a morphable Mesh where primitives' attributes are obtained by adding the original attributes to a weighted sum of targets attributes.

primitives[i].attributes.POSITION + 
  weights[0] * primitives[i].targets[0].POSITION +
  weights[1] * primitives[i].targets[1].POSITION +
  weights[2] * primitives[i].targets[2].POSITION + ...

primitives[i].targets[0].POSITION has relative position from original primitive position.

Implementations

OnLoading

https://github.com/BabylonJS/Babylon.js/blob/master/loaders/src/glTF/2.0/glTFLoader.ts#L898

        loadAttribute("POSITION", VertexBuffer.PositionKind, (babylonVertexBuffer, data) => {
            babylonVertexBuffer.forEach(data.length, (value, index) => {
                data[index] += value;
            });

            babylonMorphTarget.setPositions(data);
        });

babylonMorphTarget: MorphTarget has sum of original primitive position and morph relative position.

On GLSL

https://github.com/BabylonJS/Babylon.js/blob/master/src/Shaders/ShadersInclude/morphTargetsVertex.fx

// positionUpdated is original primitive position plus previous morphed position
// position is original primitive position
// position{X} is MorphTarget position
// morphTargetInfluences[{X}] is weight
	positionUpdated += (position{X} - position) * morphTargetInfluences[{X}];

Tests

glTF Sample

• https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0/MorphPrimitivesTest ok
• https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0/AnimatedMorphSphere ok

AliciaSolid from https://3d.nicovideo.jp/works/td32797

NG: All face.baked Mesh's MorphTarget

OK: other.baked Mesh's MorphTarget

VRoid Studio from https://studio.vroid.com/

NG

ref. three.js

https://rdrgn.github.io/three-vrm/

All of above models are OK.

Implementations

OnLoading

https://github.com/mrdoob/three.js/blob/dev/examples/js/loaders/GLTFLoader.js#L1403

					// Three.js morph position is absolute value. The formula is
					//   basePosition
					//     + weight0 * ( morphPosition0 - basePosition )
					//     + weight1 * ( morphPosition1 - basePosition )
					//     ...
					// while the glTF one is relative
					//   basePosition
					//     + weight0 * glTFmorphPosition0
					//     + weight1 * glTFmorphPosition1
					//     ...
					// then we need to convert from relative to absolute here.

					if ( target.POSITION !== undefined ) {

						var positionAttribute = morphPositions[ i ];
						positionAttribute.name = attributeName;

						var position = geometry.attributes.position;

						for ( var j = 0, jl = positionAttribute.count; j < jl; j ++ ) {

							positionAttribute.setXYZ(
								j,
								positionAttribute.getX( j ) + position.getX( j ),
								positionAttribute.getY( j ) + position.getY( j ),
								positionAttribute.getZ( j ) + position.getZ( j )
							);

						}

					}

Implementations: On GLSL

https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/ShaderChunk/morphtarget_vertex.glsl.js

	transformed += ( morphTarget0 - position ) * morphTargetInfluences[ 0 ];

Where is implementation difference between babylon.js and three.js?

refs

• https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl/WebGLMorphtargets.js
• https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl/WebGLGeometries.js#L99
• https://github.com/BabylonJS/Babylon.js/blob/master/src/Meshes/mesh.ts#L2848
• https://github.com/BabylonJS/Babylon.js/blob/master/src/Materials/materialHelper.ts#L681

Too many physics bones caused browser crash.

https://hub.vroid.com/characters/8167838262685725742/models/910412214503422810
をExampleのサイトでロードするとブラウザがクラッシュしました。
心当たりとしてはspring boneに属するボーンが１６００本弱あります。

環境
win10
Chrome
SurfacePro 4 i5 RAM8GB

An in-range update of webpack is breaking the build 🚨

The devDependency webpack was updated from 4.35.2 to 4.35.3.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

webpack is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ ci/circleci: build: Your tests failed on CircleCI (Details).
• ❌ build_and_test: * build - Failed

• lint - Blocked

---

Release Notes for v4.35.3

Bugfixes

• update acorn to include import()

Performance

• Improved runtime performance for large JSON modules

Commits

The new version differs by 20 commits.

• 4ec90ce 4.35.3
• 91a211e Merge pull request #9378 from webpack/dependabot/npm_and_yarn/ajv-6.10.1
• 0db2c54 Merge pull request #9377 from webpack/dependabot/npm_and_yarn/ajv-keywords-3.4.1
• 30ddec0 chore(deps): bump ajv from 6.10.0 to 6.10.1
• 1b54416 chore(deps): bump ajv-keywords from 3.4.0 to 3.4.1
• f092150 Merge pull request #9370 from DRoet/dynamic-import
• b56c3ec feat: update acorn for dynamic import
• 0976bd3 Merge pull request #9363 from webpack/dependabot/npm_and_yarn/eslint-plugin-jest-22.7.2
• 45acc86 Merge pull request #9361 from webpack/dependabot/npm_and_yarn/acorn-6.2.0
• dfeb9a9 Merge pull request #9359 from webpack/dependabot/npm_and_yarn/types/node-10.14.12
• db85be0 chore(deps-dev): bump eslint-plugin-jest from 22.7.1 to 22.7.2
• 219f773 chore(deps): bump acorn from 6.1.1 to 6.2.0
• 72e0540 chore(deps-dev): bump @types/node from 10.14.10 to 10.14.12
• 0af4549 Merge pull request #9356 from webpack/dependabot/npm_and_yarn/simple-git-1.118.0
• 1ce0c21 chore(deps-dev): bump simple-git from 1.117.0 to 1.118.0

There are 20 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Add GitHub Pages Runtime

• build test page
• publish with gh-pages package

Supports VRM 1.0

https://github.com/vrm-c/vrm-specification/tree/master/specification

• VRMC_materials_hdr_emissiveMultiplier-1.0
  • emissiveMultiplier
• VRMC_materials_mtoon-1.0
• VRMC_node_constraint-1.0
  • specVersion
  • constraint
• VRMC_springBone-1.0
  • specVersion
  • colliders
  • colliderGroups
  • springs
• VRMC_vrm-1.0
  • specVersion
  • humanoid
  • meta
  • firstPerson
  • expression
  • lookAt

Implements Secondary Animation

• ref. VRMSpringBone - VRM
• ref. three-vrm/VRMPhysics.ts at master · rdrgn/three-vrm
• ref. Use a Physics Engine - Babylon.js Documentation

boneIKController boneLookController をまねて SpringBoneController を作るのが良さそう？

[Bug Report] partial transparent clothes can not display normally

I download this official model from VRoid.

It should be:

I load it in babylon.

It be:

It seems have a problem about alpha channel of clothes.

MToon: Texture transform seems not to be handled properly

UniVRMに付属するMToonでは、すべてのテクスチャのSTに対して
materialProperties[i].vectorProperties._MainTex を適用する実装となっていますが、
一部テクスチャに対して適切にSTが実装されていないようです。

以下はテスト用のモデルです。
_MainTex ・ _ShadeTexture ・ _EmissionMap に対して同様の形状のテクスチャが入っていますが、
_MainTex 以外の2つのテクスチャについて、STの当たり方がデフォルトのままになっている気がします。

color-space-test.zip

こちらのSandboxで確認しました: https://virtual-cast.github.io/babylon-vrm-loader/

環境はWindows 10・Chrome 74です。

VRMAnimation file support?

Are there any plans to support VRMA files?

packages from npmjs.com don't have some `.d.ts` files

I've noticed that the package from npmjs.com is missing some *.d.ts files. This seems to cause importing problems when we import babylon-vrm-loader from a TypeScript project (this shouldn't affect JavaScript projects). Could you have a look at this?

I've investigated that in this way:

$ mkdir test
$ cd test
$ yarn add babylon-vrm-loader
yarn add v1.19.2
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "babylon-vrm-loader > [email protected]" has unmet peer dependency "@babylonjs/core@^4.0.0".
warning " > [email protected]" has unmet peer dependency "@babylonjs/core@^4.0.0".
warning " > [email protected]" has unmet peer dependency "@babylonjs/loaders@^4.0.0".
[4/4] Building fresh packages...

success Saved lockfile.
success Saved 2 new dependencies.
info Direct dependencies
└─ [email protected]
info All dependencies
├─ [email protected]
└─ [email protected]
Done in 2.27s.

And look into the directory:

$ tree node_modules/babylon-vrm-loader
node_modules/babylon-vrm-loader
├── dist
│   ├── index.d.ts
│   ├── index.js
│   ├── index.module.js
│   ├── vcast-vci-material-unity.d.ts
│   ├── vrm-extension.d.ts
│   ├── vrm-file-loader.d.ts
│   ├── vrm-interfaces.d.ts
│   ├── vrm-manager.d.ts
│   └── vrm-material-generator.d.ts
├── LICENSE
├── package.json
└── README.md

1 directory, 12 files

Here, for example, we don't have errors.d.ts in dist directory.

On the other hand, if I clone the source from GitHub and build it,

$ git clone https://github.com/virtual-cast/babylon-vrm-loader
$ cd babylon-vrm-loader
$ yarn
$ yarn build

then dist directory contains more files (including errors.d.ts):

$ tree dist
dist
├── errors.d.ts
├── humanoid-bone.d.ts
├── index.d.ts
├── index.js
├── index.module.js
├── secondary-animation
│   ├── collider.d.ts
│   ├── collider-group.d.ts
│   ├── quaternion-helper.d.ts
│   ├── sphere-collider.d.ts
│   ├── spring-bone-controller.d.ts
│   ├── vector3-helper.d.ts
│   ├── vrm-spring-bone.d.ts
│   └── vrm-spring-bone-logic.d.ts
├── test
│   └── index.d.ts
├── vcast-vci-material-unity.d.ts
├── vci-interfaces.d.ts
├── vrm-extension.d.ts
├── vrm-file-loader.d.ts
├── vrm-interfaces.d.ts
├── vrm-manager.d.ts
└── vrm-material-generator.d.ts

I'm not very sure but I guess this is a kind of packaging (or releasing) problem. I'm sorry not to have a patch; my understanding of the release procedure is not enough to do that.

Could we fix the problem specifying the missing files at files section in package.json?

Apply MToon Rim parametric Lighting

relates virtual-cast/babylon-mtoon-material#3

[BUILD] Generation for browser

https://doc.babylonjs.com/#getting-started

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html" charset="utf-8"/>
    <title>Babylon - Getting Started</title>
    <!-- Link to the last version of BabylonJS -->
    <script src="https://preview.babylonjs.com/babylon.js"></script>
    <!-- Link to the last version of BabylonJS loaders to enable loading filetypes such as .gltf -->
    <script src="https://preview.babylonjs.com/loaders/babylonjs.loaders.min.js"></script>
    <!-- Link to pep.js to ensure pointer events work consistently in all browsers -->
    <script src="https://code.jquery.com/pep/0.4.1/pep.js"></script>
    <script src="https://unpkg.com/[email protected]/dist/index.js"></script>
</head>
<body>
    <canvas id="renderCanvas"></canvas>
</body>
</html>

• Getting Started のコードに追記するだけでコンパイルなしで動くファイルを出力する(今のは UMD ビルドなので上記コードが動かない)
  • mjs ビルドを生成する
  • js(not minimized) ビルドを生成する
  • min.js ビルドを生成する
• 最小コードのサンプルを公開する
• /dist 以下のビルドを自動 push 出来るようにする
• GitHub Releases に mjs, js, min.js の三種類を含めるようにする

Invalid SpringBone Rotation when ancestor has rotated

Z -1 を向いている時は正しい SpringBone 挙動を示すが、 rootMesh や hips を回転させてから移動させると、移動先とは異なる異常な方向に揺れる。

親の rotationQuaternion が子に伝播していないことが原因か。

Unity の Transform.rotation はワールド座標に対する Quaternion を取得する。

Add targetNames morphMapping

VRM BlendShape には記述されていないが、 glTF としては設定されている MorphTarget が存在する場合がある。それらも MorphTargetManager としてインスタンス化されているので、操作出来るようにしたい。

mesh.metadata.vrmTargetNames に追記されている。

The vrm model fails to be loaded

link: https://a-cdn.qbox.net/test/models/vrm/Kevin.vrm