googlevr/tilt-brush-toolkit

Scripts and assets that help you use Tilt Brush data in your creative projects.

Tilt Brush Toolkit

The Tilt Brush Toolkit is a collection of scripts and assets that allow you to use Tilt Brush data in your creative projects. If you simply want to import Tilt Brush geometry into Unity, this package is mostly superseded by Poly Toolkit.

Contents

Unity SDK

Scripts, shaders and tools for importing and manipulating Tilt Brush .fbx exports in Unity. Unless you specifically need to use .fbx -- for instance, because Maya is part of your workflow -- try Poly Toolkit first.

• Easily import sketches into Unity
• Brush shaders and materials
• Audio reactive features
• Examples and reusable scripts to create animations and add interactivity

Click here to download the latest version of the Tilt Brush Unity SDK

Check out the Documentation to get started !

Command Line Tools

Python 2.7 code and scripts for advanced Tilt Brush data manipulation.

• bin - command-line tools
  • dump_tilt.py - Sample code that uses the tiltbrush.tilt module to view raw Tilt Brush data.
  • geometry_json_to_fbx.py - Sample code that shows how to postprocess the raw per-stroke geometry in various ways that might be needed for more-sophisticated workflows involving DCC tools and raytracers. This variant packages the result as a .fbx file.
  • geometry_json_to_obj.py - Sample code that shows how to postprocess the raw per-stroke geometry in various ways that might be needed for more-sophisticated workflows involving DCC tools and raytracers. This variant packages the result as a .obj file.
  • tilt_to_strokes_dae.py - Converts .tilt files to a Collada .dae containing spline data.
  • unpack_tilt.py - Converts .tilt files from packed format (zip) to unpacked format (directory) and vice versa, optionally applying compression.
• Python - Put this in your PYTHONPATH
  • tiltbrush - Python package for manipulating Tilt Brush data.
    • export.py - Parse the legacy .json export format. This format contains the raw per-stroke geometry in a form intended to be easy to postprocess.
    • tilt.py - Read and write .tilt files. This format contains no geometry, but does contain timestamps, pressure, controller position and orientation, metadata, and so on -- everything Tilt Brush needs to regenerate the geometry.
    • unpack.py - Convert .tilt files from packed format to unpacked format and vice versa.

tilt-brush-toolkit's Issues

TiltBrush SDK Unity Shader Problems

I encounter problems with the following shaders from the TiltBrush SDK v23 (latest version to date):

Bubbles Failed to open source file: 'Assets/ThirdParty/Noise/Shaders/Noise.cginc' at line 26 (on d3d11)
Diamond Hull: failed to open source files 'HLSLsupport.cginc' 'UnityShadersVariables.cginc' 'Lightning.cginc' and many more...
Smoke: Failed to open source file: 'Assets/ThirdParty/Noise/Shaders/Noise.cginc'
Toon: Failed to open source file: 'Assets/ThirdParty/Noise/Shaders/Noise.cginc'
WigglyGraphite: same as Diamond Hall

Anyone encountering the same? Anyone got a solution? It would be awesome to get this going. Also I am curious if there is a way to import an export from tiltbrush into unity so I will have the obj file with it. My situation is this: I export an obj from Medium and can import it into TiltBrush just fine. After doing the fancy design thing I want to export the whole sketch/project so I can import it in Unity. Until know I only could import the obj itself (without TiltBrush design) and I managed to import the TiltBrush sketch, but having problems with the shaders. They are not assigned automatically and the above mentioned are not showing up in the list so I could assign them manually.

In the documentation about importing Sketches from Tiltbrush in Unity it says I only need to import the fbx from TB into unity, unfortunately it doesn't work for me. I am using Unity 2019.4.f121

Update:
I also got problems with all the other shaders. Somehow Unity will not apply them to the TB sketch although I import it through Poly. I can assign another shader, I am using Universal Rendering Pipleline, but of course it will not create the same effect. I would be very happy to get a good tutorial on how to import TB sketches correctly into Unity 2019.4.12f1 to fully make us of these two great and powerful softwares.

Any help is much appreciated, as I am diving deep into this wonderfully creative workflow. It's been 10 years since I worked as Dev and am getting back to it these Lockdown days. I am amazed how much changed and with VR the DevGame evolved to a completely new dimension. Super excited to witness this :D

Thanks for helping out
Thomas

Mic/line-in input for audio reactive brushes

Hi there!
I'd like to use my computer line in input as an audio source for the TiltBrush Audio Reactive prefab. I'm aware it uses Keijiro Reaktion which uses mic input (so it should be possible) but just don't know how to modify the Tiltbrush script, my scripting knowledge is quite limited. Could you point me a solution?

Thanks!

thick paint shader not connecting/ missing

I've set up the toolkit as instructed but shaders are missing/ bright pink and its simple things like 'Spike' or 'Thick Paint' that are missing - attached are photos of the composition in tilt brush and in unity

Missing uv-attributes from the FBX export ?

Hello there !
Some brushes need more than 2 component of some UV channels ( like the electricity one's )
Is there any way to deal with that without going by the JSON export ?
Found out FBX dont export more than 2 component of UV per channel.

SDK

Hello, I'm having a problem with the SDK recently in unity 7.04 build. For some reason I'm getting a strange resolution, pixelated type problem with all of my sketches I import. It's happened before in 7.03 and I couldn't figure out why but I know for sure I never experienced it in the .03 builds. I've attached some screenshots of console errors that are completely greek to me that I'm hoping will help locate the problem. And I think older builds of the sdk work with Unity 7 builds which will be my experiment today. But thanks for making this and hope you can help me sort this out. Apologies if the format of me asking this is off. I'm new to GitHub

HDRP Support

Any suggestions on how to import the brushes correctly in Unity when using HDRP?

More of a question then an issue

When importing a tilt brush file into unity, would it be possible to change the color inside of unity easily? Or is all the colors calculated in tilt brush and saved to the exported file?

Tilt Brush Fbx Import Null Reference Error

Hello,

I'm using TiltBrushToolkit with Unity 2019.3.7f1. Whenever I import an fbx file that I exported from Tilt Brush, I get a bunch of lines in the Console with the error:

"NullReferenceException: Object reference not set to an instance of an object
TiltBrushToolkit.ModelImportSettings.GetDescriptorForStroke (System.String oldMaterialName) (at Assets/tilt-brush-toolkit/UnitySDK/Assets/TiltBrush/Scripts/Editor/ModelImportSettings.cs:440)
TiltBrushToolkit.ModelImportSettings.OnAssignMaterialModel (UnityEngine.Material material, UnityEngine.Renderer renderer) (at Assets/tilt-brush-toolkit/UnitySDK/Assets/TiltBrush/Scripts/Editor/ModelImportSettings.cs:218)
System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at <437ba245d8404784b9fbab9b439ac908>:0)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation."

Any thoughts on how to fix this?

Line 440 of ModelImportSettings.cs is:
return TbtSettings.BrushManifest.BrushesByGuid[brushGuid];

Line 218 of ModelImportSettings.cs is:
BrushDescriptor desc = GetDescriptorForStroke(material.name);

Thank you!

Cinema Director dependency

It seems a bit strange to have a dependency on a paid asset store component. I guess the Unity Timeline was released after this functionality was developed.

Have the devs looked into replacing Cinema Director with the built-in timeline?

Change brush colour in Unity

I'm having an impossible time locating where in the code I can possibly make a change to the colour of the disco brush after it's already imported into Unity. This got me thinking how convenient it would be to be to have this added to the toolkit overall. If this isn't something thought to be added to future releases, can I at least get pointed in the right direction where I could make this change myself?

export sequence

It is written in your documentation: "Export each frame in Tilt Brush and copy them into the project" in order to create an animation.
How to do it? I can export only a final result FBX from the TiltBrush inteface. Please provide a step-by-step instruction.

Tilt brush imports not creating baked lightmaps

I cannot bake light into the imported materials in my scene. Standard shader still accepts baked lighting. Attached are my lighting settings, scene examples, and import settings. Please let me know if this is on my end, I just never experienced anything like this.

and for reference, here is an image with the SDK not imported:

Modifying exported fbx and importing to Unity

Hello,

I'm trying to add rigging and animation (and import them to Unity) to meshes created with TiltBrush.

When I export a fbx from TiltBrush and import it to Unity everything works fine. The problem is when I import and export in 3DS Max, all the material are lost inside of Unity. How are the materials stored in the FBX? Is there a 3D CG program (Blender, Maya, etc) that keeps the material linking for Unity?

Also I'm wondering if I'll have to modify the brushes vertex shaders to support skinned animation.

Thanks!

Is there a Python 3 port of this script?

definition of ZIsDistance missing

Receive this error just opening in unity (2017.2.0p2) the toolkit (tiltbrushtoolkit-UnitySDK-v14.0.0.unitypackage)

Assets/TiltBrush/Scripts/Editor/ModelImportSettings.cs(226,46): error CS0117: TiltBrushToolkit.BrushDescriptor.Semantic' does not contain a definition for ZIsDistance'

URP support

Hello, I'm making a gallery for a mobile app using the Tilt Brush, if anyone could share a version of the URP shaders I'd be very happy.
Or tell me how to do it.
Mostly problems with animated materials such as Hypercolor

Toon Shader, Unity Export

Hi team,

I am trying to use the Toon material with my export from Tilt Brush (drawn with the Toon brush); however, I am not getting any outline. I've tried using various texture images and exports but nothing is working. Any support would be much appreciated!

Cheers

V1

.glb import error

When I try to import .glb file made by Tilt Brush, I meet an error:

Asset import failed, "Assets/glb/kbk.glb" > GlbError: bin length%4
TiltBrushToolkit.GlbParser.Err (System.String fmt, System.Object[] args) (at Assets/TiltBrush/Scripts/Gltf/GlbParser.cs:32)
TiltBrushToolkit.GlbParser.GetBinChunk (System.String glbPath) (at Assets/TiltBrush/Scripts/Gltf/GlbParser.cs:126)
TiltBrushToolkit.BufferedStreamLoader.Load (System.String uri) (at Assets/TiltBrush/Scripts/Gltf/IUriLoader.cs:126)
TiltBrushToolkit.Gltf2Root.Dereference (System.Boolean isGlb, TiltBrushToolkit.IUriLoader uriLoader) (at Assets/TiltBrush/Scripts/Gltf/Gltf2Schema.cs:103)
TiltBrushToolkit.ImportGltf.BeginImport (System.String gltfOrGlbPath, TiltBrushToolkit.IUriLoader uriLoader, TiltBrushToolkit.GltfImportOptions options) (at Assets/TiltBrush/Scripts/Gltf/ImportGltf.cs:253)
TiltBrushToolkit.ImportGltf.Import (System.String gltfOrGlbPath, TiltBrushToolkit.IUriLoader uriLoader, TiltBrushToolkit.IImportMaterialCollector materialCollector, TiltBrushToolkit.GltfImportOptions options) (at Assets/TiltBrush/Scripts/Gltf/ImportGltf.cs:166)
TiltBrushToolkit.Glb2Importer.OnImportAsset (UnityEditor.Experimental.AssetImporters.AssetImportContext ctx) (at Assets/TiltBrush/Scripts/Editor/Glb2Importer.cs:45)
UnityEditor.Experimental.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.Experimental.AssetImporters.AssetImportContext ctx) (at /Users/builduser/buildslave/unity/build/Modules/AssetPipelineEditor/Public/ScriptedImporter.cs:22)

.glb file size is 1,523,711,658 bytes, and it cannot be divided by 4. I guess this problem is of the Tilt Brush app.

Occulus / HTC

Thank you so much for this incredible work- fantastic!

However, I just failed to implement your teleporting example into a oculus experience (instead of using SteamVR cameraRig). Any idea how to proceed? Thank you so much!

Light brush

light brush import to unity error. doesn't make the glow action. Any comments on this?

Embers material scaling

When scaling geo using 'Brush/Particles/Embers' the motion of the 'particles' don't scale correctly.
There is a comment in there with an alternate that also doesn't quite scale correctly.

I locally added a scale term to the displacement function output, similar to what's done in other shaders
ie:
return dispVec * kDecimetersToWorldUnits * length(unity_ObjectToWorld[0].xyz);

how to use dump_tilt.py for noobies

hey guys!
i wanna to stream the color of a selected brush in tiltbrush. i saw that dump_tilt.py display the color of the brush selected.
the question:
how to activate the dump_tilt.py while i am playing tilt brush?
just do 'python dump_tilt.py' on DOS? i have to tweak something?

Create PyPi package for the Python code

I suggest to move bin and tests under Python directory and create setup.py to turn the Python part of the toolkit to a PyPi package.

Even if we don't want to publish the package right now it would be useful to install it locally.

Here is the draft commit.

Using code to dynamically load FBX exported by Tilt Brush in Unity

Hello, there is a question I want to ask
Tilt brush unity sdk, this plug-in imported into unit, dynamic loading of FBX exported by Tilt Brush into the project through code in unit, error occurred. Is there a feasible way?

WebGL Build and Microphone Script Issue

I'm struggling to build using WebGL due to the GenericAudioInput script referring to the Microphone, which WebGL doesn't support.

Example error:
Assets\ThirdParty\Reaktion\Utility\GenericAudioInput.cs(69,40): error CS0103: The name 'Microphone' does not exist in the current context

I've tried #if !UNITY WEBGL code to not use this reference, as well as deleting scripts, but both come back with further errors.

Anyone know how to resolve this?

Thanks.

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

• UnitySDK/Assets/ThirdParty/CSCore/CSCore.dll
• UnitySDK/Assets/ThirdParty/Json-NET-for-Unity/Assemblies/AOT/Newtonsoft.Json.dll
• UnitySDK/Assets/ThirdParty/Json-NET-for-Unity/Assemblies/Standalone/Newtonsoft.Json.dll
• UnitySDK/Assets/ThirdParty/Json-NET-for-Unity/Assemblies/Windows/Newtonsoft.Json.dll

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.

---

Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Reactivity recording with a song in Unity for 360 video export

So, these audio reactivity brushes are a real pain to capture in Unity. Tilt Brush documentations' technical way of downloading 360 videos doesn't allow for real-time sync with audio. By bringing Tilt Brush into Unity and I'm able to create a flawless app and custom interactions. But, try to capture everything in real-time 360 video from VR?!? Yeah right. All the programs, free and purchased, I've found cannot complete this task in real-time. The best only previews in real-time (OliVR). The programs are in alpha or beta on real-time 360 capture. I'm not even trying to move the camera through the painting. I just want one point through a 3 1/2 minute song.

Any ideas? Or will Tilt Brush/Unity allow for sync rendering with these brushes in some way I haven't discovered yet?

If you need more details about what I'm doing let me know. Normal people just animate objects, which are easy to capture. But these brushes are a new beast. Recording the animations in Unity doesn't seem to work, though most recorders just used transformation settings and the brushed don't touch those settings.

Susan

Electricity shader is always out of focus when using Depth of Field standard effect asset

When applying the Depth of Field image effect from the Unity Effects asset package, all electricity elements are out of focus.

A sample project exhibiting this issue can be found here https://github.com/kaze0/Tilt-Brush-Toolkit-Issue
This sample porject has a Main Camera, with the modern Depth of Field image effect applied.
A small cube was placed inside of the electricity element that was exported from tilt brush, then the depth of field effect was set to target the cube as a way of ensuring that a useful transform was being targeted. You'll notice that the cube is in focus, but the electricity is not.

I have also moved a splatter paint element, that was exported from tilt brush, into that area as well. The splatter element is in focus as well.

As set up in this test project, I would expect the electricity to be in focus.

Most brush strokes seem to work fine. Disco might be another problematic one, but I haven't had time to take a second look.

Please see project at https://github.com/kaze0/Tilt-Brush-Toolkit-Issue

Sample showing the in focus cube + splatter and out of focus electricity. Zoomed in for effect.

SteamVR 2.x

The current version of SteamVR (2.x.x) available on the Unity asset store is not compatible with v16 release of tiltbrush toolkit

Current release exports Project Settings, which overrides Project Settings in imported project

This may be fixable by changing usage of ExportPackageOptions.IncludeLibraryAssets, which I see we're not sure about anyway. I haven't tried different options yet.

Brushes with Light material are not rendered as in Tiltbrush.

When exported a scene with Light brush to unity, they are overexposed and colors are clamped to white. Any solutions to this?

Can I connect a HTC tracker with Tiltbrush camera ?

I want connect and synchronize HTC tracker and Tiltbrush camera (spectator camera). Is it possible with Python script or other solution ? I tried in mixed Reality with LIV (on Steam), but it doesn't work...
Thank you

Relationship between this repo and poly-toolkit-unity

There's considerable overlap of functionality between this repo and poly-toolkit-unity (I'm assuming that poly-toolkit-unity is a superset of the functionality here).

Will they always be updated in sync? If I want the most up to date Tilt Brush import code should I check here?

Import Errors

When upgrading unity, or opening the project for the first time, tiltbrush models always fail to import properly. After initial import has completed, we then need to reimport all the tiltbrush assets again.
I assume it's an order of import with the models importing before the brush definitions are available.

InvalidOperationException: Cannot find TiltBrushToolkitSettings.asset TiltBrushToolkit.TbtSettings.get_Instance () (at Assets/ThirdParty/TiltBrush/Scripts/TbtSettings.cs:30) TiltBrushToolkit.TbtSettings.get_BrushManifest () (at Assets/ThirdParty/TiltBrush/Scripts/TbtSettings.cs:42) TiltBrushToolkit.ModelImportSettings.GetDescriptorForStroke (System.String oldMaterialName) (at Assets/ThirdParty/TiltBrush/Scripts/Editor/ModelImportSettings.cs:451) TiltBrushToolkit.ModelImportSettings.OnAssignMaterialModel (UnityEngine.Material material, UnityEngine.Renderer renderer) (at Assets/ThirdParty/TiltBrush/Scripts/Editor/ModelImportSettings.cs:228) System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at <437ba245d8404784b9fbab9b439ac908>:0) Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation. System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at <437ba245d8404784b9fbab9b439ac908>:0) System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) (at <437ba245d8404784b9fbab9b439ac908>:0) UnityEditor.AssetPostprocessingInternal.InvokeMethodIfAvailable (System.Object target, System.String methodName, System.Object[] args) (at <6f28216fea9f453abf2e05b770ed3ee4>:0) UnityEditor.AssetPostprocessingInternal.ProcessMeshAssignMaterial (UnityEngine.Renderer renderer, UnityEngine.Material material) (at <6f28216fea9f453abf2e05b770ed3ee4>:0)

Shaders not assigned

Hi, I still have the problem that the shaders are not assigned when importing the export folder from TiltBrush into unity. Unity breaks up the meshes and the toolkit then assigns the shaders only to the first mesh. I also tried reimport all assets, still didn't work.

The display of some brushes are different in u3d and tilt brush

I have use what you point to me in my scirpt. See the following:
https://support.google.com/tiltbrush/thread/27647682?hl=en

But today ,I found the question:

The display of brushes ,such as light,Steamers,waveform,seem too white in unity3d editor,which this is totally differenct from what they look like in tilt brush

You can see the situation in the following:
https://www.youtube.com/watch?v=aL_o44lG2zk

Can you give me some advice on how to improve it to make it behave like in tilt brush.

Thank you very much!

Last Tiltbrush SDK with what version of Unity in 2019

Hi, there are a lot of Unity version in 2019. With the first 2019.1.0f2 it's OK, I have all my brushes and colors, but with 2019.3.5f1 I lost all of them. What's the last version who work ?
Thank you

Smoke shader in Linear colourspace is very different from in tiltbrush

with running assets in linear colourspace there's quite a large difference between what we see in tiltbrush and in unity.
We ended up multiplying the alpha just by eye and landed about ~2.8x
Not sure if there's a more logical way to figure what the difference is

Play mode error when SteamVR is imported

Hi, an issue I am having is that when I import the SteamVR plugin, there are a number of compiler errors in the console which prevent me from playing the scene. All I have in the scene is a camera rig, a directional light, and a Tilt brush export. My apologies, I am new to coding, VR, Unity and Github. It seems to work fine before I import the SteamVR plugin.

Thank you in advance!

some strokes import with no color

so I'm having this issue with the Light and Wavelength brushes. within a group, only some strokes will import with the correct color used in TB, the others in the same group won't have any color at all.

I'm a noob a Unity, but I really don't understand how that happens, and I can't manage to fix the colors myself in the Inspector for instance. I tried to tinker with the shader but I'm failing badly.

is there a quick way to assign a tint to the brushes that lose their colors during the import?

or perhaps you are working on fix already for this import bug?

great SDK by the way!

'AssetImporters' does not exist

Hello,

I've been using Unity 2019.2.0f1 for one of my VR projects. Since recently I downloaded the TiltBrush SDK, I've been having compiler errors and none of the builds would go through (but it plays perfectly in the editor). Here are some examples of the error in the console window:

Assets\TiltBrush\Scripts\Editor\Glb1Importer.cs(23,32): error CS0234: The type or namespace name 'AssetImporters' does not exist in the namespace 'UnityEditor.Experimental' (are you missing an assembly reference?)
Assets\TiltBrush\Scripts\Editor\Glb1Importer.cs(28,29): error CS0246: The type or namespace name 'ScriptedImporter' could not be found (are you missing a using directive or an assembly reference?)
Assets\TiltBrush\Scripts\Editor\GammaSettings.cs(22,30): error CS0246: The type or namespace name 'EditorWindow' could not be found (are you missing a using directive or an assembly reference?)

Was the 'AssetImporters' taken out of this version of 'Experimental'? Or did I save something under the wrong directory?

Would really appreciate help from you.

Best,
Naomi

p.s. Here is a section of the error code in Glb1Importer.cs

using Object = UnityEngine.Object;
#if UNITY_2017_1_OR_NEWER
using System;
using System.IO;

using UnityEngine;
using UnityEditor;
using UnityEditor.Experimental.AssetImporters;

streaming Selected color of a brush into unity

hey guys. little question:
has some way to get the HEXcolor code of a brush, and display like a variable in unity?