ultraleap/unityplugin

Ultraleap SDK for Unity.

unityplugin's Issues

Deal with hot reload

PlugInLeapNotice prefab is broken

3 things:

1. On the LeapC side, IsServiceConnected isn't working because of a bug in LeapC. This has been fixed in platform. -- the fix should be in the assets by now.
2. I am getting null object errors when the prefab is in the scene, so it seems that something might be missing from the prefab? -- further investigation reveals that at some point, some code was changed to expect a UI.Image instead of GUITexture, but other scripts still expect GUITexture.
3. We shouldn't need to check for a connection every frame. Probably a coroutine that runs every couple of seconds is more efficient.

Hand misaligned to images in Unity Visualizer

If the service is not running when the scene is started, and then started, the hands and images become misaligned. Unity also throws an exception. This only seems to happen in the Visualizer scene, not, for example, the AR scene.

MissingReferenceException: The object of type 'LeapImageRetriever' has been destroyed but you are still trying to access it.
Your script should either check if it is null or you should not destroy the object.
UnityEngine.Component.GetComponentCamera
Leap.Unity.LeapImageRetriever.ApplyCameraProjectionValues () (at Assets/LeapMotion/Scripts/LeapImageRetriever.cs:351)
Leap.Unity.LeapImageRetriever.HandleOnValidCameraParams (CameraParams camParams) (at Assets/LeapMotion/Scripts/LeapImageRetriever.cs:287)
Leap.Unity.LeapVRCameraControl.OnPreCull () (at Assets/LeapMotion/Scripts/VR/LeapVRCameraControl.cs:81)

Address comment questions in #45

#45

Accommodate instances or prefabs in HandPool

Enabled vs. Disabled IHandModels gameObjects may satisfy switch

Leap.Hand.SphereCenter does not work after Leap.Hand.TransformedCopy

The code inside calculateSphere() does not account for the scaling suffered in TransformedCopy when calculating the _sphereRadius (since _maxSphereradius and _minSphereRadius are always in millimeters), and then _sphereCenter is wrong too.

GetLeapMatrix does not belong in UnityMatrixExtension

UnityMatrixExtension by it's name is for extension methods, which GetLeapMatrix is not. Either we should move GetLeapMatrix to a different class, or rename UnityMatrixExtension to allow non-extension methods to make sense inside it.

Hide HandPool.HandPool at editor time - show at runtime

Do this to avoid confusing developers

Add component for notifying the user about disabled Images

When an app wants images but is not receiving them, we should be able to display a warning to the user warning them that the application will not function correctly until images are enabled. Optionally, we might be able to provide some sort of interface (a button for example) to enable images for the app for the duration of the application.

Multiple IHandModels for HandProxy

Physics hand lags if LeapImageRetriever not present

GetFixedFrame transforms more frames than it needs to

The current implementation of LeapProvider.GetFixedFrame calls GetTransformedFrame every time it obtains a new frame inside the inner loop. All but one of these frames is going to be discarded, and so the transformation is wasteful. It should acquire un-transformed frame objects, and only transform the once that is selected.

Switch to using text meta files

We should be using text meta files instead of binary. It will reduce repository size in the long run because we will only need to store patches upon commit instead of recommitting the entire binary file each time. It will also make merges easier for things like scenes or prefabs. It also makes it easier to tell what kind of modifications have been made to an object, since the format is fairly descriptive.

There is no tracking in built scenes.

Using the most recent develop core assests and the develop service as of 3/9/16 tracking appears not to work in built scenes.

This issue also occurs when using the public builds of the service.

Capsule hand wrist collapses at scale .3 or less

At a scale of .3 or less, the two wrist spheres collapse to the same location (the palm becomes a triangle rather than a rectangle). The rigid hand palm also stops rotating with the hand.

The lack of palm collider rotation is probably an error with the quaternion conversion --also getting this error:

Look rotation viewing vector is zero
UnityEngine.Quaternion:LookRotation(Vector3, Vector3)
Leap.Unity.UnityMatrixExtension:Rotation(Matrix) (at Assets/LeapMotion/Scripts/Utils/LeapUnityExtensions.cs:54)
Leap.Unity.HandModel:GetPalmRotation() (at Assets/LeapMotion/Scripts/Hands/HandModel.cs:99)
Leap.Unity.RigidHand:UpdateHand() (at Assets/LeapMotion/Scripts/Hands/RigidHand.cs:37)
Leap.Unity.HandProxy:UpdateRepresentation(Hand, ModelType) (at Assets/LeapMotion/Scripts/HandProxy.cs:43)
Leap.Unity.LeapHandController:UpdateHandRepresentations(Dictionary`2, ModelType, Frame) (at Assets/LeapMotion/Scripts/LeapHandController.cs:111)
Leap.Unity.LeapHandController:FixedUpdate() (at Assets/LeapMotion/Scripts/LeapHandController.cs:85)

---

The CapsuleHands class is throwing a similar, but independent error, which probably causes the wrist issue:

Look rotation viewing vector is zero
UnityEngine.Quaternion:LookRotation(Vector3, Vector3)
Leap.Unity.CapsuleHand:updateCapsules() (at Assets/LeapMotion/Scripts/Hands/CapsuleHand.cs:175)
Leap.Unity.CapsuleHand:UpdateHand() (at Assets/LeapMotion/Scripts/Hands/CapsuleHand.cs:101)
Leap.Unity.HandProxy:UpdateRepresentation(Hand, ModelType) (at Assets/LeapMotion/Scripts/HandProxy.cs:43)
Leap.Unity.LeapHandController:UpdateHandRepresentations(Dictionary`2, ModelType, Frame) (at Assets/LeapMotion/Scripts/LeapHandController.cs:111)
Leap.Unity.LeapHandController:Update() (at Assets/LeapMotion/Scripts/LeapHandController.cs:74)

HandFader throws a null reference exception.

Every frame.

Remove ToUnity and ToUnityScaled

Currently ToUnity and ToUnityScaled don't do anything different than ToVector3. Either they should be changed so that they have a use, or they should be removed.

Virtual Hand Event methods in IHandModel

• make sure HandTransitionBehaviors respond to these

Incorrect constant names or values defined for time conversion

LeapHandController defines constants for time conversion:

const float     NS_TO_S = 1e-6f
const float     S_TO_NS = 1e6f

Be that as it may, a nanosecond is 1e-9 seconds not 1e-6 seconds. Either the constant name is wrong or the value is wrong.

Physics hand lagging behind Graphics hand

Playing and stopping hangs Editor if service not running

Deal with not using physics hands

Remove LeapImageRetriever dependency from LeapVRCameraControl

Remove warnings from LeapHandController

There are warnings about unused variables coming from LeapHandController

Judder when moving hands in VR

See https://community.leapmotion.com/t/judder-in-vr-when-using-rigged-hands/4712

Developer reports horrible judder when using SkeletalHand-based hand models. Hand motion itself is fine, the judder occurs when the camera is moving (controlled by head tracking).

This sounds similar to the problem we had some time ago and which was fixed by parenting the hand objects to the HandController. In this earlier issue, attaching the HandController to a moving object caused the hands to judder when ever the parent object moved.

Remove debug logs in LeapServiceProvider

Somehow some debug logs made it into LeapServiceProvider, they should be removed.

HandDrop destination seems fairly random

The position to which the hand model returns when tracking is lost seems fairly random. It isn't the starting point in the scene and it isn't the first tracked position of the hand. Just holding a hand steady in the Leap field of view and then covering the leap with the other hand so that the tracking is abruptly lost will result in the tracked hand model "dropping" to different positions on successive repetitions.

There is also a problem when hand re-identification occurs. If the hand switches chirality, the old hand remains stuck in the position of the switch. Oddly, in this case, the destination is consistent -- it doesn't move at all.

Build settings should not be building anrdoid by default

The build settings should either be empty, or set to something like the LeapDesktop scene. Currently on develop the editor build settings are building the android scene.

Fix HandController graphicsEnabled / physicsEnabled

Currently turning either one of those off does not destroy the representation, just causes them to stop getting updated and freeze.

Debug Hands are broken

Only a few lines of the fingers get drawn in Orion.

Issues in Leap.Hand.Basis and Leap.Hand.TransformedCopy

There are two issues in this method:

//TODO verify this calculation for both hands
_basis.zBasis = -Direction;
_basis.yBasis = -PalmNormal;
_basis.xBasis = _basis.zBasis.Cross(_basis.yBasis);

The first is that, as the comment present in code suggests, the cross product must be different for each hand, if a left-handed basis is expected for left hands, and a right-handed basis is expected for right hands.
The second is much more subtle. When you call the method Hand.TransformedCopy, a new Hand instance is created where the vectors Direction and PalmNormal are multiplied by a matrix, usually the LeapMatrix. Now, the LeapMatrix M, as currently provided by the method Controller.GetLeapMatrix, has a reflection on the Z axis, and thus we have that:

M(-Direction x -PalmNormal) != M(-Direction) x M(-PalmNormal)

Which is to say that the value of Hand.TransformedCopy(...).Basis.xBasis is wrong.

The following snippet illustrates the issue:

      Matrix leapMat = GetLeapMatrix();
      Frame frame = leap_controller_.Frame(); 
      Frame transformed = frame.TransformedCopy( leapMat );

      if (frame.Hands.Count > 0)
      {
          Assert.AreEqual(
              // _M_(-Direction x -PalmNormal)
              leapMat.TransformDirection(frame.Hands[0].Basis.xBasis).Normalized,

              // _M_(-Direction) x _M_(-PalmNormal)
              transformed.Hands[0].Basis.xBasis
              );
      }

The assertion always fails.

Hand always reports it is invalid

The hand constructor never set _isValid to true.

The IsValid concept has been removed from LeapC in platform. Mark this fixed when LeapC code is next merged here.

Update ReadMe for UnityModules repo

GetFixedFrame does not cache results for the same FixedUpdate call

GetFixedFrame currently recalculates the fixed frame every time it is called, but the result will remain the same during the same FixedUpdate call. We should only perform the operation once per FixedUpdate cycle since the operation is expensive.

Desktop Mode text should be changed to reflect HMD mode toggle

Description:

The Desktop and HMD Tracking Modes toggle but the text at the top does not.

Disabling Images Reduces FPS to 50

If you disable images via control panel or in Unity it reduces the core assets fps to 50 and produces weird judder artifacts.

LeapUnityExtensions need some fixing

FlipZ was changed to not flip at all. It should be set to (1, 1, -1).
ToUnityScaled() doesn't scale. In its current form, it is equivalent to ToVector3(), simply converting a Leap.Vector object to a Unity Vector3 object.

There are several uses of ToUnityScaled() in the Orion assets. These also need to be changed to ToVector3().

Update VR Visualizer to latest Core Assets

Just a ticket to track this...

Make Abstract class for LeapProvider

We should add an abstract base for LeapProvider, that way we can provide alternate ways to provide frames into the scene. Currently there is no way to play back recordings for example.

Add custom editor helper

In our custom editor scripts we often keep doing similar tasks. For example: Specifying a custom editor for a single variable, or hiding/showing a field based on a condition.

I feel that we can consolidate these separate efforts into a single subclass of Editor to both reduce boilerplate and reduce errors.

RigidHand prefabs don't scale well

If you scale the a parent of the hand objects, the capsule colliders get over scaled: https://community.leapmotion.com/t/unity-orion-physics-model-to-big/4752

In addition, the palm object's collider is set to be a few meters in size and then scaled down by its transform. This, apparently, leads to problems with interactions involving Unity physics: https://community.leapmotion.com/t/scaled-palm-bug/4816. Setting the proper size for the collider (in mm) and resetting the scale to (1,1,1) fixes the over scaling problem for the palm.

All Android files should be together

It is totally fine to have multiple different Plugins folders, and they do not need to be located at the root of the Assets. For the android files currently located in the root Plugins folder, we should move those to a new Plugins folder at Assets/LeapMotionModules/Android/Plugins

Add Updated RiggedHand.cs back into core assets

Add FPS counter to the Unity VR Visualizer

Description:

Hyoung recently added a Data FPS and Render FPS to the SamsungVR demo (#88). This should be easily ported to the VR Visualizer.

Requirements:

Investigate ways to profile time/memory on worker thread

Currently we have no way of easily profiling performance or garbage on the worker thread, which can make it difficult to track down issues. It would be nice to be able to have a nice process for this to be able to catch issues like #30 sooner.

Physics hands separate from graphics hands

See this post: https://community.leapmotion.com/t/unity3d-orion-rigidhand-not-following-capsulehand-during-movement/4780

"I have a project setup where I interact with the controls inside of a spaceship using Orion.
Problem is that the RigidHand hitbox does not follow the model of the hand as it should when the ship starts moving forward.

When the ship is idle everything works perfectly, but as soon as it starts moving, the RigidHand always drags up behind. Problem is probably due to a simple timing between updating the model of the hand and its hitboxes."

GetFixedFrame does not calculate timeline offset properly.

The timeline synchronization method previously used in HandController has not been properly ported over to LeapProvider.

There is a inspector-side variable called PerFrameFixedUpdateOffset but I am not sure why it exists. It is a constant value being passed into a smoothing structure (which can do nothing with a constant) and then used to calculate the correctedTimestamp. As far as I can see this is causing the physics hands to behave strangely and lag behind the graphical hands.

Leap.Finger.TransformedCopy doesn't propagate the fingerId correctly

The Finger constructor sets

_id = (handId *10) + fingerId

The fingerId itself is not stored in the Finger class. Later, when TransformedCopy is called, this _id is passed as fingerId to the constructor, which once again adds handId * 10.
The result is that the Id of the transformed copy is different from the original object.

Doxygen comments for Core Asset scripts

Thumb rotations in TestFactory Hands

Example can be found in https://github.com/leapmotion/UnityModules/tree/feature-LeapHandsModule

Test scene with rigged hands
/Assets/LeapMotionModules/Hands/Scenes/Rigged_Hands_VR.unity