unity-technologies/articulations-robot-demo

Unity Robotics Demos

The recent integration of Nvidia's PhysX 4 into Unity has dramatically improved the quality of robotics simulation that is possible in Unity.

• The new articulation joint system (available in 2020.1) is much better suited to building things like robot arms than the older joint types available in Unity. It uses Featherstone's algorithm and a reduced coordinate representation to gaurantee no unwanted stretch in the joints. In practice, this means that we can now chain many joints in a row and still achieve stable and precise movement.

• The new Temporal Gauss Seidel (TGS) solver also supports more accurate simulation.

Installation

Unity 2020.10b1 or later is needed for the new joint system.

Install Unity

If you do not have Unity 2020.1.0b1 or later, add the latest 2020.1 beta release through Unity Hub. This demo has been last tested on Unity 2020.1.0b5.

Clone the Articulations Robot Demo Repo

Clone this repository:

git clone https://github.com/Unity-Technologies/articulations-robot-demo.git

Then, open the ArmRobot project in Unity.

UR3 Robot Arm

This is a simulation of the Universal Robotics UR3e robot. You can steer it by directly rotating all six joints of the arm. You can also rotate the end effector, and open and close the pincher.

Open Scenes > ArticulationRobot, and press play.

Manual Controls

You can move the robot around manually using the following keyboard commands:

A/D - rotate base joint
S/W - rotate shoulder joint
Q/E - rotate elbow joint
O/P - rotate wrist1
K/L - rotate wrist2
N/M - rotate wrist3
V/B - rotate hand
X - close pincher
Z - open pincher

All manual control is handled through the scripts on the ManualInput object. To disable manual input, just uncheck this object in the Hierarchy window.

You can learn more about how this robot was built with articulations by following our guide here.

Robotiq Hand-E Gripper

This simulation focuses on picking up objects with the Robotiq Hand-E Gripper.

Open Scenes > GripperScene, and press play. Try to pick up the cube and drop it!

Manual Controls

Use the following keyboard commands:

G - down
H - up
X - close pinhcer
Z - open pincher

ML-Agents Integration

The Articulation Robot Demo has been integrated with the ML-Agents Toolkit on this branch. Note: The integration is not up to date with the latest master branch.

Survey

Your opinion matters a great deal to us. Only by hearing your thoughts can we continue to make Unity a better simulator for robotics. Please take a few minutes to let us know about it.

Fill out the survey

License

Apache License 2.0

articulations-robot-demo's Issues

There is no AiticulatonBody

Hi I got this error

keyboard control change

Hi, Nice work! currently I am trying to import another robot (Ur5) using the urdf importer package, so I wonder if it will be possible to use the robotcontroller script on that robot with few changes.
Also In the case of manual controller it is not clear to me how the joints and gripper are related to the keys from the:

float input = Input.GetAxis("Fingers"); ArticulationHandManualInput.cs
float inputVal = Input.GetAxis(robotController.joints[i].inputAxis); RobotManualInput.cs

I am new to unity and c# so I don't know if I didn't see a script or module who deals with it in the project.

Thank you in advance

Extra joint in UR3e

Hello, thanks for the great demos!

I think your UR3e model has an extra joint that does not exist in real UR robots. Comparing with actual UR robots, including UR3e (see, e.g., this UR brochure):

1. base - ok
2. shoulder - ok
3. elbow - ok
4. wrist1 - does not exist
5. wrist2 - should be wrist1
6. wrist3 - should be wrist2
7. "rotate hand" - should be wrist3 (as a small extra part between UR3e and gripper - it's not the (exchangeable) gripper that rotates on its own but the last robot link that it is attached to)

Is this difference intentional?
If the model was closer to reality, that would be valuable as a reference for projects showing industrial scenes.

Kind regards
Bernhard

Invalid AABB Articulations Solver Error

Arm part will break out when colliding table , the whole articulation body will break when doing more then 360 degrees turn of the 3rd wrist ( wrist3 - M button )

Note : Failed to use 2020.1.0a25 due to invalid entitlement ( ? ) to some packages after clone , rolled back to 2020.1.0a17

a17 Don't have this error, but before the downgrade from a25 after a fresh clone i was facing this issue on a fresh project clone:

Assets\Scripts\ArticulationJointController.cs(13,13): 
    error CS1069: The type name 'ArticulationBody' could not be found in the namespace 'UnityEngine'. 
    This type has been forwarded to assembly 'UnityEngine.PhysicsModule, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Enable the built in package 'Physics' in the Package Manager window to fix this error

Edit :
Did install the Physics package but still the same error is present in a25...

Note i got a fresh a25 install via unity hub.

How to separate the tool from the robot and bind the articulationbody at runtime

I have multiple tools and use different tools at different times, so I can't just install the tools on the robot using articulationbody at first, I try to install the tools at runtime, but some weird issues happen

How to set property "Motion" to "Limited" in C# script

How to set property "Motion" to "Limited" in C# script？
My object is dynamically loaded，so I can't set this through interface.

Demo UR3 robot arm crashes in MacOS

Trying to open the ArticulationRobot demo through Unity Hub, the system crashes immediately, seems like the required version 2020.1.0b5 isn't supported.

I want the robot to move according to the target position, not by keyboard control, is there any good solution？？？

Articulation not movable via Transform

Not sure where to report this, so here goes -- sorry if wrong place.

Made a figure using Articulation Bodies and it works fine for everything except direct setting of its transform upon game Initialize or OnEpisodeBegin when training under ML-Agents. Any non-Articulation Body can be moved without problem, so it is not a coding issue but seems to be a bug in Unity.

Different physics outcomes when using the gripper on different scales

The gripper gives the desired behavior when used at its original size, but when it is scaled down, it doesn't seem to be able to grip other objects properly.

gripperScaling.mp4

Missing script on Cube object

Interesting repo! Excited to see Unity diving into robotics applications 😄

My scene appears to be missing a script on the cube object? Is this a required script?
Additionally, I'm finding it impossible to grasp the cube unless I drastically reduce it's mass (default is 200?). Is this related to the missing script? Or a parameter tuning thing? I've noticed the joint parameters seem very large when compared to real world values (e.g. maximum force of 3.4E38 is terrifying in the real world)

Joint Torques Availability

Is it possible to derive joint torques from the ArticulationBody? ArticulationBody.jointForce looks to be a similar but it only returns zero for revolute joints in all the tests I have conducted. I am using Unity 2020.3.19.

Disable Collisions on Articulation body

Hello,

as a first trial, I would like to use Unity only for visualizations of trajectories given by ROS.
I want no physics (gravity, friction, damping) and no collisions with external objects to be calculated by Unity as this is all done in ROS already.

Is there a way to achieve this by using the articulation body which is created automatically when importing the URDF of UR10?

As every joint of the articulation body has its own parameters I hope I will not need to set the parameters of every joint to zero. Would it e.g. be enough to turn off gravity and set mass to zero in the root body or does this option need to be disabled for every single joint if I want no physic effects on the whole robot?
Also I couldn't find any option to disable collisions on the articulation body.

about Hand-E Gripper can I move it in another axis

about Hand-E Gripper can I move it in another axis by not use UR3 Robot Arm

It seems that YAML configuration file format has been changed in ML-Agents 1.0

Hello, I had encountered a minor issue that has been solved following your links to ml-agents repository.

mlagents.trainers.exception.TrainerConfigError: The option num_update was specified in your YAML file for TrainerSettings, but is invalid.

It seems that configuration.yaml file format has been changed in the lastest ml-agents, and your ur3_config.yml file is already properly modified in the hh/develop/robo-arm-continuous branch.

To make the mlagents-learn running, I organized your parameters and commented the following.

# num_update: 1
# train_interval: 1
# use_recurrent: false

2020-06-20 14:39:08 INFO [stats.py:100] TouchCube: Step: 5000. Time Elapsed: 271.120 s Mean Reward: -11.866. Std of Reward: 5.555. Training.
2020-06-20 14:43:57 INFO [stats.py:100] TouchCube: Step: 10000. Time Elapsed: 560.016 s Mean Reward: -9.497. Std of Reward: 5.626. Training.
2020-06-20 14:49:40 INFO [stats.py:100] TouchCube: Step: 15000. Time Elapsed: 903.239 s Mean Reward: -8.487. Std of Reward: 4.681. Training.
2020-06-20 14:54:46 INFO [stats.py:100] TouchCube: Step: 20000. Time Elapsed: 1209.069 s Mean Reward: -3.008. Std of Reward: 4.634. Training.
2020-06-20 14:59:54 INFO [stats.py:100] TouchCube: Step: 25000. Time Elapsed: 1516.457 s Mean Reward: -1.882. Std of Reward: 3.949. Training.
2020-06-20 15:04:58 INFO [stats.py:100] TouchCube: Step: 30000. Time Elapsed: 1820.897 s Mean Reward: -0.098. Std of Reward: 2.458. Training.

It seems ml-agents is working properly. Yet I am not sure is it OK to just comment those parameters.

I would be appreciated if you could check the attached config file and let me know whether it is correctly modified as you originally intended.

Thanks.

ur3_config.txt

Nvidia's PhysX 4 implementation fro AMD GPUs ?

Does the implementation of PhysX 4work also for AMD GPUs or Apple/ARM GPUs (iPhone, iPad)?

I'm especially interested in articulation joint system (available in 2020.1) Temporal Gauss-Seidel (TGS)

Check if any parts can move

Hello,

I can move my robot's parts with keypresses.
And I can play a sound loop. But I need to know the body is moving, or not. So how can I check if my robot arm reached the max and min angle or just it is moving? I need to stop the sound at this moment

Thanks

Some questions about project settings.

When I used the gripper demo, I found that fingers will not move through a collider. If two fingers both touch the collider, which locates at the middle of two fingers, the fingers will automatically stop moving, although their zDrive.target is increasing. I think this feature is very useful in simulation environments when doing grasping tasks.

However, if I create a new project and copy the assets to the new project, everything changed. Fingers will not stop moving when touching the collider, until they reach the limit.

After checking, I found that your Unity project in this repo modify some project settings in unity. Thus, I copied the project settings folder from your project to my new created project, then everything works well. Although this problem has been solved, I am very curious about what modifications have you made to project settings. I will be very grateful if you tell me the answer. Thanks in advance!

My Unity information: Windows 10, Unity 2020.1.17f1c1.

Get jointForce

Hello. I am trying to read the joint force and for that I am using the following code:

public class PincherCollision : MonoBehaviour
{

public ArticulationReducedSpace force;
// Update is called once per frame
void FixedUpdate()
{
    ArticulationBody fingerA_articulation = GetComponent<ArticulationBody>();
    ArticulationReducedSpace force = fingerA_articulation.jointForce;
   
    Debug.Log("Force: ");
    Debug.Log(force);
}

}
However I don't see any force being displayed and I'm not sure if I should convert the variable "force"to a different dataype. This is a script that I have attached to the FingerA object

Errors occurred when building environment executable of the mlagents branch

Hi,
I was using the mlagents branch, and trying to build the environment executable following the instructions on this website: https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Learning-Environment-Executable.md
But several errors occurred in the build process.
Could anyone please help? Thanks in advance!

Unity version: 2020.1.8b

Inference and Time.TimeScale

I tried to train the model and then to run an inference giving the new .nn trained model in input, but seem to have problem when putting a timeScale = 1. The robot moves slowly and discontinuously, making a small rotation of the joints, followed by a freezing of the whole robot, then a new small rotation, new freezing and so on.

The problem completely disappears putting a slightly higher value of timeScale. With timeScale = 2 all movements are smooth and the robot is well behaving, but I don't understant the problem with value = 1.

The branch of inverse kinematics

It seems that there appears a 404 page when I switch to the inverse kinematics branch. I'm wondering what happens to it.

Issue with articulation-bodies in OnEpisodeBegin

Describe the bug
I'm unable to move an articulation body to be in a specific configuration in OnEpisodeBegin.

Using the gripper scene in the Articulation-robots-demo project to illustrate the issue, I'm attempting to grip the cube in the air before training i.e. in the OnEpisodeBegin, but I'm unable to do so since this causes Unity to freeze. I tried using both a simple while loop, but also a coroutine, but did not seem to work.

To Reproduce
Steps to reproduce the behaviour:

1. Clone the Unity Articulation-robots-demo project.
2. Add ML-Agents package and open the GripperScene
3. Create a new script called GripperAgent
4. Insert either in the newly created script:
   a.

public class GripperAgent : Agent
{
    public GameObject cube;
    Rigidbody rb;
    public GameObject hand;
    PincherController pincherControler;

    void Start() {
        pincherControler = hand.GetComponent<PincherController>();
        rb = cube.GetComponent<Rigidbody>();
    }

    public override void OnEpisodeBegin()
    {
        rb.useGravity = false;
        rb.position = new Vector3(0.01f,0.6f,0);
        while(pincherControler.grip<0.122f){
            pincherControler.gripState =  GripState.Closing;
            Debug.Log(pincherControler.grip);
        }
    }
    public override void CollectObservations(VectorSensor sensor){}
    public override void OnActionReceived(ActionBuffers actionBuffers){}
}

b.

public class GripperAgent : Agent
{
    public GameObject cube;
    Rigidbody rb;
    public GameObject hand;
    PincherController pincherControler;
    public bool finished = false;

    void Start() {
        pincherControler = hand.GetComponent<PincherController>();
        rb = cube.GetComponent<Rigidbody>();
    }

    public override void OnEpisodeBegin()
    {
        rb.useGravity = false;
        rb.position = new Vector3(0.01f,0.6f,0);
        StartCoroutine(MyCoroutine());
        while(!finished){}
    }
    IEnumerator MyCoroutine()
    {
        while(pincherControler.grip<0.122f){
            pincherControler.gripState =  GripState.Closing;
            Debug.Log(pincherControler.grip);
            yield return new WaitForSeconds(Time.fixedDeltaTime);
        }
        finished = true;
    }
    public override void CollectObservations(VectorSensor sensor){}
    public override void OnActionReceived(ActionBuffers actionBuffers){}
}

5. Attach a GripperAgent script to the root containing the gripper gameobject and set Max Step to 1000
6. Attach the cube and the HandE(under root) to the GripperAgent script
7. Attach Decision Requester script to the root
8. Starting the Unity Player, then causes Unity to freeze.

Environment

• Unity Version: Unity 2021.1.2f1
• OS + version: Windows 10, 19042.928
• ML-Agents version: ML-Agents v1,9.1 (Release 16)

cube slide down when try to grab using automatic scripts

I have tried to change gripperscene to grab cube by some automatic scripts, but meet some problem. The original code is controlled by keyboard which can grab the cube up and down. My code has changed the scripts into three steps to grab the cude, 1. move down the gripper , 2. grab the cube , 3. move up the gripper. And I use the same code from PincherController to control the gripper, but when the code try to grab the cube , the cube can't move, just slide down from fingerA fingerB. Can anyone help ?

MLAgents GameObject not found

Hi,
I've just started with this project and followed every step correctly as I see. (ML Agents is installed and works fine in it's example project). However, as I've clone this project from github can't see any MLAgents GameObject in the scene to use the pre-trained model.
Since I'm not going to work with machine learning/training and I'm trying to use this Project for other developments I don't want to train the arm again, but use the projects model.
I would be really thankful if you could help me through.

Moving two joints at the same time

The problem

After setting up the ArticulationDemo scene, I noticed that you can't move two joints of UR3 simultaneously.

For example, pressing W and E at the same time should result in a shoulder and elbow rotation, but it only results in motion for the first pressed key's joint. This is likely due to L13-24 of RobotManualInput.cs

for (int i = 0; i < robotController.joints.Length; i++)
{
    float inputVal = Input.GetAxis(robotController.joints[i].inputAxis);

    if (Mathf.Abs(inputVal) > 0)
    {
        RotationDirection direction = GetRotationDirection(inputVal);
        robotController.RotateJoint(i, direction);
        return;
    }
}
robotController.StopAllJointRotations();

Will there be support for optional simultaneous movement in an upcoming release? I am working on a solution for my project (I can open a PR when I finish it) but it would be good to know if it was planned to be added.

System info

• Unity version 2020.1.0b8.3654
• Windows 10

Also the new articulation system is great, been waiting for something like this for a long time. So thanks!

What's the recommended method for locking a joint temporarily

We've tried adjusting the dampening to really high levels, and it almost solves this, but there's still some movement.

We also tried setting the joint type to fixed, and that creates a requirement for saving and restoring the position, however this an awkward solution and at many times falls short.

What would be the suggested method for locking a single joint on a chain?