innosoft-pro/label-them

LabelThem

LabelThem is an online markup tool aimed at building image datasets for computer vision research, and integrated with Yandex.Toloka crowdsourcing system.

With LabelThem you can quickly get a markup, with classes and parameters of objects you are interested in, by providing images, lists of classes and parameters of objects, and, if you don't have the resources to label images by yourself, a short description of which types of objects you are interested in, and money to reward Mechanical Turks for performed markup.

Important Notice

At the moment LabelThem works only with Yandex.Toloka crowdsourcing system in Chromium and Chrome web browsers.

Newest version of LabelThem source files, compatible with Yandex.Toloka crowdsourcing platform are located in develop-toloka branch.

If you are only interested in using the system for getting a markup, you can skip the following text and go to the instructions on how to start working with LabelThem system on Yandex.Toloka.

System Prerequisites

In order to get started, you'll need to install NodeJS runtime, Bower package manager and Grunt's command line interface (CLI) globally. You may need to use sudo (for OSX, *nix, BSD etc) or run your command shell as Administrator (for Windows) to do this.

Building the system / Generating source files for Yandex.Toloka

In order to test the system on the local machine, or to generate source files needed to run the system on Yandex Toloka crowdsourcing platform you need to open a directory, which contains LabelThem system source code in a terminal (command shell on Windows), and execute the following commands:

1. bower update // to download libraries used by the system
2. npm install grunt-contrib-cssmin --save-dev // to install css minification plugin
3. npm install grunt-contrib-concat --save-dev // to install contatenation plugin
4. npm update // to install all the pre-defined dependencies (plugins) by going through the package.json file
5. grunt // to generate source files needed to run the system on Yandex Toloka crowdsourcing platform (to build systems minified and concatenated file)

The source files that are needed to run the system on Yandex Toloka crowdsourcing platform are:

1. /front/main.html
2. /build/app.js
3. /build/css/concat.min.css

Instructions on how to start working with LabelThem system on Yandex.Toloka can be found on the corresponding wiki page.

label-them's Issues

Docker env

Can't we build a docker image which will allow building needed files without the installation of extra libraries?

generate_tsv silently generates wrong data

Given invalid json such as:

{
  "classes":
  [
    "car",
    "person",
    "other""
  ],
  "parameters":
  [
    {
      "type": "boolean",
      "name": "is_occluded"
    }
  ]
}

generate_tsv.py silently generates invalid tsv file, which leads to empty space in Toloka.

Pictures with no objects can't be submitted

There are pictures where are no cars available for markup (empty road). Now they can't be marked up

Why do we have npm, bower and grunt in the very same application?

http://stackoverflow.com/questions/21198977/difference-between-grunt-npm-and-bower-package-json-vs-bower-json

Looks like it gets a little bit outdated right now. Probably we should consider reducing the number of tools needed.

Can't stop current drawing

Moreover, it's not possible to skip once you've started to draw. Line deletion goes here as well.

Stats for picture in acceptance mode

It would be useful to add statistics for a picture like following:

Number of polygons: x
Number of objects types: y
Number of object type A: z
Number of object type C: ...

This will help understand whether any classes at all are presented on the labeled picture with one glance.

Stylesheets are not loading in Yandex.Toloka in Chrome browser

Error message from console:

Refused to load the stylesheet 'https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css' because it violates the following Content Security Policy directive: "style-src 'self' 'unsafe-inline'".

Can we add some interactive manuals for toloka-users?

Many of them did strange things and mark things very bad

Zoom works to the top left corner, while it has to zoom to the centre

It is unexpected behaviour of the system. Usually when zoom is pressed picture became bigger in such way, that centre pixel of window stays on its place.

Strange naming

Somehow in JSON output we have entities called such as jsonPoints or jsonParams, while it sounds obvious.

Selection messed up with labelling

Screencast in gif on pretty random hosting :)
https://ezgif.com/optimize/ezgif-3-9d4bcadb3e.gif

Show blob classes near each marked blob

In delayed acceptance mod it can be very useful to have such feature. Now I have to manually click on each blob to check its class.
This will be also good at marking stage. Mechanical Turks will see what is already marked and how and what is not marked yet.
I think also would be good to have button which will be turn on and off such labels.
For example, that way this feature looks in TensorFlow:

Poligons move instead of zoom (when they should zoom as well)

Issue presented on the attached screencast

Offline approvement is not working

Basically, I can't see contours when I'm trying to understand should I approve or rejected the image.

Nevertheless, when I'm downloading results rejected by me they actually have something inside. In the archive below you can find example .tsv results - all and rejected only.

ExampleTSV.zip

Characteristics moves away

Also, it's not possible to know which object is being marked.
https://drive.google.com/file/d/0BxiHDYc6uk9PbFF4OEc3RWEwX28/view?usp=sharing

Is it possible to mark pictures without any marks in Y.Toloka as rejected automatically?

May be I should add some quality control activity or something like this

Removing isFirst in results

Given output result:

{
  "dataEntities": [
    {
      "jsonPoints": [
        {
          "x": 252,
          "y": 300,
          "isFirst": true
        },
        {
          "x": 250,
          "y": 286,
          "isFirst": false
        },
        {
          "x": 245,
          "y": 281,
          "isFirst": false
        },
        {
          "x": 245,
          "y": 243,
          "isFirst": false
        },
        {
          "x": 291,
          "y": 235,
          "isFirst": false
        },
        {
          "x": 313,
          "y": 242,
          "isFirst": false
        },
        {
          "x": 317,
          "y": 258,
          "isFirst": false
        },
        {
          "x": 325,
          "y": 285,
          "isFirst": false
        },
        {
          "x": 327,
          "y": 301,
          "isFirst": false
        },
        {
          "x": 327,
          "y": 313,
          "isFirst": false
        },
        {
          "x": 305,
          "y": 319,
          "isFirst": false
        },
        {
          "x": 282,
          "y": 319,
          "isFirst": false
        }
      ],
      "jsonParams": {}
    },
    {
      "jsonPoints": [
        {
          "x": 179,
          "y": 257,
          "isFirst": true
        },
        {
          "x": 177,
          "y": 246,
          "isFirst": false
        },
        {
          "x": 171,
          "y": 240,
          "isFirst": false
        },
        {
          "x": 173,
          "y": 194,
          "isFirst": false
        },
        {
          "x": 204,
          "y": 191,
          "isFirst": false
        },
        {
          "x": 262,
          "y": 199,
          "isFirst": false
        },
        {
          "x": 277,
          "y": 223,
          "isFirst": false
        },
        {
          "x": 279,
          "y": 254,
          "isFirst": false
        },
        {
          "x": 277,
          "y": 281,
          "isFirst": false
        },
        {
          "x": 260,
          "y": 286,
          "isFirst": false
        },
        {
          "x": 233,
          "y": 288,
          "isFirst": false
        },
        {
          "x": 218,
          "y": 276,
          "isFirst": false
        },
        {
          "x": 199,
          "y": 264,
          "isFirst": false
        }
      ],
      "jsonParams": {
        "class": "bigtruck"
      }
    },
    {
      "jsonPoints": [
        {
          "x": 120,
          "y": 215,
          "isFirst": true
        },
        {
          "x": 134,
          "y": 212,
          "isFirst": false
        },
        {
          "x": 134,
          "y": 196,
          "isFirst": false
        },
        {
          "x": 132,
          "y": 187,
          "isFirst": false
        },
        {
          "x": 113,
          "y": 183,
          "isFirst": false
        },
        {
          "x": 107,
          "y": 202,
          "isFirst": false
        }
      ],
      "jsonParams": {}
    },
    {
      "jsonPoints": [
        {
          "x": 393,
          "y": 128,
          "isFirst": true
        },
        {
          "x": 399,
          "y": 110,
          "isFirst": false
        },
        {
          "x": 422,
          "y": 101,
          "isFirst": false
        },
        {
          "x": 457,
          "y": 109,
          "isFirst": false
        },
        {
          "x": 483,
          "y": 123,
          "isFirst": false
        },
        {
          "x": 476,
          "y": 133,
          "isFirst": false
        }
      ],
      "jsonParams": {
        "class": "truck5-12",
        "is_occluded": true
      }
    },
    {
      "jsonPoints": [
        {
          "x": 198,
          "y": 260,
          "isFirst": true
        }
      ],
      "jsonParams": {}
    },
    {
      "jsonPoints": [
        {
          "x": 202,
          "y": 207,
          "isFirst": true
        },
        {
          "x": 247,
          "y": 195,
          "isFirst": false
        },
        {
          "x": 297,
          "y": 209,
          "isFirst": false
        },
        {
          "x": 307,
          "y": 263,
          "isFirst": false
        },
        {
          "x": 306,
          "y": 294,
          "isFirst": false
        },
        {
          "x": 253,
          "y": 299,
          "isFirst": false
        },
        {
          "x": 198,
          "y": 262,
          "isFirst": false
        }
      ],
      "jsonParams": {
        "class": "truck5-12"
      }
    },
    {
      "jsonPoints": [
        {
          "x": 130,
          "y": 213,
          "isFirst": true
        },
        {
          "x": 129,
          "y": 186,
          "isFirst": false
        },
        {
          "x": 109,
          "y": 183,
          "isFirst": false
        },
        {
          "x": 104,
          "y": 201,
          "isFirst": false
        },
        {
          "x": 113,
          "y": 212,
          "isFirst": false
        }
      ],
      "jsonParams": {
        "is_occluded": true
      }
    },
    {
      "jsonPoints": [
        {
          "x": 446,
          "y": 51,
          "isFirst": true
        },
        {
          "x": 420,
          "y": 92,
          "isFirst": false
        },
        {
          "x": 474,
          "y": 113,
          "isFirst": false
        },
        {
          "x": 487,
          "y": 65,
          "isFirst": false
        }
      ],
      "jsonParams": {}
    }
  ]
}

We have a lot of extra isFirst there. I think that this is misleading since you've already got the list order in which element with isFirst == true is always on the first position. Therefore, it could be eliminated.

Rename "Isoccluded" to "Partially covered"

Based on the first commercial run, almost nobody uses Isoccluded flag during labeling. Thus I propose to change flag naming to more general and simple.

Don't want to click save and/or choose instrument

We have only one instrument. Moreover, saving should be triggered upon "Next" button click.

Fix Getting started instruction

Hi guys! I've just tried your new release from branch develop and have some concerns:

• In getting started is better to mention that there is possible took release version and not to build yourself.
• During the building there is two missing steps in instruction: install grunt-contrib-concat and grunt-contrib-cssmin. Without them isn't working.
• Very hard to understand what is necessary to generate the .tsv file, how to use the python script. It isn't mentioned that files has to be placed locally on your computer, not in Yandex.disk. Also not mentioned how to get proper .json file.

script generate_tsv does not work on python 3 (but this is stated in readme)

ashepelev@ashepelev-Aspire-E1-571G:~/PycharmProjects/label-them$ python3 utils/generate_tsv.py -d img/ -p front/json/classesandparameters.json -i LT-Data1
  File "utils/generate_tsv.py", line 11
    except ValueError, e:
                     ^
SyntaxError: invalid syntax

on branch develop-toloka

Navigation using hand instrument isn't working at acceptance

Hello!
Yesterday I found out that there is no possible to move picture while zooming at delayed acceptance mode. I attach gif with screencast of this bug.

Seeing closed project

In case when project is already closed it still could be seen by other users.

On 100% scale picture goes besides the place for it

Review assignments instruction

I couldn't find Yandex.Toloka assignments review instruction. Could you please direct me to the appropriate documentation? If there is no such documentation, I think you should consider adding it.

output of generate_tsv is not accepted by Y.Toloka

tested on branch develop-toloka.
ya-disk:

proof that picture is opened:

tsv-file:

INPUT:image_rel	INPUT:json_params	GOLDEN:result
/LT-Data1/20161106_230949_7777_00408CACB891.mkv_004.jpg		true
/LT-Data1/20161106_230949_7777_00408CACB891.mkv_005.jpg		true
/LT-Data1/20161106_230949_7777_00408CACB891.mkv_003.jpg		true
/LT-Data1/20161106_230949_7777_00408CACB891.mkv_006.jpg		true

toloka errors:

{
    "withFailure": true,
    "results": {
        "2": {
            "success": false,
            "payload": {
                "tasks.0.input_values.json_params": {
                    "code": "VALUE_REQUIRED",
                    "message": "Value must be present and not equal to null"
                }
            }
        },
        "3": {
            "success": false,
            "payload": {
                "tasks.0.input_values.json_params": {
                    "code": "VALUE_REQUIRED",
                    "message": "Value must be present and not equal to null"
                }
            }
        },
        "4": {
            "success": false,
            "payload": {
                "tasks.0.input_values.json_params": {
                    "code": "VALUE_REQUIRED",
                    "message": "Value must be present and not equal to null"
                }
            }
        },
        "5": {
            "success": false,
            "payload": {
                "tasks.0.input_values.json_params": {
                    "code": "VALUE_REQUIRED",
                    "message": "Value must be present and not equal to null"
                }
            }
        }
    }
}

Accidental "Skip" button click is allowed

I marked up an image, trying to press "Next", forgot about clicking "Save" and then accidentally clicked on "Skip" button and then lost everything I had on image. How can we avoid such experience?

Remove contour which was already created

Maybe there is already some button for it, but I failed to find it.

Cannot invite trusted users to sandbox if they email is different from @yandex.ru

I'm trying to invite my buddy but he can't be invited by form

Nevertheless, mailbox with @yandex.ru works fine.

Built files inconsistency

/front/main.html
/build/app.js
/build/css/concat.min.css

All of them are in the different folders and their names seem to be random. Can't we build it to the very same output folder with names like "toloka.html", "toloka.js" and "toloka.min.css"? Other names could fit aswell.

In Y.Toloka markup is not displayed after rejecting or approving image.

I cannot check again rejected images if somebody complains because of this issue. Can we do something with it?

Reset current marks by "esc" key

I know that there is Ctrl+Z buttons for this, but I did not think about it first time

Russian language support

Polygon could not be deleted

During a review, I couldn't delete polygon using Backspace, Delete or any other keys. Is it possible to delete polygon during assignment review?