masashi-hatano/trajectories_prediction

trajectories_prediction's Issues

Extracting the required images from the dataset

The images are collected in approximately 30 fps, whereas the input of the model is supposed to be in 2.5 fps. Therefore, we need to extract the required images from the original dataset.

• Write a code to extract images in 2.5 fps from the dataset.
• apply the code to get a timestamp which contains the required images' filename.

Replacing the code of segformer

The current code for semantic segmentation is from the official code of segformer, but they provides with the pre-trained model, which is not enough trained. So, using a pre-trained model from huggingface is a better way to use the model because we will have less code.

• Write a code to use a pre-trained model of segformer from huggingface.
• Check its quality.

Sort input data

Input data need to be sorted according to timestamp and pedestrian ID number.
Priorities should be timestamp and then pedestrian ID.

Extracting terrain information nearby pedestrian at the beginning

It might be better to use not a generated timestamp but an original one

Creating input data for the model

After extracting the required images, input data is ready to be created.

• run the existing code of ctrans.py and get the input data.

Creating input data with the terrain information

The frequency of applying semantic segmentation needs to be decided.

experiment: average of 100 samples at once

• edit prediction.py to increase the number of samples and apply mean of them.

Filling pedestrian's body

For mask image, the part of pedestrian needs to be whitened.

• edit segmentation.py

Uncertainty of coordinate transformation from far-shot

The coordinate transformation of points which are far from recorder seems to be uncertain and incorrect.
Maybe this is because the parameters are not so exact or the depth value is not used, but I am not sure about the reasons.
It might be possible to improve if the distortion factor is taken into account.

Plot the predicted trajectories

Using a json file which contains the information about all trajectories prediction, we can now plot them in the original images and see if it is appropriate or not.

• Write a code to read contents of json file and run ctrans_inv.py.
• Save each image in output file.

Experiment: comparison of the prediction accuracy with and without SS

• create branch for this experiment.
• compare the prediction accuracy: ADE, FDE, and count inhibited area.

Experiment: comparison of prediction accuracy with and without ctrans

• create branch for this experiment.
• compare the prediction accuracy: ADE and FDE.

Inconsistency of predicted trajectories

Trajectories predicted by the model is not sorted in an ascending or descending order; therefore, they are to be carefully treated.

• Consider an algorithm that matches to the output of model.
• Realize it.

Converting prediction data to json file

Socialgan/prediction.py returns torch array, which is of [number of prediction length, number of predicted trajectories, 2D coordinate].
This array is needed to be saved as either a text file or json file.
So far, I am thinking about saving as a json file as follow:

{
    "PredTimeList": [
    {
        "time_start": 1430,
        "PedList": [
        {
            "index": 0,
            "pred_traj": [(pred1_x, pred1_y),...,(pred8_x,pred8_y)]
        },
        {
            "index": 1,
            "pred_traj": [(pred1_x, pred1_y),...,(pred8_x,pred8_y)]
        }]
    },
    {
        "time_start": 1830,
        "PedList": [
        {
            "index": 0,
            "pred_traj": [(pred1_x, pred1_y),...,(pred8_x,pred8_y)]
        },
        {
            "index": 1,
            "pred_traj": [(pred1_x, pred1_y),...,(pred8_x,pred8_y)]
        }]
    }
}

• Create a new function to save predicted trajectories in a json file.