Computing ADE/FDE when compared with other methods

[abduallahmohamed]

Computing ADE/FDE when compared with other methods

[abduallahmohamed]

Hi @pedro-mgb

Thanks for this.

First, social-gan can not use our evaluation method because of the way it generates the results. Social-gan is a generative model in which the generated samples are correlated, thus judging it as best scene is suitable. In our case, we generate a distribution parameters, then we sample from these. The CV might be valid to these datasets (not fully aware of it) because the datasets are old and not complex enough. I'd prefer for any upcoming work to use https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge which is rich enough with more complex situations and better annotations. I think this answers your first bullet point.
For the second point, I think the only way to evaluate this is by qualitative analysis. Also, if you want to use these models in a real-life applications you will need lot of conditions around it. I don't believe it makes things worse, all of them are approaches to a complex problems with each method has it is own shortcomings.
For the third point, The best of N metrics (FDE -20 , ADE -20) are not suitable to judge the performance. Why 20? ...etc?
This article http://ai.stanford.edu/blog/trajectory-forecasting/ discuss this point extensively.

Let me know if you have more questions

[pedro-mgb]

Thank you for the response. I agree with what you said.

Multimodal CV may have the "best" prediction among 20 samples, but if we look at the errors from the other predictions (e.g. on average, the top-X samples), or use a NLL loss, like it is discussed on that article -> We will see that CV looks much worse than your model or social gan or an LSTM.

Regarding Trajnet++, I think it's a step in the right direction to having some form of standard. But I believe the trajectory forecasting problem using data-driven models is still just taking its first steps.

I don't really have any other questions. Thank you, once gain.
Feel free to close the issue, if you want.