Problem about evaluation

[autonomousvision]

Problem about evaluation

[Kait0]

The controller creeping described in the extended paper is a useful workaround to this problem.
https://arxiv.org/pdf/2205.15997.pdf
If you slightly force the model to creep forward it will usually start driving again, if there is no causal reason to stop.

[ap229997]

We are aware of this issue. While we don't have an exact explanation or solution for this, there can be multiple reasons why this is happening:

• the model learns some spurious correlations about stopping behavior from the data generated by the conservative expert policy, this could be addressed to some extent by designing a better expert policy
• we have noticed that this happens more when there is a dirt patch on the road which the model most likely interprets as an obstacle, one way to address this could be to train the model with semantic loss as an auxiliary objective (with road and dirt classes mapped together)

[Kin-Zhang]

Oh! Finally I found some one also notice the lastest model checkpoint have this problem! I think it's quite serious, escpecially you just run it at your computer, here is a picture from the running:

[ap229997]

We are aware of this issue and we currently don't have a good solution for it.

[FitashUlHaq]

any update on this? :)

[Hither1]

The controller creeping described in the extended paper is a useful workaround to this problem. https://arxiv.org/pdf/2205.15997.pdf If you slightly force the model to creep forward it will usually start driving again, if there is no causal reason to stop.

Hi Just want to clarify this is not implemented in the current git repo code right?

[Kait0]

We have released the code last week in the 2022 branch of the github.
You can find it in team_code_latest/submission_agent.py