Restore functionality takes a lot of time to start for a couple of endpoints

[cilium]

Restore functionality takes a lot of time to start for a couple of endpoints

[aanm]

Found the guilty. The restore functionality creates the endpoints by reading the header stored on the endpoint's directory. (1) At that time the endpoint program is regenerated to see if everything is alright. After that it allocates the endpoints IP and recompiles the program again (2). After the endpoints were restored, cilium syncs with the docker daemon to see which containers are still up and running.
While doing that check it reaches the line docker_watcher.go#L371:d.triggerPolicyUpdates([]policy.NumericIdentity{secCtxlabels.ID}) where, inside that function, the whole cache is invalidated (policy.go#L105:d.invalidateCache()) and triggers a policy update for every endpoint in the daemon in policy.go#L109:err := ep.TriggerPolicyUpdates(d).

The FIXME in policy.go#L104:// FIXME: Invalidate only cache that is affected needs to be addressed to solve this issue.

[tgraf]

We can solve it by implementing partial policy resolution but I don't think it addresses the main issue. How about we ensure the restore functionality is run before we start accepting any external API calls and events and lock the daemon down. Individually restore the endpoints using an API which does not generate the endpoints. Then finally, trigger a full policy calculation followed by generation of all the programs.

[aanm]

The lock down already happens

Individually restore the endpoints using an API which does not generate the endpoints.

Yep, sounds good to me.