Context Deadline Exceeded in Google Cloud Run

[palantir]

Context Deadline Exceeded in Google Cloud Run

[bluekeyes]

This error is happening on what is most likely the first HTTP request made by your handler, when the ghinstallation package tries to get an authentication token. This request uses the context that you pass to CreateCheckRun, which in turn is the context passed to your handler. As far as I know, none of these packages modify the context, so the timeout you are seeing is either a default value or is set before your handler is called.

I've tried to manually update the deadline to something longer, but that didn't seem to work.

Once a context has a deadline set, it is not possible to extend the deadline without creating a new, unrelated context.

It's a bit hard to say exactly what's happening from the information provided, but here are some things to consider:

1. Can you make any outbound requests to GitHub (or other hosts)? Maybe you could make one during server startup as a test. A timeout could indicate that something is wrong with your networking configuration in Google Cloud Run.
2. Do you get the same error if you use context.Background() instead of the context passed to the handler? If yes, then the problem is a timeout (or environment issue leading to a timeout) unrelated to the context passed to the handler.
3. Assuming that you shared the code for a githubapp.Handler, how have you configured the event dispatcher and client creator? Are you using synchronous or asynchronous dispatch? Are you setting a timeout on the client by using the WithClientTimeout option?

[ex0dus-0x]

Will drop some more details, apologies for the lack of clarity!

1. Yes, outbound requests are possible, and shouldn't be blocked by default by anything on GCP's end. In fact, looking at the PR, the Check Run does successfully start executing, but the error still shows up after.

2. Using a context.Background() alleviates the problem slightly, where we're able create the check run, and even successfully complete the execution logic itself. However, this is a rare occurrence, and when I try redelivering the request with the PR event to test, the context deadline error still pops up regularly in different parts of the code that interacts with the GitHub API (ie, listing changed files).

3. Yes, we're encapsulating the behavior in a githubapp.Handler. Here is what the event dispatcher looks like:

	cc, err := githubapp.NewDefaultCachingClientCreator(
		config,
		githubapp.WithClientUserAgent("bot/prerelease"),
		githubapp.WithClientTimeout(3*time.Second),  // ... changing or removing this does not get rid of the error
		githubapp.WithClientCaching(false, func() httpcache.Cache {
			return httpcache.NewMemoryCache()
		}),
	)
	if err != nil {
		panic(err)
	}

	dispatcher := githubapp.NewEventDispatcher(
		[]githubapp.EventHandler{
			&handler.PullRequest{ClientCreator: cc},
		},
		config.App.WebhookSecret,
		githubapp.WithErrorCallback(githubapp.MetricsErrorCallback(server.Registry())),
		githubapp.WithScheduler(
			githubapp.QueueAsyncScheduler(
				QueueSize, Workers,
				githubapp.WithSchedulingMetrics(server.Registry()),
				githubapp.WithAsyncErrorCallback(githubapp.MetricsAsyncErrorCallback(server.Registry())),
			),
		),
	)
	server.Mux().Handle(pat.Post(githubapp.DefaultWebhookRoute), dispatcher)

Let me know if there's any other useful info that could help fix this.

[bluekeyes]

Thanks for the additional information. Since this seems to work locally and the failures sound a bit random, I suspect this problem is ultimately caused by the runtime environment.

I haven't used Google Cloud Run before, but this requirement stood out to me:

Computation is scoped to a request

After startup, you should only expect to be able to do computation within the scope of a request: a container instance does not have any CPU allocated if it is not processing a request.

When using the githubapp.QueueAsyncScheduler, the server responds to webhook requests immediately and then a background goroutine executes the handler code. This means the majority of work is happening outside the scope of a request, which may cause Cloud Run to terminate or restrict your container in a way that leads to this error.

I suggest trying the default synchronous scheduler (you can leave out the whole githubapp.WithScheduler option) to see if that helps. If that fixes the problem but you need asynchronous processing for other reasons (say to avoid GitHub's hook delivery timeout), it sounds like you may need to use a different deployment platform or use something like Cloud Tasks to safely schedule asynchronous work.

[ex0dus-0x]

Yes, this seems to fix the issue, thanks for the help!

It turns out another solution that allows you to keep using an async scheduler is enabling a new Cloud Run feature that keeps it running perpetually without scaling down, meaning background goroutines work too.