Lower contribution barriers on c++/cuda PRs

[bhack]

Lower contribution barriers on c++/cuda PRs

[drisspg]

@albanD I think this is a little tangential, the .devcontainer stuff is here: https://github.com/pytorch/pytorch/tree/main/.devcontainer

Currently this doesn't use a distributed sccache. In fact one of the main reasons that make it hard to develop on codespaces is because the default machines are so weak and compiling c++/cuda takes forever. I think if we were able to hook in sscache into the devcontainer it would be a silver bullet feature!

[bhack]

I don't know if the env changes are going to invalidate the cache hit. In other projects the past experience with bazel, it was very fragile without exactly the same reproducible env. But probably sscache is more robust.

[bhack]

/cc @ezyang in the case you have a feedback on this.

[ezyang]

If you are willing to do your builds in a Docker environment that exactly matches the CI env, you can technically use the sccache bucket. But IME it's more trouble than it's worth.

[bhack]

I suppose that with a little effort in the CI we could make a build (and the related sccache) quite aligned to the nightly container we are publishing every day for users/contributors.

So that the cache it will be valid. Also the pytorch team will need to check if the bucket has public access in read only mode.

[albanD]

cc @drisspg would the codespace env work for this? Where is the doc on how to use it, I can't find it.

[huydhn]

The cache bucket is currently open as read-only and we need to consider the risk to expose the write permission to the public for their local reproducibility. Sccache supposedly has a read-only mode in https://github.com/mozilla/sccache/blob/main/docs/Local.md#read-only-cache-mode, but we haven't been able to make it work with our S3 bucket IIRC (it keeps asking for AWS credentials).

[bhack]

but we haven't been able to make it work with our S3 bucket IIRC (it keeps asking for AWS credentials).

Is there any news on this? As I saw this comment yesterday: mozilla/sccache#1753 (comment)

[bhack]

So if that comment is correct we could use SCCACHE_S3_NO_CREDENTIALS=true. The main point is to populate the bucket from a regularly distributed image like nightly-devel.

[bhack]

I have some doubts about the current devcontainer case as we are building the images on demand. It would work if we reference nightly images in devcontainer

[drisspg]

Does the image matter? I though that what matters is the state of the PyTorch c++ files when attempting to compile. If someone is developing off a recent enough main branch then presumably there would alot more cache hits?

[albanD]

I think this is the same here, but we can re-use the ci docker images for the devcontainer. So we can actually put you in exactly the same setting to make sure you're going to properly hit the cache.

[huydhn]

So if that comment is correct we could use SCCACHE_S3_NO_CREDENTIALS=true. The main point is to populate the bucket from a regularly distributed image like nightly-devel.

This might not be as easy as it sounds. We explicitly don't use sccache for nightly and release builds for security reason to avoid cache poisoning attack. Theoretically, after building nightly, we could upload the build cache in an one-way direction to S3, but it does sound complicated.

[bhack]

Theoretically, after building nightly, we could upload the build cache in an one-way direction to S3, but it does sound complicated.

As we compile nightlies from scratch it could be relatively easy to push it in a new nightly bucket totally distinct from the current CI bucket or not?

[bhack]

But in any case, if you think that the CI image is pretty clean to be distributed it is the same.
I was talking about the nightly docker images cause I had the impression that the nightly-dev image is more curated to be distributed to users/developers but I've not checked all the details to compare it to the CI one.