Speed up `grpc` test on py3.10

[sentrivana]

Speed up `grpc` test on py3.10

[arr-ee]

TL;DR: caching alone might be tricky due to tox; pre-built wheels are super-easy to add, especially as a bandaid (patch below); long-term, pre-built wheels + exploring caching seems to be the way.

---

I've been eyeing this issue since I've been poking at how test envs are managed. Below is mostly my attempt to document current state and options — @sentrivana please correct what I got wrong!

So, we are forced to rebuild (older versions of) grpcio-tools against python versions released since, which is a costly operation. Current state (14 minutes for 1 python version x 3 grpcio-tools versions) is already a result of quite aggressive matrix trimming.

My understanding is that we're still trying to test against python 3.10 since it's the last release old versions of grpcio-tools build against, and we assume that if tests pass on the oldest and newest versions of python possible, they'd pass on intermediate versions as well (?)

Potential solutions:

1. Just Don't Test It
   grpcio-1.21 is from 2019. We can try assuming that users stuck with such an old version are equally hesitant to upgrade python, and/or are okay with not using the latest version of the SDK.

   This is by far the easiest solution, but it seems to go against the whole testing approach here :)

2. Cache things
   As mentioned above, this can be done relatively easily with Github Actions, and I'm sure there are other approaches.

   Some potential shortcomings:

   • Due to tox, different CI jobs install different sets of packages => more cache keys => cache churn (depending on whether cache action treats cached data as an opaque blob or dedupes based on hashes or smth). Github's total cache allowance is 10Gb, we have 354 tox environments, with pip cache size being around 100Mb per env. Actual number of keys would be lower, but might cause issues anyway.
   • Dependencies info is spread across (at least) three files => ensuring we're actually using cache might require some additional monitoring of job runtimes.
   • As a result, we'll still have 14min jobs from time to time. Might not be end of the world, but definitely not ideal.

   That said, caching can help, and trying alternative approaches (e.g. having a full dep install job to cache everything, then reuse that cache for other jobs, assuming github lets us to) might be worth it.

   It should also be the most hands-off approach, assuming everything works: packages without wheels for new versions of python would be cached without manual intervention.

3. Backfill wheels ourselves
   pip has several ways of providing alternative sources for packages, and combining that with either manually built wheels or something like cibuildwheel should give us consistently fast builds regardless of matrix size.

   I ran a quick-n-dirty check, and it works with barest minimum of changes:

   (sentry-py) m@reproduntu:~/local/sentry-python$ git diff
   diff --git a/tox.ini b/tox.ini
   index 34870b1a..bf961e2c 100644
   --- a/tox.ini
   +++ b/tox.ini
   @@ -234,6 +234,7 @@ deps =
       # in what's installed by tox (when running tox locally), try running tox
       # with the -r flag
       -r test-requirements.txt
   +    --find-links ../wheelhouse
   
       linters: -r linter-requirements.txt
       linters: werkzeug<2.3.0

   where ../wheelhouse houses wheels from pip wheel 'grpcio-tools==1.21.1', and can be replaced with a URL containing those. tox -e py3.10-grpc-v1.21 -r went from 210s to just under 15s.

   Of course, there are issues:

   • We'd need to house wheels somewhere. Any CDN, or even github repo would do, but it is still an additional bit.
   • New versions of python, new versions of dependencies, new dependencies == manual intervention in the form of building wheels or updating tooling config to build wheels.

4. An actually good solution that I have missed.

I am partial to the combination of caching and pre-built wheels, with patch above and grpcio-tooling wheel being hosted somewhere as a quick win for now.

I'm happy to look into this further — let me know what you think.

[Cheapshot003]

Sooo, what if we just cache dependencies for that suite? Like this:

name: Cache dependencies
        uses: actions/cache@v3
        id: cache
        with:
          path: |
            ~/.cache/pip
            !~/.cache/pip/log
          key: ${{ matrix.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('**/tox.ini') }}
          restore-keys: |
            ${{ runner.os }}-pip-${{ matrix.python-version }}-
            ${{ runner.os }}-pip-

That way we don't have to recompile the grpcio package. The implementation is probably wrong, don't know much about ci/cd yet. Or we can tell pip to use a prebuilt wheel.
Ah, I love dependency management \s