Update flutter/packages macOS stable tests to Mac-13 or Mac-14

[vashworth]

Update flutter/packages macOS stable tests to Mac-13 or Mac-14

[jmagman]

I think these are builds that show those macOS sandbox-related failures:
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20custom_package_tests%20stable/2/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20master%20-%20packages/2/overview

@vashworth instead, can we disable the sandbox for the url_launcher_macos and pigeon example app, then have a TODO and tracking issue to turn it back on when that tools change gets to stable?

[vashworth]

I think these are builds that show those macOS sandbox-related failures: https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20custom_package_tests%20stable/2/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20master%20-%20packages/2/overview

@vashworth instead, can we disable the sandbox for the url_launcher_macos and pigeon example app, then have a TODO and tracking issue to turn it back on when that tools change gets to stable?

Sure, I'll try that

[vashworth]

I think these are builds that show those macOS sandbox-related failures: https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20custom_package_tests%20stable/2/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20master%20-%20packages/2/overview

@vashworth instead, can we disable the sandbox for the url_launcher_macos and pigeon example app, then have a TODO and tracking issue to turn it back on when that tools change gets to stable?

So doing this for pigeon will fix the custom_package_tests stable.

However to fix for macos_platform_tests stable, we'd actually have to disable sandboxing for the following package example apps: flutter_image, google_sign_in, google_sign_in_ios, image_picker, in_app_purchase, in_app_purchase_storekit, path_provider, path_provider_foundation, shared_preferences, shared_preferences_foundation, url_launcher, url_launcher_macos, video_player, video_player_avfoundation. The test failure doesn't show because it fails before it reaches, but there's a later part of the test that does flutter test -d macos on each of those, which will fail without #149618.

@jmagman Do you think it'd still be better to disable them directly rather than just keep them on Mac-13 for now? I think it's probably fine and chances of a new one being added are super low, so I'm okay to just directly disable all of them for now, but it is a lot more than just url_launcher_macos.

[jmagman]

I think we should disable sandboxing on all of them temporarily, as long as we have an action item to turn it back on after next stable.
@stuartmorgan do you have any concerns about this?

[stuartmorgan]

I guess I'm okay with that.

Long term, do we think we're just stuck with this though? Never testing any our macOS plugins the same way they will run for most users is... not ideal.

[jmagman]

It's definitely not ideal, but it would require more work to clear out the app containers between test runs (some more details in the referenced PR #149618). I think turning the sandbox off for tests is the best bad solution we have to unblock the macOS 14 update... 😞

[vashworth]

Closing in favor of #149844