Source Generation Issue

[thomhurst]

Source Generation Issue

[thomhurst]

Also, as 17.10 preview 4 just came out, can you give it a spin. There have been lots of changes in this area, so it's possible this was fixed (or got worse =-o). Would be good to know if it still repros to you on that build. Thanks!

Will give it a go when I get home and update

[CyrusNajmabadi]

My only downside I can think of is I want to generate source code for tests which might be defined in a base class, and there's no easy way to view inherited classes without searching through the entire compilation, right?

Correct, there is absolutely no efficient way to do that. Our strong recommendation is that you not design your domain space in this fashion. We ourselves never do this. The runtime team never does this. Exactly because it just cannot be done 'incrementally' (today at least). So far, all of our first party consumers are good with this, and the 3rd party ecosystem has been doing the same.

It might seem unfortunate, but if you can't model the domain around what ISGs can actually provide well, we recommend not using SGs as the tool here, and you should use something else instead.

[CyrusNajmabadi]

That's this line:

roslyn/src/Workspaces/Core/Portable/Workspace/Solution/TextDocumentStates.cs

Line 77 in d21bece

=> _map.TryGetValue(documentId, out var state) ? state : throw ExceptionUtilities.Unreachable();

Which is... bad.

That's being called here:

So we're asking for the doc state for a generated document, and we're finding that it no longer exists. Thsi sholud not be possible in our snapshot model. I will try to repro this with the provided branch tomorrow.

[CyrusNajmabadi]

Note: i did'nt look deeply (about to go to bed), but your generators are not properly written:

https://github.com/thomhurst/TUnit/blob/935deb2bf4165ec0193cab9f6bd0fd35b08b180b/TUnit.Engine.SourceGenerator/TUnit.Engine.SourceGenerator/CodeGenerators/BeforeAfterAllTestsInClassGenerator.cs#L15-L20

This captures IMethodSymbols in the incremental pipeline adn will never be incremental. Your generator will run virtuall on every node in the entire compilation virtually on every keystroke.

[thomhurst]

I thought the idea is it'd just run on edited nodes, so keystrokes on methods? What do I need to do if this is incorrect?

[sharwell]

I thought the idea is it'd just run on edited nodes, so keystrokes on methods? What do I need to do if this is incorrect?

The problem is related to the identification of "edited nodes". The IMethodSymbol implementation captures all context about a method, including large amounts of context that is unrelated to your source generator. If any of this context changes, it will be treated as "edited" and cause those nodes to run again through the source generator. In practice, any edit anywhere in the project causes one or more changes to every IMethodSymbol in the project.

The solution to this is to create a custom data structure, e.g. MyMethodData, which contains properties for only the aspect of IMethodSymbol you need in your source generator. This data object should be equatable, either as a record, or by implementing IEquatable<T>, or by providing an IEqualityComparer<MyMethodData> to the portions of the source generator pipeline that produce it. Using a custom data model will allow the incrementality to take effect by only running steps in the pipeline when the symbol changes as it relates to what you are using from it.

I generally recommend users start at the end of the pipeline for defining the data models, and then work backwards towards the beginning to create steps that generate them.

[thomhurst]

I thought the idea is it'd just run on edited nodes, so keystrokes on methods? What do I need to do if this is incorrect?

The problem is related to the identification of "edited nodes". The IMethodSymbol implementation captures all context about a method, including large amounts of context that is unrelated to your source generator. If any of this context changes, it will be treated as "edited" and cause those nodes to run again through the source generator. In practice, any edit anywhere in the project causes one or more changes to every IMethodSymbol in the project.

The solution to this is to create a custom data structure, e.g. MyMethodData, which contains properties for only the aspect of IMethodSymbol you need in your source generator. This data object should be equatable, either as a record, or by implementing IEquatable<T>, or by providing an IEqualityComparer<MyMethodData> to the portions of the source generator pipeline that produce it. Using a custom data model will allow the incrementality to take effect by only running steps in the pipeline when the symbol changes as it relates to what you are using from it.

I generally recommend users start at the end of the pipeline for defining the data models, and then work backwards towards the beginning to create steps that generate them.

Ah I did not know that! Thanks for the information, I'll try and make it more efficient!

[CyrusNajmabadi]

@thomhurst I highly recommend https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.cookbook.md. note that the use of var testMethods = context.SyntaxProvider.CreateSyntaxProvider is high discouraged as it has no incrementality and will on every node on ever edit. Our recommended pattern is to use the ForAttributeWithMetadataName to register pipeline steps to run on symbols that have a particular attribute on it that you decide. This is (no exaggeration), generally around 1000x faster than .CreateSyntaxProvider and can often take around 1000x less memory (depending on what the callbacks passed into CreateSyntaxProvider do).

This is the only pattern we now use in the runtime itself for our own generators as it's the only system we've found fast enough, with a good balance of expressivity, and reasonableness around user experiences (for people directly using teh generator, or indirectly using it when they pull it in through a package).

[CyrusNajmabadi]

@thomhurst Having a little trouble repro'ing this. When i try, all the steps work up to the last step Right click TUnit.TestProject inside VS. Click run tests. When i do that, test-explorer gives me the following message:

========== Starting test discovery ==========
Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Could not find testhost
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.GetTestHostProcessStartInfo(IEnumerable`1 sources, IDictionary`2 environmentVariables, TestRunnerConnectionInfo connectionInfo) in /_/src/Microsoft.TestPlatform.TestHostProvider/Hosting/DotnetTestHostManager.cs:line 410
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyOperationManager.cs:line 223
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.InitializeDiscovery(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler, Boolean skipDefaultAdapters) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyDiscoveryManager.cs:line 149
========== Test discovery aborted: 0 Tests found in 1.3 sec ==========
Executing all tests in project: TUnit.TestProject

Is there some prereq/dependency i need here? Thanks!

[CyrusNajmabadi]

Also, as 17.10 preview 4 just came out, can you give it a spin. There have been lots of changes in this area, so it's possible this was fixed (or got worse =-o). Would be good to know if it still repros to you on that build. Thanks!

[thomhurst]

@thomhurst Having a little trouble repro'ing this. When i try, all the steps work up to the last step Right click TUnit.TestProject inside VS. Click run tests. When i do that, test-explorer gives me the following message:

========== Starting test discovery ==========
Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Could not find testhost
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.GetTestHostProcessStartInfo(IEnumerable`1 sources, IDictionary`2 environmentVariables, TestRunnerConnectionInfo connectionInfo) in /_/src/Microsoft.TestPlatform.TestHostProvider/Hosting/DotnetTestHostManager.cs:line 410
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyOperationManager.cs:line 223
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.InitializeDiscovery(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler, Boolean skipDefaultAdapters) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyDiscoveryManager.cs:line 149
========== Test discovery aborted: 0 Tests found in 1.3 sec ==========
Executing all tests in project: TUnit.TestProject

Is there some prereq/dependency i need here? Thanks!

Sorry you have to go into the Visual Studio experimental features and switch it on

tools -> options -> preview features -> testing platform server

[CyrusNajmabadi]

Sorry you have to go into the Visual Studio experimental features and switch it on

Thanks!

[thomhurst]

@thomhurst I highly recommend https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.cookbook.md. note that the use of var testMethods = context.SyntaxProvider.CreateSyntaxProvider is high discouraged as it has no incrementality and will on every node on ever edit. Our recommended pattern is to use the ForAttributeWithMetadataName to register pipeline steps to run on symbols that have a particular attribute on it that you decide. This is (no exaggeration), generally around 1000x faster than .CreateSyntaxProvider and can often take around 1000x less memory (depending on what the callbacks passed into CreateSyntaxProvider do).

This is the only pattern we now use in the runtime itself for our own generators as it's the only system we've found fast enough, with a good balance of expressivity, and reasonableness around user experiences (for people directly using teh generator, or indirectly using it when they pull it in through a package).

This is very useful, thank you!

My only downside I can think of is I want to generate source code for tests which might be defined in a base class, and there's no easy way to view inherited classes without searching through the entire compilation, right?

So even if I found my test with an attribute marker, I can't easily find inherited members.
Because of this I'm having to search through ClassDeclarationSyntaxes and INamedTypeSymbols and iterate through it's methods and then check for my test markers which I know is going to be crazy heavy work

[CyrusNajmabadi]

Ok. Now i get:

Log level is set to Informational (Default).
Connected to test environment '< Local Windows Environment >'
Test data store opened in 0.109 sec.
Building Test Projects
Starting test discovery for requested test run
========== Starting test discovery ==========
Starting server mode
Telemetry collector provider: 'Microsoft.Testing.Platform.Telemetry.NopTelemetryService'
Telemetry is 'ENABLED'
DOTNET_NOLOGO environment variable: ''
TESTINGPLATFORM_NOBANNER environment variable: ''
DOTNET_CLI_TELEMETRY_OPTOUT environment variable: ''
TESTINGPLATFORM_TELEMETRY_OPTOUT environment variable: '0'
TestApplicationOptions.EnableTelemetry: True
Received initialize request
Connection established with 'Microsoft Visual Studio Testing Platform Client', protocol version 1.0.0
Received testing/discoverTests request
TestFrameworkAdapter UID: 'TUnitExtension' Version: '1.0.0.0' DisplayName: 'TUnit' Description: 'TUnit Framework for Microsoft Testing Platform'
Time elapsed: 00:00:00
Server requested to shutdown
========== Test discovery finished: 0 Tests found in 747.5 ms ==========
Executing all tests in project: TUnit.TestProject
========== Starting test run ==========
========== Test run aborted: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========
========== Starting test discovery ==========
========== Test discovery skipped: All test containers are up to date ==========

But no yellowbars. LMK what you run into.

[thomhurst]

My only downside I can think of is I want to generate source code for tests which might be defined in a base class, and there's no easy way to view inherited classes without searching through the entire compilation, right?

Correct, there is absolutely no efficient way to do that. Our strong recommendation is that you not design your domain space in this fashion. We ourselves never do this. The runtime team never does this. Exactly because it just cannot be done 'incrementally' (today at least). So far, all of our first party consumers are good with this, and the 3rd party ecosystem has been doing the same.

It might seem unfortunate, but if you can't model the domain around what ISGs can actually provide well, we recommend not using SGs as the tool here, and you should use something else instead.

Thanks! I might have to settle from some marker attributes then. Not what I wanted ideally but if it's a necessary evil then so be it. Really appreciate you explaining these to me. I feel a bit more clued up about source generators now 😄

[thomhurst]

Ok. Now i get:

Log level is set to Informational (Default).
Connected to test environment '< Local Windows Environment >'
Test data store opened in 0.109 sec.
Building Test Projects
Starting test discovery for requested test run
========== Starting test discovery ==========
Starting server mode
Telemetry collector provider: 'Microsoft.Testing.Platform.Telemetry.NopTelemetryService'
Telemetry is 'ENABLED'
DOTNET_NOLOGO environment variable: ''
TESTINGPLATFORM_NOBANNER environment variable: ''
DOTNET_CLI_TELEMETRY_OPTOUT environment variable: ''
TESTINGPLATFORM_TELEMETRY_OPTOUT environment variable: '0'
TestApplicationOptions.EnableTelemetry: True
Received initialize request
Connection established with 'Microsoft Visual Studio Testing Platform Client', protocol version 1.0.0
Received testing/discoverTests request
TestFrameworkAdapter UID: 'TUnitExtension' Version: '1.0.0.0' DisplayName: 'TUnit' Description: 'TUnit Framework for Microsoft Testing Platform'
Time elapsed: 00:00:00
Server requested to shutdown
========== Test discovery finished: 0 Tests found in 747.5 ms ==========
Executing all tests in project: TUnit.TestProject
========== Starting test run ==========
========== Test run aborted: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========
========== Starting test discovery ==========
========== Test discovery skipped: All test containers are up to date ==========

But no yellowbars. LMK what you run into.

Just tried on Preview 4 and got the same as you. The source generation error seems to have disappeared.

Although I'm not sure why the test run is aborted. It doesn't seem to give any more information to try and tell why. Weird seeing as it runs via dotnet run or dotnet test

[CyrusNajmabadi]

Goign to close out for now. Please let us know if you run into this again. We can reopen if there's a repro. Tnx.