Comments (15)
Hi @michael-hawker. Thank you for reporting, we will investigate this.
from runner-images.
Hi @michael-hawker - I think we can move you to the DNF GitHub Enterprise to allow access. We can investigate
from runner-images.
Hey @michael-hawker. The behavior you have described may be caused by OOM errors sometimes. May I ask you to extend your workflow with a task that runs always at the end and outputs system events? E.g. using powershell: Get-EventLog -LogName System -EntryType Error | Format-List
.
Also it would be very useful for us to try to reproduce this in a fork. Is it possible? What credentials or secrets have to be defined?
from runner-images.
@vpolikarpov-akvelon not familiar with OOM
term, but can look at adding a step. Only credential is to push packages to the feed when integrating into main
, so nothing should be needed in the standard case of just running the workflow.
I was able to run into the issue locally in release mode in one case, so there may be something going on with .NET Native, investigating a bit more on my side too. Or that's an orthogonal issue...
from runner-images.
We've been tracking down other known halts to our test process, but here was definitely an interim case where the build succeeded eventually on this run: https://github.com/CommunityToolkit/Windows/actions/runs/5684332115/job/15413161632?pr=157#step:18:895
Where it failed previously here:
https://github.com/CommunityToolkit/Windows/actions/runs/5684332115/job/15412342108?pr=157#step:18:894
from runner-images.
I tried collecting crash dump. Test suite fails on task FrameworkElementExtension_RelativeAncestor_FreePageNavigation
(link) due to an exception with text Security check failure or stack buffer overrun
that occurs within Microsoft.ui.xaml.dll
. It doesn't seem like something is wrong with runner image, but with some software.
If you want to inspect dump yourself, you should add a couple of steps to your workflow. Add this after checkout
to enable dumps collecting:
- name: Enable User-Mode Dumps collecting
shell: powershell
run: |
New-Item '${{ github.workspace }}\CrashDumps' -Type Directory
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps' -Name 'DumpFolder' -Type ExpandString -Value '${{ github.workspace }}\CrashDumps'
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps' -Name 'DumpCount' -Type DWord -Value '10'
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps' -Name 'DumpType' -Type DWord -Value '2'
And this at the end of the workflow to upload created dumps as an artifacts:
- name: Artifact - CrashDumps
uses: actions/upload-artifact@v3
if: ${{ always() }}
with:
name: CrashDumps-${{ matrix.platform }}
path: './CrashDumps'
from runner-images.
Hey @michael-hawker. Do you have any news on this issue? Do you experience these problems still? Do you have any new information that can help with investigation as we are in a dead end currently.
from runner-images.
Thanks for checking in @vpolikarpov-akvelon we're still noticing issues. I've added the crash dump collection to our CI in one of my PRs currently facing issues, though I'm having trouble digging into the UWP minidump as my .NETCore CLR version doesn't match the one of the CLI, so I'm not sure how to install/resolve having the right one to dig deeper - any suggestions? Seems like the dump used Microsoft.NET.CoreRuntime.2.2_2.2.29713.2_x64__8wekyb3d8bbwe
, but I have Microsoft.NET.CoreRuntime.2.2_2.2.31331.1_x64__8wekyb3d8bbwe
. I wanted to compare the exceptions across runtimes. (Though the WindowsAppSDK highlighted a potential issue with the test in that PR, so I've fixed that and am re-running to recollect a clean setup again.)
I added a microsoft/vstest#2952 (comment) to hopefully poke that along about the test process failing more gracefully and providing more information about failures from the test process itself when it does crash.
I also raised the issue with the platform team since the error you pointed out seemed to come from within the system's UI dll, but it didn't seem familiar to them, so will need more investigation. Hoping if I get two dumps failing in the same place that it'll help to pass them along, though I'd like to crack the managed stack open on the UWP one first.
from runner-images.
Was able to get dumps/stacks again on both WinUI2/3 pipelines even after ignoring that test (not sure how executing test was gotten from stack though before), however I think in a different test based on last passed test gets the same overflow message in the IScrollViewer
, from DispatcherQueueHelper_FuncOfTaskOfT_Exception_DispatcherQueueExtensionTests_Test
in this run - WinUI 2 Dump - WinUI 3 Dump
The test here doesn't really do much that the test that passed previously does, I think it just happens to be the one where whatever buffer happens to overflow is based on the number of run tests? Considering it's the same stack and exception we saw earlier.
Do builds with merges to main
receive more resources to the runner than those in a PR? As we do see tend to see this issue more in PRs than we do with merges to main
(though we see it there randomly still too). Like I don't think there's anything a particular test is doing here, it seems more related to either something we're manipulating on the test app harness or bubbling up from the platform... I'll pass these new stacks to the platform team.
from runner-images.
There is no any difference between runners that handle on-push and on-pr builds.
If you suspect that failures depend on the amount of resources available, then you may try running your build on larger runners.
from runner-images.
@vpolikarpov-akvelon good to know. Yes, we've been waiting a while for the .NET Foundation to be approved for larger runners so we can try that. FYI @ChrisSfanos
from runner-images.
@ChrisSfanos I believe we should already be connected to that? I'll ping you offline so we can follow-up.
from runner-images.
Hey @michael-hawker. Do you have any updates on this? Did you test your workflow on larger runners?
from runner-images.
Well, I'm closing this issue for now due to inactivity. Feel free to contact our team using internal channel (e.g. Teams) if you still need help with this issue.
from runner-images.
@vpolikarpov-akvelon was just about to respond, the larger runners didn't help (though it took longer to see it hit again, but it did just happen this morning since we flipped over to them). Happened in another random test (a string converter test). Really seems like a hiccup in the test process or platform or something. I'll follow-up with the platform team and the dumps I provided them.
from runner-images.
Related Issues (20)
- Degrade cocoapod version to 1.14 HOT 2
- Workflow Failures Due to Inconsistent macOS Version Selection on macos-13 Runner HOT 1
- Fastlane broken on macos-14 HOT 6
- Update to Android NDK r26c HOT 1
- android emulator issue in windows 2019 and 2022 runner HOT 4
- `No space left on device` regression on `ubuntu-latest` since February 5th HOT 4
- Different screen sizes in one workflow HOT 3
- macOS runners: Connection refused by host '127.0.0.1' port '8086' HOT 4
- Docker runc, BuildKit, and Moby CVE not fixed in Ubuntu runner image HOT 3
- Update Cocoapods HOT 1
- Java cannot find certification path HOT 15
- CmdLine@2 script: 'dotnet workload install maui maui-ios' installs old versions of SDKs using vmImage: macos-13 HOT 2
- [Windows 2022] CodeQL install fails HOT 6
- Update/Add [tool name] HOT 1
- Update CMake HOT 3
- Update/Add [Strawberry Perl] HOT 1
- windows-latest: 7-Zip not found HOT 1
- Large macOS runners are pointing to unpaid versions of macOS runners HOT 3
- Update on LCM for Xcode15 HOT 1
- [windows-latest] Failing Chocolatey install of WinSCP HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from runner-images.