Always 0%

[fastlane-community]

Always 0%

[thelvis4]

After some investigations, here is what I've noticed:

• Running tests from Xcode IDE differs from running tests using xcodebuild:

Comparing Logs/Test/XXX/Session-<some_uid>.log:

Xcode:

Launching with Xcode.DebuggerFoundation.Launcher.LLDB
...
"wait_for_debugger" = 1;

xcodebuild:

Launching with Xcode.IDEFoundation.Launcher.PosixSpawn
...
 "wait_for_debugger" = 0;

• LauncherIdentifier depends on Debug Executable flag (in Xcode scheme):

Debug Executable enabled:

selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"

Debug Executable disabled:

selectedDebuggerIdentifier = ""
selectedLauncherIdentifier = "Xcode.IDEFoundation.Launcher.PosixSpawn"

So, when running tests using xcodebuild, it behaves as when Debug Executable is disabled, no matter what the actual state of Debug Executable flag is.

• When Debug Executable flag is disabled, Unit tests give correct coverage data while UI Tests do not.

Running Unit tests with Xcode.IDEFoundation.Launcher.PosixSpawn Launcher Identifier (similar with having Debug Executable flag disabled) gives proper coverage data. Running UI Tests with these configurations gives invalid coverage data (0% for Target application).
I suppose the differences come from different mechanisms used for starting UI Tests and Unit Tests.
Screenshot from a WWDC '16 presentation
WWDC '16 Advanced Testing and Continuous Integration video

P.S. If we could figure out how to run UI Tests using Xcode.DebuggerFoundation.Launcher.LLDB Launcher Identifier, I think we would get valid coverage data. So far, I couldn't find any info about how to do this.

[Vkt0r]

@iwllyu Any update in this?? I'm running Xcode 9 and after ran all the tests the UI Tests are missed from the xcoverage report. Xcov is only reporting the unit tests.

[thelvis4]

@neonichu told me that the ticket is under investigation. 🤞

[JohnCoates]

I looked into fixing this. Found the difference in logs that @thelvis4 did. Figured I'd see if there was an undocumented flag I could pass to xcodebuild to use the LLDB launcher instead of PosixSpawn. Decompiling didn't find anything, so I decided to force it myself to see if it would work. Modifying xcode build to use LLDB instead of PosixSpawn indeed works, but its fragility won't be an ideal solution to implement into most projects. Most of us are probably stuck waiting for Apple to implement a fix, so file a Radar!

Proof of Concept commands:

# we'll run xcodebuild through a debugger so we can alter its execution
lldb

# set our debugging target
target create /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild

# launch process with arguments and stop at entry point
process launch -s -- -sdk iphonesimulator -destination 'name=iPhone SE' test

# set a breakpoint on an argument where we can alter the process launcher & debugger
# the method won't be loaded into the process yet, so don't worry about the warning lldb gives, the breakpoint will be resolved automatically as soon as the method is available
breakpoint set -F "+[IDELaunchParametersSnapshot launchParametersWithSchemeIdentifier:launcherIdentifier:debuggerIdentifier:launchStyle:runnableLocation:debugProcessAsUID:workingDirectory:commandLineArgs:environmentVariables:architecture:platformIdentifier:buildConfiguration:buildableProduct:deviceAppDataPackage:allowLocationSimulation:locationScenarioReference:showNonLocalizedStrings:language:region:routingCoverageFileReference:enableGPUFrameCaptureMode:enableGPUValidationMode:debugXPCServices:debugAppExtensions:internalIOSLaunchStyle:internalIOSSubstitutionApp:launchAutomaticallySubstyle:]"

# now let's add some commands to run when the breakpoint hits
# we'll replace the launcher & debugger arguments passed to the function and then continue execution
break command add
po $rcx = (unsigned long)IDEDefaultLauncherIdentifier
po $r8 = (unsigned long)IDEDefaultDebuggerIdentifier
continue
DONE

# resume execution from the entry point. coverage should be correct now
continue

Proof of concept code for injection as a library: https://gist.github.com/JohnCoates/605319ab31c0aa05c77973481aeae4b2

[vgorloff]

Xcode Version 9.3 beta (9Q98q) seems resolves issue with missed UI Tests coverage as long as Debug executable not enabled in Test schema. Details: https://stackoverflow.com/a/48477190/1418981

[nakiostudio]

Hi @redok016 👋

I'm looking at the problem, I'll try to get a fix out soon.

Thanks @Streel88 for the additional information 👍

[nakiostudio]

Great stuff @thelvis4 👍 I've been digging a bit into it and there is very little information. I will test a couple of things during the weekend though.

Thanks!

[iwllyu]

Oops we may have jumped the gun.

iwllyu/UITestCoverageBug#4 (comment)

tl;dr: it looks like both xcode and xcodebuild are not covering UI tests. xcode could have erroneously reporting coverage before and Apple intends for UI tests not to be part of code coverage.

or it's a bug

Well, at least they're consistent now

[mathcarignani]

I have the same issue with my UI tests, when I run scan (use xcodebuild) I don't have the coverage from those tests

[redok016]

@thelvis4,@nakiostudio thanks for your great effort

[iwllyu]

https://github.com/coryalder is reporting that this is fixed in XCode9

iwllyu/UITestCoverageBug#4

[stevepeak]

At Codecov we have resolved this issue with the help of customers. It appears that xcpretty corrupted the coverage reports. https://github.com/codecov/example-swift/issues/14#issuecomment-249427948

Hope this is related and find it useful. Thank you!

[iwllyu]

I created a project that demonstrates this bug

https://github.com/iwllyu/UITestCoverageBug

If you run the tests via XCode IDE you get 80% of ViewController.swift coverage. If you run the tests via xcodebuild CLI you only get 40% coverage for ViewController.swift.

This was done with Xcode 7.3.1 - but I read that it's the same for XCode 8. I can test tonight when I get on my home machine. I'll file a radar if I confirm it on XCode 8

I filed a radar and also submitted to open rader here - https://openradar.appspot.com/29033348

not sure what you guys mean by "dupe", hopefully this is what you meant!

[raven]

Xcode Version 9.3 beta (9Q98q) seems resolves issue with missed UI Tests coverage as long as Debug executable not enabled in Test schema. Details: https://stackoverflow.com/a/48477190/1418981

I've verified this is indeed working again 🎉

[raven]

@thelvis4 have you filed a radar? Just wanting to dupe it if you have one already :)
Also, have you had a chance to check if the behaviour is different with Xcode 8?

[nakiostudio]

Hi @stevepeak 👋

Thanks for the heads up, however the issue we are facing is pretty different. xcov processes swift 3 coverage reports without any problems. In addition, xcov works directly with .xccoverage files therefore it has nothing to do with xcpretty 😞

Thank you once again and keep up the good work with Codecov!

[iwllyu]

I got this on on 12/16/2016 @ 3:17AM

Engineering has determined that your bug report (29033348) is a duplicate of another issue (28900146) and will be closed.

The open or closed status of the original bug report your issue was duplicated to appears in the yellow "Duplicate of XXXXXXXX" section of the bug reporter user interface. This section appears near the top of the right column's bug detail view just under the bug number, title, state, product and rank.

An example of the duplicate section from the bug reporter user interface with your bug and the duplicate bug info is included below:

29033348 Test Coverage reports from xcodebuild CLI ignore coverage from UI tests

State: Closed Product:
Rank: No Value

Duplicate of 28900146 (Open)

[asos-petergoldsmith]

OMG

[Streel88]

So I found a difference between using xcodebuild and the IDE

So I get a correctly generated report if I run the test with the Xcode IDE, but if I use xcodebuild it produces nothing, what params need to be passed in with xcodebuild?

My current command is as follows
xcodebuild clean test -project ./AutoTest.xcodeproj/ -scheme AutoTest -destination 'platform=iOS Simulator,name=iPhone 5s,OS=9.3' -enableCodeCoverage YES | xcpretty --report junit --report html

Update:
When I edit the scheme and disable Debug Executable, it fails to generate a coverage report

+-----------------------------+-----------------------------+
| Summary for xcov 0.7 |
+-----------------------------+-----------------------------+
| scheme | AutoTest |
| project | ./AutoTest.xcodeproj |
| output_directory | ./build/reports/xcov_output |
| minimum_coverage_percentage | 0.0 |
| ignore_file_path | ./.xcovignore |
| include_test_targets | false |
| skip_slack | false |
+-----------------------------+-----------------------------+

[16:34:48]: $ /Library/Ruby/Gems/2.0.0/gems/xcov-0.7/lib/xcov-core/bin/xcov-core -s /Users/christopher.streel/Library/Developer/Xcode/DerivedData/AutoTest-ayuflxopsdabfycvkjxznehpvvip/Logs/Test/6914B9BE-D82A-46AB-BE66-1C95EC821CF8.xccoverage -o /var/folders/wz/p18ghh6561v49_s7mgdsvhg1lyr51y/T/report.json20160419-51816-vuau4
[16:34:48]: ▸ Loading...
[16:34:48]: ▸ ------ xCov-Core ------
[16:34:48]: ▸ Opening .xccoverage file at path: /Users/christopher.streel/Library/Developer/Xcode/DerivedData/AutoTest-ayuflxopsdabfycvkjxznehpvvip/Logs/Test/6914B9BE-D82A-46AB-BE66-1C95EC821CF8.xccoverage
[16:34:48]: ▸ Parsing .xccoverage file...
[16:34:48]: ▸ File successfully parsed
[16:34:48]: ▸ Serializing coverage report...
[16:34:48]: ▸ Report successfully serialized
[16:34:48]: ▸ Writing report on disk...
[16:34:48]: ▸ Coverage report successfully created at path: /var/folders/wz/p18ghh6561v49_s7mgdsvhg1lyr51y/T/report.json20160419-51816-vuau4
+--------------+-----------+
| xcov Coverage Report |
+--------------+-----------+
| AutoTest.app | 0% |
+--------------+-----------+

[nakiostudio]

Hi @Streel88 👋

Try adding the build argument to your xcodebuild command. That should be the only difference between running the tests from Xcode and running them from the console. I hope it helps!

[Streel88]

Hi @nakiostudio

I tried that and still no luck, to see if you can replicate my issue

1. Ensure your running Xcode 7.3
2. Purge your Derived Data, either manually or via Xcode
3. cd into your Projects directory
4. Execute the following command: "xcodebuild build test -project ./Project.xcodeproj/ -scheme Project -destination 'platform=iOS Simulator,name=iPhone 5s,OS=9.3' -enableCodeCoverage YES
5. Execute the next command xcov -s Project -p ./Project.xcodeproj/ -o ./build/reports/xcov_output

You should see the output as 0%

Note: I am seeing the exact same behaviour with Slather

Update:
You can see the exact same behaviour with the IDE if your disable Debugging

1. Edit Scheme
2. Disable Debugger (Test > Uncheck Debug Executable)
3. Run Test with IDE
4. Goto Report Navigator (cmd+8)
5. View Code Coverage Report of the Test

Disable Debugging

No Code Coverage in IDE after disabling Debug Executable

[nakiostudio]

Hi again @Streel88 👋

You are absolutely right, the Debug Executable flag must be enabled in order to get the coverage information. Could you please try adding the -configuration "Debug" argument to your command?

[Streel88]

Still no luck all that does is change the build configuration, from what I've learned about it is one of the executables it produces under ../CodeCoverage/.. needs to be executed which it doesn't seem to do with xcodebuild.

I will keep looking into it further to see if I can't figure out the correct command

[nakiostudio]

Honestly, I have no idea what's wrong... if you can publish a sample project with this problem so I can clone and test it. I haven't been able to repro this on my machine.

[Streel88]

Sorry for the delayed reply, here is the dummy Project I put together to test
AutoTest.zip

Scenarios

1. Test with IDE, code completion enabled. Run xcov -> Success
2. Test with IDE, code completion enabled, Debug Disabled. Run xcov -> Fail (But expected since IDE also shows no coverage)
3. Clean/Delete DerivedData Folder, Test with xcodebuild, Run xcov -> Fail

Commands
xcodebuild
xcodebuild build test -project ./AutoTest.xcodeproj/ -scheme AutoTest -destination 'platform=iOS Simulator,name=iPhone 5s,OS=9.3' -enableCodeCoverage YES
xcov
xcov -s AutoTest -p ./AutoTest.xcodeproj/ -o ./xcov_output

[redok016]

I have the same issue. Is there any updated information related with this?

[triage]

I had this issue till recently. We have a mostly objective-c project that we've only recently started adding swift to. All of my tests (mostly obj-c) would run, yet it appeared that only swift files were used in the computation of coverage reports -- leading to low percentages like 7%.

Turned on this build setting and now my objective-c files are included in coverage reports properly ... which was not a part of the readme instructions for xcov:

[thelvis4]

@raven Sorry, haven't had the time to further investigate (with Xcode 8) and to file a radar. But I would be glad to dupe the radar if you could file one. I believe we should check this on Xcode 8, as I doubt they will release another version of Xcode 7.

[thelvis4]

@iwllyu Thank you for submitting the radar and putting it on open radar as well. 👍
I'm going to duplicate it.
btw, we use "dupe" as a short form of "duplicate".

[nakiostudio]

Thank you @iwllyu! I will take a look this weekend. Hopefully we will come up with a final solution for this issue.

[EliseLeb]

Has a solution been found?

[michaellady]

@iwllyu thank you for helping me retain my sanity!

[thelvis4]

@EliseLeb This is definitely an inconsistency (bug) in Apple's tools. And it's still reproducible with Xcode 8.2.

Xcode IDE and xcodebuild use different UITests launchers (and debuggers):

• Xcode IDE uses this plugin Xcode.app/Contents/PlugIns/DebuggerLLDB.ideplugin
  whereas
• xcodebuild uses IDEFoundation framework Xcode.app/Contents/Frameworks/IDEFoundation.framework (and does't attach any debugger)

I couldn't make (force) xcodebuild to take Xcode.DebuggerFoundation.Launcher.LLDB as a launcher, although I tried half of a day in different undocumented (and probably stupid) ways. Probably it's an implementation detail and cannot be influenced externally.

At this point, I will wait for Apple to fix this bug/inconsistency. BTW, my radar was also marked as Duplicate of 28900146 (Open).

[EliseLeb]

@thelvis4 Thanks for the update! I also filed a bug with apple but not holding my breath. What a bummer for UI tests.

[earltedly]

I've filed a radar too - thanks everyone in this thread for saving me a few extra hours working out what was going on! Has anyone tried in the latest beta yet?

[earltedly]

@thelvis4 @neonichu it also might relate to rdar://problem/28571841 which I filed about UI tests always reporting 0% coverage for linked frameworks. There's a simple example project attached.

[michaellady]

Anyone try this on 8.3.2 yet?

[iwllyu]

@michaellady you're welcome to clone the project here https://github.com/iwllyu/UITestCoverageBug and try it out on 8.3.2

[michaellady]

Hi @iwllyu, I'm still seeing the command line issue on 8.3.2

[iwllyu]

thanks, I've updated the readme in that project

[michaellady]

Woot!

…

On Tue, Jun 13, 2017 at 12:09 PM iwllyu ***@***.***> wrote: https://github.com/coryalder is reporting that this is fixed in XCode9 iwllyu/UITestCoverageBug#4 <iwllyu/UITestCoverageBug#4> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA5JKml8y7RqQN81LL9qN3_YHjEWV16Xks5sDt6AgaJpZM4IKWuQ> .

[nakiostudio]

Amazing work by the dev tools team in this release 👏

[stephen-oconnor-cp]

Sorry to comment on this after so long, but this is the only issue that relates to the problem I'm having.

Not even unit tests are running right now. If I run them in Xcode, it generates coverage and does everything as expected. If it's run via run_tests / scan, it doesn't appear that unit tests are even running, even though it has output like this:

[10:13:55]: Running Tests: ▸ Touching CameraDummyTests.xctest (in target 'CameraDummyTests' from project 'CameraDummy')
[10:13:55]: ▸ Processing empty-BuildPipeSandbox.plist
[10:13:55]: ▸ Processing empty-BuildPipeSandboxTests.plist
[10:13:56]: ▸ Linking BuildPipeSandboxTests
[10:13:56]: Running Tests: ▸ Touching BuildPipeSandboxTests.xctest (in target 'BuildPipeSandboxTests' from project 'BuildPipeSandbox')
[10:13:56]: Running Tests: ▸ Touching BuildPipeSandbox.app (in target 'BuildPipeSandbox' from project 'BuildPipeSandbox')
[10:13:56]: ▸ Linking BuildPipeSandboxUITests
[10:13:56]: ▸ Processing empty-BuildPipeSandboxUITests.plist
[10:13:56]: ▸ Copying Info.plist
[10:13:56]: Running Tests: ▸ Touching BuildPipeSandboxUITests.xctest (in target 'BuildPipeSandboxUITests' from project 'BuildPipeSandbox')
[10:13:56]: ▸ Test build Succeeded
[10:13:56]: Copying .xctestrun file

Can anybody help with this?

[michaellady]

This sounds like you should open a new issue if this was closed that long ago. I don’t even do mobile devops anymore, lol.

…

On Fri, Jan 26, 2024 at 1:20 AM stephen-oconnor-cp ***@***.***> wrote: Sorry to comment on this after so long, but this is the only issue that relates to the problem I'm having. Not even unit tests are running right now. If I run them in Xcode, it generates coverage and does everything as expected. If it's run via run_tests / scan, it doesn't appear that unit tests are even running, even though it has output like this: [10:13:55]: Running Tests: ▸ Touching CameraDummyTests.xctest (in target 'CameraDummyTests' from project 'CameraDummy') [10:13:55]: ▸ Processing empty-BuildPipeSandbox.plist [10:13:55]: ▸ Processing empty-BuildPipeSandboxTests.plist [10:13:56]: ▸ Linking BuildPipeSandboxTests [10:13:56]: Running Tests: ▸ Touching BuildPipeSandboxTests.xctest (in target 'BuildPipeSandboxTests' from project 'BuildPipeSandbox') [10:13:56]: Running Tests: ▸ Touching BuildPipeSandbox.app (in target 'BuildPipeSandbox' from project 'BuildPipeSandbox') [10:13:56]: ▸ Linking BuildPipeSandboxUITests [10:13:56]: ▸ Processing empty-BuildPipeSandboxUITests.plist [10:13:56]: ▸ Copying Info.plist [10:13:56]: Running Tests: ▸ Touching BuildPipeSandboxUITests.xctest (in target 'BuildPipeSandboxUITests' from project 'BuildPipeSandbox') [10:13:56]: ▸ Test build Succeeded [10:13:56]: Copying .xctestrun file Can anybody help with this? — Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHESKX33DYIMN437XYJUI3YQNYPFAVCNFSM4CBJNOIKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJRGE3TENRZHE4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>