lombiq/ui-testing-toolbox

Web UI testing toolbox for Orchard Core applications with Selenium. It discovers so many issues that you'll get impostor syndrome.

Lombiq UI Testing Toolbox for Orchard Core

About

Web UI testing toolbox mostly for Orchard Core applications. Everything you need to do UI testing with Selenium for an Orchard app is here.

Highlights:

• Builds on proven libraries like Selenium, Atata, and xUnit. See all the tools we use here.
• Execute fully self-contained, repeatable, parallelizable automated UI tests on Orchard Core apps, running locally or remotely.
• Do cross-browser testing with all current browsers, both in normal and headless modes.
• Check the HTML structure and behavior of the app, check for errors in the Orchard logs and browser logs. Start troubleshooting from the detailed full application dumps and test logs if a test fails.
• Start tests with a setup using recipes, start with an existing Orchard Core app or take snapshots in between tests and resume from there. Use SQLite or SQL Server database snapshots (but you can also use PostgreSQL or MySQL too without snapshots, see this proof of concept).
• Use local file storage or Azure Blob Storage for Media.
• Test e-mail sending with a local SMTP server too. Everything just works.
• Built-in tests to check that basic Orchard Core features work, like login, registration, and content management. Demo video here. And a demo video about a proof of concept to add UI testing to Orchard.
• Built-in monkey testing functionality to try to break your app with random user interactions. Demo video here.
• Check for web content accessibility so people with disabilities can use your app properly too. You can also create accessibility reports for all pages.
• Check for the validity of the HTML markup either explicitly or automatically on all page changes.
• Reliability is built in, so you won't get false negatives.
• Use shortcuts for common Orchard Core operations like logging in or enabling features instead of going through the UI so you only test what you want, and it's also faster.
• Ready to use GitHub Actions workflows for CI builds and support for test grouping and error annotations in Lombiq GitHub Actions. This feature is automatically enabled if a GitHub environment is detected.
• Support for TeamCity test metadata reporting so you can see the important details and metrics of a test at a glance in a TeamCity CI/CD server.
• Visual verification testing: You can make the test fail if the look of the app changes. Demo video here.
• If your app uses a camera, a fake video capture source in Chrome is supported. Here's a demo video of the feature, and check out the docs here.
• Interactive mode for debugging the app while the test is paused. Here's a demo of the feature, and a demo video here.
• Security scanning with Zed Attack Proxy (ZAP), the world's most widely used web app security scanner, right from UI tests. See a demo video here.

See a demo video of the project here, and the Orchard Harvest 2023 conference talk about automated QA in Orchard Core here. Also, see our Testing Toolbox for similar features for lower-level tests.

Looking not just for dynamic testing but also static code analysis? Check out our .NET Analyzers project.

We at Lombiq also used this module for the following projects:

• The new City of Santa Monica website when migrating it from Orchard 1 to Orchard Core (see case study).
• The new Smithsonian Folkways Recordings website when migrating it from Orchard 1 to Orchard Core (see case study).
• The new Lombiq website when migrating it from Orchard 1 to Orchard Core (see case study).
• The new client portal for WTW (see case study).
• It also makes DotNest, the Orchard Core SaaS better.

Do you want to quickly try out this project and see it in action? Check it out in our Open-Source Orchard Core Extensions full Orchard Core solution and also see our other useful Orchard Core-related open-source projects!

Documentation

• Tutorial and samples
• Projects in the UI Testing Toolbox
• Making an Orchard Core app testable
• Creating tests
• Configuration
• Executing tests
• Fake video capture source
• Security scanning
• Troubleshooting
• Limits
• Tools we use
• Linux-specific considerations
• Version migration guide

Contributing and support

Bug reports, feature requests, comments, questions, code contributions and love letters are warmly welcome. You can send them to us via GitHub issues and pull requests. Please adhere to our open-source guidelines while doing so.

This project is developed by Lombiq Technologies. Commercial-grade support is available through Lombiq.

ui-testing-toolbox's Issues

Performance testing (OSOE-360)

Performance testing (both server and client-side) at least on the level of "this is much slower than usual".

• We could also use Chrome performance logging. (This was previously broken in Selenium but is supposed to work with the v4 WebDriver we're using; this would also allow creating HAR files.) However, Selenium advises against using it for performance testing and we'd need a more complex setup with the app deployed to a separate server. As a low-complexity alternative that can perhaps be good enough is the following: Put perf tests into their own project, set parallelizeAssembly and parallelizeTestCollections to false, so each perf test runs alone. This way, while we'll run the test on the same machine as the app, we minimize interference and even if the absolute numbers wouldn't be indicative of the prod environment, the results would be comparable with each other.
• We could also run Google Lighthouse, also from a .NET library (but probably this will be included in Atata too). A CLI is also available.

Jira issue

Support for running tests without a browser (OSOE-365)

In certain cases, we need a running app but not a browser (or at least not a full one), like for testing web APIs. For such cases, we could start tests without a browser and then use something like RestEase for APIs or ScrapySharp as a barebone browser. The browser is launched when AtataFactory.CreateDriver() runs, this is the only thing necessary to switch off. However, a browser would still sometimes be needed for the initial setup (though we could add a feature to only open a browser when the setup is run but not for the test itself, or utilize Auto Setup).

Jira issue

Enable full parallelization in samples (OSOE-361)

Enable full parallelization once xUnit v3 is out and has support for parallelization within test collections.

Until then we could implement this hack to achieve the same. Gérald Barré also created a NuGet for this. However, this doesn't seem to offer perf benefits, see: #125

Jira issue

IsUITesting() always returns false (OSOE-412)

As reported by @DemeSzabolcs.

Jira issue

Remove Edge workaround from WebDriverFactory (OSOE-347)

Once rosolko/WebDriverManager.Net#196 is implemented in WebDriverManager.Net, remove the now necessary workaround from WebDriverFactory.

Note that due to the NuGet dependency chain, this needs an upgrade to OC 1.5.

Jira issue

v3 to later migration docs clean-up (OSOE-401)

Jira issue

Fix that TestAdminAsMonkeyRecursivelyAsync() doesn't navigate anywhere by default (OSOE-419)

TestAdminAsMonkeyRecursivelyAsync() doesn't navigate away from the login page by default. Let's fix this oversight.

Jira issue

Use a version of CultureScope that doesn't have user overrides (OSOE-350)

This is related to OrchardCMS/OrchardCore#11228
The useUserOverride in CultureInfo(string Culture, bool useUserOverride) should be always false in UI tests to make them more reproducible and portable. Depending on how the above linked issue goes, we should either force default configuration or replace it with a custom implementation that uses the method suggested in the issue description.

After this, this line in Data Tables needs to be changed too.

Jira issue

Reusable tests for basic Orchard features

We're thinking of adding a couple of reusable tests (or perhaps just bases of tests but more than just Page classes) to the UI Testing Toolbox to check some basic, important features built into Orchard. E.g.:

• Setup (we have one for it but we'd need to fix this hack.
• Registration
• Login
• Basic content operations
• Turning features on or off

The point is not to test Orchard itself, but rather, make sure that the most important features of any Orchard site are not broken. It sometimes happens, for example, that even if you don't outright break content management from your own code but your code is executed when saving a content item and throws a caught but logged exception.

Ability to use automatically created LocalDB instances (OSOE-348)

SQL Server Express LocalDB is an SQL Server-compatible DB with reduced administrative overhead. Its primary advantage for Orchard development and UI testing is that it doesn't need SQL Server to be properly configured (for HTTP connections) and its service running. However, it still needs SQL Server to be installed. Thus, its utility is limited but let's evaluate whether we can add support for automatically managed LocalDB instances (just as we do with SQL Server DBs).

Jira issue

In GitHub Actions workflows, surface log message if a UI test is retried (OSOE-471)

When a UI test fails, it's retried, by default two times. Thus, the test may succeed even if it's flaky. Let's add a warning (or if it's annoying, a notice) log message for such retries when the tests are executed in GitHub Actions, so one can be better aware.

• See how GitHubAnnotationWriter.Annotate(), we can use it to write the message. We'd need to be able to turn these on or off similarly to ErrorInTest(), see how it's used.
• We already have logging for retries here, this needs to be extended.

Jira issue

Deeper integration with xUnit (OSOE-354)

Deeper integration with xUnit via XunitContext. Also see: xunit/xunit#2457

Jira issue

Downloading the Chrome WebDriver fails with Chrome >=15 (OSOE-664)

This is because WebDriverManager.Net didn't follow up with the URL of ChromeDriver packages having a different URL starting with v15: rosolko/WebDriverManager.Net#253 The linked issue also has potential solutions. The task here is to fix our issue, in the following ways:

• Add a temporary fix first that we can get in to dev and out via NuGet ASAP, most possibly this one (here's some code that shows most possibly this in a wider context; we need to use it here). Publish a new NuGet release.
• Implement a long-term fix:
  • Evaluate using Selenium Manager instead of WebDriverManager.Net: that had this issue too (SeleniumHQ/selenium#12208), which might not be released yet.
  • Evaluate using Atata.WebDriverSetup.
  • If WebDriverManager.Net is still the best choice, then fork it and submit a fix.

Jira issue

Capture screenshot after each page load

This would also help monkey testing.

Migrate away from System.Drawing and update Microsoft.SqlServer.DacFx (OSOE-403)

v160.6296.0 of Microsoft.SqlServer.DacFx depends on System.Drawing.Common >= 6.0.0, which deliberately dropped Unix support.. This prevents us from updating Microsoft.SqlServer.DacFx because while we don't use anything from it that would require System.Drawing.Common, a lot of things related to visual verification testing uses it.

So, let's move away from System.Drawing.* types completely, and once done, update Microsoft.SqlServer.DacFx to latest.

Jira issue

Provide recommendation on how to use coverage tools (OSOE-352)

Since with UI testing, this is not trivial. Most possibly only documentation is enough.

Altcover may be a good open-source option.

Jira issue

Invalid character in artifact path (OSOE-601)

When a test fails, it seems like the browser type is included in the artifact path:

Error: Artifact path is not valid: /test/Project.Tests.UI/bin/Debug/net6.0/artifacts/20230327T100017/Project.Tests.UI.Tests.ProjectTests.VisualVerificationTests.VisualVerificationTests.VerifyProjectPage(browser: Chrome)/01 - FailureDumpPageSnapshot.mhtml. Contains the following character: Colon :

That causes an error because it includes a colon.

This most likely only happens with visual verification tests.

Here is a sample in OSOCE for this problem: https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/actions/runs/4534487655/jobs/7988726362?pr=438#step:9:12434

Log archive if the logs are deleted:
logs_8706.zip

Jira issue

UI testing randomly getting stuck in CI (OSOE-464)

See e.g. this build. The NuGetTest UI testing of OSOCE got stuck and thus timed out in 30 minutes, while all the other builds worked fine. Rerunning just that build fixed the issue. This problem started way before these recent examples and the Orchard Core 1.5 upgrade. Reruns resolve the issue every time.

Other examples unrelated to the 1.5 upgrade: here, here, here, here.

Troubleshooting this is made harder by it apparently not being possible to finish the test-dotnet action and upload artifacts when the build is canceled, see Lombiq/GitHub-Actions#77.

It seems this bug didn't solve itself in the end: Lombiq/Open-Source-Orchard-Core-Extensions#126

Jira issue

See comments for notes.

To be done:

• Check if this only happens if tests fail and are then retired, due to Task.Delay() potentially having an effect. - No, this may also happen if there are no failing tests, see here.
• Test fix for snyc-over-async in WebApplicationFactory. See:
  • Lombiq/Open-Source-Orchard-Core-Extensions#317
  • #243
  • Lombiq/Helpful-Libraries#172
• Test input redirection.
  • Lombiq/Open-Source-Orchard-Core-Extensions@206ff3e
  • Lombiq/GitHub-Actions@83f47fd
• Test disabled node reuse.
  • Lombiq/Open-Source-Orchard-Core-Extensions#273
  • Lombiq/GitHub-Actions#105
• Test ui-test-parallelism: 0 for NuGetTest builds. Note though that this is already the case for root builds using the larger runners, and it still happened there under Ubuntu.
  • Lombiq/Open-Source-Orchard-Core-Extensions#303
• Test xUnit "maxParallelThreads": -1.
  • Lombiq/Open-Source-Orchard-Core-Extensions#309
  • Lombiq/GitHub-Actions#133
• Investigate hangs with dump file, see this blog post for tips.
  • Lombiq/GitHub-Actions#215
  • Investigation: #228 (comment)
• Fix sync over async in OC RecipeHarvester
  • #263
  • Here is an open issue for this in OC: OrchardCMS/OrchardCore#10329
• Check all processes (eg. smtp4dev) stopped properly. Hunting for zombie processes...
  • It looks like the hang timeout does not help
  • Maybe we can use Wait-Process or something similar

Stuck tests outside the scope of blame-hang

Comment	OS	Runner	Build
#228 (comment)	Windows	standard	NuGet
#228 (comment), #228 (comment)	Windows	standard	root
#228 (comment)	Windows	standard	NuGet

Use a portable Chromium exe instead of a local Chrome instance

UI test crashes if the web project doesn't reference Lombiq.Tests.UI.Shortcuts (OSOE-626)

Even if it's not yet used. It throws the exception:

InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.Identity.ILookupNormalizer' while attempting to activate 'Microsoft.AspNetCore.Identity.RoleManager`1[OrchardCore.Security.IRole]'.

This dependency should be optional, and UI tests should work without it if no shortcuts are actually used.

Jira issue

Evaluate using the async Chrome driver AsyncChromeDriver (OSOE-357)

Using the async Chrome WebDriver AsyncChromeDriver. The performance advantage of this needs to be tested. Also, this is a non-standard WebDriver incompatible with everything else (including other browsers), and maintained by a single developer, as well as used by much fewer people than the synchronous one we use.

Jira issue

Support multi-tenancy (OSOE-364)

Currently, the Toolbox has some references to the Default site. If we want to test multi-tenancy, i.e. creating and launching tenants, then this needs to be extended.

• Update TenantTests too.
• Needs prefixing in GoToPageAsync() and in GoToRelativeUrlAsync().
• Needs refactoring of CreateAndEnterTenantAsync().
• Support multi-tenancy with SQL Server. The logic changing the SQL Server connection string in the appsettings.json file can be substituted with using the [Database Shells Configuration Provider](https://docs.orchardcore.net/en/latest/docs/reference/core/Shells/#database-shells-configuration-provider and changing all connection strings in a query.
• Support multi-tenancy with Azure Blob Storage. The logic in AzureBlobStorageManager can be changed to use a templated {{ ShellSettings.Name }} BasePath.
• Support multi-tenancy for snapshots.
• Refactor Lombiq.Hosting.Tenants.FeaturesGuard.Tests.UI.Extensions.TestCaseUITestContextExtensions.

Upgrade to latest Selenium.Axe and Selenium.WebDriver packages (OSOE-49)

We're one major version behind, the current version is Selenium 4. We need to update Atata to get these.

• New features and differences: https://www.browserstack.com/guide/selenium-4-features
• For Atata v2 changes see: https://github.com/atata-framework/atata/releases/tag/v2.0.0 Migration guide: https://atata.io/tutorials/migrating-to-atata-v2/
• Add Driver.ExecuteScript() and Driver.GetScreenshot() shortcuts.
• Perhaps the newer Selenium.Axe version fixes that it accepts the title of an img instead of (also) insisting on alt.
• Search the codebase for "Selenium 4" and apply fixes appropriately.
• Also update all other packages that the Toolbox uses.

Create NuGet package for Helpful Libraries

Seems UI Testing uses Helpful Libraries which is created by Lombiq, it would be nice to publish them as NuGet or GitHub packages, so everyone can run this repo without need to clone the Helpful Libraries repo

/cc @Piedone

Add fake video capture sample test (OSOE-609)

While this awesome feature has corresponding docs, for the code, it'd be better to have the code among the samples. And then we can also demo it :).

• Keep the docs page with the rest of the tips. Link it from the TOC in the Readme too.
• Mention for what kind of features this is useful (e.g. reading QR codes, real-time identity validation).

Jira issue

Offer Media management testing as part of TestBasicOrchardFeaturesAsync() too (OSOE-423)

To include more tests of basic Media file management (in addition to the setup recipe potentially referencing files), let's add a test for Media operations too.

• Test the basics of Media Library: Upload file in root (check if the file is now properly accessible), create folder, upload file to folder, delete file, delete folder. Do this with JPGs and PDFs.
• Add a new method for this to BasicOrchardFeaturesTestingUITestContextExtensions, like TestMediaOperations().
• Execute this as part of TestBasicOrchardFeaturesExceptSetupAsync() too.
• This needn't use Atata page classes like the other similar tests.

Jira issue

Test doesn't fail if the visual verification baseline image is not found but the test is retried (OSOE-591)

When visual verification testing can't find the baseline image, it'll create it, which is good. However, in conjunction with test retries, this makes those tests passing that don't have a corresponding baseline image. This should be handled somehow, to make the tests fail even on retries.

The development-time experience here should be what we have today, but when tests are executed "for real" (for example, but not necessarily during CI builds), the newly created files shouldn't be considered for retries.

An approach would be to check if the current run is in GitHub Actions, and not create baseline images then, but this ties this fix to GHA. Perhaps better would be not to create baseline images if MaxRetryCount > 0, and update the exception message to indicate when a baseline images was not created and why (since during development you shouldn't really use retries but otherwise you should).

Jira issue

Make setup compatible with any and environment-configured DB provider (OSOE-424)

OrchardCoreSetupPage.SetupOrchardCoreAsync() needs the DB provider supplied as a parameter. If it comes from the environment, setup will fail. Fix this to allow environmental DB configuration.

Furthermore, add what else may be necessary for OrchardCMS/OrchardCore#11194 to support all the DB providers, without snapshotting.

Jira issue

Handle cancellations gracefully (OSOE-362)

This xUnit bug was fixed for this but will only be part of xUnit v3. Once that's released, TestContext will contain a CancellationToken.

Hooking into the console cancel event won't work with dotnet test.

Jira issue

Migrate from `Selenium.Axe` to `TWP.Selenium.Axe.Html` (OSOE-537)

Selenium.Axe for .NET is being deprecated since Axe now provides official .NET bindings, also for Selenium, in form of Deque.AxeCore.Selenium. [TWP.Selenium.Axe.Html](https://github.com/TroyWalshProf/SeleniumAxeHtmlDotnet) provides HTML reporting for it in the same way as Selenium.Axe` did. So, let's migrate to it.

• Follow the migration instructions here. Pretty much nothing of substance should change for us, but check this.
• Accessibility checking should still work as before. Check if there are any new accessibility issues in the sample and if yes, change the configuration of the test so it doesn't fail (since it checks a page in the Blog theme, we needn't fix it).

Jira issue

Add duplicate SQL query detector (OSOE-353)

When one implements a SELECT N+1 query how it manifests is that the same query (apart from the parameters) will be executed multiple times during page load. While not every duplicated query is a problem, we could add a feature to detect such queries and fail the given test. A per-test configurable threshold would be a good start (e.g. "fail if any query is executed >3 times").

An approach would be to use the Orchard Core Mini Profiler feature for this. A better one would be to implement our similar profiling logic to tap into the OC app since that wouldn't potentially break the UI (as the Mini Profile toolbar can). Perhaps it's possible to get the info out of Mini Profiler without showing the UI.

Similarly, we could have a mechanism to detect queries that fetch too many rows but probably there are no such simple heuristics for that (otherwise, we could do the same approach and have a configurable threshold).

Common APIs for OrchardCMS/OrchardCore#12251 can help here too.

Jira issue

Retrieve admin URL prefix from AdminOptions (OSOE-659)

The admin URL prefix is currently configurable, see here. Rather, this should be retrieved from Orchard from AdminOptions.

Unless somehow TypedRoute can also access this value from a UI testing project, see: Lombiq/Helpful-Libraries#209, also pass the value to TypedRoute in TypedRouteUITestContextExtensions.

Jira issue

Migrate RestEase usage to Refit (OSOE-627)

Refit has some advantages over RestEase, so in accordance with our general team recommendation, let's move this project to Refit too.

Jira issue

Record the video of tests too (OSOE-358)

However, this seems to be mostly possible with external tools like Expression Encoder.

Jira issue

Detect visual errors (visual verification testing) (OSOE-111)

Detecting visual errors with something like Applitools, or generally detecting regressions and validation errors automatically somehow.

We could also use verification testing with Verify. Or perhaps Shouldly offers this too? Verify has support for Selenium. This may only compare HTML though (it depends on the use-case if a change in the HTML is an issue or not; sometimes we want to change the HTML without the visuals being affected, sometimes with just CSS changes the other way around). Verify.ImageSharp might be interesting for screenshots but seems to only compare metadata. Ideally, we'd offer a visual diff of screenshots.

APIs can use Verify.AspNetCore or Verify.Http too.

If using Verify or Shouldly this is what the workflow would look like:

1. Turn on visual verification testing globally or for a given test, or call an assertion method. I.e. the same how we do this for HTML validation and accessibility checks.
2. The first time you run the test you get an error and need to accept the output as the correct one.
3. Subsequently, you only get an error if the output changes. Then you have the option to again accept it if the change is expected, or fix the app.

Verification testing is not a silver bullet though: Verification testing is super simple to set up but if there's a mismatch in the approved and received output, it's sometimes hard to figure out what causes it (since it's a very high-level test). Manual assertions would still be necessary but visual verification testing can be run as another layer of checks.

Use WebApplicationFactory directly instead of running the tested app with the dotnet CLI (OSOE-46)

You can run an ASP.NET Core app directly with WebApplicationFactory, see this blog post and the corresponding code. Why?

• This can potentially make things faster.
• Reduce the app-level configuration you have to do, so no need to add the Shortcuts module, for example, or ConfigureUITesting().
• Since this would require a hard reference from the UI test project and the web app, we'd get rid of the necessity to manually build the web app.
• This would allow us to debug the app being tested too, since it would be the same as the test process.
• After this, this note should be adjusted.

See the linked PR for the beginnings of this and comments.

Note that in the linked demo stylesheets didn't load so that may be something to resolve.

This needs to happen after a .NET 6 upgrade (Lombiq/Open-Source-Orchard-Core-Extensions#30).

Making WebDriver less fragile with browser version changes (OSOE-359)

While WebDriverManager.Net now supports Chrome version selection too we can still improve on this by e.g. bundling a browser with the project instead of relying Chrome and others to be installed on the given machine (and thus taking a hard dependency on a specific Chrome version that goes away with a random auto-update).

Some alternatives:

• Use a portable Chromium exe instead (and somebody else did this from C# too). There is no NuGet package for this, however (we could create one but there might be licensing issues, or somehow download it on the first run).
• Use Selenoid to package up browsers into Docker containers.
• EO.WebBrowser is a self-contained Chromium browser engine in a NuGet package. Supposedly this can work with ChromeDriver but there might be pitfalls and it might be slower.
• Install Chrome automatically (like this unmaintained package does). This is obtrusive, however, and may not work in all environments.

Jira issue

Test should halt on error page

When the site returns an ASP.Net Core error page the test should be failed without going further. Similarly to existing HTML validation. Currently if the site returns an error page it will fail on the following steps, becuase the elements that correspond to the selectors won't exist. This creates an opaque test failure message that doesn't indicate the actual problem making it harder to diagnose.

It should identify the error page, scrape the exception message and stack trace. and put that info into the test failure.

Offer security checks (OSOE-351)

Perhaps integrate with some tool that can test a form for XSS vulnerability or something like that. Need to look around.

Jira issue

Save not just the HTML page source but also all the referenced files on a test failure (OSOE-363)

On test failure, we could save not just the HTML page source but also all the referenced files (CSS, JS, graphics...). We could possibly build an MHTML archive of the page. Atata now has this functionality, so let's use it: atata-framework/atata#698

Note that this requires an update of Atata packages to the latest.

Jira issue

Add interactive mode for debugging (OSOE-634)

An updated copy of my original post on Teams:

Back when the UI tests started the web app in a separate process, it was possible to just drop a breakpoint in your code and do stuff in the browser window interactively, because the program was still running. So you could still look around on other pages, inspect the current state of the admin dashboard, etc. It was only needed occasionally but then it was incredibly useful in debugging problems that crop up during UI tests.

Right now this is not possible. Currently the only hack where I can approximate this effect is by adding something like await Task.Delay(120000); before the line where I'd like to "break", and then I have 2 minutes to look around in another tab but that's super inconvenient.

I'd like to have the ability to switch into an interactive mode programmatically, similar to how you can switch into debug mode with Debugger.Break(). I envision a separate module that stores a mode state and exposes an MVC action for a UI to leave interactive mode and continue operation. This module would be added to the webapp just like the Shortcuts module now. And an await context.SwitchToInteractiveAsync() extension method that updates the mode, opens this action in a new tab and then waits until the state is restored, e.g. via task cancellation.

Jira issue

Allow testing Orchard 1 apps (OSOE-356)

Since non-Core ASP.NET apps are not self-hosted, this is much more complex. However, it's possible to kind of self-host them still by running IIS ourselves with IIS Hostable (Hosted?) Web Core.

• See this blog post. However, not sure where WebServer is coming from.
• The feature can be enabled from PowerShell like this.
• The corresponding native DLL is %systemdrive%\Windows\System32\inetsrv\hwebcore.dll.
• Here's a sample that interacts with the native DLL.

Jira issue

Broken Link on Readme Page

The link in this readme under the second tutorial bullet point is broken. I believe it should now point to this file.

Ability to automatically run visual verification testing on every page load (OSOE-349)

We could do this in exactly the same way as HTML validation and accessibility checking: Optionally run on every page load (or explicitly with an assertion call, which we have implemented under #166).

Note that this needs a bit more consideration than HTML validation and accessibility checking, see the comment here.

Jira issue

Use BDD testing frameworks (OSOE-355)

Use BDD testing frameworks with user-legible tests like Cucumber.

Jira issue

Add images to failure dump that fail assertions, e.g. of dimensions (OSOE-497)

When running BasicVisualVerificationTests.VerifyNavbar() all the while the .navbar-brand element has some missing or broken styles, the test will fail like this:

Error:
  Shouldly.ShouldAssertException : elementImageOriginal.Height
      should be greater than or equal to
  28
      but was
  18

See https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/actions/runs/3694863540, Annotations, 6th error for an example.

The above error is not very helpful in case you're not already deep inside the concerned area of code. It would be more helpful to have the offending element screenshots, the expected and the actual ones, inside the failure dump in order to more quickly and easily identify the problematic element on the page.

Jira issue

Randomly failing UI tests, when upgrading to OC 1.5 (OSOE-483)

In some cases, the browser navigation results in HTTP 500. In this state, there are no logs under FaliureDump folder, and looks like the web app won't start.

Related:

• #225
• #225 (comment)

Jira issue

Remove references to Internet Explorer (OSOE-663)

Internet Explorer has been dead for little over a year. Do we use [InternetExplorer] anywhere?
If not, I suggest removing related code & documentation, and marking the attribute obsolete. We wouldn't want any support obligations for a discontinued browser.

Jira issue

Custom logging instead of NLog-based one (OSOE-402)

Currently, the UI Testing Toolbox depends on NLog. In OrchardApplicationFactory we configure NLog for logging, and OrchardCoreInstance.GetLogs() we load the log files. Instead, while retaining support for the app under test to use NLog logging for its own reasons, we could have our custom in-memory logger just for testing. That'd collect log messages about errors and warnings, and we'd use that to check for log entries to fail the test.

Decoupling the project from NLog would also prevent issues like this when Orchard updates its NLog version: OrchardCMS/OrchardCore#11194 (comment)

Related: Lombiq/Open-Source-Orchard-Core-Extensions#264 (comment)

Jira issue

TestOutput.log not included in failure dump when run in GitHub Actions (OSOE-466)

When UI tests are run (and fail) in a GitHub Actions build, the usual TestOutput.log file is not included in the dump, in the folder of the test.

Jira issue