fhir-crucible/crucible

🔥 Crucible web application for testing FHIR servers

crucible's Issues

Tasks container doesn't work out of box in docker.

Help me, please. I run it using docker for windows at my developing machine.

I think something wrong with chmod options.

This line in delayed_job.rb blows up container.

Delayed::Worker.logger = Logger.new(Rails.root.join('log', 'delayed_job.log'))

Also there is an error in log:

tasks_1  | Rails Error: Unable to access log file.
Please ensure that /crucible/log/development.log exists and is writable (ie, make it writable for user and group: chmod 0664 /crucible/log/development.log).
The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.

I tried to fix it by adding at docker file end this line:

RUN /bin/bash -c "cd /crucible/log ; ls -la; chmod 777 *; ls -la"

And recive this output:

Step 10/10 : RUN /bin/bash -c "cd /crucible/log ; ls -la; chmod 777 development.log; ls -la"
 ---> Running in dd3569d2c413
total 1560
drwxrwxrwx 2 root root    4096 Jan 23 18:40 .
drwxr-xr-x 1 root root    4096 Jan 23 19:12 ..
-rwxrwxrwx 1 root root       0 Jan 22 15:11 .keep
-rwxrwxrwx 1 root root      66 Jan 22 16:48 delayed_job.log
-rwxrwxrwx 1 root root 1581538 Jan 23 19:00 development.log
total 1560
drwxrwxrwx 1 root root    4096 Jan 23 18:40 .
drwxr-xr-x 1 root root    4096 Jan 23 19:12 ..
-rwxrwxrwx 1 root root       0 Jan 22 15:11 .keep
-rwxrwxrwx 1 root root      66 Jan 22 16:48 delayed_job.log
-rwxrwxrwx 1 root root 1581538 Jan 23 19:00 development.log
Removing intermediate container dd3569d2c413

But it didn't help.

Full error log:

tasks_1  | I, [2019-01-23T19:00:21.578263 #15]  INFO -- : TestScriptEngine.parse_testscripts: Loaded /usr/local/bundle/bundler/gems/plan_executor-68901ddabdfc/lib/tests/testscripts/scripts/connectathon/Patient-02-Formal/FHIRClient/04-PatientHistory/connectathon-15-patient-fhirclient-04-history-xml.xml
tasks_1  | I, [2019-01-23T19:00:21.583244 #15]  INFO -- : TestScriptEngine.parse_testscripts: Loaded /usr/local/bundle/bundler/gems/plan_executor-68901ddabdfc/lib/tests/testscripts/scripts/connectathon/Patient-02-Formal/FHIRClient/04-PatientHistory/connectathon-15-patient-fhirclient-04-history-json.xml
tasks_1  | Rails Error: Unable to access log file. Please ensure that /crucible/log/development.log exists and is writable (ie, make it writable for user and group: chmod 0664 /crucible/log/development.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
mongo_1  | 2019-01-23T19:00:27.359+0000 [initandlisten] connection accepted from 172.18.0.4:60796 #2 (2 connections now open)
tasks_1  |   MOPED: 172.18.0.2:27017 COMMAND      database=admin command={:ismaster=>1} runtime: 7.6277ms
tasks_1  |   MOPED: 172.18.0.2:27017 COMMAND      database=crucible_development command={:count=>"badges", :query=>{}} runtime: 2.6655ms
mongo_1  | 2019-01-23T19:00:37.376+0000 [PeriodicTaskRunner] task: DBConnectionPool-cleaner took: 4ms
mongo_1  | 2019-01-23T19:00:37.437+0000 [clientcursormon] mem (MB) res:20 virt:503
mongo_1  | 2019-01-23T19:00:37.437+0000 [clientcursormon]  mapped (incl journal view):320
mongo_1  | 2019-01-23T19:00:37.437+0000 [clientcursormon]  connections:2
tasks_1  | rake aborted!
tasks_1  | SystemStackError: stack level too deep
tasks_1  | /crucible/config/initializers/delayed_job.rb:1:in `new'
tasks_1  | /crucible/config/initializers/delayed_job.rb:1:in `<top (required)>'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/engine.rb:652:in `block in load_config_initializer'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/engine.rb:651:in `load_config_initializer'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/engine.rb:615:in `each'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/engine.rb:615:in `block in <class:Engine>'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `each'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `tsort_each_child'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
tasks_1  | /crucible/config/environment.rb:5:in `<top (required)>'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
tasks_1  | /usr/local/bundle/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/application.rb:328:in `require_environment!'
tasks_1  | /usr/local/bundle/gems/railties-4.2.10/lib/rails/application.rb:457:in `block in run_tasks_blocks'
tasks_1  | /usr/local/bundle/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
tasks_1  | /usr/local/bin/bundle:30:in `block in <main>'
tasks_1  | /usr/local/bin/bundle:22:in `<main>'
tasks_1  | Tasks: TOP => jobs:work => jobs:environment_options => environment
tasks_1  | (See full trace by running task with --trace)
mongo_1  | 2019-01-23T19:00:39.604+0000 [conn2] end connection 172.18.0.4:60796 (1 connection now open)
tasks_1  | `health-data-standards` failed to load: C-CDA export disabled.
crucible_tasks_1 exited with code 1

Remove a FHIR Server from testing?

Hi,

Is there a way from the https://projectcrucible.org website on how to remove a FHIR server from testing?

We deployed a new version of our FHIR server aligned to version 3.0.1 in a separate URL. The old one on FHIR 1.8.0 is still publicly available in its original URL but we want to remove that since https://projectcrucible.org doesn't really test with 1.8.0 anymore.

Unhandled error condition in aggregate_run

The return statement at

crucible/app/controllers/servers_controller.rb

Line 123 in 8c86c3c

return unless aggregate_run

triggers a 500 error because no template is provided. Rendering a JSON error response might make more sense?

(I don't yet understand why @erikwiffin and I hitting this code path in the first place; I'm just 30 minutes into exploring the code base.)

Supported Only Filter not working?

See https://chat.fhir.org/#narrow/stream/crucible/subject/.E2.80.9Csupported.20only.E2.80.9D.20filter

Skipped tests give no reason why

When a test is skipped, there is no indication to the user why the test was skipped, only that it was skipped. It would be nice to see a reason why a test was skipped.

Update references to point to 1.8.0

Crucible - FHIR Testing ... Test Validation and Test Verification

The following example is a single occurrence and a random deep dive on a single test case (it is representative of the types of issues being located and reported on - be advised we cannot report on or provide documented input to all the related/similar issues being identified throughout Crucible). Our report is more to provide an overall assessment and software quality review into the test "process" than this particular error submission.

For the purpose of this report we will be speaking to Crucible production release against STU3 version 1.6.

For the purpose of this report the SUT (System Under Test) could be any HL7 FHIR STU3 version 1.6 implementation with server side validation turned on (so the server is validating the exchange of messages between a client and server).

For our report we will be using the test - "X010 Condition: Create New" (found under the Clinical Resource Test Suite) (Crucible message attached as an example).

---

1. Value sets and coded values - FHIR defines Languages Value Set here (http://hl7.org/fhir/2016Sep/valueset-languages.html)
Sample Crucible message ... denotes en-US as being used.
xml version="1.0" encoding="utf-8"
Condition xmlns="http://hl7.org/fhir"
language value="en-US"

Sample of valid FHIR Client message ... according to FHIR published standards
xml version="1.0" encoding="utf-8"
Condition xmlns="http://hl7.org/fhir"
language value="en"

---

2. Value sets and coded values identifier and identifier-type (http://hl7.org/fhir/valueset-identifier-type.html) - includes coded values for display value
Sample Crucible message ... denotes random generated values as being used.
display value="LBCEub3uT+9Du/AjZ5+4Ww=="

Sample of valid FHIR Client message ... according to FHIR published standards
display value="Filler Identifier"

---

3. Value sets and coded values category (http://hl7.org/fhir/valueset-condition-category.html)
Sample Crucible message ... denotes random generated values as being used.
display value="lo7PNT44qxChlTvBu8rmEA=="

Sample of valid FHIR Client message ... according to FHIR published standards
display value="Finding"

---

4. Value sets and coded values severity (http://snomed.info/sct)
Sample Crucible message ... denotes random generated values as being used.
display value="Xl7yj5HCtO+8/O7tW+/B4g=="

Sample of valid FHIR Client message ... according to FHIR published standards
display value="Mild"

---

5. Value sets and coded values onsetRange (http://hl7.org/fhir/2016Sep/datatypes.html#Range)
Sample Crucible message ... denotes random generated values as being used.
unit value="TdXRLbDXmoEdj58YZag4Nw=="

Sample of valid FHIR Client message ... according to FHIR published standards
unit value="drams"

---

6. Complex but related data values such as abatementRange (low and high values)
e.g. the low value is higher than the high value (fails simple validation rules)
Sample Crucible message ... denotes random generated values as being used.
low value value="0.14087090599202667"
high value value="0.0935924878064125"

A common adage among Testers is our need to ensure we "Test-the-Test" - (ISO refers to this as Test Calibration) as a custodian of quality assurance reporting - "we" as the testing community are held to higher degree and a level of quality and assurances above all others. Reporting either a false-positive or a false-negative can have dire consequences.

ConditionCreate_Original.txt

TS Connectathon 15 search tests fail when Bundle is returned

TS Connectathon 15 tests fail when a Bundle of type searchset is returned. For example, the test TS Connectathon 15 Location Locate Endpoint Json expects a Location as response type. But a search always returns a Bundle of type searchset, even if the number of found instances is only 1.

Crucible is using "GET [base]/[resource]?_search= " but this type of search is not documented in the FHIR specification for DSTU2

The specification (http://hl7.org/fhir/http.html#search) says we should support search using "_search" only for POST.

I am not sure if this issue should be added on this project or the "plan_executor" one. Please let me know if I should change the project.

Unable to complete CentOS 7 installation

Hi,

I followed the instructions you have for installing Crucible on CentOS 7. Everything looked good until the part where I have to open Firefox and navigate to port 80 (i.e. http://localhost). At this point, Firefox attempts to redirect me to https://localhost and then an "Unable to connect" message is shown. I'm stuck at this point.

Any idea what's wrong? Or logs I should look check or things I should revisit to make sure the installation completes?

BTW, I'm running CentOS on VirtualBox and I'm using Windows 7.

Make scores for partial runs clearer

If some user runs a partial test on tests that happen to fail, a server could get a very low score and keep that until someone else runs a complete test (or the opposite; run only a subset that succeeds).
For novice users it is very unclear what the score is based on, plus it is used for the ranking on the homepage without any weight on the nr. of tests run.

Suggestions:

• ranking on the homepage only on full test runs
• separate list for partial runs, mentioning the % of tests that were run
• add clear explanation of the scores, and make it stand out.

POST search tests send incorrect Content-Type header

All of the POST search tests send the request message with a Content-Type HTTP Header value of a FHIR mime-type; e.g. "application/fhir+xml". According to the FHIR specification for the search operation, http://hl7.org/fhir/http.html#search, the definition of the POST search is that the Content-Type will support HTML forms which is defined as "application/x-www-form-urlencoded".

It also appears that Crucible is not sending any payload with the POST search tests. In that case I would expect the Content-Type to not be sent.

FHIR servers that only allow for the HTML form mime-type, "application/x-www-form-urlencoded", for the POST search operation should send back an HTTP response code of 415 "Unsupported Media Type" which is the correct behavior but will fail these tests.

Test result permalink broken

Errors when launching crucible app and trying to add first server for testing

I am trying to set up the crucible app in order to test a FHIR server implementation for conformance. However, on Windows and CentOS after starting the app and putting in a URL the server logs the following error message from the URL (as found in the logs)

http://localhost:3000/servers/568ed8fde13823084c000001/aggregate_run?only_failures=true

Missing template servers/aggregate_run, application/aggregate_run with {:locale=>[:en], :formats=>[:json], :variants=>[], :handlers=>[:erb, :builder, :raw, :ruby, :coffee, :jbuilder]}. Searched in: * "/home/zmecklai/apps/crucible/crucible/app/views" * "/home/zmecklai/.rvm/gems/ruby-2.3.0/gems/magic_lamp-1.5.2/app/views" * "/home/zmecklai/.rvm/gems/ruby-2.3.0/gems/teaspoon-1.1.1/app/views" * "/home/zmecklai/.rvm/gems/ruby-2.3.0/gems/devise-3.4.0/app/views"

The app was launched like this:
bundle exec rails server -b0.0.0.0

Here's another statement of the same issue as reported by the app

Mongoid::Errors::DocumentNotFound in ServersController#aggregate_run
Problem: Document(s) not found for class Server with id(s) 568ed8fde13823084c000001. Summary: When calling Server.find with an id or array of ids, each parameter must match a document in the database or this error will be raised. The search was for the id(s): 568ed8fde13823084c000001 ... (1 total) and the following ids were not found: 568ed8fde13823084c000001. Resolution: Search for an id that is in the database or set the Mongoid.raise_not_found_error configuration option to false, which will cause a nil to be returned instead of raising this error when searching for a single id, or only the matched documents when searching for multiples.

Extracted source (around line #106):
104
105  def aggregate_run
106      server = Server.find(params[:server_id])
107    aggregate_run = server.aggregate_run      
108    return unless aggregate_run
109    if (params[:only_failures])

Rails.root: /home/zmecklai/apps/crucible/crucible

Application Trace | Framework Trace | Full Trace
app/controllers/servers_controller.rb:106:in `aggregate_run'
Request

Parameters:

{"only_failures"=>"true",
"format"=>:json,
"server_id"=>"568ed8fde13823084c000001"}

Trailing Slashes

See fhir-crucible/fhir_client#57

No way to run other tests suites after running a test suite without going back to server through front page.

When you are testing a FHIR server, running one suite at-a-time, after a suite completes there is no way to run a different suite, you can only see previous test runs. In order to run a dofferent suite, you need to go back to the main page, and then click on the target server, and then select and run the desired suite.

SUGGESTION: It might make more sense to add a new tab "Test Runs", instead of munging that under the "Tests" tab.

Instructions for Windows

I've got this project running on a Windows server and was wondering if anyone would be interested in the instructions for getting set up there. If so please let me know and I'll be happy to document what I've found.

STU 3.0.1 AdverseEvent fails due to non-standard 'kind' element

Hi Guys,

I have test case 'X010: AdverseEvent: Create New' failing due to the AdverseEvent resource crucible is trying to create has an unknown element within it. It has a 'kind' element. As far as i can tell this element is not documented in the FHIR spec. So I believe the root cause of my test case failing is that the resource Crucible is using is non-conformant.

X010: AdverseEvent: Create New
line 43 in AdverseEvent resource:
<kind value="ProblemDifferentManufacturer"/>

And great work guys with the site guys, keep up the good work.

Angus Millar
pyrohealth.net

Programmatic usage

Hi!

First off, Thanks for the awesome project!

---

I would like to suggest some enhancements based on what we're doing. We're running this in our own infrastructure, so it shouldn't place a burden on your servers.

1. An API to submit a request to test a server. Presently we're using the UI to submit the server URL and then on the next page selecting test suites.
2. An API to find out if the test is complete, and if complete provide a test result using the API.
3. Possibly a callback to some other URL submitted along with the request.

---

Does this seem reasonable?
Do you have any advice for me to start implementing this?

STU 3.0.1 ImagingManifest fails due to non-standard 'frameNumber' element

Hi Guys,

I have test case 'X010: ImagingManifest: Create New' failing due to the ImagingManifest resource crucible is trying to create has an unknown element within it. It has a 'frameNumber' element. As far as I can tell this element is not documented in the FHIR spec. So I believe the root cause of my test case failing is that the resource Crucible is using is non-conformant.

TestCase: X010: ImagingManifest: Create New
Resource Line number: 107
<frameNumber value="66"/>

Ubuntu installation instructions need new /etc/init.d/delayed-job script so that can be made to auto-start.

When you attempt to make the current script auto-startable (sudo update-rc.d delayed-job defaults) you get this warning, and the service will fail to start upon reboot:
update-rc.d: warning: /etc/init.d/delayed-job missing LSB information
update-rc.d: see http://wiki.debian.org/LSBInitScripts

Incorrect matching of reference URL in tests SE05.0P and SE05.0FP

Tests SE05.0P and SE05.0FP fail, while the returned Condition does match the Patient. Could the reason be that the server contains a full url e.g. http://fire.ly/Patient/xxx while the test expects Patient/xxx?

Test results:
SE05.0P: Search condition by patient reference url (partial)
SE05.0FP: Search condition by patient reference url (full)
Message
The search returned a Condition that doesn't match the Patient.

Test link:
https://projectcrucible.org/servers/5aa9896504ebd06bbb000006#5ab3b33d04ebd078e6000002/search001/SE05.0FP

Chef cookbooks for installation

Some people have expressed an interest in building and running their own crucible server. A chef cookbook can be created to simplify the installation and deployment of this tool.

Escaping Plus Sign in Query Strings

Submitted via email:

Hi,
My server is failing test “AS4006_dateWritten: Search for MedicationOrder by dateWritten”:
    
    GET https://.../fhir/MedicationOrder?datewritten=2013-05-25T19:32:52+01:00
 
The underlying technology my server is built on automatically escapes the “+” in the query to a space, which makes the value not a valid date parameter:
    2013-05-25T19:32:52 01:00
 
Which is causing my server to error.
 
I couldn’t find anything really definitive that says a plus sign in a URL is not valid, but I found many things that say that a literal “+” in a query string is shorthand for a space character, and an actual plus sign in a query value should be URL escaped: “%2B”.  The most definitive thing I found is this “recommendation” from w3.org:
    https://www.w3.org/Addressing/URL/4_URI_Recommentations.html
 
Section “Query strings”.
 
Can the test be changed to URL escape the plus sign?

Suite "Argonaut Sprint4 Test" in group "Argonaut Phase 1" does no setup

This is one of the suites that requires Crucible to authenticate to the FHIR server using the SMART on FHIR authorization workflow. So I entered the client ID and client secret that I have for the Crucible app on my authentication server and clicked “Authorize App”. Then my auth server’s authentication page comes up, I enter some credentials, then my auth server asks me to approve the permissions the app is requesting and it asks me for a patient ID. I enter the ID of a patient resource on my FHIR server, and all the tests in this suite use that patient.

I was getting some errors in this suite because, for example, Crucible will first query for MedicationOrders linked to that patient:
GET https://argonaut.intersystems.com/fhir/MedicationOrder?patient=f001

Then it will look at those results and query for one of the MedicationOrders again by code:
GET https://argonaut.intersystems.com/fhir/MedicationOrder?code=http://snomed.info/sct%7C320442002

But if there are no MedicationOrders linked to that patient, or if there are but they don’t have a code, the test fails. I was able to pass this test and others in the suite by creating the resources that Crucible is looking for, but it seems like Crucible should be doing its own setup.

https with self signed certificate

When using https with self signed certificates we are facing issues as ssl verification is failing. Is there any way to bypass ssl verification ?

STU 3.0.1 Appointment test uses invalid resource and problem with conditional interactions

• X010: Appointment: Create New: Failed Fhir Schema Validation because Reason element in Appointment resource (https://www.hl7.org/fhir/appointment.html#resource) is supposed to be an array of codeablConcept, but data used in X010 test has reason as a single codeableConcept which causes all subsequent tests to fail.

• Even though server declares that it supports conditional interactions such as conditional create, the corresponding test cases won't be included.

UI doesn't support URLs that are IP address-based

When you attempt to provide a FHIR server URL using an IP address, the UI refuses to accept it, showing this error: "Please enter a valid URL."

WORKAROUND: create a fake domain mapping in the Crucible server's /etc/hosts file to point to the desired IP address.

Non-unique id's on server summary links

In crucible/app/views/components/_server_summary.html.erb, every anchor element is given the same id="carouselPanel". id attributes on HTML elements must be unique on a given document/page.

Test SE21P incorrectly assumes UCUM unit

Test SE21P does a quantity precision test on unit 'mmol'. The search is without system, so UCUM is assumed (as default).
However, mmol is not a formal UCUM unit. Therefore it is not possible to reliably compare the quantity or it's precision.

Test payload with AdverseEvent doesn't match with FHIR 3.0.1

Test:
ClinicalResources/Resource Test Adverse Event/AdverseEvent: Create New

Received:

{
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "error",
"code": "invalid",
"diagnostics": "{:path [:suspectEntity 0 :causality], :message "expected type of string"}"
},
{
"severity": "error",
"code": "invalid",
"diagnostics": "{:path [:event], :message "extra property"}"
},
{
"severity": "error",
"code": "invalid",
"diagnostics": "{:path [:kind], :message "extra property"}"
},
{
"severity": "error",
"code": "invalid",
"diagnostics": "{:path [:severity], :message "extra property"}"
},
{
"severity": "error",
"code": "invalid",
"diagnostics": "{:path [:resultingCondition], :message "extra property"}"
}
]
}

please use the github 'Releases' feature

For those of us who want to run our own crucible server for internal use, it would be very helpful to have git tags set for specific release versions. Using github's 'Releases' feature handles this nicely.

Tests should add a common tag to all resources they create in order to simplify cleanup

The crucible test executor has a tendency to leave behind resources like this one: http://fhirtest.uhn.ca/baseDstu3/Questionnaire/29870

These tend to be filled with blocks like the following:

 "system": "http://projectcrucible.org/T8ggfo88CsW/TCBMD0d/Ig==",
      "value": "PIuwu0RYu5Lrj2nlUBW8XA==",
      "assigner": {
        "identifier": {
          "use": "usual",
          "system": "http://projectcrucible.org/6rrAksJJK+JcULS37d4cmA==",
          "value": "cumvCmOPy0HZRA8Sb/iQSg=="
        },
        "display": "Organization Zkc4doWzpbK7eyWFan/ZLg=="
      }

Unfortunately if these don't get cleaned up by the test executor they get annoying for other users of the public server because they are filled with nonsense.

Would it be possible to have a common tag added to all resources created by the test executor in order to simplify cleanup by server owners periodically?

Test XFER0 contains invalid resource

Test XFER0: Create a Patient Record as Transaction contains a Condition as the third entry that fails validation: "Instance failed constraint con-3 "Condition.clinicalStatus SHALL be present if verificationStatus is not entered-in-error""

That also prohibits the execution of the rest of the Transaction tests.

Several tests in suite "Argonaut Resprint3 Test" getting errors

I populated my server with resources to pass the tests in the suite "Argonaut Resprint3 Test". I am able to pass the first three tests just fine, but the last four are all getting errors:

• ARS304: GET Immunization with Patient ID
  Message
  Fatal Error: undefined local variable or method `med' for #Crucible::Tests::ArgonautResprint3Test:0x0000000fad8ae0
• ARS305: GET Goals with Patient ID
  Message
  Fatal Error: undefined method `patient' for #FHIR::Goal:0x0000000c7ffd80
• ARS306: GET Goals with Patient ID and Date
  Message
  Fatal Error: undefined method `patient' for #FHIR::Goal:0x0000000e780650
• ARS307: GET Devices with Patient ID
  Message
  Fatal Error: undefined method `type' for #FHIR::Device:0x0000000f19b4c8

I think my server's responses are good, but even if they aren't, I should be getting more helpful messages than this so I can fix what's wrong on my end.

DSTU2 tests failing with STU3 content

DSTU2 tests such as the format tests CT01 are checking for STU3 content and therefore failing against an DSTU2 server

Clear out authentication tokens

There is no way in the interface to remove oauth tokens once the oauth server has been authenticated.

select/deselect all suites widget icon does not revert back to checkmark after clearing past test run in cases where it was used to select all tests

here is how it looks initially:

here is how it looks after running all tests in a suite:

and here is how it looks after clearing the just-ran past test data:

it should revert back to the checkmark icon, which is does do if only a single test is done.

"Transaction and Batch Test" tests all getting "undefined method `each' for nil:NilClass "

My server is failing all of the tests in the "Transaction and Batch Test" suite under Core Functionality and I'm not sure why. It could be a problem with Crucible.

For example for test "XFER0: Create a Patient Record as Transaction" Crucible posts a Bundle with type.value="transaction" (attached) to my endpoint. After creating each resource my server responds with a Bundle with type.value="transaction-response" (also attached). However Crucible seems to be getting an error trying to parse the response. Rather than giving a reason for the test failing, the message is just: "undefined method `each' for nil:NilClass". The same goes for every other test in that suite.

If this is a problem with my server's response, the message should be fixed to be more helpful in conveying the reason for failure. Otherwise the error should be fixed.

Also, it seems that when Crucible gets an error parsing a response, we lose the ability to inspect the request and response in the Crucible UI. Ideally this would be fixed.

transaction-request.txt
transaction-response.txt

Extremely large responses

Crucible currently does not save extremely large server responses to the database due to Mongo limitations.

However, it does not consider the size of entire test runs, where each individual test is below the size limit, but taken together they exceed the size limitations.

This line causes the error:

crucible/app/models/test_run.rb

Line 146 in de28541

self.test_results << result

I have been able to recreate this bug repeatedly. Let me know if you need a demonstration.

Forbidden: Following installation and at the end when I try to browse to server

Forbidden

You don't have permission to access / on this server.
Apache/2.4.7 (Ubuntu) Server at 192.168.40.8 Port 80

Problem during setup of "Sprinkler Search Test" suite

In the "Sprinkler Search Test" suite, Crucible does a parameter-less Patient search as a setup step. Then, whatever was the first resource returned by that search, Crucible creates a Condition that references that patient. Only the actual reference is kind of messed up. For example:

<patient id="701">
    <reference value="/Patient/701?_format=application/xml+fhir"/>
    <display value="dXmhj6xWW3qSW6N6yTvg6g=="/>
</patient>

I'm not certain that the reference value is invalid, but at the very least, it's unusual, for two reasons. Obviously the query parameter, but also because of the leading slash. Every other example I've seen of a relative URL in a reference value has been formatted as "[type]/[id]". The way this reference value is formatted is causing my server to fail several subsequent tests in this suite. For example "SE05.0P: Search condition by patient reference url (partial)":
POST https://argonaut.intersystems.com/fhir/Condition/_search?_format=application/xml+fhir&patient=Patient/701

Can the reference value be simplified to (for example) "Patient/701"?

Test "X010: ImagingManifest: Create New" payload does't match FHIR 3.0.1 resource content

Resource Test Imaging Manifest

http://hl7.org/fhir/STU3/imagingmanifest.html#resource

"diagnostics": "{:path [:study 0 :series 0 :instance 0 :frameNumber], :message "extra property"}"

Test C8T1_1B assumes profile present

Test C8T1_1B assumes the profile http://hl7.org/fhir/StructureDefinition/us-core-race to be present to validate against. But at least on Vonk this first has to be posted to the /administration/StructureDefinition endpoint. Or am I missing something?

trailing "/" in history interaction

The history interaction signatures are defined as follows:
GET [base]/[type]/[id]/_history{?[parameters]&_format=[mime-type]}
GET [base]/[type]/_history{?[parameters]&_format=[mime-type]}
GET [base]/_history{?[parameters]&_format=[mime-type]}

However, all crucible history interactions without any specific parameter or format have an extra trailing "/", e.g. [base]/EligibilityResponse/61cba955-4474-41ec-8e60-b8cbed78dabc/_history/

Could you please remove the "/" at the end? Thanks.

Legacy Crucible Tests prevent 100% conformance and success

As a long standing Crucible user, our FHIR (WildFHIR) implementation has undergone a lot of development and retesting. During this time, the Crucible project has equally undergone some change, updates and revisions.

When the Crucible Team deprecates legacy test case(s), no matter the reason. The Crucible project starburst, conformance history and conformance percentage being reported going forward include reference to those now none existent test cases. Simple navigation will show those test, while we attempt to navigate to the test themselves and there is no longer any test definition.

The result of this issue is that many long supported test systems will be prevented from ever getting to 100% success or 100% conformance. Where new systems or systems with extremely limited conformance capability can quickly get to 100%. This could mislead the viewer into believing the system with 100% is higher quality or a more capable implementation than a longer supported test system which has 95%.

Request: If the Crucible Project team deprecates any test cases (or historically any test cases previously deprecated), future conformance testing should immediately remove those deprecated tests from counting forward towards the conformance percentage.

Current work-around (not a favored solution) - delete all test history for the test system and start fresh. This is a destructive operation, and should be a means of last resort. As test history is valuable information to the standards community and the test system owners.

application/fhir+xml Accept header in JSON tests

Hi, I noticed that in all the JSON specific tests I've tried so far, DELETE requests are sent with XML (i.e. "application/fhir+xml") Accept header. Can you please change them to JSON?

In fact, even in more generic resource tests like those for clinical resources, almost all read, create, update operations are done with JSON Accept header; but only the delete operation is done with XML Accept header.

P.S. I first introduced this issue in zulip, but you can ignore it. github is more appropriate for issue reports.

R005 fails while resource does not contain narrative

Test R005 fails as requested summary narrative was not provided, but if we run the same request without _summary the original resource does not contain a narrative.

Test link:
https://projectcrucible.org/servers/5aa9896504ebd06bbb000006#5ab4f4cc04ebd05db1000004/readtest/R005

Neither of your installations work on Unix

Can you please look at updating your installation guides - there are some out of date libraries - and dependancies on Ruby 2.2.5 required

How to clear out "Recent common failures" list?

Hi,

This is a dumb question and I might be missing something, but how do I tell Crucible to clear out the "Recent common failures" list?

I saw the projectcrucible.org was updated to support FHIR 1.8.0. I reexecuted tests on our server. I navigated to the Server Summary tab to look at the failed tests. It says there that the server still has "Unknown resource type MedicationOrder" error. As far as I know, this resource existed in FHIR 1.6.0 but was removed in later FHIR versions.

I wanted to clear out this error as it doesn't apply anymore to the latest FHIR version.

Best Regards,