Add New Zealand endpoint

Following HISO10033 the NZ Ministry of Health is providing SNOMED CT resources, including a snowstorm endpoint https://snomednz.digital.health.nz/

Note: This endpoint will fail httr::http_error(), possibly because HEAD() leads to error due to redirection to https://snomednz.digital.health.nz/swagger-ui.html. In any case this requires adaptation of

snomed_public_endpoint_suggest()
snomedizer_options_set()

pkgcheck results - pkgcheck-slog

Checks for snomedizer (v0.3.0)

git hash: 159c6d51

✔️ Package name is available
✔️ has a 'codemeta.json' file.
✔️ has a 'contributing' file.
✔️ uses 'roxygen2'.
✔️ 'DESCRIPTION' has a URL field.
✔️ 'DESCRIPTION' has a BugReports field.
✖️ Package has no HTML vignettes
✖️ These functions do not have examples: [release_version, snomed_public_endpoint_list, snomed_public_endpoint_suggest, snomedizer_version_compatibility].
✔️ Package has continuous integration checks.
✔️ Package coverage is 97%.
✔️ R CMD check found no errors.
✔️ R CMD check found no warnings.

Important: All failing checks above must be addressed prior to proceeding

Package License: MIT + file LICENSE

api_operations_backlog

SNOWSTORM operators left to implement

Backlog spreadsheet

Next to come

/{branch}/concepts | findConcepts | api_concepts
/{branch}/concepts/{conceptId} | findConcept | api_concept
/branches | Retrieve all branches | api_all_branches
/branches/{path} | Retrieve a single branch | api_branch
/branches/{path}/children | Retrieve branch descendants | api_branch_descendants
/{branch}/descriptions | findDescriptions | api_descriptions
/version | Software build version and timestamp. | api_version
/browser/{branch}/concepts | Load concepts in the browser format. | api_browser_concept
/browser/{branch}/concepts/{conceptId} | Load a concept in the browser format. | api_browser_concepts
/browser/{branch}/concepts/{conceptId}/ancestors | findConceptAncestors | api_browser_concept_ancestors
/browser/{branch}/concepts/{conceptId}/children | findConceptChildren | api_browser_concept_children
/browser/{branch}/concepts/{conceptId}/history | View the history of a Concept. | api_browser_concept_history
/browser/{branch}/concepts/{conceptId}/parents | findConceptParents | api_browser_concept_parents
/{branch}/concepts/{conceptId}/descendants | findConceptDescendants | api_concept_descendants
/{branch}/concepts/{conceptId}/descriptions | findConceptDescriptions | api_concept_descriptions
/browser/{branch}/descriptions | Search for concept descriptions. | api_browser_concept_descriptions
/{branch}/authoring-stats | Calculate statistics for unreleased/unversioned content to be used in daily build browser. | api_authoring_stats
/{branch}/authoring-stats/changed-fully-specified-names | getChangedFSNs | api_changed_fsn
/{branch}/authoring-stats/inactivated-concepts | getInactivatedConcepts | api_inactivated_concepts
/{branch}/authoring-stats/inactivated-synonyms | getInactivatedSynonyms | api_inactivated_synonyms
/{branch}/authoring-stats/new-concepts | getNewConcepts | api_new_concepts
/{branch}/authoring-stats/new-synonyms-on-existing-concepts | getNewSynonymsOnExistingConcepts | api_new_synonyms
/{branch}/authoring-stats/reactivated-concepts | getReactivatedConcepts | api_reactivated_concepts
/{branch}/authoring-stats/reactivated-synonyms | getReactivatedSynonyms | api_reactivated_synonyms
/codesystems | Retrieve all code systems | api_all_code_systems
/codesystems/{shortName} | Retrieve a code system | api_code_system
/codesystems/{shortName}/versions | Retrieve all code system versions | api_code_system_all_versions
/browser/{branch}/concepts/bulk/{bulkChangeId} | Fetch the status of a bulk concept creation or update. | api_browser_concepts_bulk_change
/{branch}/concepts/{conceptId}/authoring-form | getConceptAuthoringForm | api_concept_authoring_form
/{branch}/concepts/{conceptId}/inbound-relationships | findConceptInboundRelationships | api_concept_inbound_relationships
/{branch}/concepts/{conceptId}/normal-form | getConceptNormalForm | api_concept_normal_form
/{branch}/concepts/{conceptId}/references | Find concepts which reference this concept in the inferred or stated form (including stated axioms). | api_concept_references
/{branch}/descriptions/semantictags | List semantic tags of all active concepts together with a count of concepts using each.| api_descriptions_semantic_tags
/{branch}/descriptions/{descriptionId} | fetchDescription | api_description
/mrcm/{branch}/attribute-values/{attributeId} | Retrieve valid values for the given attribute and term prefix. | api_mrcm_attribute_value
/mrcm/{branch}/concept-model-attribute-hierarchy | Retrieve all active concept model attributes in a hierarchical structure. | api_mrcm_attribute_hierarchy
/mrcm/{branch}/domain-attributes | Retrieve MRCM domain attributes applicable for the given stated parents. | api_mrcm_attribute_domain
/multisearch/concepts | Search concepts across multiple Code Systems. | api_multi_concept
/multisearch/descriptions | Search descriptions across multiple Code Systems. | api_multi_descriptions
/browser/{branch}/members | Search for reference set ids. | api_browser_refset_members
/{branch}/members | Search for reference set members. | api_refset_members
/{branch}/members/{uuid} | fetchMember | api_member
/{branch}/relationships | findRelationships | api_relationships
/{branch}/relationships/{relationshipId} | fetchRelationship | api_relationship
/{branch}/report/inactive-concepts-without-association | Find inactive concepts with no historical association grouped by inactivation type. | api_inactive_concepts_no_association

pkgcheck results - develop

Checks for snomedizer (v0.4.0)

git hash: d55a0283

✔️ Package name is available
✔️ has a 'codemeta.json' file.
✔️ has a 'contributing' file.
✔️ uses 'roxygen2'.
✔️ 'DESCRIPTION' has a URL field.
✔️ 'DESCRIPTION' has a BugReports field.
✔️ Package has at least one HTML vignette
✔️ All functions have examples.
✔️ Package has continuous integration checks.
✖️ Package coverage failed
✖️ R CMD check found 1 error.
✔️ R CMD check found no warnings.

Important: All failing checks above must be addressed prior to proceeding

Package License: MIT + file LICENSE

Add api operation path under api_*() function doc

Use Snowstorm trademark in doc

Problem conect URL down

Hi,

I try connect but the function send this error:

Warning message:
Status 500 INTERNAL_SERVER_ERROR
Index [branch] not found.; nested exception is [branch] ElasticsearchStatusException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [branch]]]

I´m looking for your URL https://snowstorm.test-nictiz.nl but is down.

¿is it possible change the URL into the configuration function?

Thankds

url too long

set batches of 300 conceptIds

infection_concepts <- concepts_descendants(
  conceptIds = "40733004",
  direct_descendants = FALSE,
  activeFilter = TRUE,
  endpoint = local_endpoint,
  branch = local_branch,
  limit = 10000
)[[1]]

infection_descriptions <- concepts_descriptions(
  conceptIds = infection_concepts$conceptId,
  endpoint = local_endpoint,
  branch = local_branch,
  limit = 10000)

> No encoding supplied: defaulting to UTF-8.
> Warning message:
> Status 414

Bug - requests to Snowstorm returning status 423

Solution: change user agent and identify snomedizer library and version in the REST request.

README first example gives: `Status 500 INTERNAL_SERVER_ERROR`

I tried running the first example in the README but it gives an error:

> api_concepts(term = "pneumonia", activeFilter = TRUE)
Response [https://snowstorm.test-nictiz.nl/MAIN/concepts?term=pneumonia&limit=50&offset=0&activeFilter=TRUE]
  Date: 2023-09-25 22:24
  Status: 500
  Content-Type: application/json
  Size: 230 B
{
  "error" : "INTERNAL_SERVER_ERROR",
  "message" : "Index [branch] not found.; nested exception is [branch] ElasticsearchStatusException[Elasticsearch exception [type=index_not_foun...
Warning message:
Status 500 INTERNAL_SERVER_ERROR
Index [branch] not found.; nested exception is [branch] ElasticsearchStatusException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [branch]]]

functions to validate endpoint and branch

refactor validations in snomedizer_options_set
apply this validations in api_operations

may need to also separate lower-level REST operations from checks - at the moment wrapper functions that use purrr map() would carry out validations at every step.

roxygen example too long

Pneumonia example too long

httr::content(api_concepts(term = "pneumonia"))
https://temp-snomedizer.netlify.app/reference/api_operations.html

Upgrades for snowstorm 7.3.0

Release notes

create showInternalReleases in api_code_system_all_versions() as per FRI-183 Add internalRelease flag to CodeSystemVersion. By default this is hidden from the listing by default, this is configurable.

This won't be tested because public endpoints are currently not using the internalRelease flag.
The internalRelease flag is currently optional in snowstorm

Invalid concept_is behaviour

concept_is does not allow the narrowing down of an ECL query.

Current behaviour:

# check that bacterial pneumoniae (53084003) are excluded from the whole set of pneumoniae
# 10625071000119104  | Bronchopneumonia caused by bacteria (disorder) |
> concept_is("10625071000119104", target_ecl = "<<233604007 MINUS <<53084003") 
[1] TRUE

Expected behaviour:

> concept_is("10625071000119104", target_ecl = "<<233604007 MINUS <<53084003") 
[1] FALSE

This is caused by original flaw in #12 #21.
The revised function should no longer use the "<<" operator and expect the user to instead specify it in their query.

Upgrades for Snowstorm 7.4.0

Release notes

Upgrades for Snowstorm 7.11.0

Release notes

API functions

Implementation of the most important API operations from https://snowstorm.ihtsdotools.org/snowstorm/snomed-ct/swagger-ui.html

Upgrades for Snowstorm 7.7.0

Release notes

introduce searchAfter parameter in all relevant 'api_*()' operations (not a breaking change, just an opportunity to document this parameter)

pkgcheck results - fix/attempt_pkgcheck

Checks for snomedizer (v0.3.0)

git hash: 5e047d4d

✔️ Package name is available
✔️ has a 'codemeta.json' file.
✔️ has a 'contributing' file.
✔️ uses 'roxygen2'.
✔️ 'DESCRIPTION' has a URL field.
✔️ 'DESCRIPTION' has a BugReports field.
✖️ Package has no HTML vignettes
✔️ All functions have examples.
✔️ Package has continuous integration checks.
✔️ Package coverage is 96.9%.
✔️ R CMD check found no errors.
✖️ R CMD check found 2 warnings.

Important: All failing checks above must be addressed prior to proceeding

Package License: MIT + file LICENSE

api_concepts http error

httr::content(api_concepts(term = "blurgh", branch="*(*£"))
prints some html stuff, not json!

User feature request: searching descriptions

Several requests have come through for the following features which depend on Snowstorm/Elastic:

fuzzy searches: see request to Snowstorm developers
boolean searches: e.g. if building a clinical codelist for 'primary hypertension' (i.e. high blood pressure that's not secondary to some other pathology), my search strategy may be something like use concept_find() for any descriptions matching (('essential' OR primary') AND ('hypertension')) BUT NOT ('pulmonary' OR 'intracranial')
- -> workaround with grepl is possible and could be explained in vignette
- -> Snowstorm 7.9 will support ECL 1.5 and description filters

Main vignette: add example on ICD10 classif map

Thanks to @nels for the use case!

Summary

The main vignette should contain a fifth example for

users wishing to create codelists using ICD-10 codes
users wishing to recode concepts into ICD-10

This will require implementing the snowstorm RefSet members endpoints.

Note: At the moment this seems to be yet to be implemented into ECL, but considered for ECL 1.7 according to SLPG meeting 23/09/2021. This may involve a new operator to query the mapTarget field in RF2 refsets.

Test cases

Find all the SNOMED CT concepts that map to ICD-10 code "L56.2"

    ^ 447562003 |ICD-10 complex map refset| {{ M mapTarget = "L56.2" }}
    ^ [referencedComponentId] 447562003 |ICD-10 complex map refset| {{ M mapTarget = "L56.2" }}

Find the SNOMED CT concept that maps to ICD-10 code "L56.2" with a mapRule = 1 and mapGroup = 1

^ [referencedComponentId] 447562003 |ICD-10 complex map refset| {{ M mapGroup = #1, mapPriority = #1, mapTarget = "L56.2" }}

Upgrades for Snowstorm 7.9.3

Release notes

Update infection_concepts to RF2 June 2020

snowstorm update 5.0.6

sort out social media share preview

twitter card
picture
og:image

Function to add parents in columns of data frame

As a dplyr user, I want to create a column indicating whether a concept stored in a row is a descendant of another concept, so that I don't have to waste time using the concept_descendants() wrapper

data.frame %>% 
    mutate(desc_123038009_Specimen = concept_descendant_indicator(
          conceptIds = .$concept_code,
          parent = "123038009",
          ... )

concept_infections

Build concept_infections function

Upgrades for Snowstorm 7.2.0

FRI-158 Add effectiveTime and isPublished filters to concept search ( api_concepts() )
add new tests

see release notes
see effectiveTime

Raise limit to 100 in wrapper functions

@AnikaC-git would you mind sense-checking the below before I implementing the proposed change in wrapper functions that use .split_into_chunks?

Source of the problem

concept_find() is designed to retrieve concepts by chunks of 99 conceptIds (see CHUNK_SIZE=100 in wrapper.R) from GET /{branch}/concepts. This is due to the URL character limit constraint of GET (:arrow_right: error 414 Request-URI Too Large)

But often the limit on the number of results sent back by snowstorm will be lower than this -- the default is 50, and a user cannot set limit above a hard threshold of 10,000.

Example

# Get 221 concept IDs
concepts <- concept_find(ecl = "<233604007", limit = 300)

# Retrieve those 
concepts_batch <- concept_find(
    conceptIds = concepts$conceptId,
    limit = 50
)

Warning messages:                                                                                                        
1: 
This server request returned just 50 of a total 99 results.
Please increase the server `limit` to fetch all results. 
2: 
This server request returned just 50 of a total 100 results.
Please increase the server `limit` to fetch all results.

Currently, the function will display a warning with every chunk.
Say you are asking the function to retried 153 concepts, that means you have 2 chunks, so 2 warnings.
Neither elegant nor self-explanatory!
Users will not understand why this happens.

Note: The same thing will apply to:

concepts_included_in
concepts_descriptions
concepts_map
For the latter 2, it's impossible to predict how many results will be returned for every chunk of size = 100.
Hence forcing the limit to 10000. I don't anticipate this being insufficient, and even if it were, the user would see the warning (though they'd be unable to do much about it).