trouble parsing an HL7 message with a repeating group

[microsoft]

trouble parsing an HL7 message with a repeating group

[BoyaWu10]

Hi @ppc001, could you please share more context of your question? Does the converter work fail to get the OBX segment or the NTE segments? BTW, could you please share the raw liquid template you created instead of the snapshot?

[ppc001]

Attached is the liquid template From: Boya Wu ***@***.***> Sent: Wednesday, October 20, 2021 2:05 AM To: microsoft/FHIR-Converter ***@***.***> Cc: Costner, Phil C. ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/FHIR-Converter] trouble parsing an HL7 message with a repeating group (#325) External Email: Use Caution opening links or attachments Hi @ppc001<https://github.com/ppc001>, could you please share more context of your question? Does the converter work fail to get the OBX segment or the NTE segments? BTW, could you please share the raw liquid template you created instead of the snapshot? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#325 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APSI2WNNFTMASR43UNVIU5DUHZLYHANCNFSM5F7UTKKA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. We are here to help you get the care you need. Visit Novant Health<http://novanthealth.org/coronavirus> or Novant Health UVA<https://www.novanthealthuva.org/about-us/newsroom/coronavirus.aspx> for up-to-date information. Estamos aquí para ayudarle con el cuidado que usted necesita. Visite Novant Health<http://novanthealth.org/coronavirus> o Novant Health UVA<https://www.novanthealthuva.org/about-us/newsroom/coronavirus.aspx> para información actualizada. This message and any included attachments are from NOVANT HEALTH INC. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by email. If you believe that any information contained in this message is disparaging or harassing or if you find it objectionable please contact Novant Health, Inc. at 1-844-266-8268 or forward the email to ***@***.******@***.***>.

[ppc001]

ORU_R01.txt

[ppc001]

I attached the liquid template as ORU_R01.txt to the issue on GitHub. The converter is able to process the first OBX but any of the OBX or NTE segments following it. From: Boya Wu ***@***.***> Sent: Wednesday, October 20, 2021 2:05 AM To: microsoft/FHIR-Converter ***@***.***> Cc: Costner, Phil C. ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/FHIR-Converter] trouble parsing an HL7 message with a repeating group (#325) External Email: Use Caution opening links or attachments Hi @ppc001<https://github.com/ppc001>, could you please share more context of your question? Does the converter work fail to get the OBX segment or the NTE segments? BTW, could you please share the raw liquid template you created instead of the snapshot? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#325 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APSI2WNNFTMASR43UNVIU5DUHZLYHANCNFSM5F7UTKKA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. We are here to help you get the care you need. Visit Novant Health<http://novanthealth.org/coronavirus> or Novant Health UVA<https://www.novanthealthuva.org/about-us/newsroom/coronavirus.aspx> for up-to-date information. Estamos aquí para ayudarle con el cuidado que usted necesita. Visite Novant Health<http://novanthealth.org/coronavirus> o Novant Health UVA<https://www.novanthealthuva.org/about-us/newsroom/coronavirus.aspx> para información actualizada. This message and any included attachments are from NOVANT HEALTH INC. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by email. If you believe that any information contained in this message is disparaging or harassing or if you find it objectionable please contact Novant Health, Inc. at 1-844-266-8268 or forward the email to ***@***.******@***.***>.

[BoyaWu10]

Got it. The current behavior of get_related_segment_list returns a series of consecutive segments following a specific segment. When we obtain the OBX segments which follow OBR segment with this filter, as they are not consecutive (each OBX is followed by several NTE segments), the filter only returns the first OBX segment.

Currently, the segment filters in converter lack the context of grouping and have limited functions. We will investigate more about how to support it in future.

As a workaround for this special case, we can use get_segment_lists to get the OBX segments instead. After that, we can get the NTE segments with get_related_segment_list, as they are consecutive.

{% assign obxSegmentLists = hl7v2Data | get_segment_lists: 'OBX' -%}
    {% for obxSegment in obxSegmentLists.OBX -%}
        {% assign nteSegmentLists = hl7v2Data | get_related_segment_list: obxSegment, 'NTE' -%}
        {% for nteSegment in nteSegmentLists.NTE -%}
            {% if nteSegment.1.Value == 1 -%}
                {% assign comments = nteSegment.3.Value -%}
            {% else %}
                {% assign comments = comments | append: "^" -%}
                {% assign comments = comments | append: nteSegment.3.Value -%}
            {% endif %}
        {% endfor -%}
...

[ydequin21]

I've noticed that after OBX, the function will search for an NTE segment(s) until it gets to one rather than only looking at the next line. As you can see below, there are multiple lines between OBX and NTE but it still gets converted and added as a comment.

.

I have also attached a correct scenario for reference.

Optimally, we would want the function to only look at one line after the OBX line. If the next few consecutive lines are NTE then they should be added as a comment and appended to each other but if it is anything other than NTE there should not be a comment at all.

The liquid code surrounding this issue is the same one you recommended which I have reattached and the ORU_R01.hl7 is still a valid file to reference.

[ydequin21]

Hey @yankunhuang-pku,

Wanted to check-in on the progress of this issue.

[yankunhuang-pku]

We are implementing a new filter to solve this problem. I'll let you know if it's ready.

[ydequin21]

good afternoon,
checking on progress

[yankunhuang-pku]

We plan to release it this week or next week if nothing goes wrong.

[ydequin21]

great to hear! do you have a more precise timeline at this time?

[yankunhuang-pku]

We released a filter called split_data_by_segments for now. I think you can try using this filter to remove the confusing segments to solve this problem.

[ginalee-dotcom]

Closing per Yankun's solution above. Please re-open if you have additional questions, thanks!

[ydequin21]

Thank you for creating the changes and releasing the filter @yankunhuang-pku.
I have been trying to use the filter to fix my current issue but I am not succeeding. I have attached two sample HL7v2 files that I am using to test the filter, bkr_998302 and bkr_998330_2.

For bkr_998302, the intended result would be to have the NTE segments(notes/comments) of line 23-32 in the sample data file to follow OBX |9| (Observation) from line 22 as seen in MSFT_2 below

Currently, with the most recent version of the FHIR converter, the NTE segments follow every single OBX instances from line 14 through 22 plus an additional instance in the ServiceRequest resource group. (line 12).

We added the filter at the obrSegment level as you can see in MSFT_4 below on line 453. This deleted all nteSegments under Observations and only kept the nteSegments in the ServiceRequest resource group as seen in MSFT_3 below

For bkr_998330_2, the intended result is for the NTE segments to populate the correct Observation resources and for all the Observations to get converted.

With the most recent converter update, we have noticed that the notes/comments are populating the correct Observation resources as seen in MSFT_5 below. However, the following OBX and NTE segments (lines 14-42) end up getting lost and not converted. I believe this issue also stems from the NTE filtering that ends up ignoring OBXsegments that follow NTE segments.

please advise on how we can achieve our intended results and move forward. Please let me know if further documentation would be necessary or helpful.

MSFT_issue.zip

[yankunhuang-pku]

Are these tests based on the VS Code extension? At present, we can only use the command line tool in this project to try the new filter, which has not been released on the VS Code side.

Could you test these cases with the command line tool first?

[ppc001]

We tried testing the split_data_by_segments filter from the command line and got the following error

C:\tmp\Microsoft.Health.Fhir.Liquid.Converter.Tool>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d C:\Users\ppc001\source\repos\fhir_liquidtemplates -r ORU_R01 -i c:\tmp\fhir_input -o c:\tmp\fhir_output
Processing c:\tmp\fhir_input\bkr_998302.hl7
Process failed: Error happened when rendering templates: Error - Filter 'split_data_by_segments' does not have a default value for 'segmentIdSeparators' and no value was supplied

Here is the code snippet that was called.

{% assign obxSegmentLists = hl7v2Data | split_data_by_segments 'OBX|NTE' -%}

Not sure if we did something wrong there

[ydequin21]

Are these tests based on the VS Code extension? At present, we can only use the command line tool in this project to try the new filter, which has not been released on the VS Code side.

Could you test these cases with the command line tool first?

Do you have a timeline for the VS Code release?
In future updates, will the VS Code release always be delayed? How long is that delay?
thank you

[yankunhuang-pku]

We tried testing the split_data_by_segments filter from the command line and got the following error

C:\tmp\Microsoft.Health.Fhir.Liquid.Converter.Tool>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d C:\Users\ppc001\source\repos\fhir_liquidtemplates -r ORU_R01 -i c:\tmp\fhir_input -o c:\tmp\fhir_output Processing c:\tmp\fhir_input\bkr_998302.hl7 Process failed: Error happened when rendering templates: Error - Filter 'split_data_by_segments' does not have a default value for 'segmentIdSeparators' and no value was supplied

Here is the code snippet that was called.

{% assign obxSegmentLists = hl7v2Data | split_data_by_segments 'OBX|NTE' -%}

Not sure if we did something wrong there

This case may be because you wrote it wrong. Missing symbol : after split_data_by_segments

[yankunhuang-pku]

Are these tests based on the VS Code extension? At present, we can only use the command line tool in this project to try the new filter, which has not been released on the VS Code side.
Could you test these cases with the command line tool first?

Do you have a timeline for the VS Code release? In future updates, will the VS Code release always be delayed? How long is that delay? thank you

Yes, it will be released later than this tool. The specific delay time depends on other work.

[ppc001]

Thank you for that correction. I totally missed that and it completed successfully after I added the :. However, it is still not producing the results that I am expecting. Do you have an example of how you tested this new filter? From: yankunhuang-pku ***@***.***> Sent: Thursday, January 20, 2022 10:10 PM To: microsoft/FHIR-Converter ***@***.***> Cc: Costner, Phil C. ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/FHIR-Converter] trouble parsing an HL7 message with a repeating group (#325) External Email: Use Caution opening links or attachments We tried testing the split_data_by_segments filter from the command line and got the following error C:\tmp\Microsoft.Health.Fhir.Liquid.Converter.Tool>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d C:\Users\ppc001\source\repos\fhir_liquidtemplates -r ORU_R01 -i c:\tmp\fhir_input -o c:\tmp\fhir_output Processing c:\tmp\fhir_input\bkr_998302.hl7 Process failed: Error happened when rendering templates: Error - Filter 'split_data_by_segments' does not have a default value for 'segmentIdSeparators' and no value was supplied Here is the code snippet that was called. {% assign obxSegmentLists = hl7v2Data | split_data_by_segments 'OBX|NTE' -%} Not sure if we did something wrong there This case may be because you wrote it wrong. Missing symbol : after split_data_by_segments — Reply to this email directly, view it on GitHub<#325 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APSI2WLPHLKUPBJMQJXGFT3UXDFBPANCNFSM5F7UTKKA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> We are here to help you get the care you need. Visit Novant Health<http://novanthealth.org/coronavirus> or Novant Health UVA<https://www.novanthealthuva.org/about-us/newsroom/coronavirus.aspx> for up-to-date information. Estamos aquí para ayudarle con el cuidado que usted necesita. Visite Novant Health<http://novanthealth.org/coronavirus> o Novant Health UVA<https://www.novanthealthuva.org/about-us/newsroom/coronavirus.aspx> para información actualizada. This message and any included attachments are from NOVANT HEALTH INC. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by email. If you believe that any information contained in this message is disparaging or harassing or if you find it objectionable please contact Novant Health, Inc. at 1-844-266-8268 or forward the email to ***@***.******@***.***>.

[ydequin21]

we have tested with command line and we are seeing the same issues. I have attached the input sample data files and some images detailing what we are seeing and what we wish to see. I have also attached what current outputs we currently see.
thank you for your assistance.

please re-open this issue thread
sample_data_outputs.zip

[yankunhuang-pku]

@ppc001 @ydequin21 I think it may be the error of your template logic. Here I can provide some usage for your reference. You can try to write your template logic like this sample. Please note that you should use command line tool instead of VS Code extension.

If you have other questions, please provide the templates corresponding to your input data and outputs.

[ydequin21]

@yankunhuang-pku, thank you for looking into our previous issues.

We originally thought the issue above pertaining to bkr_998330 had to do with the new filter but the issue persists without the filter when using the most recent FHIR Converter template and resources.

I have included the ORU_R01.liquid template. it is your current most updated version without custom changes. The input(bkr_998330 input) has NTE under OBR, OBX |1|, and OBX |2|. In our JSON output we clearly see the note under Service Request(OBR) and one Observation (OBX |1|) but there is not another Observation (OBX |2|) so we lose the entire observation with all the notes that followed it.

I have also reattached the expected output for a clearer perspective on what the issue is.

Moving on with our project we are not using the latest upgrade because of these code-breaking issues.
Please look into this issue.
sample_data_outputs.zip

[yankunhuang-pku]

The reason for this problem is that the templates are still old and the behavior is the same as before if you haven't modified the templates. We have not replaced the old filter get_related_segment_list with the new filter split_data_by_segments in the current templates.
@ginalee-dotcom This seems to be a template problem. Could you help to refine the templates? I think we can not use get_related_segment_list: obrsegment, 'OBX', but split the data based on the segments OBR and OBX by using the new filter split_data_by_segments.

[ginalee-dotcom]

Closing this as we have the filter split_data_by_segments as a solution. Will be refining the templates to use this filter.