phatboyg/machete

Cut through the Crap, with Machete, a text parser, object mapper, and query engine.

Hi

• 🔭 I’m always working on MassTransit

machete's Issues

Packaging in OS X and Linux

This issue is for the long term idea of being able to publish packages. Largely we have been developing Machete with the Mono runtime for it to be compatible with Linux, OS X, and Windows we need to come up with a similar packaging technology to NuGet. I am sure there is one for both Linux and OS X but wanted create this issue as a reminder.

Need a way to query segment by Id

We need a way to query the message by Id. Its not all that useful to be able to find a segment by its relative position in a file because in most use cases that is not known information. What is known, however, is the segment Id.

A perfect use case is when you are using Machete within a UI. In such a scenario, the user may be provided a segment name (e.g. PID) and need to be able to return the segment object and from there be able to access the field by index.

Weird syntax for EntityType

I am able to do this...

msh.EntityType.EntityType

This is sort of weird syntax, where the last EntityType returns Type.

TimeZoneInfo.FindSystemTimeZoneById method does not seem to work on none Windows OS

The method TimeZoneInfo.FindSystemTimeZoneById does not seem to work on none Windows machines because it relies on registry keys to lookup time zones and of course the registry is a Windows concept.

https://msdn.microsoft.com/en-us/library/system.timezoneinfo.findsystemtimezonebyid(v=vs.110).aspx

Excerpt from the above link,

"FindSystemTimeZoneById tries to match id to the subkey names of the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Time Zones branch of the registry under Windows XP and Windows Vista. This branch does not necessarily contain a comprehensive list of time zone identifiers. If required by an application, you can create a particular time zone either by calling one of the overloads of the CreateCustomTimeZone method or by calling FromSerializedString to deserialize a TimeZoneInfo object that represents the required time zone. However, time zones created by these method calls are not included in the registry and cannot be retrieved using the FindSystemTimeZoneById method. These custom time zones can be accessed only through the object reference returned by the CreateCustomTimeZone or FromSerializedString method call."

There might be a different way to achieve the same functionality by using the CreateCustomTimeZone method in some way.

How can i create x12 messages from clr objects?

First of all thanks for creating this project! It looks awesome!

I played around with the code, and it seems that the API is all tailored around parsing existing formatted messages into clr objects.
I'm wondering how to go about creating a well formatted x12/hl7 message from clr types.

Is that possible?

Handling of "Not Used" Elements

I'm trying to parse an X12 835 document, and the NM1 Segment has space for "Name Prefix" - NM106, which is labeled as "Not Used" in the documentation, however, I see that in your code you are ignoring this field.

For reference sake, I am getting my information from here: https://reviewer.x12.org/x322publicreviewweb
Then navigating to 2.4 \ Table 2 - Detail \ Loop 2100 - Claim Payment Information \ NM1 - Patient Name.
And I see in your code: Generated\V5010\Segments\Maps\NM1Map.cs - Line 20 sets position 6 to Name Suffix instead of Name Prefix.

Thank you!

Changing CreateTranslator signature

Currently, CreateTranslator looks like this...

Machete.Schema.CreateTranslator(typeof(MessageTranslate), () => new MessageTranslate());

Perhaps we could refactor so that devs don't have to pass in the type. It could look something like this...

Machete.Schema.CreateTranslator(() => new MessageTranslate());

or it can get the type from the input translate like this...

Machete.Schema.CreateTranslator(() => new MessageTranslate());

Map condition doesn't work when SegmentList is followed by Segment of same entity type

If you have the following data:

...

PER*CX*PROVIDER SERVICES*TE*321321321*UR*www.default.org
PER*BL*PROVIDER SERVICES*TE*321321321*UR*www.default.org
PER*BL*PROVIDER SERVICES*TE*321321321*UR*www.default.org
PER*BL*PROVIDER SERVICES*TE*321321321*UR*www.default.org
PER*IC*PROVIDER SERVICES*TE*321321321*UR*www.default.org

...

...and a map that looks like this...

...
Segment(x => x.BusinessContactInformation, 4,
x => x.Condition = parser => parser.Where(p => p.ContactFunctionCode.IsEqualTo("CX")));
Segment(x => x.TechnicalContactInformation, 5,
x => x.IsRequired().Condition = parser => parser.Where(p => p.ContactFunctionCode.IsEqualTo("BL")));
Segment(x => x.PayerWebsite, 6,
x => x.Condition = parser => parser.Where(p => p.ContactFunctionCode.IsEqualTo("IC")));
...

...then the expectation is that I should be able to match on all PER segments that have a ContactFunctionCode of "BL" and return a SegmentList on the TechnicalContactInformation property. This works but when it is followed by another segment of the same entity type (e.g. PER) with different condition (e.g. ContactFunctionCode = "IC") it will return that segment as part of the previous segment list. This is similar to issue #58 but for segments of the same type coming after a segment list.

I have a failing unit test, Layout835Tests.Verify_can_parse_PayerWebsite_when_multiple_TechnicalContactInformation_segments_present

IsUnknown should probably be IsKnown

Was looking through the following test code to skip segments and came across a few confusing things:

var result = parsed.Query(q => from ignored in q.Except<HL7Segment, EVNSegment>().ZeroOrMore() from segment in q.Select<EVNSegment>() where segment.EntityType.IsUnknown == false select new {segment, ignored});

First off I noticed the IsUnknown property hanging off of the EntityType interface. I think it would be more natural to developers to be IsKnown. That way if were wanting to skip the segment I could do as the code snippet above indicates segment.EntityType.IsKnown == false.

Create a way to register necessary components to use within applications

Need a means to pull in artifacts to ease the use of Machete. Without this developers would have to register components on their own. Out of the box Machete should have Autofac support for the following things:

• Registering each parser type (e.g. layout, text, LINQ, etc.)
• Registering each schema
• Registering each transformer
• Registering each validator

Other IoCs can be added as we move along but Autofac currently has the biggest market share. Of course, people might not want to register their components using any IoC so for those folks you still have option of not using. I think it might make sense to put this in the Machete.* project under the folder Integration. For instance, Machete.HL7 would have one specific for HL7. There would be an interface in the Machete project though and the specific implementations would live where the subsequent schemas live.

Syntax for defining required segments within layout needs rethink

This is a bit weird syntax when defining a segment within a layout as required. Below is the current syntax...

Segment(x => x.ORC, 0, x => x.Required = true);

I propose we change this to...

Segment(x => x.ORC, 0, x => x.IsRequired());

...that way by default it is not required. Another, even more simple, syntax would be to simply this parameter with a boolean like so,

Segment(x => x.ORC, 0, true);

...this would be an optional parameter set to false by default.

ConvertValue<T> returned on empty fields

Given the following HL7,

MSH|^~&|LIFTLAB||MACHETE||201701131234|||K113|P|

I have a test using the Or extension method returning ConvertValue for MSH-9 (field right before K113) when you clearly see that it is missing. The test is asserting on HasValue and IsPresent and the expectation is that these would be both false but they come back as true. Also, I am asserting on Value.Missing(), which should come back as MissingValue() and this is coming back as ConvertValue() instead.

Interestingly, when I do a Console.WriteLine(actual.Value.MessageCode.Value) before the asserts I get the following:

Machete.HL7.Tests.OrTests.Verify_can_return_missing_value_on_complex_typed_field

Machete.ValueMissingException : The value is missing.
at Machete.Values.MissingValue1[TValue].Machete.Value<TValue>.get_Value () [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Values/MissingValue.cs:18 at Machete.Values.ConvertValue1[TValue].Machete.Value.get_Value () [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Values/ConvertValue.cs:32
at Machete.HL7.Tests.OrTests.Verify_can_return_missing_value_on_complex_typed_field () [0x00070] in /Users/albert/Documents/Git/Machete/src/Machete.HL7.Tests/OrTests.cs:49
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in /private/tmp/source-mono-d15-3/bockbuild-d15-3/profiles/mono-mac-xamarin/build-root/mono-x64/mcs/class/corlib/System.Reflection/MonoMethod.cs:305

This seems to indicate that MissingValue is indeed called first but then ConvertValue is called and subsequently returned. Not sure why this. Perhaps a condition is missing somewhere.

Implement X12 Schema

Implement the X12 Schema (i.e. Loops, Segments, and Components).

X12 parser failing on segment delimiter

Currently, the X12 parser is failing on segment delimiter when you add a space before the Tilda, ~

Let's look at an example for parsing the GS segment.

This is fine...
ISA036327 01NFMC01 ZZMACHETE ZZPERSEPVS 0906010406*^005010000265310P*:~GSHBFL0738PERSEPVS200906010406121X005010X279

This fails...
ISA036327 01NFMC01 ZZMACHETE ZZPERSEPVS 0906010406*^005010000265310P*: ~GSHBFL0738PERSEPVS200906010406121X005010X279

There is a white space before the tilda on the failing data above.

Documentation and Project Page Update

I'd love to start using this in a project I'm working on however I haven't been able to find documentation on this? If there is documentation maybe a link could be put in the README to direct to it.

Also, the link in the .Net Nuget Package goes to a page that doesn't exist on the MassTransit site. (Please see https://www.nuget.org/packages/Machete.HL7/1.0.233-develop). Updating this would be helpful as well.

ValueList.TryGetValue not working

ValueList.TryGetValue does not seem to be working. Please check unit test ValueListTests.Should_be_possible.

@phatboyg please check #22

Bug in EntityValueList getting too many values

So, given the following developer defined segment...

VL1|ABCXYZ123

...say we want to iterate over the ValueList to determine the number of values, which we can clearly see as being 3. However, what was being returned was 4 because the EntityValueList.TryGetValue method was returning a null for index 3, which translates into the fourth item.

Handling unknown segments

Sometimes there will be segments defined by the source system that need to be rolled up into a LINQ to HL7 query. In such a case we need a way to select unknown segments.

I might want to query all the unknown segments skipping over all others. Here is a possible code fragment on how that query might look,

from unknown in query.Select<HL7Segment>().Where(x => x.SegmentId.HasValue && x.SegmentId.Value != "ORC")

Dynamic field parsing problem

I am calling this the dynamic field parsing problem because of the following usage scenario:

In the HL7 specification it states that the OBX-2 field's value determines how you will need to handle the OBX-5 field. This has been difficult in the past to deal with especially in statically generated specifications. In such cases, implementations usually represent both fields as string. However, traditionally the developer is left with the responsibility of having to parse out the OBX-5 field on their own. This means that parser is rather useless when it comes to OBX-5 if the developer is responsible for then building a mini parser on top. Consider the following example:

OBX|...|XPN|...|...|Doe^Jane^^^Dr^MD|...

So, in the above scenario the OBX-2 field is telling us that OBX-5 should be parsed as an XPN data type. In this case I would want to apply the parse to OBX-5 as a component field on extraction of the data rather than when the field is defined since you would not know in all cases what the data type is beforehand.

Perhaps the verb could something like:

As(string dataType)

This would hang off of the field itself.

possible example,

.Select<OBX>().Select(x => x.ObservationValue).As("XPN")

This would, of course, blow up if no data type with the given ID is found.

LayoutList and EntityList enumerators not working

Add enumerators for LayoutList and EntityList including tests.

Add an alias field for each property

Create an alias off of each Value and Segment so that we can attach meta to the field to be used for UI use cases.

Seems weird passing in IParser<TSchema> to HL7StructureConfigurator in order create a HL7 structure

It seems a bit weird passing in IParser to HL7StructureConfigurator in order create a HL7 structure with the extension method CreateHL7. The StructureConfigurator class is only using it to call Parser.Schema to return ISchema. Why not just pass in ISchema if it is needed?

Also, would it not make sense to initialize it the same as with normal segment maps? Usually the parser is created from the schema. This seems like a cart before the horse situation where the parser is need to create the schema.

Do not use HL7MessageParser in non-streaming scenario

Currently the HL7MessageParser is being used in the non-streaming scenario and we do not enforce the parser blowing up if there is no MSH segment. Below is the conversation we had on a recent PR:

phatboyg commented 4 days ago
But this is for parsing a stream of messages, right?
It should just return Unmatched after the first matching message, instead of throwing an exception.

ahives commented 4 days ago • edited
Hmm, you are partially correct. It gets called for both and maybe that is the problem. For stream this might be bad but the question is that for stream parsing do we retain knowledge of the delimiters? If we do then it is a useless check for both, which means we probably should just hard code the delimiters.

ahives commented 4 days ago
That said, if I have a malformed message that was discovered on the first read then why would I want to continue parsing? Moreover, why would that be considered a successful parse if I find the data I am looking for downstream on a malformed message?

Words Structure and Layout are confusing

Instead of Layout about something like DocumentSection instead. The reason is because layout is definition of a section of a document. Also, instead Structure what about something like Document. I think the words Layout and Structure don't really capture the meaning and can lead to confusion.

Create ability to parse files with multiple messages

This covered under the HL7 spec 2.10.3 called HL7 batch protocol. The structure looks like this...

FHS|^&|LCAMC|LABCORP|ATLANTIC HEALTH|ATLANTIC HEALTH|2016011209225417|||TEST|F1601209541701
BHS|^&|LCAMC|LABCORP|ATLANTIC HEALTH|ATLANTIC HEALTH|2016011209225417||||B000001
MSH|^&|LCAMC|LABCORP|ATLANTIC HEALTH|ATLANTIC HEALTH|20160112092254||ORU^R01|M16012000000000001|T|2.3|||ER|ER
...
MSH|^&|LCAMC|LABCORP|ATLANTIC HEALTH|ATLANTIC HEALTH|20160112092254||ORU^R01|M16012000000000002|T|2.3|||ER|ER
...
BTS|0000000006|B000001|00000187
FTS|1|F160120954170~0000189

We need to be able to write a query that iterates the cursor to each message (i.e. MSH segment) and then we should be able to query off of the individual message. I think this is a perfect stream parser use case as well.

IsPresent is true all the time

When you call IsPresent property in any situation it always evaluate to true even if it returns an Value.Missing.

Need to make debugging experience for loaded Schema better

Right now when a entity map is missing there is no useful data presented to the developer in the form of which map the engine is choking on trying to apply. Developers are relegated to have to search each one by hand. This is a extremely painful experience.

All default Machete maps should be DateTimeOsset instead of DateTime

All maps that are defined within Machete should be DateTimeOffset instead of DateTime. Imagine parsing the following:

20170502173021.1212-0800

Using DateTime the time zone component (i.e. -0800) would be lost, thus, going against the primary tenant of the parser not being destructive. However, with using DateTimeOffset you get the same functionality of DateTime and the accuracy of the time zone component.

For developers wanting to go against this recommendation in user-define maps we should provide the following extension methods:

DateTimeOffset ConvertTo(this Value< DateTime > value)
DateTime ConvertTo(this Value< DateTimeOffset > value)

Slightly confusing syntax for querying documents

Currently I can query the document like this:

Parsed parsed = _parser.Parse(message);

var query = Query.Create(q =>
from x in q.Select()
select new {x.MessageType});

var result = query.Parse(parsed);

This is a bit unnatural from a developers perspective. Immediately I am thrown off by the different usages of the word Parse. From my perspective when I get a returned value from Query.Create I should be done. That statement should return my result. Right now the above code snippet reads:

1. Parse message
2. Create a query
3. Parse the query

I propose the following syntax,

var result = Query.Create(q =>
from x in q.Select()
select new {x.MessageType}).ExecuteAgainst(parsed);

or...

var query = Query.Create(q =>
from x in q.Select()
select new {x.MessageType});
var result = query.ExecuteAgainst(parsed);

This reads a little different,

1. Parse message
2. Create a query
3. Execute query against parsed message

Having a method called Parse hang off of the query can throw a developer off as to the purpose of staging the query to be executed against the parsed document.

NuGet package available?

Hi,
is there a nuget package available? I can't build Machete. I get following error.

C:\Program Files\Git\cmd\git.exe status -s -b
System.Reflection.TargetInvocationException: Ein Aufrufziel hat einen Ausnahmefehler verursacht. ---> System.ArgumentException: The input sequence was empty.
Parametername: source
   bei Microsoft.FSharp.Collections.SeqModule.Head[T](IEnumerable`1 source)
   bei Fake.Git.Information.getBranchName(String repositoryDir) in D:\code\fake\src\app\FakeLib\Git\Information.fs:Zeile 51.
   bei FSI_0005.Build.informationalVersion[a](a _arg1)
   bei <StartupCode$FSI_0005>.$FSI_0005_Build$fsx.main@()
   --- Ende der internen Ausnahmestapelüberwachung ---
   bei System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   bei System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   bei System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   bei [email protected](Unit unitVar0) in D:\code\fake\src\app\FakeLib\FSIHelper.fs:Zeile 323.
System.Exception: Start of process git.exe failed. Das System kann die angegebene Datei nicht finden
   at [email protected](String message) in D:\code\fake\src\app\FakeLib\ProcessHelper.fs:line 103
   at Fake.ProcessHelper.ExecProcessWithLambdas(FSharpFunc`2 configProcessStartInfoF, TimeSpan timeOut, Boolean silent, FSharpFunc`2 errorF, FSharpFunc`2 messageF) in D:\code\fake\src\app\FakeLib\ProcessHelper.fs:line 103
   at Fake.ProcessHelper.ExecProcessAndReturnMessages(FSharpFunc`2 configProcessStartInfoF, TimeSpan timeOut) in D:\code\fake\src\app\FakeLib\ProcessHelper.fs:line 130
   at Fake.Git.CommandHelper.runGitCommand(String repositoryDir, String command) in D:\code\fake\src\app\FakeLib\Git\CommandHelper.fs:line 28
   at Fake.Git.Information.getBranchName(String repositoryDir) in D:\code\fake\src\app\FakeLib\Git\Information.fs:line 51
   at FSI_0005.Build.informationalVersion[a](a _arg1) in D:\development\HL7MacheteLibTest\Machete\build.fsx:line 28
   at <StartupCode$FSI_0005>.$FSI_0005_Build$fsx.main@() in D:\development\HL7MacheteLibTest\Machete\build.fsx:line 40
Stopped due to error

Change TryGetLayout methods to return none null

Change all TryGetLayout method to return a none null when the object cannot be safely returned.

Create test helper package

Create a package for helping developers test the applications they build using Machete. Some things that are immediate needs are as follows:

• Setting up the parser
• Formatting HL7 messages for comparisons
• Comparing messages/documents
• Reading/loading messages/documents

Translated Entities Should Not Copy Internal Properties

When an entity is translated, there are values which should not be copied from the original entity, but instead, set to the values which are set on a new entity.

The following properties should be Set instead of copied:

• Id
• IsEmpty
• Values

Will also need the ability to obtain the TextSlice from the values, in order, based upon the schema, to fill the Fields. This will likely get more real once the formatter work is completed.

HL7Entity causing naming collisions

There is a naming collision for the HL7Entity interface that is causing problems with HL7Entity. This interface can be found both in Machete.HL7 and Machete.HL7Schema. Why is Machete.HL7Schema.HL7Entity needed when the project Machete.HL7Schema already references Machete.HL7 where it lives? Seems like a problem with code gen.

Layouts failing because maps are incorrect for ORed segments

Consider the following HL7 2.6 specification fragment...

ORC
[
<OBR|RQD|RQ1|RXO|ODS|ODT>
[{ NTE }]
[ CTD ]
[{ DG1 }]
[{
OBX
[{ NTE }]
}]
]

...the current Machete schema and map for this is as follows:

public class ORM_O01_ORDER_DETAILMap :
    HL7V26LayoutMap<ORM_O01_ORDER_DETAIL>
{
    public ORM_O01_ORDER_DETAILMap()
    {
        Segment(x => x.OBR, 0, x => x.Required = true);
        Segment(x => x.RQD, 1, x => x.Required = true);
        Segment(x => x.RQ1, 2, x => x.Required = true);
        Segment(x => x.RXO, 3, x => x.Required = true);
        Segment(x => x.ODS, 4, x => x.Required = true);
        Segment(x => x.ODT, 5, x => x.Required = true);
        Segment(x => x.NTE, 6);
        Segment(x => x.CTD, 7);
        Segment(x => x.DG1, 8);
        Layout(x => x.Observation, 9);
    }
}

The problem is that when the spec identifies segments as <OBR|RQD|RQ1|RXO|ODS|ODT> we are interpreting this to mean all segments are required rather than one, which is the intent of the spec specifying the OR operator (i.e. |). Also, because we have defined this as being 6 segments the indexes are off, which means it will never match a real ORM message.

NA type

Below is from the HL7 spec.

Definition: This data type is used to represent a series (array) of numeric values. A field of this type may contain a one-dimensional array (vector or row) of numbers. Also, by allowing the field to repeat, a two-dimensional array (table) of numbers may be transmitted using this format, with each row of the table represented as one repetition of the field. Arrays that have one or more values not present may be transmitted using this data type. "Not present" values are represented as two adjacent component delimiters. If the absent values occur at the end of a row, the trailing component delimiters may be omitted. If an entire row of a table has no values, no component delimiters are necessary (in this case, there will be two adjacent repetition delimiters).

Example 1: vector of 8 numbers |125^34^-22^-234^569^442^-212^6|
Example 2: 3 x 3 array of numbers |1.2^-3.5^5.22.0^3.1^-6.23.5^7.8^-1.3|
Example 3: 5 x 4 array of numbers with the values in positions (1,1), (2,2), (2,3), (3,3), (3,4), (4,1), (4,2), (4,3), and (4,4) not present
|^2^3^45^^^89^10~~17^18^19^20|

X12 parser blows up when attempting to parse fields defined as ValueList

When attempting to parse fields defined as ValueList the X12 parser blows up with the below message. There is a failing test for this called Should_be_able_to_parse_simple_list in Machete.X12.Tests.EntityFieldAccessTests.ValueListTests

Machete.X12.Tests.EntityFieldAccessTests.ValueListTests.Should_be_able_to_parse_simple_list

Machete.MacheteParserException : The slice is not a list: Machete.X12.Tests.TestSchema.TestSegment.Field2 (type => ValueList<System.String>)
at Machete.SchemaConfiguration.Specifications.PropertySpecification3[TEntity,TSchema,TValue].List (Machete.TextSlice slice, System.Int32 position) [0x00027] in /Users/albert/Documents/Git/Machete/src/Machete/Configuration/SchemaConfiguration/Specifications/PropertySpecification.cs:130 at Machete.Entities.EntityProperties.ValueListEntityPropertyConverter2[TEntity,TValue].Convert (TEntity entity, Machete.TextSlice slice) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Entities/EntityProperties/ValueListEntityPropertyConverter.cs:31
at Machete.Entities.DynamicEntityConverter2[TEntity,TSchema].Convert (Machete.TextSlice slice) [0x00011] in /Users/albert/Documents/Git/Machete/src/Machete/Entities/DynamicEntityConverter.cs:53 at Machete.Entities.DynamicEntityConverter2[TEntity,TSchema].Convert[T] (Machete.TextSlice slice) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Entities/DynamicEntityConverter.cs:33
at Machete.Entities.UnbuiltEntityConverter1[TEntity].Convert[T] (Machete.TextSlice slice) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Entities/UnbuiltEntityConverter.cs:26 at Machete.Schema1[TSchema].TryConvertEntity[T] (Machete.TextSlice slice, T& entity) [0x00025] in /Users/albert/Documents/Git/Machete/src/Machete/Schema.cs:124
at Machete.X12.Slices.X12ParseResult1[TSchema].TryGetEntity[T] (System.Int32 index, T& entity) [0x0000b] in /Users/albert/Documents/Git/Machete/src/Machete.X12/Slices/X12ParseResult.cs:31 at Machete.Cursors.EntityResultCursor1[TSchema].GetNext () [0x00009] in /Users/albert/Documents/Git/Machete/src/Machete/Cursors/EntityResultCursor.cs:76
at Machete.Cursors.EntityResultCursor1[TSchema].get_HasNext () [0x00012] in /Users/albert/Documents/Git/Machete/src/Machete/Cursors/EntityResultCursor.cs:58 at Machete.Parsers.AnyParser1[T].Parse (Machete.Cursor1[TInput] input) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Parsers/AnyParser.cs:12 at Machete.Parsers.EntityParser2[TSchema,TEntity].Parse (Machete.Cursor1[TInput] input) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Parsers/EntityParser.cs:25 at Machete.Parsers.EntityQueryParser2[TSchema,T].Parse (Machete.Cursor1[TInput] input) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Parsers/EntityQueryParser.cs:29 at Machete.Parsers.SelectManyParser4[TInput,T,TSelect,TResult].Parse (Machete.Cursor1[TInput] input) [0x0002d] in /Users/albert/Documents/Git/Machete/src/Machete/Parsers/SelectManyParser.cs:34 at Machete.Parsers.EntityQueryParser2[TSchema,T].Parse (Machete.Cursor1[TInput] input) [0x00000] in /Users/albert/Documents/Git/Machete/src/Machete/Parsers/EntityQueryParser.cs:29 at Machete.ParsedCursorExtensions.Query[TSchema,TResult] (Machete.EntityResult1[TSchema] entityResult, Machete.IParser`2[TInput,TResult] query) [0x00007] in /Users/albert/Documents/Git/Machete/src/Machete/Querying/ParsedCursorExtensions.cs:53
at Machete.X12.Tests.EntityFieldAccessTests.ValueListTests.Should_be_able_to_parse_simple_list () [0x00036] in /Users/albert/Documents/Git/Machete/src/Machete.X12.Tests/EntityFieldAccessTests/ValueListTests.cs:33
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in /private/tmp/source-mono-d15-3/bockbuild-d15-3/profiles/mono-mac-xamarin/build-root/mono-x64/mcs/class/corlib/System.Reflection/MonoMethod.cs:305

Add Apache 2.0 license on source code

Add Apache 2.0 license on source code

Fields property off of segment is empty

I am not able to access the Fields to get segment fields raw string values. This is essential for parsing unknown segments. Often times you get segments from source that are Z segments whose definition is unknown, yet, we still need to handle. Imagine the following use case:

Ex,

ZCP|1|923|Blow^Joe|...

While you don't have an official definition for the ZCP segment you need to be able to access ZCP-3. I should be able to access it via the Fields property and then use the converter facility to parse it into XPN.

Should be able to do something like this,

result.Value.Fields[2].Value

Cannot stream parse through multiple layouts in the same transaction in X12

When stream parsing a file with multiple X12 transactions the Layout parser returns not matched because all of the transactions don't match.

Lazy Initialization

Let's face it, startup time to compile the entire schema is long (relatively).

And it's rare that the entire schema is used immediately. So validate it, and then wait until entity converters are used to initialize the property converters, etc. That way, all the expression compilation is happening in the background using the TPL, instead of occurring inline.

• Cache all type/property setters and getters
• Defer expression compilation to background Task
• Use basic reflection until fast expression-based accessor is compiled
• Use cached accessors for translators
• Use cached accessors for creators

This should significantly reduce the startup time of using the schema.

Need ability to format a list of segments

We need the ability to format a query or message out into a text string. In the case of segments coming from queries, it should not require an MSH segment to be present. This functionality should also allow for other formats as well. We will add different formatters as we go.

We cannot depend on defined segments such as PID. We need be able to process HL7Segment because we will get unknown segments from time to time.

Generated file name not consistent with class name for HL7Version26

The generated class HL7Version26 file name is called HL7Version.cs, which does not match. Need to change the generator to correct this.

X12 - incorrect mapping

There is an issue with the L1000A_835Map. It is incorrectly marking the TechnicalContactInformation as required when it is only situational.

As well, all of the PER types segments on this layout should all be SegmentLists as they can repeat up to twice.

X12 - 835 - will not parse when missing section

I have an 835 file which does not contain the SVC (Service Payment Information) loops (L2110_835), and the file is not parsing b/c of this.

This is valid for an 835 to not contain this loop.

Add ability to LayoutParserOptions to be able skip certain segments

@phatboyg Currently, when using layouts we are unable to create a layout that will skip over segments that are not defined in the layout. This functionality is key when parsing messages that may have user defined segments that are not known at the time of parsing.

Query not yielding any orders

@phatboyg I don't understand why the below query is not yielding any orders. What am I missing here?

Given the following HL7...

MSH|^~&|MACHETELAB|^DOSC|MACHETE|18779|20130405125146269||ORM^O01|1999077678|P|2.3|||AL|AL
NTE|1||KOPASD
NTE|2||A3RJ
NTE|3||7ADS
NTE|4||G46DG
PID|1|000000000026|60043^^^MACHETE^MRN||MACHETE^JOE||19890909|F|||123 SEASAME STREET^^Oakland^CA^94600||5101234567|5101234567||||||||||||||||N
PD1|M|F|N||||F|
NTE|1||IN42
PV1|1|O|||||92383^Machete^Janice||||||||||||12345|||||||||||||||||||||||||201304051104
PV2||||||||20150615|20150616|1||||||||||||||||||||||||||N
IN1|1|||MACHETE INC|1234 Fruitvale ave^^Oakland^CA^94601^USA||5101234567^^^^^510^1234567|074394|||||||A1|MACHETE^JOE||19890909|123 SEASAME STREET^^Oakland^CA^94600||||||||||||N|||||666889999|0||||||F||||T||60043^^^MACHETE^MRN
GT1|1|60043^^^MACHETE^MRN|MACHETE^JOE||123 SEASAME STREET^^Oakland^CA^94600|5416666666|5418888888|19890909|F|P
AL1|1|FA|^pollen allergy|SV|jalubu daggu||
ORC|NW|PRO2350||XO934N|||^^^^^R||20130405125144|91238^Machete^Joe||92383^Machete^Janice
OBR|1|PRO2350||11636^Urinalysis, with Culture if Indicated^L|||20130405135133||||N|||||92383^Machete^Janice|||||||||||^^^^^R
DG1|1|I9|788.64^URINARY HESITANCY^I9|URINARY HESITANCY
OBX|1||URST^Urine Specimen Type^^^||URN
NTE|1||abc
NTE|2||dsa
ORC|NW|PRO2351||XO934N|||^^^^^R||20130405125144|91238^Machete^Joe||92383^Machete^Janice
OBR|1|PRO2350||11637^Urinalysis, with Culture if Indicated^L|||20130405135133||||N|||||92383^Machete^Janice|||||||||||^^^^^R
DG1|1|I9|788.64^URINARY HESITANCY^I9|URINARY HESITANCY
OBX|1||URST^Urine Specimen Type^^^||URN
NTE|1||abc
NTE|2||dsa
ORC|NW|PRO2352||XO934N|||^^^^^R||20130405125144|91238^Machete^Joe||92383^Machete^Janice
OBR|1|PRO2350||11638^Urinalysis, with Culture if Indicated^L|||20130405135133||||N|||||92383^Machete^Janice|||||||||||^^^^^R
DG1|1|I9|788.64^URINARY HESITANCY^I9|URINARY HESITANCY
OBX|1||URST^Urine Specimen Type^^^||URN
NTE|1||abc
NTE|2||dsa

        var result = parsed.Query(q =>
        {
            var obxQuery = from obx in q.Select<OBX>()
                from nte in q.Select<NTE>().ZeroOrMore()
                select new
                {
                    OBX = obx,
                    NTE = nte
                };

            var obrQuery = from obr in q.Select<OBR>()
                from dg1 in q.Select<DG1>().Optional()
                from obx in obxQuery.Optional()
                select new
                {
                    OBR = obr,
                    DG1 = dg1,
                    OBX = obx,
                };

            var testQuery = from orc in q.Select<ORC>()
                from obr in obrQuery.ZeroOrMore()
                select new
                {
                    ORC = orc,
                    Tests = obr
                };

            return from msh in q.Select<MSH>()
                from ignored in q.Except<HL7Segment, ORC>().ZeroOrMore()
                from orders in testQuery.ZeroOrMore()
                select new
                {
                    MSH = msh,
                    Orders = orders
                };
        });`

The unit test is AdvancedQueryTests.Should_be_able_to_query_blocks

Need a way to represent optional parsing

Currently, in the LINQ to Machete functionality we have a way to tell the parser to only select those segments of a particular type zero or more times.

from obr in query.Select<OBR>().ZeroOrMore()

Per the HL7 specification, there are times where we would want to only select a particular segment either zero or one times. In such a case we could syntax that looks like this,

from obr in query.Select<OBR>().ZeroOrOne

or

from obr in query.Select<OBR>().Optionally()

Layout parsing inconsistent with LINQ parser

When I call Structure.TryGetLayout(out layout) to return a layout it returns

     ILayoutParserFactory<T, HL7Entity>

...where T is the layout. For starters I wouldn't call this ILayoutParserFactory and I will explain why a little later after I give more context.

Consider the following syntax for querying a parsed message...

        ParseResult<HL7Entity> parse = Parser.Parse(message);
        var result = parse.Query(q =>
        {
            var obxQuery = from obx in q.Select<OBX>()
                from nte in q.Select<NTE>().ZeroOrMore()
                select new
                {
                    OBX = obx,
                    NTE = nte
                };

            var obrQuery = from obr in q.Select<OBR>()
                from dg1 in q.Select<DG1>().Optional()
                from obx in obxQuery.Optional()
                select new
                {
                    OBR = obr,
                    DG1 = dg1,
                    OBX = obx
                };

            var testQuery = from orc in q.Select<ORC>()
                from obr in obrQuery.ZeroOrMore()
                select new
                {
                    ORC = orc,
                    OBR = obr
                };

            return from msh in q.Select<MSH>()
                from nte in q.Select<NTE>().ZeroOrMore()
                from skip in q.Except<HL7Segment, ORC>().ZeroOrMore()
                from tests in testQuery.ZeroOrMore()
                select new
                {
                    MSH = msh,
                    Notes = nte,
                    Tests = tests
                };
        });

...as you can see I parse the message then query it. However, layouts I have to do the following...

 EntityResult<HL7Entity> entityResult = Parser.Parse(message);
 ILayoutParserFactory<SomeLayout, HL7Entity> layout;
 Structure.TryGetLayout(out layout)
 Parser<HL7Entity, SomeLayout> query = entityResult.CreateQuery(q => layout.CreateParser(LayoutParserOptions.None, q));
 Result<Cursor<HL7Entity>, SomeLayout> result = entityResult.Query(query);

When compared, using layouts takes at least 4 steps to do what I can with LINQ in 2. Perhaps we can combine the first two steps of parsing out a layout into something like this...

 LayoutResult<SomeLayout, HL7Entity> parse = Structure.Parse(message)

This would be more natural syntax to what the developer has now been used to with the LINQ parser, which by this time they'd probably be well versed.

Nullable Types

It seems it would be a good idea to make some of the values nullable. In particular, you have the ValueOrDefault extension method, which is great, but if for example, I need to get a DateTime, which happens to be empty, I will end up getting the 1/1/1, which I will then have to handle properly.
This would be even more important for ints & decimals, since they would be returning a 0, which I don't think would have the same meaning as no value.

Thank you!