nullbucket/edi-debatcher

ASC-X12 EDI Debatcher. Work in progress (99% complete, but probably good enough to use).

edi-debatcher's Issues

Wrapped/Unwrapped defined backwards

The enumerations are wrong (confusing):

edi-debatcher/src/main/java/org/null0/edi/debatcher/Delimiters.java

Lines 9 to 11 in d511812

	enum EdiWrapStyle {
	Unknown, Unwrapped, Unix, Windows
	};

They should be this instead: Unknown, Wrapped, UnwrappedForLinux, UnwrappedForWindows. "Wrapped" means no whitespace between segments. Double-check the logic (use) to be sure nothing changes.

Fix unit tests

🐛 Today I got all the code to compile, but there are lots of failures in the junit tests (when run either from Maven or Eclipse).

1. Some fail partially, such as "expected:<0[06]> but was:<0[19]" Could be code or test data.
2. Some fail due to "java.io.FileNotFoundException: \home\developer\lnxshare\output<filename> (The system cannot find the path specified)." Note that this was was a hard-coded Linux path.
3. Some fail due to "System Environment Variable is not found.: data"

There are a lot of things that should be cleaned up in the code, but before we go too far down that path, let's get these tests working. Not just passing, but working in cases where the test data is just off. With that in mind, the hard-coded path and system environment errors should be easy to fix. The data-related errors will take a bit longer because some domain knowledge of the EDI test is required (I can help wit those).

SEE test-log.txt

FileProperties class does not appear to be utilitized

FileProperties class does not appear to be used other than in a Unit Test. Could this possibly be an attempt to read all delimiters from a properties file, but was left un-finished?

FileProperties.java

Extractor

Extractor, Extractor1, TestExtractor1 aren't really being used. What were they supposed to be for? See why.

Extractor and TestExtractor are mostly empty implementations.

TestExtractor1 looks like an experiment or attempt to debatch another way that involves heavy file I/O. This could have someone's earlier attempt to write a debatcher before I wrote up the algorithm to debatch sequentially without using heavy file I/O or lots of memory.

Handle non-claim EDI

Debatcher has only been tested on 837i/p claims . While addressing issue #27 I've noticed that non-edi claims will cause a null pointer exception error, and even if that is fixed, further down we would get an unexpected segment error. Fix the debatcher to handle non-claim EDI messages. It should at least debatch the envelopes and write out the transaction sets. Use EDI Notepad to write a valid non-837 EDI message and verify the debatcher's results.

Create envelope-only test message

We need an envelope-only test message (ISA-IEA, GS-GE, ST-SE) so that we can determine if our debatcher would work generically with non-claims EDI file types. We need this test before we decouple claims-specific logic.

Inspect output results

Now that we have the unit tests working, take a moment to manually re-inspect the output using a simple EDI editor to be sure we haven't corrupted anything or to see if we need to write any new tests. Write something to inspect the file integrity that is similar to the manual check.

NullPointerException reading past last segment in file

This test throws an unhandled exception, fails the build:

edi-debatcher/src/test/java/org/null0/edi/debatcher/test/DebatcherTest.java

Lines 46 to 50 in d511812

	// TODO: Begin to uncomment these tests and get them to work. This one reads past the last segment (IEA) and throws null pointer exception comparing IEA-02 to ISA-13.
	@org.junit.Test
	public void test_01() throws Exception {
	test ("1 ISA - 1 GS - 1 ST - 1 HL 20 - 1 HL 22 - 1 CLM");
	}

The Debtcher is reading past the last segment (IEA) and throws null pointer exception comparing IEA-02 to ISA-13. Line 162, ReadInterchangeControls: if (!isa13.equals(iea02))

Improve Whitespace handling

To clean up and simplify code that handles white space between segments, implement and use two distinct options (config properties): ignore_whitespace_between_segments, output_edi_wrap. The first option is for reading, defaults to true in order to handle Linux or Windows wrapping; the latter specifies the output wrapping style: Unwrapped, WrappedForLinux ("\r"), WrappedForWindows ("\r\n").

SCA to remove dead code

Install and run the following Eclipse static code analysis plugins to detect dead or suspect code:

1. UCDetector - dead/unused code, bad scope. Eclipse Marketplace.
2. PMD - SCA (bugs, style, duplication). Eclipse "add new": https://dl.bintray.com/pmd/pmd-eclipse-plugin/updates/
3. SpotBugs - Successor to dead FindBugs project. Eclipse Marketplace.
4. SonarLint - real time SCA, popular. Eclipse Marketplace.

Remove dead code and serious issues.

(Later we may do CheckStyle, but need a good template, so don't run it at the same time.)

Debatcher837

After the "Decouple to interfaces" milestone is achieved, move the claims-specific logic to a Debatcher837 class. It should have a separate Maven POM that uses the core debatcher as a dependency.

Decouple to interfaces

The main Debatcher class is directly instantiating Config, EdiValidatorDefault, and MetaDataLoggerDefault instances. Create a greedy constructor so that these dependencies are injected.

SegmentReader class

🎨 Factor out code form the Debatcher class to a SegmentReader helper class with the following public methods: String getCurrentSegment(); boolean fileReadCompleted(); boolean isIeaFound(); boolean isEndOfFile(); void setCurrentSegment(String newValue); String readField(int position); String getNextSegment(). Move the private method getDataChunk() over there too (most of the work is done there). We would probably want to get more of the unit tests working first, but this should be a low-risk refactor.

FindBugs Problem (Scary): thread safety: unsynchronized lazy initialization of a static field

LOCATION:

edi-debatcher/src/main/java/org/null0/edi/debatcher/Config.java

Line 37 in 1c864cb

if (staticAndDynamicProperties != null) {

ERROR:
Incorrect lazy initialization and update of static field org.null0.edi.debatcher.Config.staticAndDynamicProperties in org.null0.edi.debatcher.Config.loadConfiguration() [Scary(8), Normal confidence]

EXPLANATION:
From http://findbugs.sourceforge.net/bugDescriptions.html#LI_LAZY_INIT_UPDATE_STATIC:

LI: Incorrect lazy initialization and update of static field (LI_LAZY_INIT_UPDATE_STATIC)
This method contains an unsynchronized lazy initialization of a static field. After the field is set, the object stored into that location is further updated or accessed. The setting of the field is visible to other threads as soon as it is set. If the futher accesses in the method that set the field serve to initialize the object, then you have a very serious multithreading bug, unless something else prevents any other thread from accessing the stored object until it is fully initialized.

Even if you feel confident that the method is never called by multiple threads, it might be better to not set the static field until the value you are setting it to is fully populated/initialized.

Debatcher client: console example

In addition to the unit tests, write a console app that demonstrates how to call the debatcher. I/O should be file-oriented.

From milestone: "We'll also need an example of a file system implementation that uses our API: a separate console (command-line) debatcher application. The app would know about directories and filesystems, but our core debatcher API would not."