Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/walmartlabs/gozer

Open source library to parse various X12 file formats for retail/supply chain
https://github.com/walmartlabs/gozer

Last synced: about 1 month ago
JSON representation

Open source library to parse various X12 file formats for retail/supply chain

Host: GitHub
URL: https://github.com/walmartlabs/gozer
Owner: walmartlabs
License: other
Created: 2018-04-09T17:53:20.000Z (over 6 years ago)
Default Branch: master
Last Pushed: 2023-12-06T19:44:21.000Z (about 1 year ago)
Last Synced: 2024-11-06T12:45:44.650Z (about 1 month ago)
Language: Java
Homepage:
Size: 815 KB
Stars: 67
Watchers: 7
Forks: 25
Open Issues: 7
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

Electronic-Interchange-Github-Resources - walmartlabs/gozer - The EDI X12 Standard provides a uniform way for companies to exchange information across different sectors. (Libraries / Java)

README

[![Build Status](https://github.com/walmartlabs/gozer/workflows/Gozer-CI/badge.svg)](https://github.com/walmartlabs/gozer/actions)

[![codecov](https://codecov.io/gh/walmartlabs/gozer/branch/master/graph/badge.svg)](https://codecov.io/gh/walmartlabs/gozer)

# EDI X12 Standard Parsing Library

The EDI X12 Standard provides a uniform way for companies to exchange information across different sectors.

This open source library, available through @WalmartLabs, provides Java based classes that can parse various X12 file formats into a representative Java object model. It's primary focus is on those formats related to the Supply Chain sector.

## Supported X12 Format Parsers and Transaction Set Parsers

## Why the name Gozer

The Agile development team at @WalmartLabs that is responsible for the design and development of "inbound processing" products is called the GhostBusters. Inbound Processing covers the broad array of micro-services involved with moving and receiving merchandise between locations, including the supplier to the distribution center, the supplier to the store (DSD), and the distribution center to the store.

## Basic Design Approach

Gozer seeks to provide more than a generic X12 parsing capability that turns an EDI X12 message into a list of segments and elements, although it does have that capability. Gozer hopes to provide a library of easy to use classes, that can transform a parsed EDI message into a set of POJOs that corresponds with a specific X12 format. Attributes are labeled with their business names rather than the more cryptic segment identifiers. In addition, there are numerous interfaces and various extension points allowing users to customize some of the parsing approaches that are offered in Gozer.

The initial design provided some basic validation capabilities but most of that was updated in 0.3.0. It is now recommended that users of the framework build out validation capabilities so that they can enforce compliance with the guides that are agreed upon between the various parties. This also allows users to decide how best to handle the generation of 824 warnings and errors when validation fails.

### Practical Approach to Parsing
Each X12 format that is supported will have a Java implementation of the `X12Parser`. The parser will be responsible for parsing the message in the given format to an `X12Document`, which is the representative Java object model (POJO).

```java
public interface X12Parser {
/**
* parse the X12 transmission into a representative Java object
*
* @return the representative Java object
* @throws X12ParserException
*/
T parse(String sourceData);
```

The Gozer parsers that implement `X12Parser` are designed to parse a specific EDI X12 document type. In order for a document to be considered successfully parsed the message MUST be well-formed. That means that the segments must have the correct segment identifiers, which must be nested correctly and appear in the proper order. If the X12 message is not well-formed and can't be parsed the parser will throw an `X12ParserException`.

If the EDI message was well-formed and successfully parsed that does not mean that it is valid. The parsers, intentionally, do not attempt to validate any values within the segment elements (other than segment identifiers). They are intended to be as loose as possible regarding validation during the parsing of different versions of the EDI X12 message. Validation can be performed against the `X12Document` that is returned by the parser when it is successful.

Gozer currently provides two implementations of the `X12Parser`.

- The `StandardX12Parser` for most EDI documents wrapped in ISA/ISE envelopes and contain transaction sets.
- The `DefaultDex894Parser` to handle DEX 894 documents.

Using Gozer is as easy as passing the contents of the EDI message to an instantiated parser:

X12Document x12Doc = x12Parser.parse(x12Message);

Creating an instance of the `X12Parser` will vary depending on which one is being used.

# Getting Started: The Standard EDI Parser and Document in Gozer

Most EDI/X12 messages adhere to a common format that wraps transaction sets in an envelope. These can all be parsed using the `StandardX12Parser`.

Creating an instance of the `StandardX12Parser` is very simple:

```java
try {
StandardX12Parser x12Parser = new StandardX12Parser();
StandardX12Document x12Document = x12Parser.parse(new String(ediMessage));
} catch (X12ParserException e) {
// parsing did not go well
}
```

However, please check out the section **Handling Transaction Sets** to make sure the `StandardX12Parser` is setup correctly.

## Exploring the Standard EDI message format

Each message will have an envelope that contains a set of groups. Each group will contain a set of transaction sets.

```
ISA*...
GS*...*12345*X*005010
ST*856*00001
...
SE*100*00001
GE*1*12345
ISE*...
```

These types of messages can be parsed using the `StandardX12Parser`. This is an implementation of the `X12Parser` interface that returns a `StandardX12Document`.

Valid, well-formed messages will be wrapped in an interchange control envelope. This envelope will require the first segment to start with `ISA` and the last segment to end with an `ISE`.

```
ISA*...
...
ISE*...
```

The parser will place the elements found in the interchange control envelope in the object `InterchangeControlEnvelope`.

Inside the interchange control envelope, the message can contain one or more groups. Each group will start with a `GS` segment line and end with the first occurrence of a segment that has the identifier `GE`.

```
ISA*...
GS*...*12345*X*005010
...
GE*1*12345
ISE*...
```

The `StandardX12Document` will hold a list of `X12Group`s. Each instance in the list will map to a group in the message.

Inside each group, the message can contain one or more transaction sets. Each transaction set will start with an `ST` segment line and end with the first occurrence of a segment that has the identifier `SE`. It is the transaction set that contains an EDI document.

```
ISA*...
GS*...*12345*X*005010
ST*856*00001
...
SE*100*00001
GE*1*12345
ISE*...
```

The attributes in each transaction set will be stored in an object that implements the `X12TransactionSet` interface (more on that later). All of the transaction sets for a particular a group will be placed in the corresponding `X12Group` as a list.

Therefore this EDI message:

```
ISA*...
GS*...*12345*X*005010
ST*856*00001
BSN*00*001*20190823*2112*0001
SE*100*00001
ST*856*00002
BSN*00*002*20190823*2112*0001
SE*100*00002
GE*1*12345
GS*...*12346*X*005010
ST*856*00003
BSN*00*003*20190823*2112*0001
SE*100*00003
GE*2*12346
ISE*...
```

would be stored in a `StandardX12Document` as follows:
* One `InterchangeControlEnvelope`
* list with two `X12Group` objects
* the first `X12Group` would have a list with two `X12TransactionSet`s
* the second `X12Group` would have a list with one `X12TransactionSet`

## Handling Transaction Sets
Each EDI transmission can contain one or more transaction sets. Each transaction set can represent a different document type. For example the ASN (856) or a Purchase Order (850). Each of these document types are comprised of a variety of different segments.

When the `StandardX12Parser` is instantiated it will not be able to parse any EDI document types.

StandardX12Parser x12Parser = new StandardX12Parser();

One or more `TransactionSetParser` implementations must be registered so that the parser will work correctly. Each `TransactionSetParser` will understand a different document type.

The `GenericTransactionSetParser` is designed to work with any document type but will only be able to return a list of generic `X12Segment`s instead of a more user-friendly domain object.

StandardX12Parser x12Parser = new StandardX12Parser();
x12Parser.registerTransactionSetParser(new GenericTransactionSetParser());

In order to take advantage of the set of POJOs that corresponds with a specific X12 format, each individual `TransactionSetParser` implementation that a user is interested in must be registered individually.

// setting up the standard X12 parser to work
// with ASN 856 and PO 850 documents
// all other documents will be ignored
StandardX12Parser x12Parser = new StandardX12Parser();
x12Parser.registerTransactionSetParser(new DefaultAsn856TransactionSetParser());
x12Parser.registerTransactionSetParser(new DefaultPo950TransactionSetParser());

When a transaction set does not have a registered parser the `X12StandardParser` will send the list of segments associated with it to an extension point. By default, the extension point will ignore any transaction set that it receives. When it is important to interact with a transaction set that had no `TransactionSetParser` implementation registered a custom class that implements `UnhandledTransactionSet` can be registered.

```java
StandardX12Parser x12Parser = new StandardX12Parser();
x12Parser.registerTransactionSetParser(new DefaultAsn856TransactionSetParser());
x12Parser.registerUnhandledTransactionSet(new CustomHandlerForUnhandledTransactionSet());
```

Look at the [Sample](https://github.com/walmartlabs/gozer/tree/master/src/test/java/sample) to see a very simplified example of how to register a transaction set parser and use the `StandardX12Parser`.

## Using the parsed document data

The `StandardX12Document` provides access to the entire parsed EDI message which may contain one or more transaction sets.

In order to access the first transaction set on the first group:

```java
// parse an EDI file transmission
StandardX12Document x12 = x12Parser.parse(new String(ediMessage));

// access to envelope
InterchangeControlEnvelope envelope = x12.getInterchangeControlEnvelope();

// access to the groups
List groups = x12.getGroups();

// retrieve the transaction sets from the first group
List txForGroupOne = groups.get(0).getTransactions();

// retrieve the first transaction set from the first group
X12TransactionSet txSet = txForGroupOne.get(0);

// identify which EDI document type
// the transaction set represents
if ("856".equals(txSet.getTransactionSetIdentifierCode()) {
// we have an ASN
AsnTransactionSet asnTx = (AsnTransactionSet) txSet;
```

### Dealing with Hierarchy Loops

Some of the EDI documents have hierarchy loops. Each of the hierarchy segments is related to the others using a set of indexes.

HL*1**S
HL*2*1*O
HL*3*2*P
HL*4*3*I
HL*5*3*I

The basic looping structure (used on an ASN) above denotes a hierarchy of data that starts with a shipment loop.

- S(hipment)
- O(rder)
- P(ack)
- I(tem)
- I(item)

Each loop may have other segments associated with it.
For example this is a pack loop with a MAN segment providing a pack label

HL*3*2*P
MAN*GM*00001914178883300010

All looping information will be available in the `X12TransactionSet` object returned by the `TransactionSetParser`.

```java
// after parsing an EDI message
if ("856".equals(txSet.getTransactionSetIdentifierCode()) {
// we have an ASN
AsnTransactionSet asnTx = (AsnTransactionSet) txSet;
// the first loop is an X12Loop of type Shipment
Shipment shipment = asnTx.getShipment()

// get the children loops
List shipmentChildLoops = shipment.getParsedChildrenLoops();

// the Shipment has one or more Order loops
// get the first child
X12Loop loop = shipmentChildLoops.get(0);
if ("O".equals(loop.getCode()) {
// we have an Order loop
Order orderLoop = (Order) loop;
// access a segment on the order loop
order.getPrf().getPurchaseOrderNumber();

List orderChildLoops = order.getParsedChildrenLoops();
X12Loop orderChildLoop = orderChildLoops.get(0);

// examine loops on the order
switch (orderChildLoop.getCode()) {
case "T" :
// found a tare
this.processTare((Tare)orderChildLoop);
break;
case "P" :
// found a pack
this.processPack((Pack)orderChildLoop);
break;
case "I" :
// found an item
this.processItem((Item)orderChildLoop);
break;
default:
break;
}

```

Look at the [X12LoopUtilTest](https://github.com/walmartlabs/gozer/tree/master/src/test/java/com/walmartlabs/x12/util/loop/X12LoopUtilTest) to see how to walk through loops and examine the segments associated with each one.

Look at the [DefaultAsn856TransactionSetParserEntireTxSetTest](https://github.com/walmartlabs/gozer/blob/master/src/test/java/com/walmartlabs/x12/asn856/DefaultAsn856TransactionSetParserEntireTxSetTest.java) to see how to access various parts of the parsed document.

In earlier versions any error detected during looping would throw an `X12ParserException` stopping any further parsing. Starting with 0.3.0, when an error in looping occurs an exception will no longer be thrown. Instead the parser will set an attribute to indicate how that part of the parsing went. Invalid looping is now treated as a post-parsing validation error. It will be up to the user of the framework to check the value of this attribute and assess the looping issues.

### Dealing with Hierarchy Loop errors
It is expected that after parsing, the code should verify how looping was handled.

```java
X12TransactionSet txSet = txForGroupOne.get(0);
if (txSet.hasLooping()) {
if (txSet.isLoopingValid()) {
// looping ok
} else {
// handle looping errors
List loopingErrors = asnTx.getLoopingErrors();
}
```

After parsing, it is considered the responsibility of the consumer to perform any additional validations on the data that was provided in various segments and elements. That can be done through custom code.

# Getting Started: The DEX 894 Parser and Document in Gozer
The DEX EDI/X12 message format does not follow the standard message format. It has a separate parser, however it still implements the `X12Parser` interface. This allows it to be used in any product with the standard Gozer parser without any special considerations. Since there are no transaction sets in DEX there is nothing extra to register. This parser will work correctly after instantiation.

```java
String dexMessage = ...
DefaultDex894Parser dexParser = new DefaultDex894Parser();
Dex894 dexDoc = dexParser.parse(dexMessage);
```

The `DefaultDex894Parser` returns an `X12Document` of the type `Dex894`.

### Using utilities after parsing
A parsed DEX document can be used similar to a standard X12 document. Various utilities can be used for post-processing the data.
For example:

```java
Dex894 dex = dexParser.parse(dexMessage);
Dex894Item dexItem = dex.getItems().get(0);

// check CRC on each DEX transaction
CyclicRedundancyCheck crc16 = new CyclicRedundancyCheck();
List dexTxList = dex.getTransactions();
for (Dex894TransactionSet dexTx : dexTxList) {
if (crc16.verifyBlockOfText(dexTx.getIntegrityCheckValue(), dexTx.getTransactionData())) {
// passed CRC check
} else {
// failed CRC check
}
}

```

# Contributing to Gozer

## Getting Started: Writing a Parser
[Guide for writing EDI parsers](AddingParsers.md)

## Getting Started: Writing a TransactionSetParser
[Guide for writing EDI parsers](AddingTransactionSetParsers.md)

## More Information

[About EDI Standards](http://ediacademy.com/blog/edi-x12-standard/)

[EDI file format basics](https://www.xtranslator.com/prod/beginguidex12.pdf)

[ASC X12 Standards](http://edi.aaltsys.info/01_standards.html)

[EDI transaction set codes](https://www.spscommerce.com/resources/edi-documents-transactions/)

[GTIN formats](https://www.gtin.info/)

[Converting between GTIN formats](https://www.free-barcode-generator.net/ean-14/)

[ITF-14 format](https://www.free-barcode-generator.net/itf-14/)

## Build and Release

```
mvn clean install
```

See [walmartlabs-pom for more information](https://github.com/walmartlabs/walmartlabs-pom)

## Licensing

This software is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0)

Also see LICENSE file.