https://github.com/tbt-post/edi-receiver

EDI messages receiver backend
https://github.com/tbt-post/edi-receiver
4a ams bms branches dkdhub edi fms logistics saas-data tbt tbt-cloud updcon wms
Last synced: about 1 month ago
JSON representation
EDI messages receiver backend
Host: GitHub
URL: https://github.com/tbt-post/edi-receiver
Owner: tbt-post
License: epl-2.0
Archived: true
Created: 2020-03-05T08:51:13.000Z (about 5 years ago)
Default Branch: master
Last Pushed: 2022-10-12T14:26:01.000Z (over 2 years ago)
Last Synced: 2024-01-26T09:16:25.511Z (about 1 year ago)
Topics: 4a, ams, bms, branches, dkdhub, edi, fms, logistics, saas-data, tbt, tbt-cloud, updcon, wms
Language: Clojure
Homepage: https://tbt-post.net
Size: 269 KB
Stars: 3
Watchers: 2
Forks: 4
Open Issues: 0
Metadata Files:
- Readme: README.adoc
- License: LICENSE
Awesome Lists containing this project

README

        = EDI-Receiver

image::https://raw.githubusercontent.com/tbt-post/edi-receiver/master/EDI-icon.svg[EDI-Receiver,200,200]

image:https://github.com/tbt-post/edi-receiver/workflows/Clojure%20CI/badge.svg?branch=trunk[Clojure CI]

image:https://github.com/tbt-post/edi-receiver/workflows/Docker%20Image%20PG%20CI/badge.svg?branch=trunk[Docker Image PG CI]

image:https://github.com/tbt-post/edi-receiver/workflows/Docker%20Image%20MySQL%20CI/badge.svg?branch=trunk[Docker Image MySQL CI]

image:https://github.com/tbt-post/edi-receiver/workflows/Docker%20PG%20Load%20Test/badge.svg?branch=trunk[Docker PG Load Test]

A clojure application designed to receive and store EDI messages from the https://tbt-post.net/[TBT Cloud] into a customer's local database. The solution is intended to be quite universal so is actively used to integrate other platforms (like, for example, https://dkdhub.com/[DKDHUB]) and applications into the robust and flexible digital environment. 

Latest release assets available https://github.com/tbt-post/edi-receiver/releases/latest[here].

NOTE:: Default configuration implies PG and rely on upstream versioning

== Customisations

There are a lot of possible customisations available as well as adding a new schemas or changing the schema sources. On order to get more information, please use the contact form https://tbt-post.net/contact/[here]

== Makefile build

In order to build and run with Makefile you should first https://docs.docker.com/install/[install docker],

have https://leiningen.org/#install[Leiningen] installed and resolve build dependencies.

=== Install dependencies

NOTE:: Default JDK/JRE may vary between distros, but by default Java 11+ is intended

----

sudo apt-get install make default-jdk-headless

----

=== Run standalone

Standalone run means you have external PostgreSQL server and successfully prepared your configuration

----

## 1st time to compile

make compile

## 1st time to synchronize schemas and initialize tables

make run-sync

## normally just

make run

----

=== Run as a dock'ed bundle

.PG

----

sudo make docker-run

----

.MySQL

----

sudo make docker-run-mysql

----

=== Benchmark

----

sudo make docker-run-load-pg

----

== Manual build

=== Running with docker

https://docs.docker.com/install/[Install docker] first, then run

.PG

----

sudo docker build -t edi-receiver -f docker/Dockerfile.pg .

----

.MySQL

----

sudo docker build -t edi-receiver -f docker/Dockerfile.mysql .

----

.Run

----

sudo docker run -it edi-receiver

----

=== Installing clojure

----

sudo apt-get update

sudo apt-get install wget default-jdk

sudo wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -P /bin

sudo chmod a+x /bin/lein

lein upgrade

----

=== Build the application

----

lein deps

lein build-all

----

=== Installing postgresql

----

sudo apt-get update

sudo apt-get install -y wget gnupg

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'

sudo apt-get update

sudo apt-get install postgresql

----

=== Creating empty PG database

----

su - postgres -c "psql --command \"ALTER USER postgres WITH PASSWORD 'postgres';\""

su - postgres -c "psql --command \"CREATE DATABASE edi;\""

----

=== Running the application

To view migration SQL:

----

java -jar target/edi-control-standalone.jar --print-deploy-sql

----

To initialize or migrate database:

----

java -jar target/edi-control-standalone.jar --deploy

----

To start receiver:

----

java -jar target/edi-receiver-standalone.jar

----

== Config

Config is java .properties file:

|===

|parameter |type |description

|upstream.topics |string |comma separated list of topics to receive

|upstream.cache |string |path where upstream schemas cached

|upstream.sync |boolean |set true to validate upstream cache on startup

|api.host |string |http server ip address

|api.port |integer |http server port, default is 8000

|api.auth.username |string |basic auth username, if auth required

|api.auth.password |string |basic auth password, if auth required

|db |string |which storage to use, "pg" or "mysql". Options from `jdbc.*` will be merged with corresponding options

|pg.host |string |postgresql host, default is "localhost" (from jdbc.host)

|pg.port |integer |postgresql port, default is 5432

|pg.database |string |postgresql database, default is "edi" (from jdbc.database)

|pg.user |string |postgresql user

|pg.password |string |postgresql password

|mysql.host |string |mysql host, default is "localhost" (from jdbc.host)

|mysql.port |integer |mysql port, default is 3306

|mysql.database |string |mysql database, default is "edi" (from jdbc.database)

|mysql.user |string |mysql user

|mysql.password |string |mysql password

|===

See link:resources/edi-receiver.properties[edi-receiver.properties] for defaults and more options.

== Customizing config

----

cp resources/edi-receiver.properties local.properties

----

Then edit local.properties file and run

----

java -jar target/edi-receiver-standalone.jar -c local.properties

----

Default properties will be updated with local.properties

Some config options can be customized from command line, see help:

----

java -jar target/edi-receiver-standalone.jar --help

----

== Proxying

Sample configuration for proxying

----

backend.0.name="dear-kafka"

backend.0.type="kafka"

backend.0.bootstrap-servers="127.0.0.1:9092"

backend.0.enabled=true

backend.1.name="weak-connection-api"

backend.1.type="http"

backend.1.method="post"

backend.1.uri="http://my.host/api/topic/{topic}" ; {topic} will be substituted with proxy.N.target

backend.1.auth.enabled=true

backend.1.auth.type="basic"

backend.1.auth.username="test"

backend.1.auth.password="pass"

backend.1.enabled=true

proxy.0.backend="dear-kafka"

proxy.0.source="document"

proxy.0.target="proxy_document" ; kafka topic name

proxy.0.enabled=true

proxy.0.reliable=true           ; means not to accept message if kafka producer fails

; optional buffer configuration, used only with reliable=true

proxy.0.buffer.enabled=true         ; when true, if producer fails, keep message in buffer

proxy.0.buffer.max-size=1000        ; maximum buffer size, will return 422 in case of buffer owerflow

proxy.0.buffer.max-tries=3          ; retries number to send message, then message will be dropped

proxy.0.buffer.expire-time-s=180    ; message expiration time, seconds, then message will be dropped, can be nil (= forever)

proxy.0.buffer.tries-interval-ms=10000     ; minimal retry interval, default is 10000

; optional logging configuration. Used to store backend responses in "log" table

proxy.0.logging.enabled=true

proxy.0.logging.reference-fields="doctype,id" ; selected field names, stored as reference

proxy.1.backend="weak-connection-api"

proxy.1.source="document"

proxy.1.target="proxy_document"

proxy.1.condition="(= sender \\"tbt\\")"

proxy.1.transform="[[body (str body) (some? body)]]"

proxy.1.enabled=true

proxy.1.reliable=false          ; means to ignore api fails

----

=== Logging backend responses

Backend responses can be stored into `log` table.

|===

|Column|Postgresql type|Mysql type|Fescription

|`created_at`|`timestamptz`|`datetime`|time record created at

|`context`|`jsonb`|`json`|Proxy configuration, e.g.: `{"key":0,"target":"proxy_document","backend":"dear-kafka"}`

|`reference`|`jsonb`|`json`|Message reference. Selected message keys, as defined in setting `reference-fields`.

|`content`|`jsonb`|`json`|Backend response

|`raw`|`bytea`|`blob`|Backend response raw content, HTTP backend only

|===

То enable logging, add following lines to proxy configuration:

----

proxy.0.logging.enabled=true

proxy.0.logging.reference-fields="field1,field2"

----

Setting `reference-fields` is mandatory. `log.reference` will be stored like:

----

{"field1": "value1", "field2": "value2"}

----

== Transform

Transformation rules:

----

[:restrictive                           ; optional modifier, makes rules restrictive

 [target.path  ]

 [another.path ]           ; condition can be optional

 ...]

----

Restrictive means to start result from empty map. For example, following rules

----

[:restrictive

 [a (str (+ aa 10)) (odd aa)]   ; aa is odd, so evaluates

 [a (str aa) (even aa)]         ; no hit as aa is not even

 [b bb]                         ; simple assignment

 [c cc (some? cc)]              ; assign c from cc if cc is not nil

 [new 777]]                     ; constant new value

----

applied to

----

{:aa 1

 :bb 2

 :cc nil}

----

will produce

----

{:a   "11"

 :b   2

 :new 777}

----

Non-restrictive rules can be used to update or drop some values, for example, rules

----

[[aa (str (+ aa 10)) (odd aa)]  ; aa is odd so update it

 [aa (str aa) (even aa)]        ; no hit as aa is not even

 [cc :dissoc]                   ; remove cc anyway

 [new 777]]                     ; constant new value

----

applied to

----

{:aa 1

 :bb 2

 :cc nil}

----

will produce

----

{:aa  "11"

 :bb  2                         ; there was no rule on bb

 :new 777}

----

== Benchmarking

Warning: benchmarking will affect your database and backends!

To send test requests to receiver, run

----

java -jar target/edi-control-standalone.jar --fire fire-config.json

----

where fire-config.json is file with content like:

----

{

  "threads": 10,

  "requests-per-thread": 1000,

  "payloads": [

    {

      "uri": "http://localhost:8000/api/topic/document",

      "method": "post",

      "auth": {

        "enabled": true,

        "type": "basic",

        "username": "test",

        "password": "pass"

      },

      "headers": {

        "Content-Type": "application/json"

      },

      "body": "{\"sender\":\"tbt\",\"timestamp\":\"2004-10-19 10:23:54+02\",\"doctype\":\"DocReturnOrders\",\"id\":\"3d9cddc1-f71d-0d6a-6343-010ccfc800a6\",\"body\":{\"a\":\"b\"},\"checksum\":\"fd6b34c54a331ccbbfacade3ea4bd225\"}"

    }

  ]

}

----

edi-control will run 10 threads with 1000 requests per thread and calculate average request time and request count per second.

If multiple payloads supplied, payload for each request will be selected randomly.

== Development

Execute (go) in repl to start development profile with autoreload and local.properties config.

== License

Copyright © 2020 Kasta Group LLC

Copyright © 2020-2022 TBT LLC

This program and the accompanying materials are made available under the

terms of the Eclipse Public License 2.0 which is available at

http://www.eclipse.org/legal/epl-2.0.

This Source Code may also be made available under the following Secondary

Licenses when the conditions for such availability set forth in the Eclipse

Public License, v. 2.0 are satisfied: GNU General Public License as published by

the Free Software Foundation, either version 2 of the License, or (at your

option) any later version, with the GNU Classpath Exception which is available

at https://www.gnu.org/software/classpath/license.html.
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/tbt-post/edi-receiver

Awesome Lists containing this project

README
