{"id":13299796,"url":"https://github.com/tbt-post/edi-receiver","last_synced_at":"2025-03-10T11:32:44.626Z","repository":{"id":44764094,"uuid":"245112210","full_name":"tbt-post/edi-receiver","owner":"tbt-post","description":"EDI messages receiver backend","archived":true,"fork":false,"pushed_at":"2022-10-12T14:26:01.000Z","size":275,"stargazers_count":3,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-01-26T09:16:25.511Z","etag":null,"topics":["4a","ams","bms","branches","dkdhub","edi","fms","logistics","saas-data","tbt","tbt-cloud","updcon","wms"],"latest_commit_sha":null,"homepage":"https://tbt-post.net","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tbt-post.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-05T08:51:13.000Z","updated_at":"2023-05-27T11:06:05.000Z","dependencies_parsed_at":"2023-01-19T22:30:12.110Z","dependency_job_id":null,"html_url":"https://github.com/tbt-post/edi-receiver","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbt-post%2Fedi-receiver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbt-post%2Fedi-receiver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbt-post%2Fedi-receiver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbt-post%2Fedi-receiver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tbt-post","download_url":"https://codeload.github.com/tbt-post/edi-receiver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242843319,"owners_count":20194361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["4a","ams","bms","branches","dkdhub","edi","fms","logistics","saas-data","tbt","tbt-cloud","updcon","wms"],"created_at":"2024-07-29T17:37:51.891Z","updated_at":"2025-03-10T11:32:44.614Z","avatar_url":"https://github.com/tbt-post.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"= EDI-Receiver\n\nimage::https://raw.githubusercontent.com/tbt-post/edi-receiver/master/EDI-icon.svg[EDI-Receiver,200,200]\n\nimage:https://github.com/tbt-post/edi-receiver/workflows/Clojure%20CI/badge.svg?branch=trunk[Clojure CI]\n\nimage:https://github.com/tbt-post/edi-receiver/workflows/Docker%20Image%20PG%20CI/badge.svg?branch=trunk[Docker Image PG CI]\nimage:https://github.com/tbt-post/edi-receiver/workflows/Docker%20Image%20MySQL%20CI/badge.svg?branch=trunk[Docker Image MySQL CI]\nimage:https://github.com/tbt-post/edi-receiver/workflows/Docker%20PG%20Load%20Test/badge.svg?branch=trunk[Docker PG Load Test]\n\nA clojure application designed to receive and store EDI messages from the https://tbt-post.net/[TBT Cloud] into a customer's local database. The solution is intended to be quite universal so is actively used to integrate other platforms (like, for example, https://dkdhub.com/[DKDHUB]) and applications into the robust and flexible digital environment. \n\nLatest release assets available https://github.com/tbt-post/edi-receiver/releases/latest[here].\n\nNOTE:: Default configuration implies PG and rely on upstream versioning\n\n== Customisations\n\nThere are a lot of possible customisations available as well as adding a new schemas or changing the schema sources. On order to get more information, please use the contact form https://tbt-post.net/contact/[here]\n\n== Makefile build\n\nIn order to build and run with Makefile you should first https://docs.docker.com/install/[install docker],\nhave https://leiningen.org/#install[Leiningen] installed and resolve build dependencies.\n\n=== Install dependencies\n\nNOTE:: Default JDK/JRE may vary between distros, but by default Java 11+ is intended\n\n----\nsudo apt-get install make default-jdk-headless\n----\n\n=== Run standalone\n\nStandalone run means you have external PostgreSQL server and successfully prepared your configuration\n\n----\n## 1st time to compile\nmake compile\n## 1st time to synchronize schemas and initialize tables\nmake run-sync\n## normally just\nmake run\n----\n\n=== Run as a dock'ed bundle\n\n.PG\n----\nsudo make docker-run\n----\n\n.MySQL\n----\nsudo make docker-run-mysql\n----\n\n=== Benchmark\n\n----\nsudo make docker-run-load-pg\n----\n\n== Manual build\n\n=== Running with docker\n\nhttps://docs.docker.com/install/[Install docker] first, then run\n\n.PG\n----\nsudo docker build -t edi-receiver -f docker/Dockerfile.pg .\n----\n\n.MySQL\n----\nsudo docker build -t edi-receiver -f docker/Dockerfile.mysql .\n----\n\n.Run\n----\nsudo docker run -it edi-receiver\n----\n\n=== Installing clojure\n\n----\nsudo apt-get update\nsudo apt-get install wget default-jdk\nsudo wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -P /bin\nsudo chmod a+x /bin/lein\nlein upgrade\n----\n\n=== Build the application\n\n----\nlein deps\nlein build-all\n----\n\n=== Installing postgresql\n\n----\nsudo apt-get update\nsudo apt-get install -y wget gnupg\nwget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -\nsudo sh -c 'echo \"deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main\" \u003e\u003e /etc/apt/sources.list.d/pgdg.list'\nsudo apt-get update\nsudo apt-get install postgresql\n----\n\n=== Creating empty PG database\n\n----\nsu - postgres -c \"psql --command \\\"ALTER USER postgres WITH PASSWORD 'postgres';\\\"\"\nsu - postgres -c \"psql --command \\\"CREATE DATABASE edi;\\\"\"\n----\n\n=== Running the application\n\nTo view migration SQL:\n\n----\njava -jar target/edi-control-standalone.jar --print-deploy-sql\n----\n\nTo initialize or migrate database:\n\n----\njava -jar target/edi-control-standalone.jar --deploy\n----\n\nTo start receiver:\n\n----\njava -jar target/edi-receiver-standalone.jar\n----\n\n== Config\n\nConfig is java .properties file:\n\n|===\n|parameter |type |description\n\n|upstream.topics |string |comma separated list of topics to receive\n|upstream.cache |string |path where upstream schemas cached\n|upstream.sync |boolean |set true to validate upstream cache on startup\n|api.host |string |http server ip address\n|api.port |integer |http server port, default is 8000\n|api.auth.username |string |basic auth username, if auth required\n|api.auth.password |string |basic auth password, if auth required\n|db |string |which storage to use, \"pg\" or \"mysql\". Options from `jdbc.*` will be merged with corresponding options\n|pg.host |string |postgresql host, default is \"localhost\" (from jdbc.host)\n|pg.port |integer |postgresql port, default is 5432\n|pg.database |string |postgresql database, default is \"edi\" (from jdbc.database)\n|pg.user |string |postgresql user\n|pg.password |string |postgresql password\n|mysql.host |string |mysql host, default is \"localhost\" (from jdbc.host)\n|mysql.port |integer |mysql port, default is 3306\n|mysql.database |string |mysql database, default is \"edi\" (from jdbc.database)\n|mysql.user |string |mysql user\n|mysql.password |string |mysql password\n|===\n\nSee link:resources/edi-receiver.properties[edi-receiver.properties] for defaults and more options.\n\n== Customizing config\n\n----\ncp resources/edi-receiver.properties local.properties\n----\n\nThen edit local.properties file and run\n\n----\njava -jar target/edi-receiver-standalone.jar -c local.properties\n----\n\nDefault properties will be updated with local.properties\n\nSome config options can be customized from command line, see help:\n\n----\njava -jar target/edi-receiver-standalone.jar --help\n----\n\n== Proxying\n\nSample configuration for proxying\n\n----\nbackend.0.name=\"dear-kafka\"\nbackend.0.type=\"kafka\"\nbackend.0.bootstrap-servers=\"127.0.0.1:9092\"\nbackend.0.enabled=true\n\nbackend.1.name=\"weak-connection-api\"\nbackend.1.type=\"http\"\nbackend.1.method=\"post\"\nbackend.1.uri=\"http://my.host/api/topic/{topic}\" ; {topic} will be substituted with proxy.N.target\nbackend.1.auth.enabled=true\nbackend.1.auth.type=\"basic\"\nbackend.1.auth.username=\"test\"\nbackend.1.auth.password=\"pass\"\nbackend.1.enabled=true\n\nproxy.0.backend=\"dear-kafka\"\nproxy.0.source=\"document\"\nproxy.0.target=\"proxy_document\" ; kafka topic name\nproxy.0.enabled=true\nproxy.0.reliable=true           ; means not to accept message if kafka producer fails\n\n; optional buffer configuration, used only with reliable=true\nproxy.0.buffer.enabled=true         ; when true, if producer fails, keep message in buffer\nproxy.0.buffer.max-size=1000        ; maximum buffer size, will return 422 in case of buffer owerflow\nproxy.0.buffer.max-tries=3          ; retries number to send message, then message will be dropped\nproxy.0.buffer.expire-time-s=180    ; message expiration time, seconds, then message will be dropped, can be nil (= forever)\nproxy.0.buffer.tries-interval-ms=10000     ; minimal retry interval, default is 10000\n\n; optional logging configuration. Used to store backend responses in \"log\" table\nproxy.0.logging.enabled=true\nproxy.0.logging.reference-fields=\"doctype,id\" ; selected field names, stored as reference\n\nproxy.1.backend=\"weak-connection-api\"\nproxy.1.source=\"document\"\nproxy.1.target=\"proxy_document\"\nproxy.1.condition=\"(= sender \\\\\"tbt\\\\\")\"\nproxy.1.transform=\"[[body (str body) (some? body)]]\"\nproxy.1.enabled=true\nproxy.1.reliable=false          ; means to ignore api fails\n----\n\n=== Logging backend responses\n\nBackend responses can be stored into `log` table.\n|===\n|Column|Postgresql type|Mysql type|Fescription\n\n|`created_at`|`timestamptz`|`datetime`|time record created at\n|`context`|`jsonb`|`json`|Proxy configuration, e.g.: `{\"key\":0,\"target\":\"proxy_document\",\"backend\":\"dear-kafka\"}`\n|`reference`|`jsonb`|`json`|Message reference. Selected message keys, as defined in setting `reference-fields`.\n|`content`|`jsonb`|`json`|Backend response\n|`raw`|`bytea`|`blob`|Backend response raw content, HTTP backend only\n|===\n\nТо enable logging, add following lines to proxy configuration:\n----\nproxy.0.logging.enabled=true\nproxy.0.logging.reference-fields=\"field1,field2\"\n----\nSetting `reference-fields` is mandatory. `log.reference` will be stored like:\n----\n{\"field1\": \"value1\", \"field2\": \"value2\"}\n----\n\n\n== Transform\n\nTransformation rules:\n\n----\n[:restrictive                           ; optional modifier, makes rules restrictive\n [target.path \u003cexpression\u003e \u003ccondition\u003e]\n [another.path \u003cexpression2\u003e]           ; condition can be optional\n ...]\n----\n\nRestrictive means to start result from empty map. For example, following rules\n\n----\n[:restrictive\n [a (str (+ aa 10)) (odd aa)]   ; aa is odd, so evaluates\n [a (str aa) (even aa)]         ; no hit as aa is not even\n [b bb]                         ; simple assignment\n [c cc (some? cc)]              ; assign c from cc if cc is not nil\n [new 777]]                     ; constant new value\n----\n\napplied to\n\n----\n{:aa 1\n :bb 2\n :cc nil}\n----\n\nwill produce\n\n----\n{:a   \"11\"\n :b   2\n :new 777}\n----\n\nNon-restrictive rules can be used to update or drop some values, for example, rules\n\n----\n[[aa (str (+ aa 10)) (odd aa)]  ; aa is odd so update it\n [aa (str aa) (even aa)]        ; no hit as aa is not even\n [cc :dissoc]                   ; remove cc anyway\n [new 777]]                     ; constant new value\n----\n\napplied to\n\n----\n{:aa 1\n :bb 2\n :cc nil}\n----\n\nwill produce\n\n----\n{:aa  \"11\"\n :bb  2                         ; there was no rule on bb\n :new 777}\n----\n\n== Benchmarking\n\nWarning: benchmarking will affect your database and backends!\n\nTo send test requests to receiver, run\n\n----\njava -jar target/edi-control-standalone.jar --fire fire-config.json\n----\n\nwhere fire-config.json is file with content like:\n\n----\n{\n  \"threads\": 10,\n  \"requests-per-thread\": 1000,\n  \"payloads\": [\n    {\n      \"uri\": \"http://localhost:8000/api/topic/document\",\n      \"method\": \"post\",\n      \"auth\": {\n        \"enabled\": true,\n        \"type\": \"basic\",\n        \"username\": \"test\",\n        \"password\": \"pass\"\n      },\n      \"headers\": {\n        \"Content-Type\": \"application/json\"\n      },\n      \"body\": \"{\\\"sender\\\":\\\"tbt\\\",\\\"timestamp\\\":\\\"2004-10-19 10:23:54+02\\\",\\\"doctype\\\":\\\"DocReturnOrders\\\",\\\"id\\\":\\\"3d9cddc1-f71d-0d6a-6343-010ccfc800a6\\\",\\\"body\\\":{\\\"a\\\":\\\"b\\\"},\\\"checksum\\\":\\\"fd6b34c54a331ccbbfacade3ea4bd225\\\"}\"\n    }\n  ]\n}\n----\n\nedi-control will run 10 threads with 1000 requests per thread and calculate average request time and request count per second.\nIf multiple payloads supplied, payload for each request will be selected randomly.\n\n== Development\n\nExecute (go) in repl to start development profile with autoreload and local.properties config.\n\n== License\n\nCopyright © 2020 Kasta Group LLC\n\nCopyright © 2020-2022 TBT LLC\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary\nLicenses when the conditions for such availability set forth in the Eclipse\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\nthe Free Software Foundation, either version 2 of the License, or (at your\noption) any later version, with the GNU Classpath Exception which is available\nat https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftbt-post%2Fedi-receiver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftbt-post%2Fedi-receiver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftbt-post%2Fedi-receiver/lists"}