{"id":19354651,"url":"https://github.com/svanteschubert/en16931-data-extractor","last_synced_at":"2025-04-23T08:31:57.618Z","repository":{"id":51098731,"uuid":"166656620","full_name":"svanteschubert/en16931-data-extractor","owner":"svanteschubert","description":"Data Extractor for the European e-Invoice Specification (EN16931)","archived":false,"fork":false,"pushed_at":"2025-03-23T12:14:35.000Z","size":139254,"stargazers_count":14,"open_issues_count":6,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-02T11:11:14.446Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://prototypefund.de/project/papierloser-alltag/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/svanteschubert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-20T12:06:19.000Z","updated_at":"2024-10-10T12:24:38.000Z","dependencies_parsed_at":"2023-12-22T10:45:18.084Z","dependency_job_id":"bde5410a-d660-4999-baa3-484f1bbb2279","html_url":"https://github.com/svanteschubert/en16931-data-extractor","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svanteschubert%2Fen16931-data-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svanteschubert%2Fen16931-data-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svanteschubert%2Fen16931-data-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svanteschubert%2Fen16931-data-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/svanteschubert","download_url":"https://codeload.github.com/svanteschubert/en16931-data-extractor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250399508,"owners_count":21424195,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T05:22:18.382Z","updated_at":"2025-04-23T08:31:52.604Z","avatar_url":"https://github.com/svanteschubert.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Extractor for the European e-Invoice Specification (EN16931)\n\n## Summary\n\nOpen sourced for the creators of the EU e-invoice specification to allow easier sanity checks of the data within the syntax binding tables (within EN16931-3-2 and EN16931-3-3) to improve the quality of the CEN specification.\nBuild as part of the [PrototypeFund project \"paperless\"](https://prototypefund.de/project/papierloser-alltag/) to generate larger parts of the software implementing the [European e-invoice specification (EN16931)](https://invoice.fans/en/EN16931-en/).\n\n## Background\n\n### The EU e-Invoice Specification (EN16931)\n\nThere is [a wonderful online introduction for EN16931](https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Compliance+with+eInvoicing+standard) given by the EU and [great presentation slides from CEF](https://ec.europa.eu/cefdigital/wiki/download/attachments/59180282/CEFeInvoicingWebinar%239UnderstandingUBL_CII_v1.0.pdf?version=1\u0026modificationDate=1520420915552\u0026api=v2).\n\n### Reason for this Software\n\nThe specification EN16931 is meant to build e-invoice software, but it is tedious and error prone to read the PDF (or paper) specification and manually copy its data.\nThe goal of this tool is to extract the main data, the so called syntax-binding (mapping) between the XML formats and the Semantic Data Model from more structured office documents. The bindings are for instance required to generate source code for a software allowing to load/save both XML formats. In addition our future software shall modify/create XML by an API based on the Semantic Data Model of EN16931.\nSome reasoning behind from a developers bird perspective:\nThere are three [graphs](https://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg) within the EU specification:\n\n1. Graph: Semantic Data Model (describes the user semantic both XML are being mapped upon)\n2. Graph: UBL XML - W3C XML grammar (describes all allowed XML files of UBL)\n3. Graph: UN/CEFACT CII XML - W3C XML grammar (describes all allowed XML files of UN/CEFACT)\n\nThe syntax bindings - that this tool is extracting - are connecting \"graph 2\" with \"graph 1\" and \"graph 2\" with \"graph 3\".\nIn addition, aside of the grammar there are more restrictions upon the XML that W3C Schema is not able to express ([given by ISO Schematron constraints](https://github.com/CenPC434/validation)), for instance an order date has to be earlier than the pay date.\nThese schematron restrictions can be seen as additional relations upon the XML grammar graphs (graph 2 \u0026 graph 3).\nIt is planned to map those XML constraints later to the \"Semantic Data model\" level (graph 1). By this it could be validated if there are constraints only for UBL or UN/CEFACT and missing for the other XML.\nAlso the same restrictions on the semantic graph might be reused for other older e-invoice formats to be mapped to the Semantic Data Model, in other words the validation artefacts could be easier reused.\n\n## Specification EN16931-3: Details on Syntax Binding\n\nThe EU e-invoice specification demands the support of two XML file formats [OASIS UBL 1.2](http://docs.oasis-open.org/ubl/UBL-2.1.html) and [UN/CEFACT XML Industry Invoice D16B](https://www.unece.org/cefact/xml_schemas/index).\n\nIn its 3rd part (EN16931-3) - the deliverable of WG 3 of CEN TC434 - the EU specification binds the XML syntax with the EU e-invoice semantic via an ![EU Syntax Binding](./docs/resources/EN16931-SyntaxBinding.png)\n\nFor each syntax exist a document, which contains at least two mapping table (normative \u0026 informative).\n\n1. The first \"normative table\" (see below) describe the syntax binding from the semantic (light grey) to XML (dark grey),\n![Normative Syntax Binding: Semantic to XML (UBL)](docs/resources/3-2-normative-table.png)\nBelow the desired access to the normative syntax binding as structured XML data instead of PDF (digital paper/stone).\n![Normative Syntax Binding (UBL) as extracted XML](docs/resources/3-2-normative-xml.png)\n2. The second \"informative table\" (see below) describes it the other way around from XML (dark grey) to semantic (light grey).\n![Informative Syntax Binding: XML to Semantic (UBL)](docs/resources/3-2-informative-table.png)\nThe informative table (here Table 3) does not add any new information compared to its preceding sibling the normative table (here Table 2). The informative table starts with the XML part (dark grey), but uses only two of the five XML attributes from the prior normative table.\n*NOTE:* In theory, the *exact complete data of the informative table* should already exist in the preceding normative table.\n\nThe above images where overtaken from [public EU webinar slides](https://ec.europa.eu/cefdigital/wiki/download/attachments/59180282/CEFeInvoicingWebinar%239UnderstandingUBL_CII_v1.0.pdf?version=1\u0026modificationDate=1520420915552\u0026api=v2).\n\n## Software: EN16931 Data Extractor\n\nThe data extractor is a simple Java tool that can be called from command line. It reads from one or more office documents of EN16931-3 the containing syntax binding (or mapping) tables.\nFor each table, each row is being split into the:\n\n- __XML part (dark grey)__\n- __Semantic part (light grey)__\n\nThe data of each table is being saved in an own XML format to ease reading and testing the data set.\nThis new XML structure is straight forward: A list of semantic entities, each containing the corresponding XML elements as their children.\n\n*NOTE:*\nTo ease comparison of a \"normative table\" from the specification with it's (hopefully identically) \"informative table\" twin, which has fewer XML columns, the normative table is being saved twice, once with all information and a second time as a subset equal to the informative table infoset, making file comparison easier.\n\n## Software Prerequisites\n\n### Running (everybody, e.g. CEN Technical Writer)\n\n- [\u003e= JDK 11](https://openjdk.java.net/install/)\n\n### Building (developers)\n\n- [\u003e= JDK 11](https://openjdk.java.net/install/)\n- [GIT](https://git-scm.com/)\n- [\u003e= Apache Maven 3.5](https://maven.apache.org/download.cgi?Preferred=ftp://mirror.reverse.net/pub/apache/)\n\n*NOTE:*\nThere is an [own chapter for software developers about the software](docs/software.md).\n\n## Usage\n\n### Extracting the Data\n\n1. Download the JAR with all dependencies includes [en16931-data-extractor-20230523-jar-with-dependencies.jar](docs/en16931-data-extractor-20230523-jar-with-dependencies.jar)\n1. To see version information via command-line call:\n\"__java -jar [en16931-data-extractor-20230523-jar-with-dependencies.jar](docs/en16931-data-extractor-20230523-jar-with-dependencies.jar)__\"\n1. Save the CEN DOCX documents of EN16931-3 as ODT (tested with [LibreOffice 6.2.5.2](https://www.libreoffice.org/download/download/) on Ubuntu 19.04 (disco) via commandline: 'libreoffice --headless --convert-to odt *.docx')\n\n*NOTE:* I do not use the DOCX files, but save them to ODT as I [worked for 20 years on the OpenDocument format (and predecessors)](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office-collab) and [maintain an ODF library](https://github.com/tdf/odftoolkit).\n\n1. To extract data from the specification via command-line call and move the output into a text file:\u003cbr/\u003e\n\"__java -jar [en16931-data-extractor-20230523-jar-with-dependencies.jar](docs/en16931-data-extractor-20230523-jar-with-dependencies.jar) specification.odt (or directory)  \u003e log.txt__\"\n\n### Data Analysis\n\n- The extracted data can be found at three locations aside the input document:\n\n    1. For each table of EN16931-3, which defines a syntax binding, an XML file was saved in a folder named equal to the specification name.\n    2. For each syntax-binding the data-set of the informative syntax-binding table and its preceding normative table data-set (which has the suffix _SUBSET as it does not save all XML columns), a folder \"\\_SAME_BINDING\\_\\\u003cFormat\\\u003e\" has been created with the identical XML files within.\n    3. Every semantic data-set of each syntax binding table is being saved as XML within the \"*SAME_SEMANTIC* folder.\n\n- Use a text file comparing tool like [Total Commander on Windows (MENU:Files -\u003e Compare By Content)](https://www.ghisler.com/download.htm) to detect the differences between the [\"informative\"](docs/resources/16931-3-3_example_informative.xml) and its twin the [\"normative SUBSET\"](docs/resources/16931-3-3_example_SUBSETnormative.xml) XML file.\n\n__Example:__\nThe picture below - using fictitious data - shows a data set with a problem, as aside of the heading bearing the two table names is different between the two tables, the XML cardinality shall not be different between normative and informative syntax binding table:\n![Example:](docs/resources/TotalCommanderComparison.png)\n\n## Participation\n\nPull requests are most welcome! :-)\n\nPlease note that this README.md is being generated. The version number and paths to the JAR are being replaced by automation during every build (ie. 'mvn install'). The README.md to edit can be found at \u0026lt;ROOT\u0026gt;/site/README.md.\n\nThe reason for this was to automate not only the build, but also the deployment of the JAR with all dependencies for the user (see usage section above). For this reason the revision number from the \u0026lt;ROOT\u0026gt;/pom.xml - I am using the date - is being added to the generated artefacts and the artefacts (JARs \u0026 documentation) are being copied to the github site directory (ie. \u0026lt;ROOT\u0026gt;/docs).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsvanteschubert%2Fen16931-data-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsvanteschubert%2Fen16931-data-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsvanteschubert%2Fen16931-data-extractor/lists"}