{"id":13592108,"url":"https://github.com/slott56/TigerShark","last_synced_at":"2025-04-08T23:30:48.030Z","repository":{"id":2777394,"uuid":"3776674","full_name":"slott56/TigerShark","owner":"slott56","description":"X12 Message Processing","archived":false,"fork":false,"pushed_at":"2023-08-21T14:56:51.000Z","size":5001,"stargazers_count":18,"open_issues_count":0,"forks_count":44,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-08-02T16:44:06.881Z","etag":null,"topics":["edi","x12"],"latest_commit_sha":null,"homepage":"http://slott56.github.io/TigerShark/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slott56.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2012-03-20T15:25:04.000Z","updated_at":"2024-07-23T10:31:22.000Z","dependencies_parsed_at":"2022-09-10T03:44:28.715Z","dependency_job_id":"f8f81ac3-46dd-4e23-8305-e7a52b4285bd","html_url":"https://github.com/slott56/TigerShark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slott56%2FTigerShark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slott56%2FTigerShark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slott56%2FTigerShark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slott56%2FTigerShark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slott56","download_url":"https://codeload.github.com/slott56/TigerShark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223346536,"owners_count":17130458,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["edi","x12"],"created_at":"2024-08-01T16:01:05.868Z","updated_at":"2024-11-06T13:30:17.278Z","avatar_url":"https://github.com/slott56.png","language":"Python","funding_links":[],"categories":["Libraries"],"sub_categories":["Python"],"readme":"TigerShark is an X12 EDI message parser that can be tailored to\na specific partner in the health care payment ecosystem.\n\nAbout Tiger Shark\n=================\n\nTigerShark is used to transform X12 schema definitions into Plain old Python class definitions.\nThe class definitions are used by an application to process an X12 data stream.\n\nTigerShark's generated code can parse X12 messages, providing Python objects that\ncan be used in a variety of ways. A web application can be built using any Python\nframework. A Juypter Lab notebook can be built to do analysis work on claim data.\nPython-based middleware can be written to process claim files.\n\nThe X12 message definitions from from the PyX12 project, see https://github.com/azoner/pyx12.\n\nAbout X12 EDI\n=============\n\nSee https://x12.org/products. \n\nThe X12 syntax defines a complex more-or-less hierarchical structure\nof files that have messages. Messages have Loops (that can repeat), Segments,\nand individual Data Elements.\n\nThe core idea is that a Segment is a group of related Data Elements. A segment\nhas an identifier, making it possible to identify repeating loops \nand any omitted segments.\n\nThe segments are often terminated with a `~` and data elements terminated with `*` or `|`.\nA complete schema is required, including details of fields that are optional.\n\nSee https://docs.informatica.com/data-integration/b2b-data-transformation/10-1/libraries-guide/descriptions-of-the-libraries/hipaa-library/hipaa-message-structure.html\n\nDependencies\n============\n\nWe rely on https://github.com/azoner/pyx12 for the definition of the message.\n\n(Ideally, we'd go to the source documents, including\nthe X12 standard and Implementation Guides. We don't know where to find these.)\n\nBuilding the Message Classes\n============================\n\n1. Clone and check out  https://github.com/azoner/pyx12.\n2. `cd tigershark3/tools`.\n3. Put `tigershark` on the `PYTHONPATH` environment variable.\n4. Run the `xml_extract.py` application to build the message classes.\n   All of the parameters are in the `__name__ == \"__main__\"` block.\n\n```shell\ncd tigershar3/tools\nPYTHONPATH=.. python xml_extract.py\n```\n\nTesting\n=======\n\nComplete test:\n\n```shell\ntox\n```\n\nRun marked tests with detailed logging.\nVery handy for debugging.\n\n```shell\nPYTHONPATH=tigershark3 pytest -m x12parser --log-cli-level=DEBUG tests\n```\n\nDependencies\n============\n\nThis project uses [pip-tools](https://pypi.org/project/pip-tools/)\nto create a detailed `requirements-dev.txt` \nfrom the `pyproject.toml`\n\n```shell\npip-compile --extra=dev --extra=test -o requirements-dev.txt\n```\n\n\n\nState of the Project\n====================\n\nVersion 3.0\n-----------\n\nA foundational rewrite. This uses Python type annotations to define message structure.\nThe ``tools/xml_extract.py`` tool converts PyX12 XML message definitions into\npure Python classes simply and directly.\n\nIt can emit a JSON Schema description of messages, also.\nThis permits transforming raw data to JSON for analysis and processing.\n\n\nVersion 0.2.5\n-------------\n**Lots** of 271 bugfixes! Several tests for 271 files have been added. A few\nparsing bugs have been fixed.\n\nA nice change is that the parser no longer crashes if there is an invalid code\nin an X12 element. This was causing me nothing but grief so I disabled it.\nValid codes are only checked when there is a single valid code for a segment,\nsince this is important in determining loop boundaries for 271 files.\n\nParseErrors return a more helpful error message so tracking down a bad line\nis much easier.\n\nImportant bugfix that prevents an early parser exit if an optional segment\nisn't found, but later optional segments are present.\n\nPyX12, which this project depends on, changed its project layout (in version\n2.0.0), so the parser generation scripts have been updated to look in the new\ndirectory.\n\nVersion 0.2.4\n-------------\nI discovered a bug that caused deductible/co-insurance/co-payments from being\nsummed if they occurred at the claims-level rather than the adjustments level.\nThis resulted in underreporting the *actual* amounts. This has been fixed and\nunit tests have been added for this case.\n\nI also made adjustments to the directory structure. Tests have been moved up\nand the latest version of PyX12 was used to generate the parsers. An old PyX12\ntarball is no longer included in this distribution, so instructions were\nadded to get PyX12 and set it up for parser generation.\n\nVersion 0.2.3\n-------------\nInitial support for reading 270/271 files. I'm not sure when I'll add support\nfor creating X12 files, since I have yet to need to do so. I haven't even\ntested creating them.\n\nSegmentAccess, SegmentSequenceAccess, and X12SegmentBridge now all work pretty\nwell. I didn't really like how the 835 facade was implemented so I spent more\ntime trying to figure out nicer ways of structuring the 270/271 facades. This\nmeant understanding and fixing the Segment[Sequence]Access classes. I was able\nto avoid a bunch of ugly multiple inheritance tricks and mostly freed myself\nfrom setting properties in __init__ (though not entirely, and I'm not sure\nif I even want to totally remove this). I may clean up the 835 facade later,\nbut I didn't want to introduce any breaking changes in this version.\n\nI am understanding TigerShark more and more as I continue implementing\nthings that S. Lott didn't get to, however this also means that I'm being\nbitten by the complexity of the project more often. There are a lot of good\nideas in this project, and I keep encountering things I didn't expect\n(non-sequential hierarchical level grouping in 271 files, wtf??). TigerShark\ncan handle most of these weird cases with minor bugfixes (which makes me more\nconfident that this design was right from the start), but I don't think\nTigerShark can fully support 270/271 files due to their weird structuring.\nI intend to re-write a good portion of TigerShark after implementing several\nmore formats, since I'll have a clear idea of the kind of requirements have\nto be met.\n\nVersion 0.2.2a\n-------------\nNothing big, just a bugfix to ElementSequenceAccess (so it actually works)\nand moved two large enum types to an enums module.\n\nEdit: Followup fix to allow unknown values in the enum x12 type, since it's\npossible that an insurance company returns an outdated remark code.\n\nVersion 0.2.1\n-------------\nI realized that a single EOB file can contain multiple EOBs. This means that\nthe f835 facade now has a list of all of its individual EOBs as a `facades`\nproperty.\n\nI also fixed a few typos, added a ClaimAdjustments common X12LoopBridge with\nthe corresponding claim adjustment reasons as an enum x12type, and improved\nthe tests for 835 files.\n\nThis package is now being used in production, and the 835 facade can be\nconsidered somewhat stable.\n\nVersion 0.2\n-----------\nI've added a setup.py script and organized the files a bit more. I'm\nconsidering this a major version bump because the inclusion of setup.py and\npregenerated parsers makes this a *lot* closer to a fully usable package. I\nmake no claim that any parser other than the 835 works as expected, since I\nhave only dealt with 835 files so far.\n\nDevelopment will probably slow down now that things are mostly working. In the\npipeline are auto-generated facades, or facades for 270/271 files, whichever\nI need to do first.\n\nIf this sort of thing interests you, the awesome biotech startup where I\nwork is hiring. I can't say much about it other than it involves genes, real\nscience, and we are currently saving lives and improving the future of\nhumanity. Do drop me a line.\n\n(Insurance billing is a painful but necessary step in this process.)\n\nVersion 0.1\n-----------\n\nTigerShark was initially developed by [S. Lott](https://github.com/slott56),\net. al. The code was recently released at my request, after I stumbled on a\nfew blog posts about the project:\n\n  1. [Python as Config Language - Forget XML and INI files (Jan 12, 2008)](http://www.itmaybeahack.com/homepage/iblog/C465799452/E20080111205451.html)\n  2. [Two Python Config-File Design Patterns (Jan 19, 2008)](http://www.itmaybeahack.com/homepage/iblog/C465799452/E20080119082306.html)\n  3. [Configuration File Scalability - Who Knew? (Revised) (Jan 26, 2008)](http://www.itmaybeahack.com/homepage/iblog/C465799452/E20080126181253.html)\n  4. [Python as Configuration Language - More Good Ideas (March 28, 2008)](http://www.itmaybeahack.com/homepage/iblog/C465799452/E20080328172746.html)\n  5. [Synchronicity and Document Object Models. (March 31, 2008)](http://www.itmaybeahack.com/homepage/iblog/C465799452/E20080331113231.html)\n  6. [POPO and GOPS - Plain Old Python Objects and Good Old Python Syntax (April 1, 2008)](http://www.itmaybeahack.com/homepage/iblog/C412398194/E20080401060105.html)\n\nBy the time I found those posts I had been struggling with X12 files for\nabout two weeks, dealing with broken parsers and PDFs that cost thousands of\ndollars that describe the spec over 750 pages in human - but not, or only\nbarely, machine - readable format. (How the healthcare industry gets away with\ngetting the government to mandate a proprietary file format which you have to\npay to read is the subject of another rant...).\n\nI was struck by the amount of good, deep thought that went into the decisions\nS. Lott made, especially as compared to everything else I had seen. If you\nwant to contribute to this project, I highly encourage you to go read those\nposts first.\n\nWhat you see in version 0.1 is a series of hacks to get TigerShark working.\nI fixed a few bugs, added a facade for 835 files, and added setup instructions\nto the readme. The facade code is a mess (I didn't have enough time to fully\nunderstand the descriptor pattern and all of the underlying data structures\nSteven used), and I'll have to come back and make it nicer. Ultimately the\nfacade should be able to be generated straight from the xml files which are\nused to build the parser. I removed a bunch of files that didn't appear to\nbe used anywhere. I didn't try to get the demo django site working, and I'll\neither remove it or add instructions for it in a later version.\n\nMany thanks to [S. Lott](https://github.com/slott56) for releasing the code\nand answering my questions, and to [John Holland](https://github.com/azoner)\nfor providing the xml files in his package [pyX12](https://github.com/azoner/pyx12).\n\nInstallation\n============\n\n```sh\npython setup.py install\n```\n\nManually Generating the Parsers\n-------------------------------\n\nThe `setup.py` script will install default parsers, but you might want to\ngenerate your own, or you're fixing the generation script and need to test.\nYou can either convert all of the 4010 xml files in `Downloads/pyx12-1.5.0.zip`\nor convert a file individually (which gives you more control over the result).\n\n### Generating All Parsers From PyX12 archive ###\n\nIf you just want to generate all of the parsers, you can use the\n`generate_all_parsers` script:\n\n```sh\ngit clone https://github.com/azoner/pyx12.git\ncd pyx12\npython setup.py sdist --formats=gztar,zip\ncd ../\npython tools/generate_all_parsers.py pyx12/dist/pyx12-*.zip -d parsers\n```\n\nThis will generate all parsers in a directory called `parsers`.\n\n### Generating A Single Parser ###\n\nYou can also just create a single parser from an unzipped pyx12 source:\n\n```sh\ngit clone https://github.com/azoner/pyx12.git\ncd parsers\npython ../tools/convertPyX12.py 835.4010.X091.A1.xml M835_4010_X091_A1.py -b ../pyx12/pyx12/map/ -n parsed_835\n```\n\nThis will generate a `M835_4010_X091_A1.py` parser in your current directory.\n\nUsage\n=====\n\nUsing a Parser\n--------------\n\n```python\nfrom x12 import msg_835_4010_X091_A1\nfrom x12.base import Source, X12Parser\nfrom pathlib import Path\n\nEXAMPLES = Path(\"/path\") / \"to\" / \"examples\"\nexample = EXAMPLES / \"835-example.txt\"\ndocument = Source(example.read_text())\n# Skip some validation rules\nerrors_here = [\n  \"*_N1:n101:Enumerated\", \"*_N1:n103:Enumerated\", \"*_REF:ref01:Enumerated\",\n  \"*_NM1:nm101:Enumerated\", \"*_NM1:nm102:Enumerated\", \"*_NM1:nm108:Enumerated\"\n]\nparser = X12Parser(msg_835_4010_X091_A1.MSG835, skip_validation=errors_here)\nmsg = parser.parse(document)\n```\n\nNow you can access the segments of the X12 file in a purely pythonic way.\n\n```python\n# Payee is in N1 in Loop 1000A. L1000A_N1 found in HEADER.ItemL1000A\n# The ISA/GS/ST loop structure is always similar. \n# Nested for statements make sense instead of always getting the first instance.\nst = msg.isa_loop[0].gs_loop[0].st_loop[0]\n\n# A good practice is the get the segment, and then process fields of the segment.\nn1 = st.header[0].l1000a[0].n1\nn4 = st.header[0].l1000a[0].n4\nts3 = st.detail[0].l2000[0].ts3\n\nassert n1.n102 == \"UNITED HEALTHCARE INSURANCE COMPANY\"\nassert n4.n403 == \"553430000\"\nassert ts3.ts304 == 2.0\n```\n\nTests\n-----\n\nIf you are kind enough to create a facade, *please* add unit tests. To run\nthe tests that currently exist, run the following in the current directory.\n\n```sh\npython -m unittest discover\n```\n\nNote that if you first `cd tests` and then run the unit tests, they will fail\nbecause the tests expect certain files to be in certain paths.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslott56%2FTigerShark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslott56%2FTigerShark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslott56%2FTigerShark/lists"}