{"id":35619788,"url":"https://github.com/scieloorg/ioisis","last_synced_at":"2026-01-05T06:04:20.976Z","repository":{"id":62571399,"uuid":"220330827","full_name":"scieloorg/ioisis","owner":"scieloorg","description":"I/O for ISIS files in Python","archived":false,"fork":false,"pushed_at":"2022-07-05T21:37:33.000Z","size":212,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-09-20T20:17:50.097Z","etag":null,"topics":["cds-isis"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scieloorg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-07T21:19:39.000Z","updated_at":"2022-12-03T23:25:13.000Z","dependencies_parsed_at":"2022-11-03T18:26:24.816Z","dependency_job_id":null,"html_url":"https://github.com/scieloorg/ioisis","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/scieloorg/ioisis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scieloorg%2Fioisis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scieloorg%2Fioisis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scieloorg%2Fioisis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scieloorg%2Fioisis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scieloorg","download_url":"https://codeload.github.com/scieloorg/ioisis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scieloorg%2Fioisis/sbom","scorecard":{"id":804594,"data":{"date":"2025-08-11","repo":{"name":"github.com/scieloorg/ioisis","commit":"5d1da6c9be0f234956dfbc1d6e4f1b83d1a9404b"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.5,"checks":[{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: BSD 2-Clause \"Simplified\" License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":6,"reason":"4 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-fh56-85cw-5pq6","Warn: Project is vulnerable to: GHSA-fm67-cv37-96ff","Warn: Project is vulnerable to: GHSA-wpqr-jcpx-745r","Warn: Project is vulnerable to: OSV-2021-955"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T11:26:06.350Z","repository_id":62571399,"created_at":"2025-08-23T11:26:06.350Z","updated_at":"2025-08-23T11:26:06.350Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28214412,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-05T02:00:06.358Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cds-isis"],"created_at":"2026-01-05T06:02:44.101Z","updated_at":"2026-01-05T06:04:20.970Z","avatar_url":"https://github.com/scieloorg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IOISIS - I/O tools for converting ISIS data in Python\n\nThis is a Python library with a command line interface (CLI)\nintended to access data from ISIS database files\nand convert among distinct file formats.\n\nThe converters available in the CLI are:\n\n|  **Command**      | **Description**                           |\n|:-----------------:|:-----------------------------------------:|\n| `bruma-mst2csv`   | MST+XRF to CSV based on Bruma             |\n| `bruma-mst2jsonl` | MST+XRF to JSON Lines based on Bruma      |\n| `csv2iso`         | CSV to ISO2709                            |\n| `csv2jsonl`       | CSV to JSON Lines                         |\n| `csv2mst`         | CSV to ISIS/FFI Master File Format        |\n| `iso2csv`         | ISO2709 to CSV                            |\n| `iso2jsonl`       | ISO2709 to JSON Lines                     |\n| `jsonl2csv`       | JSON Lines to CSV                         |\n| `jsonl2iso`       | JSON Lines to ISO2709                     |\n| `jsonl2mst`       | JSON Lines to ISIS/FFI Master File Format |\n| `mst2csv`         | ISIS/FFI Master File Format to CSV        |\n| `mst2jsonl`       | ISIS/FFI Master File Format to JSON Lines |\n\n*Note*:\nThe `bruma-*` commands and the `bruma` module\nuse a specific pre-compiled version\nof [Bruma](https://github.com/scieloorg/Bruma)\nthrough [JPype](https://github.com/jpype-project/jpype),\nwhich requires the JVM (Java Virtual Machine).\nThe `iso` and `mst` modules, as well as\nthe other modules and CLI commands\ndon't require Bruma.\nBruma only gets downloaded in its first use.\n\nThe Python-based alternative to Bruma\nwas created from scratch,\nand it's based on [Construct](https://github.com/construct/construct),\na Python library that allows a declarative implementation\nof the binary file structures\nfor both parsing and building.\nCurrently,\nthe ISO (ISO2709-based file format),\nthe MST (ISIS/FFI Master file format) and\nthe XRF (ISIS/FFI Cross-reference file format)\nfile formats can be parsed/built with the library,\nbut the XRF files aren't used nor built\nby the Bruma-independent library/CLI.\n\nMost details regarding the parse/build process\ncan be configured in both the library and the CLI,\nincluding the several variations of the MST file\nthat are specific to CISIS.\nCISIS has a serialization behavior dependent of the architecture\nand of its compilation flags,\nbut `ioisis` can deal with most (perhaps all)\nthe distinct MST \"file formats\" that can be generated/read\nby some specific CISIS version.\n\nEverything in `ioisis` is platform-independent,\nand most of its defaults are based on the *lindG4* version of CISIS,\nand on the [isis2json](https://github.com/scieloorg/isis2json)\n*MongoDB type 1* (`-mt1`) output.\nThe `--xylose` option of several CLI commands\nswitches the JSONL defaults to use the dictionary structure\nexpected by [Xylose](https://github.com/scieloorg/xylose).\n\n\n## Installation and testing\n\nIt requires Python 3.6+,\nand it's prepared to be tested in every Python version\nwith [tox](https://github.com/tox-dev/tox)\nand [pytest](https://pytest.org).\n\n```bash\n# Installation\npip install ioisis\n\n# Testing (one can install tox with \"pip install tox\")\ntox                      # Test on all Python versions\ntox -e py38 -- -k scanf  # Run \"scanf\" tests on Python 3.8\n```\n\n\n## Command Line Interface (CLI)\n\nTo use the CLI command, use `ioisis` or `python -m ioisis`.\nExamples:\n\n```bash\n# Convert file.mst to a JSONL in the standard output stream\nioisis mst2jsonl file.mst\n\n# Convert file.iso in UTF-8 to an ASCII file.jsonl\nioisis iso2jsonl --ienc utf-8 --jenc ascii file.iso file.jsonl\n\n# Convert file.jsonl to file.iso where the JSON lines are like\n# {\"tag\": [\"field\", ...], ...}\nioisis jsonl2iso file.jsonl file.iso\n\n# Convert big-endian lindG4 MST data to CSV (one line for each field)\n# ignoring noise in the MST file that might appear between records\n# (it can access data from corrupt MST files)\nioisis mst2csv --ibp ignore --be file.mst file.csv\n\n# Convert active and logically deleted records from file.mst\n# to filtered.mst, selecting records and filtering out fields with jq,\n# using a \"v\" prefix to the field tags,\n# reseting the MFN to 1, 2, etc. while keeping its order\n# instead of using the in-file order, besides enforcing a new encoding,\n# with a file that might already have some records partially in UTF-8\nioisis bruma-mst2jsonl --all --ftf v%z --menc latin1 --utf8 file.mst \\\n| jq -c 'select(.v35 == [\"PRINT\"]) | del(.v901) | del(.v540)'\n| ioisis jsonl2mst --ftf v%z --menc latin1 - filtered.mst\n```\n\nBy default, the input and output are the standard streams,\nbut some commands require a file name, not a pipe/stream.\nBruma requires the MST input to be a file name\nsince the XRF will be found based on it\n(only the `bruma-*` commands require XRF).\nThe `*2mst` commands require a file name for the MST output\nbecause the first record of it (the control record)\nhas some information that will be available\nonly after generating the entire file (i.e., it's created at the end),\nthis makes the random access a requirement.\n\nAll commands have an alias:\ntheir names with only the first character of the extension\n(or `b` for `bruma-`).\nTry `ioisis --help` for more information about all commands\nand `ioisis csv2mst --help` for the specific `csv2mst` help\n(every command has its own help).\n\nThe encoding of all files are explicit through a `--_enc` option,\nwhere the `_` should be replaced\nby the first letter of the file extension,\nhence `--menc` has the MST encoding,\n`--cenc` the CSV encoding,\nand so on.\nFor the `bruma-*` commands, the `--menc` is handled in Java,\nall other encoding options are handled in Python.\nThe `--utf8` option forces the input to be handled as UTF-8,\nand only the parts of it that aren't in such encoding\nare handled by the specific file format encoding,\nthat is, the `--_enc` option become a fallback for UTF-8.\nThis helps loading data from databases with mixed encoding data.\n\n\n### JSON/CSV mode, field and subfield processing\n\nThere are several other options to the CLI commands\nintended to customize the process,\nperhaps the most important of these options\nis the `-m/--mode`,\nwhich regards to the field and record formats in JSONL files\n(and the `-M/--cmode`, which does the same for CSV files).\nThe valid values for it are:\n\n* `field` (*default*):\n  Use the raw field value string (ignore the subfield parsing options)\n* `pairs`:\n  Split the field string as an array of `[key, value]` subfield pairs\n* `nest`:\n  Split the field string as a `{key: value}` object,\n  keeping the last subfield value of a key\n  when the key appears more than once\n* `inest`:\n  CISIS-like subfield nesting processing, similar to the `nest`,\n  but keeps the first entry with the key instead of the last one\n  (only makes difference when `--no-number`)\n* `tidy`:\n  Tabular format where the records are splitten,\n  and each field is regarded as a single JSON line\n  like `{\"mfn\": mfn, \"index\": index, \"tag\": field_key, \"data\": value}`\n* `stidy`:\n  Subfield tidy format, it's similar to the `tidy` format\n  but the fields are themselves splitten\n  in a way that each subfield is regarded\n  as a single JSON line in the result,\n  including the subfield key in the `\"sub\"` key of the result\n\nWhen used together with `--no-number`,\nthe `field`, `pairs` and `nest` modes are respectively similar\nto the `-mt1`, `-mt2` and `-mt3` options of `isis2json`.\nThe `inest` mode isn't available in `isis2json`,\nit follows the CISIS behavior on subfield querying instead.\nFor CSV, only the `tidy` and `stidy` formats are available,\ngiven that the remaining formats aren't tabular.\n\nThe `--ftf` is an option that expects a *field tag formatter* template\nfor processing the field tag, and it's the same\nfor both JSON/CSV output (rendering/building) and input (parsing).\nThese are the interpreted sequences:\n\n* `%d`: Tag number\n* `%r`: Tag as a string in its raw format.\n* `%z`: Same to `%r`, but removes the leading zeros from ISO tags\n* `%i`: Field index number in the record, starting from zero\n* `%%`: Escape for the `%` character\n\n*Note*:\n`%d` and `%i` options might have a numeric parameter in the middle\nlike the `printf`'s `%d` (e.g. recall `\"%03d\" % 15` in Python).\n\nFor the subfield processing, there are several options available:\n\n* `--prefix`:\n  Character/string that starts a new subfield in the field text\n* `--length`:\n  Size of the subfield key/tag (number of characters)\n* `--lower/--no-lower`:\n  Toggle for the normalization of the subfield key/tag,\n  which is performed by simply lowering their case\n* `--first`:\n  The subfield key/tag to be used by the leading field data\n  before the first prefix appears\n* `--empty/--no-empty`:\n  Toggle to show/hide the subfields with no characters at all\n  (apart from the subfield key/tag)\n* `--number/--no-number`:\n  Repeated subfield keys are handled by adding a number suffix to them,\n  starting from `1` in the first repeat,\n  and this option toggles this behavior (to add the suffix or not)\n* `--zero/--no-zero`:\n  Choose if the first occurrence of each subfield key in a field\n  should have a `0` suffix\n  to follow the numbering described in the previous option\n  (it has no effect when `--no-number`)\n* `--sfcheck/--no-sfcheck` (for JSONL/CSV input only):\n  Check if the specification of the subfield parsing/unparsing rules\n  given in the previous parameters would resynthesize all input fields\n  exactly in the way they appear\n\nThe `--xylose` option\nis just an alternative way of using \"`--mode=inest --ftf=v%z`\".\nTo be more similar\nto the [isis2json](https://github.com/scieloorg/isis2json) output\nwhile still making use of the format expected by\n[Xylose](https://github.com/scieloorg/xylose),\nyou should use instead \"`--mode=nest --no-number --ftf=v%z`\".\n\n\n### Common MST/ISO input options\n\nBoth MST and ISO records have a STATUS flag,\nwhich answers this question: *is this record logically deleted*?\nSTATUS equals to 1 means True (*deleted*), 0 means False (*active*).\n\nEvery record in the MST file structure has an MFN,\na serial number/ID of the record in the database.\nA major difference between the `bruma-mst2*` commands\nand the `mst2*` ones\nis in the way they handle the MFN:\nBruma always access the MST file through the XRF file,\njumping the addresses to iterate through the records\nsorting them by MFN,\nwhereas the Python implementation gets the records\nin their block/offset order\n(i.e., the order they appear in the input file).\nFor ISO files, there's no MFN stored,\nbut `ioisis` can generate it (starting from 1, like common MST records)\nif they're required (e.g. for creating CSV files).\n\nThese options are common to several commands\nwhen reading from MST or ISO files:\n\n* `--only-active/--all`:\n  Flag to select if the STATUS=1 records (logically deleted records)\n  should be in the output or not\n* `--prepend-mfn/--no-mfn`:\n  Add an artificial field `mfn` at the beginning of each record\n  with the record MFN as a string (though it's always a number)\n* `--prepend-status/--no-status`\n  Add an artificial field `status` at the beginning of each record\n  with the record STATUS as a string\n  (though it's usually just zero or one)\n\n\n### ISO-specific options\n\nThe ISO file can be seen as just a sequence of records glued together.\nEach record has 3 parts: a *leader*, a *directory* and *field values*.\nThe *leader* has some metadata,\nmost of them only accessible through the library, not the CLI\n(only the STATUS is used by the CLI).\nThe *directory* is a sequence of constant-sized structures\n(*directory items*),\neach of them representing a single field\n(its tag, its value length and its relative offset),\nwhich is matched with its respective value\nin the last part of the record.\n\nInternally to the ISO file,\nafter the directory and between each field value,\nthere's a **field terminator**.\nAt the end of the record, there's both a **field terminator**\nand, finally, a **record terminator**.\nBy default, CISIS uses the \"`#`\" as the terminator,\nthe same one for field and record,\nand that's also the `ioisis` default.\nBut it's not always the case for input/output files.\nFor example, in the MARC21 specifications\nthe field terminator is the \"`\\x1e`\" character\nand the record terminator is the \"`\\x1d`\" character.\n\nThese are the options for ISO I/O commands:\n\n* `--ft`:\n  ISO Field terminator\n* `--rt`:\n  ISO Record terminator\n* `--line`:\n  Line length for splitting a record (not counting the EOL)\n* `--eol`:\n  End of line (EOL) character or string, ignored if `--line=0`\n\nThe default values for them are the CISIS ones,\nwhich are intended to make it possible to see the ISO file\nas a common text file.\nBy default, every ISO record (raw bytes)\nis splitten into lines of 80 bytes,\nand an EOL gets printed after the record terminator,\nso two records won't share the same line.\nThe line splitting is a CISIS-specific behavior,\nit's required in order to open the ISO files it exports,\nand it might make debugging easier.\nUsing \"`--line=0`\" disables this behavior,\njoining everything as a single huge line.\nThe terminators might have more than one character, as well as the EOL,\nand these 3 parameters (like other inputs shown as *BYTES* in the help)\nare parsed by the CLI,\nso \"`\\t`\" is recognized as the TAB character\nand \"`\\n`\" as a LF (Line Feed).\n\n\n### MST-specific options (Python/construct)\n\nThe options shown here regards to the Python implementation\nof the MST file format builder/parser,\nthese are not available for the `bruma-*` commands.\n\nThe ISIS/FFI Master File Format (MST file) structure\nis a binary file divided as joined records.\nThe overall structure of it is documented in the *Appendix G*\nof the [Mini-micro CDS/ISIS: reference manual (version 2.3)](\n  https://unesdoc.unesco.org/ark:/48223/pf0000211280\n), however it's incomplete,\nseveral enhancements had been done in the file structure\nin order to make it possible to fit more data in these databases.\nNevertheless, the MST file is still a file with joined records,\nwhere each record has 3 blocks: leader, directory and field values.\nIt's similar to an ISO file with an empty field and record terminator,\nbut the leader and directory items are binary,\nthe metadata isn't the same,\nand the padding, alignment and sizes are quite hard to properly grasp.\n\nThis is the internal structure of the leader and a directory item\nin a single record of a MST file\n(it doesn't apply to the control record):\n\n```raw\n                   -------------------------------------------------\n                  |    Format | ISIS     ISIS     FFI      FFI      |\n                  | Alignment | 2        4        2        4        |\n -----------------------------+-------------------------------------|\n|         Leader size (bytes) | 18       20       22       24       |\n| Directory item size (bytes) | 6        6        10       12       |\n|-----------------------------+-------------------------------------|\n|           |      00-01      | MFN.1    MFN.1    MFN.1    MFN.1    |\n|           |      02-03      | MFN.2    MFN.2    MFN.2    MFN.2    |\n|           |      04-05      | MFRL     MFRL     MFRL.1   MFRL.1   |\n|           |      06-07      | MFBWB.1  (filler) MFRL.2   MFRL.2   |\n|           |      08-09      | MFBWB.2  MFBWB.1  MFBWB.1  MFBWB.1  |\n|  Leader   |      10-11      | MFBWP    MFBWB.2  MFBWB.2  MFBWB.2  |\n|           |      12-13      | BASE     MFBWP    MFBWP    MFBWP    |\n|           |      14-15      | NVF      BASE     BASE.1   (filler) |\n|           |      16-17      | STATUS   NVF      BASE.2   BASE.1   |\n|           |      18-19      |          STATUS   NVF      BASE.2   |\n|           |      20-21      |                   STATUS   NVF      |\n|           |      22-23      |                            STATUS   |\n|-----------+-----------------+-------------------------------------|\n|           |      00-01      | TAG      TAG      TAG      TAG      |\n|           |      02-03      | POS      POS      POS.1    (filler) |\n| Directory |      04-05      | LEN      LEN      POS.2    POS.1    |\n|   item    |      06-07      |                   LEN.1    POS.2    |\n|           |      08-09      |                   LEN.2    LEN.1    |\n|           |      10-11      |                            LEN.2    |\n -----------+-----------------+-------------------------------------|\n            |  Offset (bytes) |              Structure              |\n             -------------------------------------------------------\n```\n\nThese structure names follow the Mini-micro CDS/ISIS reference manual,\nwhere the \"`.1`\" and \"`.2`\" suffixes are there to expose\nwhere the field has 4 bytes, otherwise the field has just 2 bytes.\nThe starting offset of every field must be\nan integer multiple of the alignment number, hence the fillers.\nThe endianness don't change the position of any of these fields,\nit just change the order of the 2 or 4 bytes of the field itself\n(where *little* endian, known as \"swapped\" in CISIS,\nmeans that the last byte of the data\nis at the *lowest* address/offset).\nMost of that structure shown up to now\ncan be controlled through three parameters:\nthe **Format**, the **Intra-record alignment** and the **Endianness**.\nThese are the two possible formats:\n\n* *ISIS file format*:\n  The original standard documented in the reference manual\n* *FFI file format*:\n  An alternative to overcome the record size of 16 bytes (MFRL),\n  doubling it and all the other fields that has something to do\n  with the internal offsets of a record\n\nThese are the MST-specific options\nthat control the main structure of its records:\n\n* `--end`:\n  Tells whether the bytes of each field are `big` or `little` endian,\n  the `--le` and `--be` are shorthands for these, respectively\n* `--format`:\n  Choose the `isis` or `ffi` file format,\n  the `--isis` and `--ffi` are shorthands for these\n* `--packed/--unpacked`:\n  These control the leader/directory alignment,\n  *packed* means that their alignment is 2,\n  whereas *unpacked* means that their aligment is 4.\n\nThe MST file has a leading record called the *Control record*,\nwhose MFN (*Master file number*, here *file* stands for a record)\nis zero.\nIt has this 32-bytes structure\n(apart from a trailing filler of 32 bytes in CISIS):\n\n```raw\n -----------------------------\n|  Offset (bytes) | Structure |\n|-----------------+-----------|\n|      00-01      | CTLMFN.1  |\n|      02-03      | CTLMFN.2  |\n|      04-05      | NXTMFN.1  |\n|      06-07      | NXTMFN.2  |\n|      08-09      | NXTMFB.1  |\n|      10-11      | NXTMFB.2  |\n|      12-13      | NXTMFP    |\n|      14-15      | TYPE      |\n|      16-17      | RECCNT.1  |\n|      18-19      | RECCNT.2  |\n|      20-21      | MFCXX1.1  |\n|      22-23      | MFCXX1.2  |\n|      24-25      | MFCXX2.1  |\n|      26-27      | MFCXX2.2  |\n|      28-29      | MFCXX3.1  |\n|      30-31      | MFCXX3.2  |\n -----------------------------\n```\n\nThe most important field in there is the TYPE shown above,\nwhich is written as MFTYPE in the CDS/ISIS reference manual,\nbut the TYPE has actually two single-byte fields in it,\nand the order of these two\nis the only multi-field scenario that depends on the endianness:\n\n* MSTXL *(most significant byte)*:\n  The offset *shift* in all XRF entries (to be discussed)\n* MFTYPE *(least significant byte)*:\n  The master file type (should always be zero for user database files)\n\nWe've already seen the intra-record differences\namong distinct MST file formats,\nbut the overall structure itself has differences.\nA really important parameter for the overall MST file structure is the\n**Inter-record alignment**.\nSome details about the overall file structure and alignment are:\n\n* The file is divided as 512-bytes *blocks*,\n  and the last block should be filled up to the end\n* The first record must be the control record\n* The records are simply stacked one after another,\n  but with alignment constraints:\n  * The BASE and MFN fields of a record must be in the same block\n  * The record itself should have an alignment of 2 bytes\n    (word alignment, the ISIS default for inter-record alignment)\n\nThe *shift* name comes from the XRF file structure,\nwhich has just 32 bytes\nto store both the block, the offset and some flags.\nThe XRF should be capable of pointing to the address of every record\nin the MST file,\nhence some \"bit twiddling\" must be done to enable larger MST files.\nThis had been done through the MSTXL field,\nwhich represents the *shift*,\nthe number of times we must *bit-shift the offset to the right*.\nDoing so we lose the least significant bits,\nhence our offsets should always be aligned to \"2^shift\"\n(two raised to the power of *shift*).\nThat's the main inter-record alignment constraint we have.\n\nThese are the MST-specific options\nregarding the inter-record alignment:\n\n* `--control-len`:\n  Length of the control record, in bytes,\n  to control the first filler size\n* `--shift` (MST file output only):\n  The MSTXL value,\n  telling the inter-record alignment should be of at least\n  *2 raised to the power of MSTXL* bytes\n* `--shift4is3/--shift4isnt3`:\n  Toggle if MSTXL equals to 3 in a file or in `--shift`\n  should be regarded as 4,\n  it's a historical behavior of CISIS\n* `--min-modulus`:\n  The minimum inter-record alignment, in bytes (2 by default).\n  This option makes it possible to bypass the standard word alignment,\n  \"`--min-modulus=1 --shift=0`\" would make MST files\n  with byte-alignment (i.e., with no inter-record padding/filler)\n\nThere are three locking mechanisms in ISIS\nthat might be stored in an MST file:\n\n- EWLOCK *(Exclusive Write Lock)*:\n  It's a flag, stored in MFCXX3 (control record)\n- DELOCK *(Data Entry Lock)*:\n  It's a counter, stored in MFCXX2 (control record),\n  of how many records are locked at once\n- RLOCK *(Record Lock)*:\n  It's the sign of the MFRL (record length) of every record\n  (the record size is actually the absolute value of MFRL)\n\nUsually these makes no difference when the ISIS is just a static file\nthat no process is modifying,\nand the `ioisis` CLI ignores the EWLOCK and DELOCK\n(they can be accessed by `ioisis` as a library, though).\nThere's one option in `ioisis` to enable/disable\nthe interpretation of all these locks,\nand it's exposed to the CLI since it affects the RLOCK:\n\n* `--lockable/--no-locks`:\n  Control if the MFRL should be signed (lockable)\n  or unsigned (no RLOCK, doubling the record length limit)\n\nSeveral are the fillers (padding characters)\nthat might appear in the MST file\ndue to the several alignment constraints.\nAnother issue with the MST file\nis that it doesn't have one single filler for all these cases,\nand perhaps some tool in some specific architecture\nmight behave differently.\nAs the parser is strict (i.e., it checks the alignment and fillers),\nsome of these might need to be tuned before loading the MST file,\nand these are the commands that makes that possible:\n\n* `--filler`:\n  Default filler for unset filler options, but the record filler\n* `--record-filler`:\n  For the trailing record data, after the last field value\n  (the default is a whitespace)\n* `--control-filler`:\n  For the trailing bytes of the control record\n* `--slack-filler`:\n  For the leader/directory when `--unpacked`\n* `--block-filler`:\n  For the last bytes in a 512-bytes block\n  that don't belong to any record\n  (end of file or due to the \"MFN+BASE in the same block\" constraint)\n\nThe filler options above have a single parameter,\nwhich should always be a 2-characters string\nwith the filler byte code in hexadecimal.\n\nFinally, sometimes the input MST file is corrupt and can't be loaded,\ne.g. because the block filler isn't clean,\nor because a MFRL is smaller than the actual record data.\nSince the overall record structure has some internal constraints\n(sizes and offsets/addresses),\n`ioisis` can go ahead\nignoring the next few bytes that makes no sense as a new record.\nTo do so, one should call it with the *Invalid block padding* option\n(`--ibp`),\nwhose value can be:\n\n* `check` (default):\n  The strict behavior, `ioisis` crashes when some invalid data appears\n  in some offset that should have a record\n* `ignore`:\n  Silently skips the invalid data\n* `store`:\n  Put the trailing information in an artificial `ibp` field\n  of the output, in hexadecimal\n\n\n## Library\n\nA common data structure in the library for representing a single record\nis the *tidy list of tag-value pairs*, or **tl**.\nIt doesn't have anything to do with the `tidy`/`stidy` JSONL/CSV modes,\nit's just a way to store the data\navoiding the scattered structure of the raw record container.\nTo load data with the library:\n\n```python\nfrom ioisis import bruma, iso, mst, fieldutils\n\n# In the mst module, you must create a StructCreator instance\nmst_sc = mst.StructCreator(ibp=\"store\")\nwith open(\"file.mst\", \"rb\") as raw_mst_file:\n    for raw_tl in mst_sc.iter_raw_tl(raw_mst_file):\n        tl = fieldutils.nest_decode(raw_tl, encoding=\"cp1252\")\n        ...\n\n# For bruma.iter_tl the input must be a file name\nfor tl in bruma.iter_tl(\"file.mst\", encoding=\"cp1252\"):\n    raw_tl = fieldutils.nest_encode(raw_tl, encoding=\"utf-8\")\n    ...\n\n# The idea is similar for an ISO file, but ...\nfor raw_tl in iso.iter_raw_tl(\"file.iso\"):\n    tl = utf8_fix_nest_decode(raw_tl, encoding=\"latin1\")\n    ...\n\n# ... for ISO files, you can always use either a file name\n# or any file-like object open in \"rb\" mode\nwith open(\"file.iso\", \"rb\") as raw_iso_file:\n    for tl in iso.iter_tl(raw_iso_file, encoding=\"latin1\"):\n        ...\n```\n\nThe following generator functions/methods\nare the ones that appeared in the example above:\n\n* `mst.StructCreator.iter_raw_tl`: Read MST keeping data in bytestrings\n* `iso.iter_raw_tl`: Read ISO keeping data in bytestrings\n* `bruma.iter_tl`: Read MST already decoding its contents\n* `iso.iter_tl`: Read ISO already decoding its contents\n\nIt's worth noting that the following functions\nfrom the `fieldutils` module\nallows encoding/decoding all record fields/subfields at once:\n\n* `nest_encode`\n* `nest_decode`\n* `utf8_fix_nest_decode`\n\nThe latter is the same to `nest_decode`,\nbut uses the given encoding as a fallback,\ntrying first to decoded all the contents as UTF-8.\n\nWhat's the content of a single decoded *tl*?\nIt's a list of `[tag, value]` pairs (as lists or tuples), like:\n\n```raw\n[[\"5\", \"S\"],\n [\"6\", \"c\"],\n [\"10\", \"br1.1\"],\n [\"62\", \"Example Institute\"]]\n```\n\nOne can generate a single ISO record from a *tl*:\n\n```python\n\u003e\u003e\u003e from ioisis import iso, fieldutils\n\u003e\u003e\u003e tl = [[\"1\", \"test\"], [\"8\", \"it\"]]\n\u003e\u003e\u003e raw_tl = fieldutils.nest_encode(tl, encoding=\"utf-8\")\n\u003e\u003e\u003e raw_tl\n[[b'1', b'test'], [b'8', b'it']]\n\u003e\u003e\u003e con = fieldutils.tl2con(raw_tl, ftf=iso.DEFAULT_ISO_FTF)\n\u003e\u003e\u003e con\n{'dir': [{'tag': b'001'}, {'tag': b'008'}], 'fields': [b'test', b'it']}\n\u003e\u003e\u003e iso.DEFAULT_RECORD_STRUCT.build(con)\nb'000580000000000490004500001000500000008000300005#test#it##\\n'\n\n```\n\nThe process to create records is to convert them to the\n*internal [construct] container format* (or simply **con**),\nwhich is done by `fieldutils.tl2con`.\nTo create an MST file,\nyou can use the `build_stream` method of the `mst.StructCreator`,\nwhose first parameter should be a generator of *con* instances,\nand the second is the seekable file object.\n\nThere's still a third format, called the *record dict* format,\nwhich is based on the JSONL \"`--mode=field`\" output format.\nIt has less resources available internally to the library\nwhen compared with the abovementioned alternative,\nbut it might be simpler to use in some cases:\n\n```python\n\u003e\u003e\u003e iso.dict2bytes({\"1\": [\"testing\"], \"8\": [\"it\"]})\nb'000610000000000490004500001000800000008000300008#testing#it##\\n'\n\n# The same, but from the tl\n\u003e\u003e\u003e tl = [[\"1\", \"testing\"], [\"8\", \"it\"]]\n\u003e\u003e\u003e record = fieldutils.tl2record(tl)\n\u003e\u003e\u003e iso.dict2bytes(record)\nb'000610000000000490004500001000800000008000300008#testing#it##\\n'\n\n```\n\nTo load ISIS data from `bruma` or `iso`,\nyou can also use the `iter_records` function\nof the respective module,\nbut it's more customizable\nif you use the `fieldutils` converter functions:\n\n* `record2tl`\n* `tl2record`\n* `tl2con`\n\nPerhaps the simplest way to understand the behavior of the library\nis to use the CLI and to check the code of the called command.\n\n\n### Modules\n\nThe modules available in the `ioisis` package are:\n\n| **Module**    | **Content**                                         |\n|:-------------:|:---------------------------------------------------:|\n| `bruma`       | Everything about MST file processing based on Bruma |\n| `ccons`       | Custom construct classes                            |\n| `fieldutils`  | Field/subfield processing functions and classes     |\n| `iso`         | ISO parsing/building stuff tools on construct       |\n| `java`        | Java interfacing resources based on JPype1          |\n| `mst`         | MST/XRF parsing/building tools based on construct   |\n| `streamutils` | Classes for precise file/pipe processing            |\n| `__main__`    | CLI (Command Line Interface)                        |\n\nUsually, the only modules one would need from `ioisis`\nto use it as a library\nare `iso`, `mst`, `bruma` and `fieldutils`,\nthe remaining modules can be seen as internal stuff.\n\nBy default, the `mst` module doesn't use/create XRF files.\nOne can create/load XRF data using the struct created by\nthe `mst.StructCreator.create_xrf_struct` method.\n\n\n### ISO construct containers (lower level data access Python API)\n\nThe `iso` module\nuses the [Construct](https://github.com/construct/construct) library,\nwhich makes it possible to create\na declarative \"structure\" object\nthat can perform bidirectional building/parsing\nof bytestrings (instances of `bytes`)\nor streams (files open in the `\"rb\"` mode)\nfrom/to construct containers (dictionaries).\n\n\n#### Building and parsing a single record\n\nThis low level data access\ndoesn't perform any string encoding/decoding,\nso every *value* in the input dictionary\nused for building some ISO data\nshould be a raw bytestring.\nLikewise, the parser doesn't decode the encoded strings\n(tags, fields and metadata),\nkeeping bytestrings in the result.\n\nHere's an example\nwith a record in the \"minimal\" format expected by the ISO builder.\nThe values are bytestrings,\nand each directory entry matches its field value based on their index.\n\n```python\n\u003e\u003e\u003e lowlevel_dict = {\n...     \"dir\": [{\"tag\": b\"001\"}, {\"tag\": b\"555\"}],\n...     \"fields\": [b\"a\", b\"test\"],\n... }\n\n# Build a single ISO record bytestring from a construct.Container/dict\n\u003e\u003e\u003e iso_data = iso.DEFAULT_RECORD_STRUCT.build(lowlevel_dict)\n\u003e\u003e\u003e iso_data\nb'000570000000000490004500001000200000555000500002#a#test##\\n'\n\n# Parse a single ISO record bytestring to a construct.Container\n\u003e\u003e\u003e con = iso.DEFAULT_RECORD_STRUCT.parse(iso_data)\n\n# The construct.Container instance inherits from dict.\n# The directory and fields are instances of construct.ListContainer,\n# a class that inherits from list.\n\u003e\u003e\u003e [directory[\"tag\"] for directory in con[\"dir\"]]\n[b'001', b'555']\n\u003e\u003e\u003e con.fields  # Its items can be accessed as attributes\nListContainer([b'a', b'test'])\n\u003e\u003e\u003e len(con.fields) == con.num_fields == 2  # A computed attribute\nTrue\n\n# This function directly converts that construct.Container object\n# to a dictionary of already decoded strings in the the more common\n# {tag: [field, ...], ..} format (default ISO encoding is cp1252):\n\u003e\u003e\u003e iso.con2dict(con).items()  # It's a defaultdict(list)\ndict_items([('1', ['a']), ('555', ['test'])])\n\n```\n\n\n#### Other record fields\n\nEach ISO record is divided in 3 parts:\n\n* Leader (24 bytes header with metadata)\n* Directory (metadata for each field value, mainly its 3-bytes *tag*)\n* Fields (the field values themselves as bytestrings)\n\nThe *leader* has:\n\n* Single character metadata (`status`, `type`, `coding`)\n* Two numeric metadata (`indicator_count` and `identifier_len`),\n  which should range only from 0 to 9\n* Free room for \"vendor-specific\" stuff as bytestrings:\n  `custom_2` and `custom_3`,\n  where the numbers are their size in bytes\n* An entry map, i.e., the size of each field of the directory:\n  `len_len`, `pos_len` and `custom_len`,\n  which should range only from 0 to 9\n* A single byte, `reserved`, literally reserved for future use\n\n```python\n\u003e\u003e\u003e con.len_len, con.pos_len, con.custom_len\n(4, 5, 0)\n\n```\n\nActually, the `reserved` is part of the entry map,\nbut it has no specific meaning there,\nand it doesn't need to be a number.\nApart from the entry map and the not included length/address fields,\nnone of these metadata has any meaning when reading the ISO content,\nand they're all filled with zeros by default\n(the ASCII zero when they're strings).\n\n```python\n\u003e\u003e\u003e con.status, con.type, con.coding, con.indicator_count\n(b'0', b'0', b'0', 0)\n\n```\n\nLength and position fields that are stored in the record\n(`total_len`, `base_addr`, `dir.len`, `dir.pos`)\nare computed in build time and checked on parsing.\nWe don't need to worry about these fields,\nbut we can read them if needed.\nFor example, one directory record (a dictionary) has this:\n\n```python\n\u003e\u003e\u003e con.dir[1]\nContainer(tag=b'555', len=5, pos=2, custom=b'')\n\n```\n\nAs the default `dir.custom` field has zero length,\nit's not really useful for most use cases.\nGiven that, we've already seen all the fields there are\nin the low level ISO representation of a single record.\n\n\n#### Tweaking the field lengths\n\nThe ISO2709 specification tells us\nthat a directory entry should have exactly 12 bytes,\nwhich means that `len_len + pos_len + custom_len` should be 9.\nHowever, that's not an actual restriction for this library,\nso we don't need to worry about that,\nas long as the entry map have the correct information.\n\nLet's customize the length to get a smaller ISO\nwith some data in the `custom` field of the directory,\nusing a 8 bytes directory:\n\n```python\n\u003e\u003e\u003e dir8_dict = {\n...     \"len_len\": 1,\n...     \"pos_len\": 3,\n...     \"custom_len\": 1,\n...     \"dir\": [{\"tag\": b\"001\", \"custom\": b\"X\"}, {\"tag\": b\"555\"}],\n...     \"fields\": [b\"a\", b\"test\"],\n... }\n\u003e\u003e\u003e dir8_iso = iso.DEFAULT_RECORD_STRUCT.build(dir8_dict)\n\u003e\u003e\u003e dir8_iso\nb'0004900000000004100013100012000X55550020#a#test##\\n'\n\u003e\u003e\u003e dir8_con = iso.DEFAULT_RECORD_STRUCT.parse(dir8_iso)\n\u003e\u003e\u003e dir8_con.dir[0]\nContainer(tag=b'001', len=2, pos=0, custom=b'X')\n\u003e\u003e\u003e dir8_con.dir[1]  # The default is always zero!\nContainer(tag=b'555', len=5, pos=2, custom=b'0')\n\u003e\u003e\u003e dir8_con.len_len, dir8_con.pos_len, dir8_con.custom_len\n(1, 3, 1)\n\n```\n\nWhat happens if we try to build from a dictionary\nthat doesn't fit with the given sizes?\n\n```python\n\u003e\u003e\u003e invalid_dict = {\n...     \"len_len\": 1,\n...     \"pos_len\": 9,\n...     \"dir\": [{\"tag\": b\"555\"}],\n...     \"fields\": [b\"a string with more than 9 characters\"],\n... }\n\u003e\u003e\u003e iso.DEFAULT_RECORD_STRUCT.build(invalid_dict)\nTraceback (most recent call last):\n  ...\nconstruct.core.StreamError: Error in path (building) -\u003e dir -\u003e len\nbytes object of wrong length, expected 1, found 2\n\n```\n\n\n### ISO files, line breaking and delimiters\n\nThe ISO files usually have more than a single record.\nHowever, these files are created by simply concatenating ISO records.\nThat simple: concatenating two ISO files\nshould result in another valid ISO file\nwith all the records from both.\n\nAlthough that's not part of the ISO2709 specification,\nthe `iso.DEFAULT_RECORD_STRUCT` parser/builder object\nassumes that:\n\n* All lines of a given record but the last one\n  must have exactly 80 bytes,\n  and a line feed (`\\x0a`) must be included after that;\n* Every line must belong to a single record;\n* The last line of a single record must finish with a `\\x0a`.\n\nThat's the behavior of `iso.LineSplitRestreamed`,\nwhich \"wraps\" internally the record structure\nto give this \"line splitting\" behavior,\nbut that can be avoided by setting the `line_len` to `None` or zero\nwhen creating a custom record struct.\n\n\n#### Parsing/building data with meaningful line breaking characters\n\nSuppose we want to store these values:\n\n```python\n\u003e\u003e\u003e newline_info_dict = {\n...     \"dir\": [{\"tag\": b\"SIZ\"}, {\"tag\": b\"SIZ\"}, {\"tag\": b\"SIZ\"}],\n...     \"fields\": [b\"linux^c\\n^s1\", b\"win^c\\r\\n^s2\", b\"mac^c\\r^s1\"],\n... }\n\n```\n\nThat makes sense as an example of an ISO record\nwith three `SIZ` fields, each with three subfields,\nwhere the second subfield\nis the default newline character of some environment,\nand the third subfield is its size.\nAlthough can build that using the `DEFAULT_RECORD_STRUCT`\n(the end of line never gets mixed with the content),\nwe know beforehand that our values have newline characters,\nand we might want an alternative struct\nwithout that \"wrapped\" line breaking behavior:\n\n```python\n\u003e\u003e\u003e breakless_struct = iso.create_record_struct(line_len=0)\n\u003e\u003e\u003e newline_info_iso = breakless_struct.build(newline_info_dict)\n\u003e\u003e\u003e newline_info_iso\nb'000950000000000610004500SIZ001200000SIZ001100012SIZ001000023#linux^c\\n^s1#win^c\\r\\n^s2#mac^c\\r^s1##'\n\u003e\u003e\u003e newline_info_con = breakless_struct.parse(newline_info_iso)\n\u003e\u003e\u003e newline_info_simple_dict = dict(iso.con2dict(newline_info_con))\n\u003e\u003e\u003e newline_info_simple_dict\n{'SIZ': ['linux^c\\n^s1', 'win^c\\r\\n^s2', 'mac^c\\r^s1']}\n\u003e\u003e\u003e newline_info_iso == iso.dict2bytes(\n...     newline_info_simple_dict,\n...     record_struct=breakless_struct,\n... )\nTrue\n\n```\n\n\n#### Parsing/building with a custom line breaking and delimiters\n\nThe default builder/parser for a single record\nwas created with:\n\n```python\nDEFAULT_RECORD_STRUCT = iso.create_record_struct(\n    field_terminator=iso.DEFAULT_FIELD_TERMINATOR,\n    record_terminator=iso.DEFAULT_RECORD_TERMINATOR,\n    line_len=iso.DEFAULT_LINE_LEN,\n    newline=iso.DEFAULT_NEWLINE,\n)\n```\n\nWe can create a custom object using other values.\nTo use it, we'll pass that object\nas the `record_struct` keyword argument\nwhen calling the functions.\n\n\n```python\n\u003e\u003e\u003e simple_data = {\n...     \"OBJ\": [\"mouse\", \"keyboard\"],\n...     \"INF\": [\"old\"],\n...     \"SIZ\": [\"34\"],\n... }\n\u003e\u003e\u003e custom_struct = iso.create_record_struct(\n...     field_terminator=b\";\",\n...     record_terminator=b\"@\",\n...     line_len=20,\n...     newline=b\"\\n\",\n... )\n\u003e\u003e\u003e simple_data_iso = iso.dict2bytes(\n...     simple_data,\n...     record_struct=custom_struct,\n... )\n\u003e\u003e\u003e from pprint import pprint\n\u003e\u003e\u003e pprint(simple_data_iso.decode(\"ascii\"))\n('00096000000000073000\\n'\n '4500OBJ000600000OBJ0\\n'\n '00900006INF000400015\\n'\n 'SIZ000300019;mouse;k\\n'\n 'eyboard;old;34;@\\n')\n\u003e\u003e\u003e simple_data_con = custom_struct.parse(simple_data_iso)\n\u003e\u003e\u003e simple_data == iso.con2dict(simple_data_con)\nTrue\n\n```\n\nThe calculated sizes don't count the extra line breaking characters:\n\n```python\n\u003e\u003e\u003e simple_data_con.total_len, simple_data_con.base_addr\n(96, 73)\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscieloorg%2Fioisis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscieloorg%2Fioisis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscieloorg%2Fioisis/lists"}