{"id":19120245,"url":"https://github.com/i4ds/stix_python_pub023","last_synced_at":"2025-09-03T13:38:20.235Z","repository":{"id":85538381,"uuid":"176323110","full_name":"i4Ds/STIX_python_PUB023","owner":"i4Ds","description":"STIX data processing pipeline written in python","archived":false,"fork":false,"pushed_at":"2021-05-01T10:43:26.000Z","size":109617,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-18T01:51:29.070Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/i4Ds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-18T16:09:13.000Z","updated_at":"2021-05-01T10:43:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"4226798e-641f-40fd-8a19-d7d02619d513","html_url":"https://github.com/i4Ds/STIX_python_PUB023","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/i4Ds/STIX_python_PUB023","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i4Ds%2FSTIX_python_PUB023","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i4Ds%2FSTIX_python_PUB023/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i4Ds%2FSTIX_python_PUB023/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i4Ds%2FSTIX_python_PUB023/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/i4Ds","download_url":"https://codeload.github.com/i4Ds/STIX_python_PUB023/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i4Ds%2FSTIX_python_PUB023/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273453254,"owners_count":25108470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T05:13:28.103Z","updated_at":"2025-09-03T13:38:20.211Z","avatar_url":"https://github.com/i4Ds.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## Data processing pipeline running on STIX pub023 server\n\nThis package includes STIX raw data parser, ior reader, \ndata parser Qt GUI, calibration data processing code, and STIX pipeline data processing code. \n\n\n\n\n### 2. Installation\n   The package relies on python3 and some extra python modules. \n   \n\n    \n\n#### 2.1 Python3 and pip3 installation\non Linux (Ubuntu)\n\n```console\nsudo  apt-get install python3\nsudo apt install python3-pip\n```\nOn windows\n\n  - Download  python3.6 executable installer\n \n     64-bit Windows: https://www.python.org/ftp/python/3.6.7/python-3.6.7-amd64.exe \n   \n     32-bit Windows: https://www.python.org/ftp/python/3.6.7/python-3.6.7.exe \n \n  - Install python3\n \n    When installing python, choose `customize installation`, \n  `install pip` and `add python path to the system environment`. \n  \n#### 2.2 STIX parser Installation\n\n##### 2.2.1 User\nThis package is hosted on PyPI and the most recent stable version can be installed with pip:\n\n```sh\n  pip3 install stix_parser\n``` \n\nThis package can also be installed step by step:\n\n  (1) Download the source code and then unzip the file: \n \n    https://github.com/i4Ds/STIX-dataviewer/archive/master.zip\n   \n   or \n   ```sh\n   git clone https://github.com/i4Ds/STIX-python-data-parser\n   \n   ```\n   (2) Dependency installation \n  ```cmd\n   pip3 install numpy xmltodict PyQt5 pyqtchart scipy pymongo python-dateutil\n````\n  (3)Install the package to python libarary path:\n\nNote that this step is optional.\n```bash\npython3 setup.py install  \n```\n\n##### 2.2.2 Developer\n```\npip install -e .[dev]\n```\n\n### 3. Usage\n\n#### 3.1. As a command-line STIX data parser\n\nUsage:\n```bash\npython3 -m stix_parser.apps.parser\npython -m stix_parser.apps.stix_quicklook\npython -m stix_parser.apps.stix_parser_gui\n```\nIf you have installed the package to python library path, you just need to execute:\n```bash\nstix-parser\n```\nArguments:\n```console\nUsage: parser.py [-h] -i [INPUT] [-o OUTPUT] [--idb IDB] [--opf {tuple,dict}]\n                 [-t {binary,ascii,xml}] [--wdb] [--db-host DB_HOST]\n                 [--db-port DB_PORT] [--db-user DB_USER] [--db-pwd DB_PWD]\n                 [-m COMMENT] [--SPID [SPID [SPID ...]]]\n                 [--services [SERVICES [SERVICES ...]]] [-v VERBOSE]\n                 [-l LOGFILE]\n\noptional arguments:\n  -h, --help            show this help message and exit\n\nRequired arguments:\n  -i [INPUT]            Input raw data filename.\n\nOptional arguments:\n  -o OUTPUT             Output python pickle filename.\n  --idb IDB             IDB filename (sqlite3).\n  --opf {tuple,dict}    format to store output parameters. \n  -t {binary,ascii,xml}\n                        Input file type. Three types (binary, ascii or xml)\n                        are supported. Filename extensions will be used to detect file types if not specified.\n  --wdb                 Write decoded packets to local MongoDB.\n  --db-host DB_HOST     MongoDB host IP.\n  --db-port DB_PORT     MongoDB host port.\n  --db-user DB_USER     MongoDB username.\n  --db-pwd DB_PWD       MongoDB password.\n  -m COMMENT            comment\n  --SPID [SPID [SPID ...]]\n                        Only to parse packets of the given SPIDs.\n  --services [SERVICES [SERVICES ...]]\n                        Only to parse packets of the given service types.\n  -v VERBOSE            Logger verbose level\n  -l LOGFILE, --log LOGFILE\n                        Log filename\n```\n\nExample:\n```console\npython3 stix_parser/apps/parser.py -i \u003cRAW_DATA_FILENAME\u003e -o \u003cOUTPUT\u003e  -v  \u003cVerbose level\u003e\n```\n\n\n#### 3.2. Using the parser in your own code.  \n  \n  Here are several examples.\n - Example 1\n\n   Parse a raw data file and dump the decode packets to a python pickle file\n\n```python\n#!/usr/bin/python3 \nfrom stix_parser.core import stix_parser\nparser = stix_parser.StixTCTMParser()\nparser.parse_file('raw.binary', 'output.pkl')\n```\n - Example 2\n Parse a raw data file and print the content of the decoded packets:\n\n```python\n#!/usr/bin/python3 \nfrom stix_parser.core import stix_parser\n\nf=open('raw.binary','rb')\nbuffer=f.read()\n\nparser = stix_parser.StixTCTMParser()\npackets=parser.parse_binary(buffer)\nfor packet in packets:\n  print(packet['header'])\n  print(packet['parameters'])\n```\n\n - Example 3:\n \n Parse a hexadecimal  string: \n```python\n\n#!/usr/bin/python3 \nimport pprint\nfrom stix_parsr.core import stix_parser\nparser = stix_parser.StixTCTMParser()\ndata='0d e5 c3 ce 00 1a 10 03 19 0e 80 00 87 46 6e 97 04 80 00 87 46 00 00 00 00 00 00 00 00 00 00 00 00'\npackets=parser.parse_hex(data)\n\npprint.pprint(packets)\n\n```\n\nOutput:\n\n```python\n{\n 'header': {'APID': 1509,\n            'APID_pid': 94,\n            'PUS': 16,\n            'SCET': 2147518278.4319916,\n            'SPID': 54103,\n            'SSID': 4,\n            'TMTC': 'TM',\n            'TPSD': -1,\n            'UTC': '2068-01-19T12:51:18.431',\n            'category': 5,\n            'coarse_time': 2147518278,\n            'descr': 'STIX HK report - SID 4',\n            'destination': 14,\n            'fine_time': 28311,\n            'header_flag': 1,\n            'length': 17,\n            'packet_id': 3557,\n            'packet_type': 0,\n            'process_id': 222,\n            'seg_flag': 3,\n            'segmentation': 'stand-alone packet',\n            'seq_count': 974,\n            'service_subtype': 25,\n            'service_type': 3,\n            'unix_time': 3094203078.4319916,\n            'version': 0},\n 'parameters': [('NIX00020', 4, '', []),\n                ('NIX00059', 2147518278, '', []),\n                ('NIXD0059', 0, 'NotAvailable', []),\n                ('NIXD0060', 0, 'NoSignNThrFlux', []),\n                ('NIXD0061', 0, 'NoFlareDetect', []),\n                ('NIXG0020', 0, '', []),\n                ('NIX00283', 0, '', []),\n                ('NIX00284', 0, '', []),\n                ('NIX00063', 0, '', []),\n                ('NIXD0064', 0,, 'False', []),\n                ('NIXG0064', 0, '', []),\n                ('ZZPAD032', 0, '', [])]}\n```\n\nEach parameter has a structure as follows:\n\n - The first column: Parameter name,  \n - The second column: raw value, \n - The third column: engineering value, decompressed value, or an empty string\n - The fourth column:  an empty list, or its children if it is a repeater. \n\n\n\n\n\n#### 3.3 Using the GUI \n##### 3.3.1 Run the GUI\n```bash\npython stix_parser/apps/stix_parser_gui.py\n```\nor \n```bash\nstix-parser-gui\n```\nif you have installed the package to python library path.\n\n\n##### 3.3.2 GUI basic functions\n  The GUI allows parsing/loading and displaying STIX packets in the following data formats/sources:\n   - STIX raw data\n   - SOC XML format\n   - MOC ASCII format\n   - packets stored in  python pickle files (pkl and pklz)\n   - packets stored in NoSQL database MongoDB\n   - packets received from TSC via socket\n   - STIX TM binary hex string copied from the clipboard\n  The GUI also allows plotting parameters as a function of timestamp or packet number. \n  \n##### 3.3.3  GUI plugins\n  One could create plugins to analyze the packets displayed in the GUI.\n   The plugin manager can be loaded by clicking \"Tool-\u003ePlugins\" or the plugin icon in the toolbar. \n  Here is a plugin example:\n````python\n#plugin example\nimport pprint\nclass Plugin:\n    def __init__(self,  packets=[], current_row=0):\n        self.packets=packets\n        self.current_row=current_row\n        print(\"Plugin  loaded ...\")\n        \n    def run(self):\n        # your code goes here\n        print('current row')\n        print(self.current_row)\n        if len(self.packets)\u003e1:\n            pprint.pprint(self.packets[self.current_row])\n            \n````\n  More plugin examples are available in  stix_parser/plugins/\n\n\n##### 3.3.4 GUI screenshots\n\n![GU data parser GUI](screenshots/stix_parser_1.jpg)\n![GU data parser GUI](screenshots/stix_parser_2.jpg)\n\n\n\n### 4. How to convert IDB from mdb format  to a sqlite database on Linux (Ubuntu)\n\n- Install mdbtools and sqlite3\n```sh\nsudo apt-get install mdbtools sqlite3  \n```\n\n- Use the script  idb/mdb2sql.sh to convert mdb to sql\n```bash\nsh idb/mdb2sql.sh  STIX_IDB.mdb \u003e idb.sql\n\n```\n- Append the following lines to the end of the generated sql file\n```sh\nupdate PCF set PCF_WIDTH=16 where PCF_NAME=\"NIX00123\";\nupdate PCF set PCF_WIDTH=8 where PCF_NAME=\"ZZPAD008\";\nupdate PCF set PCF_WIDTH=16 where PCF_NAME='ZZPAD016';\nupdate PCF set PCF_WIDTH=24 where PCF_NAME='ZZPAD024';\nupdate PCF set PCF_WIDTH=32 where PCF_NAME='ZZPAD032';\ndrop table if exists IDB;\ncreate table IDB(\ncreation_datetime  datetime,\nversion  varchar(64) not null\n);\ninsert into IDB (creation_datetime, version) values (current_timestamp, '2.26.28');\n```\nPlease replace the string \"2.26.28\" with your actual IDB version.\n\n\n- Create IDB sqlite3 database using the sql file\n```bash\nsqlite3 idb.sqlite3 \u003c idb.sql\n```\n\nOne may see some errors. For example, \" table  Name AutoCorrect Save Failures already exists\".  \nThose tables are not used by this parser. They can be removed from the sql script. \n\n- Copy idb.sqlite3 to idb/\n\n\n\n\n### 5. Compilation with cython for distribution\nexecute \n```sh\npython3 compile.py build_ext --inplace\n```\nThe generated *.c and the original *py files can be deleted for distribution.\n\n\n\n### 6. Run daemons\n 1) Parser daemons\n  ```sh\n  nohup python3 stix/app/parser_daemon.py \u0026\n  ```\n  It checks the directories defined in stix/core/config.py if there is a new data file every minute. If so, the file will be parsed and the decoded packets will be written to MongoDB\n  2) Calibration\n  ```sh\n  nohup python3 stix/analysis/calibration.py \u0026\n  ```\n  It performs calibration analysis every 10 minutes for new calibration spectrum.\n\n\n### 7. SPICE kernel data update\n  1) download the latest SPICE kernel from STIX GFTS server\n  2) Unzip the zip file, replace the following file with the latest version\n      stix/data/SPICE/kernels/sclk/solo_ANC_soc-sclk_xxxx_V01.tsc\n  \n      \n      \n   3) Edit stix/core/config.py, update the tsc filename\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi4ds%2Fstix_python_pub023","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fi4ds%2Fstix_python_pub023","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi4ds%2Fstix_python_pub023/lists"}