{"id":19665184,"url":"https://github.com/alttch/pulr","last_synced_at":"2025-04-28T22:31:03.377Z","repository":{"id":51387903,"uuid":"290875337","full_name":"alttch/pulr","owner":"alttch","description":"pull devices and transform data into events","archived":false,"fork":false,"pushed_at":"2023-04-14T14:18:34.000Z","size":211,"stargazers_count":23,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T12:56:31.284Z","etag":null,"topics":["automation","data","data-collection","data-conversion","ethernet-ip","industiral","modbus","plc","plc-programming","snmp"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alttch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-27T20:30:42.000Z","updated_at":"2024-09-13T00:51:06.000Z","dependencies_parsed_at":"2022-09-02T06:30:54.959Z","dependency_job_id":null,"html_url":"https://github.com/alttch/pulr","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fpulr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fpulr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fpulr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alttch%2Fpulr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alttch","download_url":"https://codeload.github.com/alttch/pulr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251397576,"owners_count":21583034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","data","data-collection","data-conversion","ethernet-ip","industiral","modbus","plc","plc-programming","snmp"],"created_at":"2024-11-11T16:21:31.445Z","updated_at":"2025-04-28T22:31:02.344Z","avatar_url":"https://github.com/alttch.png","language":"Rust","readme":"# Pulr - pull fieldbus devices and generate events\n\nNOTE: the project is no longer supported. All the functionality has been\nintegrated into [EVA ICS v4](https://www.eva-ics.com/).\n\n\u003cimg src=\"https://img.shields.io/badge/license-Apache%202.0-green\" /\u003e \u003cimg\nsrc=\"https://img.shields.io/badge/rust-2018-pink.svg\" /\u003e\n\n## What is Pulr\n\nPulr is a small command-line tool, which can pull data from the device and\nconvert it into events. Meaning the data is outputted ONLY when it's changed.\n\nThe data is always being outputted to STDOUT, you can grep it or use any\npipeline processor (I use [vector](https://vector.dev/)) to push it via HTTP,\ninto databases etc.\n\nBefore the output, any data part can be transformed: converted, divided,\nmultiplied, rounded and so on.\n\nThe data can be outputted as plain text or as nd-json.\n\n```shell\npulr -F modbus.yml -L\n```\n\n```json\n{\"id\":\"sensor:axon/din1.value\",\"value\":0}\n{\"id\":\"sensor:axon/din2.value\",\"value\":1}\n{\"id\":\"sensor:axon/din3.value\",\"value\":1}\n{\"id\":\"sensor:axon/din4.value\",\"value\":0}\n{\"id\":\"unit:axon/dout1.status\",\"value\":1}\n{\"id\":\"unit:axon/dout2.status\",\"value\":1}\n{\"id\":\"unit:axon/dout3.status\",\"value\":0}\n{\"id\":\"unit:axon/dout4.status\",\"value\":0}\n{\"id\":\"unit:axon/aout.value\",\"value\":0.00619}\n{\"id\":\"sensor:axon/ain.value\",\"value\":5.2045}\n{\"id\":\"unit:tests/u1.status\",\"value\":0}\n{\"id\":\"unit:tests/u2.status\",\"value\":0}\n```\n\n## Installing\n\n[Download a static binary from releases\npage](https://github.com/alttch/pulr/releases) and enjoy.\n\n[libplctag](https://github.com/libplctag/libplctag) is statically linked with\nLinux binaries. For Windows binaries, *plctag.dll* is included into the release\nzip archive.\n\n## Building from source\n\nEthernet/IP support requires\n[libplctag](https://github.com/libplctag/libplctag), download and install it:\n\n```shell\ngit clone https://github.com/libplctag/libplctag\ncd libplctag\ncmake CMakeLists.txt\nmake\nsudo make install\nsudo ldconfig\n```\n\nAfter, either use Makefile targets or run cargo manually:\n\n```shell\ncargo build --release # or custom options\n```\n\n## Configuring\n\nLook in ./examples for the example configurations.\n\n## How does it work\n\nOne Pulr instance pulls one piece of the hardware equipment. The goal is to\npull and process the data as fast as possible, but die as soon as any errors\noccur. Pulr is built to be started by supervisor, which collects the data from\nit and restarts the process on crashes.\n\n## Is it fast enough?\n\nPulr is written in Rust. So it's rocket-fast and super memory efficient. You\ncan start a hundred of Pulr processes on a single machine and barely notice any\nload.\n\nThe first (draft) Pulr version was written in Python, it isn't supported any\nlonger, but kept in \"legacy0\" branch.\n\n## Protocols\n\nCurrently supported:\n\n* Modbus (TCP/UDP only)\n* SNMP (v2)\n* Ethernet/IP (Allen Bradley-compatible)\n\nNeed to pull data from Modbus RTU? Use our free [Modbus gateway\nserver](https://github.com/alttch/modbusgw).\n\n\n## Data transformers\n\n* **calc\\_speed** - calculate value growing speed, useful for SNMP interface\n  counters\n\n* **multiply**, **divide**, **round**\n\n## Output type\n\n* text (aliases: stdout, plain, \"-\") - output the data as plain text, default\n* ndjson (alias: json) - output the data as newline delimited JSON\n* csv - comma-separated values\n* eva/datapuller - specific type for [EVA ICS](https://www.eva-ics.com/)\n\nOptional field \"time-format\" adds time to data output. Valid values are:\n\"rfc3339\", \"timestamp\" (alias: raw).\n\n### JSON output customization\n\nBy default, data in JSON is outputted as\n\n```json\n{ \"time\": \"time rfc 3339/timestamp\", \"id\": \"metric id\", \"value\": \"event value\" }\n{ \"time\": \"time rfc 3339/timestamp\", \"id\": \"metric id\", \"value\": \"event value\" }\n{ \"time\": \"time rfc 3339/timestamp\", \"id\": \"metric id\", \"value\": \"event value\" }\n```\n\nSpecifying output format as **ndjson/short** (aliases: *ndjson/s*,\n*json/short*, *json/s*), this can be switched to \"short\" format, with only 2\nfields: \"time\" and \"metric id\":\n\n```json\n{ \"time\": \"time rfc 3339/timestamp\", \"metric id\": \"event value\" }\n{ \"time\": \"time rfc 3339/timestamp\", \"metric id\": \"event value\" }\n{ \"time\": \"time rfc 3339/timestamp\", \"metric id\": \"event value\" }\n```\n\n## Real life example\n\n### Running manually\n\nConsider you have a device, want to collect metrics from it and store them into\n[InfluxDB](https://www.influxdata.com/). Pulr comes with a tiny tool called\n**ndj2influx**, which allows parsing NDJSON data and storing it directly into\nInfluxDB.\n\nIt's highly recommended to set \"time-format\" (rfc3339 or raw, doesn't matter)\nto have metrics stored with the same timestamp the pull request has been\nperformed.\n\nAnd full command to get and store metrics will be:\n\n```shell\npulr -F /path/to/pulr-config.yml -L -O ndjson | \\\n     ndj2influx http://\u003cINFLUXDB-IP-OR-HOST\u003e:8086 \\\n         \u003cDATABASE_NAME\u003e @device1 -U \u003cDB_USERNAME\u003e:\u003cPASSWORD\u003e -M id -v\n```\n\nLet's explain all options:\n\n* pulr option *-L* tells Pulr to work in loop, continuously pulling the device.\n\n* pulr option *-O ndjson* makes sure Pulr will output data in NDJSON format.\n\n* first two ndj2influx options specify InfluxDB API URL and database name\n\n* the next option should specify base metric column (device name). As pulr\n  doesn't output it, set it for all metrics to *@device1*\n\n* option *-U* is used to pass InfluxDB authentication. If auth isn't turned on,\n  the option is not requited.\n\n* option *-M id* is to use \"id\" column as metric id.\n\n* option *-v* is for verbose output and can be omitted in production.\n\nThe same result is produced with commands:\n\n```shell\npulr -F /path/to/pulr-config.yml -L -O ndjson/short | \\\n     ndj2influx http://\u003cINFLUXDB-IP-OR-HOST\u003e:8086 \\\n         \u003cDATABASE_NAME\u003e @device1 -U \u003cDB_USERNAME\u003e:\u003cPASSWORD\u003e -v\n```\n\neverything is almost the same, except Pulr is told to produce \"short\"\n(id=value) JSON output and option *-M id* for ndj2influx can be omitted.\n\nIt's recommended to hide password from command line arguments on production\nusing *INFLUXDB_AUTH* env variable:\n\n```shell\npulr -F /path/to/pulr-config.yml -L -O ndjson | \\\n     INFLUXDB_AUTH=\u003cDB_USERNAME\u003e:\u003cPASSWORD\u003e ndj2influx \\\n        http://\u003cINFLUXDB-IP-OR-HOST\u003e:8086 \u003cDATABASE_NAME\u003e @device1 -M id -v\n```\n\n### Running with supervisor\n\nBoth tools will crash as soon as any problem occurs. They're made this way,\nbecause in production \"it's better to crash and be restarted, than freeze\".\n\nTo automatically restart the tools, let's use any supervisor, e.g.\n[Supervisord](http://supervisord.org/) (available in almost all Linux distros).\n\nCreate a simple config and put it to */etc/supervisor/conf.d/pulr-device1.conf*\n\n```ini\n[program:pulr-device1]\ncommand=sh -c \"sleep 1 \u0026\u0026 pulr -F /path/to/pulr-config.yml -L -O ndjson | ndj2influx http://\u003cdbhost\u003e:8086 \u003cDATABASE_NAME\u003e @router1 -M id -v\"\nenvironment=INFLUXDB_AUTH=\u003cDB_USERNAME\u003e:\u003cPASSWORD\u003e\nautorestart=true\nautostart=true\npriority=100\nevents=PROCESS_STATE\n```\n\nThat's all. Supervisord will monitor the processes and restart them if\nnecessary.\n\n## Rust version difference\n\nAs it was mentioned above, Rust version is fast. It's very fast and efficient.\nHowever it differs from the draft Python version:\n\n* \"version\" field in configuration file should be set to \"2\"\n\n* \"transform\" syntax was changed a little bit, see config examples. \"speed\"\n  function renamed to \"calc\\_speed\".\n\n* more command line args.\n\n* No more \"bit2int\" and \"int2bit\" transformers, it's hardly to imagine where\n  they could be useful.\n\n## Troubleshooting\n\nBefore reporting a problem, try running Pulr with verbose output (\"-v\" command\nline flag).\n\n### Handling pull timeouts\n\nRun pulr as:\n\n    PULR_VERBOSE_WARNINGS=1 pulr \u003coptions\u003e\n\n## Bugs, feature requests, patches\n\nYou welcome.\n\nJust:\n\n* Outputs. No outputs are planned, except STDOUT. Use pipeline converters.\n\n* Pulling more than one device simultaneously. Isn't planned, start another\n  Pulr process instead.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falttch%2Fpulr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falttch%2Fpulr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falttch%2Fpulr/lists"}