{"id":13399017,"url":"https://github.com/sipcapture/pastash","last_synced_at":"2025-04-09T15:04:51.317Z","repository":{"id":16667040,"uuid":"80405659","full_name":"sipcapture/paStash","owner":"sipcapture","description":"pastaʃ'ʃ = Spaghetti I/O Event Data Processing, Interpolation, Correlation and beyond :spaghetti:","archived":false,"fork":false,"pushed_at":"2024-11-05T13:49:21.000Z","size":5870,"stargazers_count":103,"open_issues_count":30,"forks_count":28,"subscribers_count":12,"default_branch":"next","last_synced_at":"2025-04-09T15:04:46.311Z","etag":null,"topics":["amqp","asterisk","avaya","elasticsearch","freeswitch","grafana-loki","gundb","hep","kafka","log","logs","logstash","loki","netflow","node","parser","realtime","redis","statsd","zeromq"],"latest_commit_sha":null,"homepage":"http://sipcapture.io","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sipcapture.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["qxip"]}},"created_at":"2017-01-30T08:48:55.000Z","updated_at":"2025-04-01T15:09:13.000Z","dependencies_parsed_at":"2024-04-13T20:21:46.309Z","dependency_job_id":"9daa8467-7524-4bd7-a8bf-a25633250e77","html_url":"https://github.com/sipcapture/paStash","commit_stats":{"total_commits":584,"total_committers":15,"mean_commits":38.93333333333333,"dds":"0.34246575342465757","last_synced_commit":"fd122f15a6a10ae5b019269241ec7a8572479d61"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sipcapture%2FpaStash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sipcapture%2FpaStash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sipcapture%2FpaStash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sipcapture%2FpaStash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sipcapture","download_url":"https://codeload.github.com/sipcapture/paStash/tar.gz/refs/heads/next","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055284,"owners_count":21040157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","asterisk","avaya","elasticsearch","freeswitch","grafana-loki","gundb","hep","kafka","log","logs","logstash","loki","netflow","node","parser","realtime","redis","statsd","zeromq"],"created_at":"2024-07-30T19:00:33.493Z","updated_at":"2025-04-09T15:04:51.295Z","avatar_url":"https://github.com/sipcapture.png","language":"JavaScript","readme":"paStash\n====\n_\"When logs give you spaghetti, make pasta\"_\n\n\u003cimg src=\"http://i.imgur.com/wYjsCqz.png\"/\u003e\n\n[![ghcr push](https://github.com/sipcapture/paStash/actions/workflows/ghcr_push.yml/badge.svg)](https://github.com/sipcapture/paStash/actions/workflows/ghcr_push.yml)\n\nWhat is paStash ?\n---\n\nPaStasH _(pastaʃ'ʃ-utta)_ is a [NodeJS](http://nodejs.org) multi I/O processor supporting ingestion, decoding, interpolation and correlation of data - be it logs, packets, events and beyond. PaStash supports the Logstash configuration format and delivers cross-functionality comparable to _\"Beats\"_ with custom modules, providing a flexible and agnostig data pipelining tool.\n\n\nWhat can I do with paStash ?\n---\n\npaStash is designed manage spaghetti I/O with input, processors and output modules for all seasons, and can be useful in many scenarios, such as parsing logs to objects, distributing data to multiple formats, interexchanging and correlating protocols and streams, while interpolating and manipulating data intransit. paStash is developed using NodeJS, which is an ideal language for applications with many IO and offers:\n\n* lower memory footprint\n* lower cpu footprint\n* faster startup delay\n* ease of extension\n\npaStash configuration is compatible with logstash. You can easily replace a logstash node by a paStash one in most cases. The data are formatted in the same way to be compatible with logstash UIs.\n\nHow does it work ?\n===\n\nThe architecture is identical to logstash architecture. You have to instanciates plugins with the paStash core. There are three type of modules:\n\n* [inputs plugins](#inputs): where datas come into paStash. Examples: file, zeromq transport layer\n* [filter plugins](#filters): extract and manipulate fields from logs, like timestamps. Example: regex plugin\n* [outputs plugins](#outputs): where datas leave from paStash: Examples: ElasticSearch , zeromq transport layer.\n\n\nA typical paStash deployement contains agents to crawl logs and a log server.\n\nOn agent, paStash is configured whith inputs plugins to get logs from your software stack, and one output plugin to send logs to log server (eg. zeromq output plugin).\n\nOn log server, logs come trough a zeromq input plugin, are processed (fields and timestamps extraction), and send to ElasticSearch.\n\n\n\nHow to use it ?\n===\n\nInstallation\n---\n### NPM install\nPaStash and its dependencies can be easily installed with NPM _(provides `pastash` as command)_ as root or sudo user:\n```\nnpm install --unsafe-perm -g @pastash/pastash\n```\n\n### NPM plugin install\nPaStash can be extended with modules from the `@pastash` NPM keyspace\n```\nnpm install --unsafe-perm -g @pastash/output_loki\n```\n\n### Plugins list\n* See [paStash plugins](https://github.com/lmangani/pastash/tree/master/plugins) for plugins lists, examples and docs\n\n------------\n\nConfiguration formats\n---\n\nThere are two format for configuration. The legacy format use urls. The new one is identical to the [logstash config format](https://www.elastic.co/guide/en/logstash/current/configuration.html).\n\nNote : multiple configuration files can be used in parallel with the ``--config_dir`` switch.\n\n\nConfiguration by logstash config files\n---\n\nExample for a simple logging pipeline:\n```\ninput {\n  file {\n    path =\u003e '/tmp/toto.log'\n  }\n}\n\noutput {\n  loki {\n    host =\u003e localhost\n    port =\u003e 3100\n    path =\u003e \"/loki/api/v1/push\"\n  }\n}\n```\n\nYou can use ``if`` to have an [event dependent configuration](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html). See [here for details](docs/common_params.md).\nAs for urls, config can be specified\n\n* directly on the command line\n* in a file (use the ``--config_file`` switch)\n* in all files in a directory (use the ``--config_dir`` switch)\n\nNote : the implementation is young, all bugs reports are welcome.\nNote : both formats can be mixed.\n\nCommand lines params\n---\n\n* ``--log_level`` to change the log level (emergency, alert, critical, error, warning, notice, info, debug)\n* ``--log_file`` to redirect log to a log file.\n* ``--patterns_directories`` to add some directories (separated by ,), for loading config for regex plugin and grok plugins. Grok patterns files must be located under a ``grok`` subdirectory for each specified directory.\n* ``--db_file`` to specify the file to use as database for file inputs (see below)\n* ``--http_max_sockets`` to specify the max sockets of [http.globalAgent.maxSockets](http://nodejs.org/api/http.html#http_agent_maxsockets). Default to 100.\n* ``--alarm_file`` to specify a file which will be created if paStash goes in alarm mode.\n\nExamples\n---\n\nConfig file for an agent:\n```\ninput {\n  file {\n    path =\u003e \"/var/log/nginx/access.log\"\n  }\n}\n\noutput {\n  zeromq {\n    address =\u003e [\"tcp://log_server:5555\"]\n  }\n}\n```\n\nConfig file for log server:\n```\ninput {\n  zeromq {\n    address =\u003e [\"tcp://0.0.0.0:5555\"]\n  }\n}\n\nfilter {\n  regex {\n    pattern =\u003e http_combined\n  }\n}\n\noutput {\n  elasticsearch {\n    host =\u003e localhost\n    port =\u003e 9200\n  }\n}\n```\nSee our [wiki](https://github.com/sipcapture/paStash/wiki) for many more [examples](https://github.com/sipcapture/paStash/wiki)\n\n\n\nLicense\n===\n\npaStash Copyright 2016 - 2020 QXIP BV\n\nnode-logstash Copyright 2012 - 2014 Bertrand Paquet\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n\n\n----\n#### Made by Humans\nThis Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons.\u003cbr\u003e\nIf you use this software in production, please consider supporting its development with contributions or [donations](https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=donation%40sipcapture%2eorg\u0026lc=US\u0026item_name=SIPCAPTURE\u0026no_note=0\u0026currency_code=EUR\u0026bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest)\n\n[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=donation%40sipcapture%2eorg\u0026lc=US\u0026item_name=SIPCAPTURE\u0026no_note=0\u0026currency_code=EUR\u0026bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) \n","funding_links":["https://github.com/sponsors/qxip","https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=donation%40sipcapture%2eorg\u0026lc=US\u0026item_name=SIPCAPTURE\u0026no_note=0\u0026currency_code=EUR\u0026bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest"],"categories":["Contribution Guidelines"],"sub_categories":["Gateway Applications"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsipcapture%2Fpastash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsipcapture%2Fpastash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsipcapture%2Fpastash/lists"}