{"id":13936833,"url":"https://github.com/sunary/ank","last_synced_at":"2025-04-28T11:51:12.356Z","repository":{"id":53525479,"uuid":"61674214","full_name":"sunary/ank","owner":"sunary","description":"Python data pipeline","archived":false,"fork":false,"pushed_at":"2023-05-29T00:22:09.000Z","size":160,"stargazers_count":57,"open_issues_count":3,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-06T11:41:58.201Z","etag":null,"topics":["ank","chain","dependency-injection","distributed-systems","docker","microservices","python-microservices","queue","schedule"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sunary.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-21T23:31:37.000Z","updated_at":"2024-05-06T08:17:38.000Z","dependencies_parsed_at":"2024-05-02T23:03:06.465Z","dependency_job_id":"c1db5671-116a-4466-9aab-de92faabf731","html_url":"https://github.com/sunary/ank","commit_stats":{"total_commits":101,"total_committers":4,"mean_commits":25.25,"dds":"0.25742574257425743","last_synced_commit":"f5c783aa51227aea9d8860c649533deed3149852"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunary%2Fank","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunary%2Fank/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunary%2Fank/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunary%2Fank/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sunary","download_url":"https://codeload.github.com/sunary/ank/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235388545,"owners_count":18981945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ank","chain","dependency-injection","distributed-systems","docker","microservices","python-microservices","queue","schedule"],"created_at":"2024-08-07T23:03:02.363Z","updated_at":"2025-01-24T05:12:12.180Z","avatar_url":"https://github.com/sunary.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## ANK - Python Streaming system ##\n\n\n### Overview: ###\n Python Streaming system, REST-API and Schedule task using queue message(rabbitMQ, zeroMQ, kafka)\n Each processor is a chain, put it together to make a pipeline.\n \n \n### Support chain model:\n* 1 - 1:\n    ```\n    processor - processor\n    ```\n* 1 - n:\n    ```\n              / processor\n    processor - processor\n              \\ processor\n    ```\n* n - 1:\n    ```\n    processor \\\n    processor - processor\n    processor /\n    ```\n* n - n:\n    ```\n    processor - processor    processor - processor    processor - processor\n    processor - processor    processor /              processor \u003c \n    processor - processor    processor - processor    processor - processor\n    ```\n* join message:\n    ```\n    message1 \\\n    message2 - [message1, message2, message3]\n    message3 /\n    ```\n* split message:\n    ```\n                                   / message1\n    [message1, message2, message3] - message2\n                                   \\ message3\n    ```\n    \n\n### Requirements: ###\n* Python 2.x\n* Python 3.x\n\n\n### How to use: ###\n* **Install:**\n    * From github:\n        - clone this project: `git clone git@github.com:sunary/ank.git`\n        - install: `python setup.py install`\n    * From pypi:\n        - `pip install ank`\n        \n* **Create new Service:**\n    ```shell\n    $ ank -c TestService\n    $ cd TestService\n    ```\n    then continue using below commands\n\n* **Edit app (processor.py):**\n    * *Example:*\n    ```python\n    from ank.components.pipe_app import PipeApp\n\n    class ExampleApp(PipeApp):\n    \n        def start(self):\n            for i in range(100):\n                self.chain_process(i)\n    \n        def process(self, message=None):\n            '''\n            Args:\n                message: {'content': (*) 'content of message',\n                          'flags': (list|tuple) 'define next process will be use'}\n                              raise TypeError if you don't declare this in return of before branching-processor\n                              if 'flags' == [True, True]: process both in next branching-processors\n                              if 'flags' == [True, False]: process 1st processor in next branching-processors\n                              if 'flags' == [False, True]: process 2nd processor in next branching-processors\n                              if 'flags' == [False, False]: no processor, stop chain\n                          is None: stop chain\n            '''\n            return message['content'] + 1\n    ```\n    \n* **Edit services and chains (services.yml):**\n    * *Syntax:*\n    ```yaml\n    services:\n      Object1:\n        - class: module.name.ClassName\n        - arguments: [$Object, %variable%] \n      \n      AnkChain2:\n        - class: ank.components.module_name.XXXApps\n        - arguments: ~\n        \n    chains:\n      - Object1\n      - AnkChain2\n    ```\n    * *Example:*\n    ```yaml\n    services:\n      StartApp:\n        class: processor.StartApp\n        arguments: [$MongoClient, $Redis, '%batch_size%']\n    \n      Mongodb:\n        class: pymongo.MongoClient\n        arguments: ['%mongo_host%', '%mongo_port%']\n    \n      Redis:\n        class: redis.client.StrictRedis\n        arguments: ['%redis_host%', '%redis_port%']\n    \n      OtherApp:\n        class: processor.OtherApp\n        arguments: ~\n    \n      LogApp:\n        class: components.log_app.LogApp\n        arguments: ~\n    \n    chains:\n      - StartApp\n      - LogApp\n      - OtherApp\n      - [OddApp, EvenApp] # will be processed depend on key `flags`\n      - OtherApp\n    ```\n    ANK will read top-down `chains`, find correspond `services` and get parameters from `settings.yml`.\n    \n* **Generate and edit setting (settings.yml):**\n\n     ```shell\n     $ ank create -c BaseApp\n     ```\n\n    * *Example:*\n\n    ```yaml\n    parameters:\n      mongo_host: localhost\n      mongo_port: 27017\n      \n      redis_host: localhost\n      redis_port: 6379\n      \n      queue_uri: 'amqp://admin:admin@localhost:5672/'\n      exchange_name: InputExchange\n      routing_key: ExchangeToQueue\n      \n      batch_size: 100\n    ```\n\n    Help you create `settings` template file. Just rename from `_settings.yml` to `settings.yml` and fill in values.\n    \n* **Build Service (create docker image) and run:**\n\n    ```shell\n    $ ank build\n    $ docker run --entrypoint /bin/sh $docker_image_id\n    ```\n    \n* **Generate processor (_processor.py):**\n    \n    ```shell\n    $ ank gen_processor -fs settings.yml\n    ```\n* **Generate setting (_setting.yml):**\n\n    ```shell\n    $ ank gen_setting -fs _setting.yml\n    ```\n* **Run Service:**\n\n    ```python\n    from ank import program_loader\n    program_loader.main(file_setting='settings.yml')\n    ```\n\n    or\n\n    ```shell\n    $ ank run -fs settings.yml\n    ```\n    \n### Base Apps: ###\n* **PipeApp:** Pipeline App.\n* **APIApp:** REST-API interface using flask.\n* **ScheduleApp:** Using crondtab-time format to set schedule.\n\n\n### Component Apps: ###\n* **LogApp:** Log every messages.\n* **JoinApp:** Join messages into one.\n* **SplitApp:** Split message.\n* **---Consumer:** Get message from queue.\n* **---Producer:** Push message to queue.\n\n\n### TODO\n\n- [x] base apps: Base, API, Schedule\n- [x] supported chain: join, split\n- [x] supported consumer/producer: redis, kafka, zmq, rabbitmq\n- [ ] supported callback success/fail\n- [x] generate: processor.py, settings.yml\n- [ ] management/admin\n- [ ] build/deploy\n- [x] sample: PipeApp, ScheduleApp, APIApp\n- [ ] test\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunary%2Fank","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsunary%2Fank","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunary%2Fank/lists"}