{"id":15031264,"url":"https://github.com/j-p-d-e-v/kafru","last_synced_at":"2025-07-04T06:39:56.086Z","repository":{"id":256078936,"uuid":"852914491","full_name":"j-p-d-e-v/kafru","owner":"j-p-d-e-v","description":"This library offers robust background task execution with parallel processing through multithreading, enabling tasks to run concurrently and efficiently. ","archived":false,"fork":false,"pushed_at":"2025-03-05T12:45:19.000Z","size":22586,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-09T22:45:27.388Z","etag":null,"topics":["crate","queueing","rust","rust-lang","scheduler","surrealdb","task"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/kafru","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j-p-d-e-v.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-05T16:41:42.000Z","updated_at":"2025-03-05T12:45:22.000Z","dependencies_parsed_at":"2024-10-12T14:01:05.830Z","dependency_job_id":"13f4c5b2-359f-4105-a703-384040b358eb","html_url":"https://github.com/j-p-d-e-v/kafru","commit_stats":{"total_commits":31,"total_committers":2,"mean_commits":15.5,"dds":0.06451612903225812,"last_synced_commit":"f570c46abc57d5f1df1edee02c2be9a22d720b65"},"previous_names":["j-p-d-e-v/kafru"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/j-p-d-e-v/kafru","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-p-d-e-v%2Fkafru","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-p-d-e-v%2Fkafru/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-p-d-e-v%2Fkafru/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-p-d-e-v%2Fkafru/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j-p-d-e-v","download_url":"https://codeload.github.com/j-p-d-e-v/kafru/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-p-d-e-v%2Fkafru/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263463890,"owners_count":23470448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crate","queueing","rust","rust-lang","scheduler","surrealdb","task"],"created_at":"2024-09-24T20:15:18.813Z","updated_at":"2025-07-04T06:39:56.058Z","avatar_url":"https://github.com/j-p-d-e-v.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafru\n\n**kafru** is a Rust crate designed for efficient task queue management, inspired by the task queue systems found in other programming environments. The name \"kafru\" is derived from the \"Kafra\" system in the popular game Ragnarok Online. \n\nThis library offers robust background task execution with parallel processing through multithreading, enabling tasks to run concurrently and efficiently. Additionally, **kafru** features advanced scheduling capabilities, utilizing cron for precise task timing. It integrates seamlessly with SurrealDB for comprehensive management of queues, metrics, and schedules, ensuring reliable and organized task execution.\n\n![Cute cat GIF](kafru-overview.gif)\n\n## Features\n\n- **Task Registry**: Manage and organize a collection of structs designed for task execution.\n- **Scheduler**: Utilize cron-based scheduling to define when tasks are pushed to the queue, with options to set start and end times.\n- **Queue**: Maintain a list of tasks awaiting execution.\n- **Worker**: Execute tasks from the queue, referencing the task registry to determine the appropriate struct for each task.\n- **SurrealDB Integration**: Store and manage scheduling details, queue information, and metrics using SurrealDB.\n\n## Installation\n\n```\ncargo add kafru\n```\n\n## Usage\n\n### 1. Run SurrealDB\n\nFor testing, you can start SurrealDB using an in-memory database:\n\n```bash\ncargo test -- --nocapture\n```\n\n```bash\nsurreal start -u kafru_admin -p kafru_password -b 0.0.0.0:4030 --allow-all memory\n```\n\nFor running SurrealDB in file mode, refer to the [SurrealDB documentation](https://surrealdb.com/docs/surrealdb/introduction/start).\n\n---\n\n### 2. Environment Variables\n\nThe following environment variables can be used to configure the database:\n\n| Key                  | Description                                                   | Default Value      |\n|----------------------|---------------------------------------------------------------|--------------------|\n| `KAFRU_DB_USERNAME`  | The database username.                                         | `kafru_admin`      |\n| `KAFRU_DB_PASSWORD`  | The database password.                                         | `kafru_password`   |\n| `KAFRU_DB_PORT`      | The port number of the database.                               | `4030`             |\n| `KAFRU_DB_HOST`      | The database host or IP address.                               | `127.0.0.1`        |\n| `KAFRU_DB_NAMESPACE` | The database namespace, useful for separating production and testing databases. | `kafru`            |\n| `KAFRU_DB_NAME`      | The database name.                                             | `kafru_db`         |\n\n---\n\n## Registering Task Structs to the Task Registry\n\nThe task registry stores a collection of task structs that contain the logic to be executed.\n\n### 3. Create a Task Struct\n\n- Your struct must implement the `TaskHandler` trait.\n- Define the code to execute within the `async fn run(\u0026self, _params: std::collections::HashMap\u003cString, Value\u003e) -\u003e Result\u003c(), String\u003e` method.\n\n#### Sample\n\n```rust\nuse async_trait::async_trait;\nuse kafru::task::TaskHandler;\nuse serde_json::Value;\nuse std::collections::HashMap;\n\npub struct MySampleStruct;\n\n#[async_trait]\nimpl TaskHandler for MySampleStruct {\n    async fn run(\u0026self, _params: HashMap\u003cString, Value\u003e) -\u003e Result\u003c(), String\u003e {\n        let x = 1; // Replace this with your logic.\n        let total = x + 1;\n        // Perform your task logic here.\n        Ok(())\n    }\n}\n```\n\n### 4. Register the Struct to the Task Registry\n\n- After defining the task struct, register it in the task registry by providing a name for the struct.\n\n#### Sample\n\n```rust\nuse async_trait::async_trait;\nuse kafru::task::{TaskHandler, TaskRegistry};\nuse serde_json::Value;\nuse std::collections::HashMap;\n\npub struct MySampleStruct;\n\n#[async_trait]\nimpl TaskHandler for MySampleStruct {\n    async fn run(\u0026self, _params: HashMap\u003cString, Value\u003e) -\u003e Result\u003c(), String\u003e {\n        // Task logic here.\n        Ok(())\n    }\n}\n\n#[tokio::main]\nasync fn main() {\n    let mut task_registry: TaskRegistry = TaskRegistry::new().await;\n    task_registry.register(\"mytesthandler\".to_string(), || Box::new(MySampleStruct)).await;\n}\n```\n\n---\n\n### 5. Running the Worker and Scheduler via the Manager\n\nTo execute tasks using both the worker (watcher) and the scheduler, follow these steps:\n\n```rust\nuse std::sync::Arc;\nuse kafru::manager::Manager;\nuse kafru::task::TaskRegistry;\nuse std::sync::Arc;\nuse crate::database::Db;\n\n// Initialize database instance\nlet db_instance = Db::new(None).await;\n\n// Wrap the database instance in an Arc\nlet db: Arc\u003cDb\u003e = Arc::new(db_instance.unwrap());\n\n// Declare the server e.g. MYLOCALHOSTABC\nlet server: String = \"MYLOCALHOSTABC\".to_string();\n\n// Declare the author name.\nlet author: String = \"Juan dela Cruz\".to_string();\n\n// Initialize the Manager struct.\nlet mut manager = Manager::new(server.clone(),author.clone()).await;\n\n// Initialize the Task Registry struct and register the task.\nlet mut task_registry: TaskRegistry = TaskRegistry::new().await;\ntask_registry.register(\"mytesthandler\".to_string(), || {\n    Box::new(MySampleStruct {\n        message: \"Hello World\".to_string(),\n    })\n}).await;\n\n// Share the task_registry using Arc for thread safety.\nlet task_registry: Arc\u003cTaskRegistry\u003e = Arc::new(task_registry);\n\n// Run the Worker, specifying the queue name, number of threads, the task registry, task poll interval (in seconds), and a database arc clone.\nlet _ = manager.worker(\"worker-default\".to_string(), 5, task_registry.clone(), 15, Some(db.clone())).await;\n\n// (Optional) Run the Scheduler by specifying the scheduler name, interval (in seconds), and a database arc clone.\nlet _ = manager.scheduler(\"scheduler-default\".to_string(), 5, Some(db.clone())).await;\n\n// Optionally, wait for both the worker and scheduler to finish. This will prevent the function from exiting prematurely.\nlet _ = manager.wait().await;\n\n```\n\nTo send command to the scheduler, queue worker, and task.\n```rust\nuse crate::agent::Agent;\nuse std::sync::Arc;\nuse crate::database::Db;\nuse crate::Command;\n\nlet author: String = \"Juan dela Cruz\".to_string();\nlet server: String = \"MYLOCALHOSTABC\".to_string();\nlet agent_name = Agent::to_name(server.clone(),\"worker-default-0\").await;\nlet agent_data = agent.get_by_name(agent_name,server.clone()).await?;\nlet agent_id = agent_data.id.unwrap();\nlet command = Command::QueueGracefulShutdown;\nlet message: String =  \"test dela cruz\".to_string();\nlet result = agent.send_command(agent_id, command, Some(author),Some(message)).await;\nassert!(result.is_ok(),\"{:?}\",result.unwrap_err());\n```\n## More Examples\n\nFor more examples, you can go to: ```src/test/``` directory.\n\n---\n\n### Explanation\n\n- **Task Registration**: Registers a task struct (`MySampleStruct`) in the task registry.\n- **Task Registry**: Shared using `Arc` to enable safe concurrent access in a multi-threaded environment.\n- **Worker**: Runs tasks from the queue, controlled by parameters such as:\n  - `queue_name`: Name of the queue to pull tasks from.\n  - `num_threads`: Number of threads to handle task execution.\n  - `task_registry`: Registry that contains the task structs.\n  - `poll_interval`: Polling interval (in seconds) to check for new tasks.\n- **Scheduler**: (Optional) Runs scheduled tasks at specified intervals, similar to cron jobs.\n- **Join**: Ensures that both the worker and scheduler keep running without the program exiting prematurely.\n\n\n## Code Documentation\n\nFor detailed documentation, visit [docs.rs/kafru](https://docs.rs/kafru).\n\n## License\n\nThis project is licensed under the Custom License Agreement. See the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for guidelines.\n\n## Contact\n\nFor questions or feedback, you can reach me at [jpmateo022@gmail.com](mailto:jpmateo022@gmail.com).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-p-d-e-v%2Fkafru","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj-p-d-e-v%2Fkafru","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-p-d-e-v%2Fkafru/lists"}