{"id":18808954,"url":"https://github.com/nabeel-ahmed-jamil/process","last_synced_at":"2026-05-12T07:38:05.370Z","repository":{"id":227259728,"uuid":"354566770","full_name":"NABEEL-AHMED-JAMIL/process","owner":"NABEEL-AHMED-JAMIL","description":"The process used for scheduler on a different time (UI-MODIFY-ANALAYTICS-SCREEN CURRENT BRANCH)","archived":false,"fork":false,"pushed_at":"2024-09-12T11:24:24.000Z","size":18599,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-29T22:36:10.483Z","etag":null,"topics":["java-8","java-8-lambda","java-8-time-api","java-executor","java-thread","jdbc","postgresql","scheduler","spring-boot","thread-pool"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NABEEL-AHMED-JAMIL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-04T14:37:37.000Z","updated_at":"2024-09-02T19:35:13.000Z","dependencies_parsed_at":"2024-09-12T21:30:03.999Z","dependency_job_id":null,"html_url":"https://github.com/NABEEL-AHMED-JAMIL/process","commit_stats":null,"previous_names":["nabeel-ahmed-jamil/process"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NABEEL-AHMED-JAMIL%2Fprocess","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NABEEL-AHMED-JAMIL%2Fprocess/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NABEEL-AHMED-JAMIL%2Fprocess/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NABEEL-AHMED-JAMIL%2Fprocess/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NABEEL-AHMED-JAMIL","download_url":"https://codeload.github.com/NABEEL-AHMED-JAMIL/process/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239742988,"owners_count":19689421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java-8","java-8-lambda","java-8-time-api","java-executor","java-thread","jdbc","postgresql","scheduler","spring-boot","thread-pool"],"created_at":"2024-11-07T23:14:48.159Z","updated_at":"2026-05-12T07:38:05.359Z","avatar_url":"https://github.com/NABEEL-AHMED-JAMIL.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scheduler Engine\n\n### Overview\nThis project is a **Scheduler Engine** designed to efficiently manage and execute source jobs.  \nIt leverages **Apache Kafka** for real-time stream processing, providing scalability, fault tolerance, and high throughput for job execution workflows.\n\n```\nThere are 5 types of schedulers:\n1. Minute (runs every few minutes, e.g., every 5 minutes)\n2. Hourly (runs every few hours, e.g., every 1 hour)\n3. Daily (runs once per day)\n4. Weekly (runs once per week)\n5. Monthly (runs once per month)\n```\n\n### Features\n- **Job Scheduling** – Supports periodic, delayed, and priority-based job execution.  \n- **Real-Time Stream Processing** – Uses Kafka producers and consumers to process jobs as messages.  \n- **Scalability** – Distributed architecture for handling a large volume of jobs concurrently.  \n- **Fault Tolerance** – Kafka ensures message durability and recovery in case of failure.  \n- **Prioritized Task Execution** – Executes tasks based on defined priority levels using a queue system.  \n- **Thread Pool Execution** – Uses `ThreadPoolExecutor` with a `PriorityBlockingQueue` for efficient parallel job handling.\n\n### Architecture\n1. **Producer** – Sends job/task messages to Kafka topics.  \n2. **Scheduler Engine** – Pulls tasks, applies scheduling logic, and enqueues them for execution.  \n3. **Consumer** – Reads processed results or error events from Kafka.  \n4. **Executor** – Runs prioritized tasks concurrently using a thread pool.\n\n### Tech Stack\n- **Java / Spring Boot** – Core backend engine  \n- **Apache Kafka** – Real-time messaging and stream processing  \n- **PostgreSQL** – (Optional) Persistence for metadata and job tracking\n\n### Use Cases\n- ETL job scheduling and execution  \n- Data pipeline orchestration  \n- Real-time stream processing  \n- Event-driven workflows\n\n### Running the Project\n```bash\n# Clone the repository\ngit clone https://github.com/NABEEL-AHMED-JAMIL/process/tree/split-mono-to-microservice\n\n# Navigate into the project directory\ncd process\n\n# Build the project\nmvn clean install\n\n# Run the application\nmvn spring-boot:run\n```\n\n### ETL Workflow Diagram\nBelow are the existing workflow details of the process.\n\n#### 1. Old ETL Workflow Diagram\n![alt text](ext-detail/old-etl.png)\n\n#### 2. New ETL Workflow Diagram\n![alt text](ext-detail/new-etl.png)\n\n---\n\n## Helper Queries for Viewing and Running the Project\n\u003e **Note:** Before running the project, execute the following SQL scripts.\n\n```sql\nINSERT INTO lookup_data VALUES\n('1001','2021-03-31 22:09:43.244','This scheduler sends source jobs into the queue','2021-04-01T00:16:34.567','SCHEDULER_LAST_RUN_TIME'),\n('1002','2021-03-31 23:06:48.744','Defines the fetch size limit for retrieving data from the database','25','QUEUE_FETCH_LIMIT');\n\nINSERT INTO source_task_type (source_task_type_id, description, queue_topic_partition, service_name)\nVALUES ('1000', '[consumer test]', 'topic=test-topic\u0026partitions=[*]', 'Test');\n```\n\n---\n\n## Process Endpoints\nList of available endpoints with their descriptions:\n\n1. **Download Batch File**  \n   `http://localhost:9098/api/v1/bulk.json/downloadBatchSchedulerTemplateFile`\n2. **Upload Batch File**  \n   `http://localhost:9098/api/v1/bulk.json/uploadBatchSchedulerFile`\n\n### Kafka Setup\n1. **Start Apache Zookeeper**\n   ```bash\n   .\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties   # Windows\n   ./zookeeper-server-start.sh ../config/zookeeper.properties                  # Linux/Mac\n   ```\n\n2. **Start the Kafka Server**\n   ```bash\n   .\\bin\\windows\\kafka-server-start.bat .\\config\\server.properties   # Windows\n   ./bin/kafka-server-start.sh ./config/server.properties                 # Linux/Mac\n   ```\n\n3. **View Cluster Details**\n   Use **Kafka Offset Explorer** to view cluster and topic details.  \n   ![alt text](ext-detail/Topic-Detail.png)\n\n---\n\n### Process Database UML\nThe image below shows the database design details for the process.  \n![alt text](ext-detail/new-dbdesing.png)\n\n---\n\n### Monitoring (Grafana \u0026 Spring Boot Actuator)\nSpring Boot Actuator is enabled in this project to provide production-ready features such as monitoring, health checks, metrics, and system information.  \nIt exposes endpoints that allow developers and operators to monitor the application, debug issues, and integrate with monitoring tools like Prometheus and Grafana.\n\nThe exposed endpoints include:\n- Application health  \n- Metrics  \n- Configuration properties  \n- Environment details  \n- Loggers  \n- Liquibase migration status  \n- Scheduled tasks  \n- Request mappings  \n\nThese endpoints make it easier to monitor the application in real time and ensure smooth operation in production.\n\n```json\n{\n  \"_links\": {\n    \"self\": { \"href\": \"http://localhost:9098/api/v1/actuator\", \"templated\": false },\n    \"beans\": { \"href\": \"http://localhost:9098/api/v1/actuator/beans\", \"templated\": false },\n    \"caches\": { \"href\": \"http://localhost:9098/api/v1/actuator/caches\", \"templated\": false },\n    \"health\": { \"href\": \"http://localhost:9098/api/v1/actuator/health\", \"templated\": false },\n    \"info\": { \"href\": \"http://localhost:9098/api/v1/actuator/info\", \"templated\": false },\n    \"liquibase\": { \"href\": \"http://localhost:9098/api/v1/actuator/liquibase\", \"templated\": false },\n    \"loggers\": { \"href\": \"http://localhost:9098/api/v1/actuator/loggers\", \"templated\": false },\n    \"metrics\": { \"href\": \"http://localhost:9098/api/v1/actuator/metrics\", \"templated\": false },\n    \"prometheus\": { \"href\": \"http://localhost:9098/api/v1/actuator/prometheus\", \"templated\": false },\n    \"scheduledtasks\": { \"href\": \"http://localhost:9098/api/v1/actuator/scheduledtasks\", \"templated\": false },\n    \"mappings\": { \"href\": \"http://localhost:9098/api/v1/actuator/mappings\", \"templated\": false }\n  }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabeel-ahmed-jamil%2Fprocess","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnabeel-ahmed-jamil%2Fprocess","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabeel-ahmed-jamil%2Fprocess/lists"}