Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/aksbuzz/clickless


https://github.com/aksbuzz/clickless

Last synced: 10 months ago
JSON representation

Awesome Lists containing this project

README 

          
1. Integrate with exisitng wokrflow app

5. Add AWS (??)



-----



That's a fantastic question. Building a workflow engine touches on so many core software engineering concepts (distributed systems, databases, APIs, message queues, reliability) that it's a perfect springboard into more advanced and impactful projects. For a developer with 4 years of experience looking to grow, the next steps should focus on either **deepening expertise** in a specific area or **broadening scope** to encompass more of the product lifecycle.



Here are some project ideas, categorized by the skills they build upon.



### Category 1: Deepening Distributed Systems Expertise



These projects take the concepts from the workflow engine and apply them at a larger scale or with more complex requirements.



1.  **Project: A Feature Flagging / A/B Testing Platform**

    *   **Description:** Build a service like LaunchDarkly or Optimizely. It would have a UI to define feature flags (e.g., `new-checkout-flow`) and rules (e.g., "enable for 10% of users in Germany"). Your applications would query this service via a high-performance SDK to decide which features to show a user.

    *   **Skills Built:**

        *   **High Availability & Low Latency:** This service must be incredibly fast and resilient. If it goes down, it can't take your main applications down with it.

        *   **Real-time Data Streaming:** You'd stream exposure events ("user X saw variation B") to a data pipeline for analysis. This involves tools like Kafka, Kinesis, or a high-throughput message queue.

        *   **SDK Design:** You'll learn how to write a good client library for other developers to use.

        *   **Complex Rule Engines:** Implementing the targeting rules (user attributes, percentages, etc.) is a challenging logic problem.



2.  **Project: A Job Queueing System with Web UI (like Celery/Sidekiq but from scratch)**

    *   **Description:** While you used Celery, building the core components yourself is a massive learning experience. Create a system where users can submit jobs, monitor their progress via a web UI (like Sidekiq's or Celery Flower's), see failures, and manually retry them.

    *   **Skills Built:**

        *   **Deeper Broker Knowledge:** You'll go beyond basic pub/sub and implement patterns like reliable queues, priority queues, and dead-letter handling directly with RabbitMQ or Redis.

        *   **Concurrency and Worker Management:** How do you write a worker process that is efficient, handles signals gracefully (like SIGTERM for shutdown), and reports its status?

        *   **WebSockets:** The web UI would use WebSockets to provide real-time updates on job statuses.

        *   **API Design for Asynchronicity:** Designing endpoints to submit a job and get back a job ID for later polling.



3.  **Project: A Pluggable Data Ingestion Pipeline**

    *   **Description:** Build a service that can pull data from various sources (e.g., a PostgreSQL database, a Salesforce account, a CSV file from an SFTP server), transform it, and load it into a destination (e.g., a data warehouse like Snowflake, BigQuery, or just another database). Think of a simplified version of Fivetran or Airbyte.

    *   **Skills Built:**

        *   **Plugin Architecture (Deep Dive):** This heavily relies on the "Level 3" connector model we discussed. You'd be building the framework for these connectors.

        *   **Data Transformation:** Handling different data formats, cleaning messy data, and mapping schemas.

        *   **Batch vs. Stream Processing:** You could design it to run on a schedule (batch) or to react to database change-data-capture (CDC) events (stream).

        *   **Resilience and State Management:** If a pipeline transferring millions of rows fails midway, how do you resume from the point of failure without duplicating data?



---



### Category 2: Broadening Scope to Product & User Experience



These projects focus less on the deep backend and more on delivering a complete, user-facing product.



4.  **Project: An Internal Developer Platform (IDP)**

    *   **Description:** Many companies are building platforms to simplify life for their developers. Create a web portal where a developer can, with a few clicks:

        *   Scaffold a new microservice from a template (e.g., a Python FastAPI service with Dockerfile, CI/CD pipeline, etc.).

        *   View the health, logs, and deployment status of their services.

        *   Manage feature flags for their services (integrating with the platform from Project #1).

    *   **Skills Built:**

        *   **"Platform as a Product" Thinking:** Your users are other developers. You need to think about their "user experience."

        *   **Infrastructure as Code:** You'd use tools like Terraform or Pulumi to provision resources.

        *   **CI/CD Automation:** You'll be dynamically creating and managing GitHub Actions or GitLab CI pipelines.

        *   **Service Catalogs:** Integrating with tools like Backstage or building a simpler version to keep track of service ownership and documentation.



5.  **Project: A "Low-Code" API Builder**

    *   **Description:** Create a UI where a less technical user can define a simple data model (e.g., a "Product" with fields: name, price, image_url) and the system automatically generates a fully functional REST and/or GraphQL API for it, complete with a database table.

    *   **Skills Built:**

        *   **Code Generation & Metaprogramming:** You'll write code that writes other code or dynamically configures an API gateway.

        *   **Database Schema Management:** Programmatically creating and migrating database tables (e.g., using Alembic or Django Migrations).

        *   **Authentication & Authorization:** You'd need to build a robust system for API keys and user permissions (e.g., "who can write to this endpoint?").

        *   **UI/UX for Complex Tasks:** Making a complex process like API creation feel simple is a major design challenge.



6.  **Project: A Real-time Notification Service**

    *   **Description:** Build a centralized service that other microservices can call to send notifications to users. The service would manage user preferences (e.g., "send me an email for critical alerts, but a push notification for comments") and handle the actual delivery across different channels (Email via SendGrid, SMS via Twilio, Push Notifications via Firebase).

    *   **Skills Built:**

        *   **Integrating Third-Party APIs:** You'll become an expert at reading API docs, handling rate limits, and managing API keys securely.

        *   **Fan-out Architectures:** A single incoming event ("user X commented on post Y") might need to be "fanned out" into multiple notifications.

        *   **Template Management:** Building a system for storing and rendering notification templates (e.g., using Jinja2 for emails).

        *   **User Preference Modeling:** Designing the database schema and API to handle complex user notification settings.