{"id":22018369,"url":"https://github.com/flavienbwk/prefect-docker-compose","last_synced_at":"2025-05-07T10:43:19.039Z","repository":{"id":39635609,"uuid":"302702290","full_name":"flavienbwk/prefect-docker-compose","owner":"flavienbwk","description":"A simple guide to understand Prefect and make it work with your own docker-compose configuration.","archived":false,"fork":false,"pushed_at":"2024-05-02T23:56:18.000Z","size":169,"stargazers_count":160,"open_issues_count":2,"forks_count":38,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-31T09:11:14.792Z","etag":null,"topics":["docker","docker-compose","guide","prefect","python3","s3","tutorial","workflow-management"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flavienbwk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-09T16:54:01.000Z","updated_at":"2024-11-27T03:19:15.000Z","dependencies_parsed_at":"2024-11-30T05:12:05.186Z","dependency_job_id":"5cb93bdc-0701-4bb0-8298-c9802129cfec","html_url":"https://github.com/flavienbwk/prefect-docker-compose","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fprefect-docker-compose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fprefect-docker-compose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fprefect-docker-compose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fprefect-docker-compose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flavienbwk","download_url":"https://codeload.github.com/flavienbwk/prefect-docker-compose/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252862298,"owners_count":21815833,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","guide","prefect","python3","s3","tutorial","workflow-management"],"created_at":"2024-11-30T05:12:00.504Z","updated_at":"2025-05-07T10:43:19.009Z","avatar_url":"https://github.com/flavienbwk.png","language":"Python","readme":"# Prefect - Docker Compose\n\nA simple guide to understand and make Prefect **2.x** work with your own docker-compose configuration.\n\nInterested in examples for Prefect **1.x** ? Switch to the [last 1.x branch of this repo](https://github.com/flavienbwk/prefect-docker-compose/tree/e758a498d5819550a9b926b0bf9bb4e9c85574d1).\n\nThis allows you to package your Prefect instance for fully-containerized environments (e.g: docker-compose, Kubernetes) or offline use.\n\n![Operating principle of Prefect](./prefect_schema_principle.jpg)\n\n- [Prefect - Docker Compose](#prefect---docker-compose)\n  - [Run the server](#run-the-server)\n  - [Run one or multiple agents](#run-one-or-multiple-agents)\n  - [Run your first flow via the Prefect API](#run-your-first-flow-via-the-prefect-api)\n    - [Principles to understand](#principles-to-understand)\n    - [Flow with Local storage (easiest)](#flow-with-local-storage-easiest)\n    - [Flow with S3 Storage (recommended)](#flow-with-s3-storage-recommended)\n    - [Flow with Docker storage](#flow-with-docker-storage)\n      - [Preparing the Registry](#preparing-the-registry)\n      - [Start the Docker in Docker agent](#start-the-docker-in-docker-agent)\n      - [Registering the flow](#registering-the-flow)\n\n## Run the server\n\n1. Optionally open and edit the [`server/.env`](./server/.env) file\n\n    :information_source: All `PREFECT_*` env variables can be [found in the Prefect settings.py file](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/settings.py#L238).\n\n2. Start the server :\n\n    ```bash\n    cd server \u0026\u0026 docker-compose up --build -d \u0026\u0026 cd -\n    ```\n\n3. Access the Orion UI at [localhost:4200](http://localhost:4200)\n\n## Run one or multiple agents\n\nAgents are services that run your scheduled flows.\n\n1. Optionally open and edit the [`agent/docker-compose.yml`](./agent/docker-compose.yml) file.\n\n    \u003e :information_source: In each `docker-compose.yml`, you will find the `PREFECT_API_URL` env variable including the `172.17.0.1` IP address. This is the IP of the Docker daemon on which are exposed all exposed ports of your containers. This allows containers launched from different docker-compose networks to communicate. Change it if yours is different (check your daemon IP by typing `ip a | grep docker0`).\n    \u003e\n    \u003e ![Docker interface IP](./docker_interface.png)\n    \u003e\n    \u003e Here, mine is `192.168.254.1` but the default is generally to `172.17.0.1`.\n\n2. Start the agent :\n\n    ```bash\n    docker-compose -f agent/docker-compose.yml up -d\n    ```\n\n    \u003e :information_source: You can run the agent on another machine than the one with the Prefect server. Edit the `PREFECT_API_URL` env variable for that.\n\n    Maybe you want to instanciate multiple agents ?\n\n    ```bash\n    docker-compose -f agent/docker-compose.yml up -d --scale agent=3 agent\n    ```\n\n3. Our agents are now starting listening the Orion server on the `flows-example-queue` queue ([see the `--work-queue` option](./agent/docker-compose.yml#L7)).\n\n## Run your first flow via the Prefect API\n\n### Principles to understand\n\n\u003e :speech_balloon: [Execution in your cloud; orchestration in ours](https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296)\n\nThis means the Prefect server never stores your code. It just orchestrates the running (optionally the scheduling) of it.\n\n1. After developing your flow, Prefect will register it to the Orion server [through a Deployment](./client/app/weather.py#L49). In that script, you may ask the server to run your flow 3 times a day, for example.\n2. Your code never lies on the Prefect server : this means the code has to be stored somewhere accessible to the agents in order to be executed.\n\n    Prefect has [a lot of storage options](https://docs.prefect.io/tutorials/storage) but the most famous are : Local, S3, Docker and git.\n    - Local : saves the flows to be run on disk. So the volume where you save the flows must be [shared among your client and your agent(s)](./client/docker-compose.yml#L9). Requires your agent to have the same environment than your client (Python modules, packages installed etc... (the same Dockerfile if your agent and client are containers))\n    - S3 : similar to local, but saves the flows to be run in S3 objects.\n    - Docker : saves the flows to be run as Docker images to your Docker Registry so your agents can easily run the code.\n\n### Flow with Local storage (easiest)\n\n:information_source: If your agents are installed among multiple machines, I recommend you to mount a shared directory with SSHFS.\n\n1. Run the following command to register your deployment and run the flow :\n\n    ```bash\n    docker-compose -f client/docker-compose.yml up # Executes weather.py\n    ```\n\n2. Access the UI to see your flow correctly run\n\n### Flow with S3 Storage (recommended)\n\n\u003cdetails\u003e\n\u003csummary\u003eTutorial for S3 Storage\u003c/summary\u003e\n\u003cbr/\u003e\n\nWe will use [MinIO](https://www.github.com/minio/minio) as our S3 server.\n\n1. Optionally open and edit the [`client_s3/.env`](./client_s3/.env) file and start MinIO\n\n    ```bash\n    docker-compose -f client_s3/docker-compose.yml up -d minio # Starts MinIO\n    ```\n\n2. Register the flow :\n\n    ```bash\n    docker-compose -f client_s3/docker-compose.yml up weather # Executes weather.py\n    ```\n\nNow your flow is registered. You can access the UI to run it.\n\n\u003c/details\u003e\n\n### Flow with Docker storage\n\nThis method requires our client AND agent containers to have access to Docker so they can package or load the image in which the flow will be executed. We use _Docker in Docker_ for that.\n\n\u003cdetails\u003e\n\u003csummary\u003eTutorial for (secure) Docker Storage\u003c/summary\u003e\n\n#### Preparing the Registry\n\nA Docker Registry is needed in order to save images that are going to be used by our agents.\n\n1. Generate the authentication credentials for our registry\n\n    ```bash\n    sudo apt install apache2-utils # required to generate basic_auth credentials\n    cd client_docker/registry/auth \u0026\u0026 htpasswd -B -c .htpasswd myusername \u0026\u0026 cd -\n    ```\n\n    \u003e To add more users, re-run the previous command **without** the -c option\n\n2. Start the registry\n\n    ```bash\n    docker-compose -f client_docker/docker-compose.yml up -d registry\n    ```\n\n3. Login to the registry\n\n    You need to allow your Docker daemon to push to this registry. Insert this in your `/etc/docker/daemon.json` (create if needed) :\n\n    ```json\n    {\n      \"insecure-registries\": [\"172.17.0.1:5000\"]\n    }\n    ```\n\n4. Start the registry\n\n    ```bash\n    docker login http://172.17.0.1:5000 # with myusername and the password you typed\n    ```\n\n    You should see : _Login Succeeded_\n\n#### Start the Docker in Docker agent\n\nOptionally edit registry credentials in [`./agent_docker/docker-compose.yml`](./agent_docker/docker-compose.yml) and run :\n\n```bash\ndocker-compose -f agent_docker/docker-compose.yml up --build -d\n```\n\n#### Registering the flow\n\nWe're going to push our Docker image with Python dependencies and register our flow.\n\n1. Build, tag and push the image\n\n    ```bash\n    docker build . -f ./client_docker/execution.Dockerfile -t 172.17.0.1:5000/weather/base_image:latest\n    ```\n\n    \u003e You **must** prefix your image with the registry URI `172.17.0.1:5000` to push it\n\n    ```bash\n    docker push 172.17.0.1:5000/weather/base_image:latest\n    ```\n\n2. Register the flow\n\n    Optionally edit registry credentials in `./client_docker/docker-compose.yml` and run :\n\n    ```bash\n    docker-compose -f ./client_docker/docker-compose.yml up --build weather\n    ```\n\nNow your flow is registered. You can access the UI to run it.\n\n\u003c/details\u003e\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fprefect-docker-compose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflavienbwk%2Fprefect-docker-compose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fprefect-docker-compose/lists"}