{"id":19865641,"url":"https://github.com/openfaas/python-fan-in-example","last_synced_at":"2025-08-23T23:04:50.861Z","repository":{"id":84492755,"uuid":"524044377","full_name":"openfaas/python-fan-in-example","owner":"openfaas","description":"Python code example for implementing a fan-out/fan-in pattern with OpenFaaS functions.","archived":false,"fork":false,"pushed_at":"2022-10-27T12:58:49.000Z","size":109,"stargazers_count":8,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-06T23:08:24.755Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.openfaas.com/blog/fan-out-and-back-in-using-functions/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openfaas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-12T10:25:14.000Z","updated_at":"2024-01-25T05:38:40.000Z","dependencies_parsed_at":"2023-07-29T17:46:31.278Z","dependency_job_id":null,"html_url":"https://github.com/openfaas/python-fan-in-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfaas%2Fpython-fan-in-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfaas%2Fpython-fan-in-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfaas%2Fpython-fan-in-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfaas%2Fpython-fan-in-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openfaas","download_url":"https://codeload.github.com/openfaas/python-fan-in-example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251992926,"owners_count":21677022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T15:23:33.128Z","updated_at":"2025-08-23T23:04:50.837Z","avatar_url":"https://github.com/openfaas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fan-out/fan-in pattern with OpenFaaS\nThis repo contains an example of how to fan out and back in using OpenFaaS.\n\n\u003e The complete write up for this example is published on the OpenFaaS blog:\n\u003e - [Exploring the Fan out and Fan in pattern with OpenFaaS](https://www.openfaas.com/blog/fan-out-and-back-in-using-functions/)\n\nIn the example a csv file containing image urls is used is the input for a batch job. The inception function is called for each url and categorizations are returned through machine learning. The result of each invocation is stored in an S3 bucket. When the batch is completed a final function is called that aggregates and summarizes the results. The summary is stored in the S3 bucket.\n\n![Screenshot of the queue-worker metrics, aws S3 console showing individual function results and a json file with the final results of the batch job.](https://pbs.twimg.com/media/FahM5rCVEAESamf?format=jpg\u0026name=medium)\n\n## How to run it?\n### Setup dependencies\nS3 is used as a data store. You will need to create an S3 bucket that the functions can use. This example uses a bucket named `of-demo-inception-data` replace any references to this bucket with your own bucket name.\n\nRedis is used to keep track of the completed work by decrementing a counter for each batch.\nYou can deploy redis using `arkade`\n```bash\narkade install redis\n```\n\n### Add secrets\nMake sure the required secrets are available for the functions.\n\nS3 credentials:\n\n```bash\necho $aws_access_key_id | faas-cli secret create s3-key\necho $aws_secret_access_key | faas-cli secret create s3-secret\n```\n\nRedis password:\n\n```bash\nexport REDIS_PASSWORD=$(kubectl get secret --namespace redis redis -o jsonpath=\"{.data.redis-password}\" | base64 --decode)\n\necho $REDIS_PASSWORD | faas-cli secret create redis-password\n```\n\n### Set env variables\nThe `env.yml` file contains some env variables used by the function like the name of the bucket that should be used and the redis hostname. Update this file with your values.\n\n```yaml\nenvironment:\n  redis_hostname: \"redis-master.redis.svc.cluster.local\"\n  redis_port: 6379\n  s3_bucket: of-demo-inception-data\n```\n\n### Deploy the functions\nDeploy the stack.\n\n```bash\nfaas-cli template pull stack\n```\n\n```bash\nfaas-cli deploy\n```\n\n### Run with example data\nThe [data](./data) folder has several csv files containing urls that can be used as data source for this example.\n```\ndata\n├── batch-200.csv # 200 records\n├── batch-500.csv # 500 records\n├── batch-1000.csv # 1000 records\n└── batch-2000.csv # 2000 records\n```\n\nThe `create-batch` function looks for the input files in the S3 bucket. Upload them to your S3 bucket.\n\n```bash\naws s3 cp data/batch-200.csv s3://of-demo-inception-data/data/batch-200.csv\n```\n\nInvoke the function `create-batch` with the name of the source file you want to start processing.\n```bash\ncurl -i  http://127.0.0.1:8080/function/create-batch -d data/batch-200.csv\n\nHTTP/1.1 201 Created\nContent-Type: text/html; charset=utf-8\nDate: Wed, 17 Aug 2022 15:19:56 GMT\nServer: waitress\nX-Call-Id: 4121651e-8bd4-470a-8ad3-70ecd68b8194\nX-Duration-Seconds: 0.387640\nX-Start-Time: 1660749596315319961\nContent-Length: 69\n\n{\"batch_id\": \"55f22778-675f-4a85-8e1c-5d777faa4399\", \"batch_size\": 200}%  \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenfaas%2Fpython-fan-in-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenfaas%2Fpython-fan-in-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenfaas%2Fpython-fan-in-example/lists"}