{"id":21006264,"url":"https://github.com/derlin/fastapi-notebook-runner","last_synced_at":"2025-06-21T07:08:38.670Z","repository":{"id":151664138,"uuid":"607624568","full_name":"derlin/fastapi-notebook-runner","owner":"derlin","description":"A small REST API to execute a Jupyter Notebook on-demand, used as an example for https://github.com/derlin/introduction-to-fastapi-and-celery","archived":false,"fork":false,"pushed_at":"2023-03-24T09:13:03.000Z","size":70,"stargazers_count":35,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-15T01:38:22.015Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-28T10:58:56.000Z","updated_at":"2025-03-05T15:46:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"bad1c56f-5a83-4a8a-8c01-5510105e96cb","html_url":"https://github.com/derlin/fastapi-notebook-runner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/derlin/fastapi-notebook-runner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Ffastapi-notebook-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Ffastapi-notebook-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Ffastapi-notebook-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Ffastapi-notebook-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derlin","download_url":"https://codeload.github.com/derlin/fastapi-notebook-runner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Ffastapi-notebook-runner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261080624,"owners_count":23106602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T08:50:28.125Z","updated_at":"2025-06-21T07:08:33.651Z","avatar_url":"https://github.com/derlin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Notebook Runner using FastAPI and Celery\n\nThis repository contains the code of a small REST API to execute a Jupyter Notebook on-demand.\nHit `/start` and it will start the execution. Hit `/progress` to know the status of the execution. In case of failure,\nhit `/output` to see the stack trace. If you want to stop the execution, hit `/kill`.\nIf you forgot all about the above, hit `/docs`.\n\nNOTE: only one execution can be requested at a time: hitting `/start` twice in a row, the second request will return an error `400`,\nsaying an execution was already planned.\n\n## Technologies\n\n* the API is implemented using [FastAPI](https://fastapi.tiangolo.com/), \"a modern, fast (high-performance), \n  web framework for building APIs with Python 3.7+ based on standard Python type hints\"\n* as Python multi-threading is crap, the background tasks are handled by [celery](https://docs.celeryq.dev/en/stable/),\n  \"*a task queue with focus on real-time processing, while also supporting task scheduling*\"\n* Celery requires a broker (to transport messages and events) and optionally a backend (to store results). It supports many implementations,\n  but [Redis](https://redis.io/) seemed the best option as it is (a) very fast and (b) very easy to set up.\n* Celery is very powerful but doesn't provide a built-in way to limit the number of tasks (to one in our case).\n  For that, and since we already have Redis installed anyway, I am using a [Redis Lock](https://redis-py.readthedocs.io/en/v4.1.2/lock.html)\n  in the FastAPI application directly (more on this later) \n* FastAPI is an ASGI application. You thus need an ASGI server. The FastAPI doc recommends uvicorn for development,\n  and I used [gunicorn](https://gunicorn.org/) with a uvicorn worker class in production (i.e. in the Dockerfile)\n* The notebook is executed using the [nbconvert Python API](https://nbconvert.readthedocs.io/en/latest/execute_api.html)\n\n## Local development\n\nThis repo uses poetry for dependency management. If you haven't installed it already, see https://python-poetry.org/docs/.\nAfter cloning the repo, simply run `poetry install`. This will create a virtual environment (`.venv`) and install all the\ndependencies (production + development).\n\n\nTo develop locally, you need a redis instance running:\n```bash\ndocker run --rm --name some-redis -p 6379:6379 -d redis\n```\n\nThen, start both celery and FastAPI in reload mode (this means you can work on the code, and every change will be hot reloaded 😍):\n```bash\n# start celery\n# ! you need watchdog installed: pip install watchdog\nwatchmedo auto-restart --directory=./nb_runner --pattern=worker.py\\\n  -- celery --app=nb_runner.worker.celery_app worker --concurrency=1 --loglevel=info\n \n# start fastapi\nuvicorn nb_runner.main:app --reload\n```\n\nYou now have access to the application on **port 8000**. Happy coding!\n\n## Before you push\n\nEnsure you have formatted your source files following PEP8.\nSimply run the following in your terminal (Dockerfile build will fail otherwise):\n```bash\npoetry run black nb_runner\n```\n\n## Running on a local k3d cluster\n\nFirst, start a k3d cluster with a local registry:\n```bash\nk3d registry create registry.localhost --port 5555\nk3d cluster create test --registry-use k3d-registry.localhost:5555 --api-port 6550 -p \"80:80@loadbalancer\"\n```\n\nTo be able to push to the registry, you need to add the following to your `/etc/hosts`:\n```bash\necho \"127.0.0.1 k3d-registry.localhost\" | sudo tee -a /etc/hosts\n```\n\nNow, build the image and push it to the local registry:\n```bash\n# Using regular docker\ndocker build k3d-registry.localhost:5555/nb-runner .\ndocker push k3d-registry.localhost:5555/nb-runner\n\n# Using buildx (you need amd64 image for k3d)\ndocker buildx build --platform=linux/amd64 --pull --push -t k3d-registry.localhost:5555/nb-runner .\n```\n\nFinal step, you need to configure a bit the Helm Chart for k3d. For that, create a `values-k3d.yaml` file and add:\n```yaml\ningress:\n  enabled: true\n  host: nb-runner  # \u003c- this needs to match an entry in /etc/hosts redirecting to 127.0.0.1\n  path: /\n\nnb_runner:\n  image:\n    name: k3d-registry.localhost:5555/nb-runner\n```\n\nWith this, you are ready to deploy to k3d:\n```bash\nhelm install nb-runner helm/nb-runner --values values-k3d.yaml\n```\n\nIf you want to test the ingress, you need to add the following to your `etc/hosts` as well (you can change the host, just don't\nforget to change it in the `values-k3d.yaml` AND the `/etc/hosts`):\n```bash\necho \"127.0.0.1 nb-runner\" | sudo tee -a /etc/hosts\n```\n\nWith this entry, you can now go to http://nb-runner/docs. Congrats, you deployed to Kubernetes!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Ffastapi-notebook-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderlin%2Ffastapi-notebook-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Ffastapi-notebook-runner/lists"}