{"id":16950577,"url":"https://github.com/mischnic/python-ci","last_synced_at":"2025-03-22T13:31:25.324Z","repository":{"id":49979523,"uuid":"102364618","full_name":"mischnic/python-ci","owner":"mischnic","description":"A lightweight CI-server with a web interface","archived":false,"fork":false,"pushed_at":"2018-06-17T09:58:26.000Z","size":744,"stargazers_count":8,"open_issues_count":34,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T11:39:18.481Z","etag":null,"topics":["badge","ci","ci-server","flask","github-webhooks","jwt","latex","lightweight","python","raspberry-pi","react"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mischnic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-04T13:29:15.000Z","updated_at":"2023-12-08T00:36:38.000Z","dependencies_parsed_at":"2022-08-03T16:30:15.466Z","dependency_job_id":null,"html_url":"https://github.com/mischnic/python-ci","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mischnic%2Fpython-ci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mischnic%2Fpython-ci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mischnic%2Fpython-ci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mischnic%2Fpython-ci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mischnic","download_url":"https://codeload.github.com/mischnic/python-ci/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244962818,"owners_count":20539229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["badge","ci","ci-server","flask","github-webhooks","jwt","latex","lightweight","python","raspberry-pi","react"],"created_at":"2024-10-13T21:58:01.236Z","updated_at":"2025-03-22T13:31:25.017Z","avatar_url":"https://github.com/mischnic.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# python-ci\n\nA lightweight CI-server written in python, originally developed for a Raspberry Pi because other existing solutions were to resource-intensive (Jenkins) or cumbersome to use.\n\n- Has a *React*-ive web interface\n\t+ Count letters and words in LaTeX documents\n\t+ Show statistics about your build\n- GitHub integration\n\t+ GitHub webhook\n\t+ Display the build status next to the commit on GitHub\n\nDrawbacks:\n\n- Builds aren't fully isolated, the same cloned repository is `git reset --hard` to the corresponding commit and then used for building.\n\t+ LaTeX: the git repo should remain untouched, as `latexmk` save the build files elsewhere\n\t+ npm: `npm build` is run and then the specified build folder is `zip`ped up.\n\n\n![List view](docs/example_web1.png)\n\n![Details view](docs/example_web2.png)\n\n## Setup\n\n- tested with Python 3.5 and 3.6\n- install required libs: `pip3 install -r requirements.txt`\n\nClone your source folder next to the script (see below), copy `start.sh.in` to `start.sh` and make `start.sh` executable. Enviroment variables in `start.sh` for the python script serve as configuration:\n\n- `OUTPUT_SUFFIX`: the `_build` below; optional (default: `_build`)\n- `SECRET`: the secret from the GitHub webhook configuration; optional\n- `JWT_SECRET`: the secret for creating a JWT token\n- `PASSWORD`: the password (username is hardcoded: `user`)\n- `PROJECT`: comma-seperated string of your projects (project folders) (e.g. `Maths` or `Maths,Name`)\n- `NGINX_ACCEL`: set to any value to use nginx's `X-Accel-Redirect` for build files\n- `CI_PATH`: additional `PATH` entries to set when executing commands (e.g `/Library/TeX/texbin` on macOS)\n- Needed to set commit statuses, otherwise optional:\n\t- `TOKEN`: a GitHub personal access token\n\t- `URL`: the URL under which the server is accessible (including `http[s]://`)\n\nIf you get the error `Permission denied (publickey)` during build, and a line for your private key in `~/.ssh/config`: `IdentityFile ~/.ssh/your_key_name` and uncomment the corresponding section in `start.sh`\n\nTo install python-ci as a systemd service, run `./install-service.sh`, this will configure the service and enable it. Then you can use commands like:\n\n- `sudo systemctl start/stop/restart python-ci` to start/stop/restart the server\n- `sudo systemctl enable/disable python-ci` to enable/disable the autostart on boot\n\nYou need the following file hierarchy: (clone your project like `Maths`)\n\n\tpython-ci\n\t |- build\n\t   |- Maths\n\t      |- .git\n\t      |- .ci.json\n\t      | - Document.tex\n\t   |- Maths_build\n\t      |- Document.pdf\n\t      |- Document.aux\n\t     - ...\n\t |- README.md\n\t |- src\n\t |- [TeXcount_3_1]\n\t |- ...\n(`Maths` and `Document` will serve as example names for the rest of this document)\n\n`.ci.json` is the project's configuration file:\n\n\t{\n\t\t\"language\": \"latex\",\n\t\t\"main\": \"Document\",\n\t\t\"stats\": [\"counts\"] // optional\n\t}\n\nCurrently implemented languages:\n- `git`: Update repository only\n- `latex`: Update repository and run `latexmk` on the `${main}.tex` file\n- `npm`: Update repository, run `yarn install` and `yarn build` (with env variables specified in the `env` dict) in the `source` folder, excepts output in the `source`/`output` folder and packages the content into a zip file.\n\nCurrently implemented \"stats\":\n- for `latex`:\n\t- `counts`: Show `main`'s letter count\n\t\n\t\nNote:\nThe `counts` stats options needs [TeXcount](http://app.uio.no/ifi/texcount/download.html) to be downloaded to a folder `TeXcount_3_1` inside `python-ci`. To count bibliography, `%TC:subst \\printbibliography \\bibliography` needs to be the first line of your document and you'll have to patch TeXcount (from [here](https://gist.github.com/mischnic/f8b0433934e046c4e6d0202d99276b82)).\n\n## Usage\n\nTo run `python-ci.py` in the background (have it exit when closing the terminal) without using systemd: `nohup ./start.sh \u0026`.\n\npython-ci delivers the following pages: (they accept **only long** commit-hashes)\n\n\n### Web Interface\n\nWith the configuration below, the web interface is served at `ci.example.com`.\n\n### API\n\n(The following links are only correct, if you use a dedicated webserver as a proxy to python-ci with a configuration as seen below. The python-ci server itself responds to requests like `/Maths/1f2a23..`, without `/api`.)\n\n- All API requests except for the last listed here need a JWT token either specified as a GET parameter (`...?token=eyJhbGciOiJIUz...`) or as a  header: `Authentification: Bearer eyJhbGciOiJIUz...`.\n- The commit-hashes in URLs can generally be replaced by `latest`\n\n| Desc | URL | Request data | Response data | \n| ---- | --- | ------------ | ------------- |\n| Login| `POST /api/login` | json: `{username: \"user\", password: \"pass\"}` | text: `jwt token...` |\n| Build| `GET /api/\u003cproj\u003e/\u003csha:not\"latest\"\u003e/build` | - | status code: 200 OK, 503 Busy |\n| List projects| `GET /api/` | - | json: `[\"Maths\", \"test\"]` |\n| List builds| `GET /api/\u003cproj\u003e/` | - | json: see below *|\n| PDF build artifact| `GET /api/\u003cproj\u003e/\u003cref\u003e/pdf` | - | `main.pdf` |\n| Compile log | `GET /api/\u003cproj\u003e/\u003cref\u003e/log` | - | `.log` |\n| Badge | `GET /api/\u003cproj\u003e/\u003cref\u003e/svg` | - | ![badge](docs/example_badge.svg) |\n| Badge | `GET /api/\u003cproj\u003e/latest/svg` | - (no auth. needed) |  |\n\n`/api/\u003cproj\u003e/`:\n\n\t{\n\t  \"id\": \"user/Maths\",\n\t  \"language\": \"latex\",\n\t  \"latest\": \"947ddfc29b39ab40619e51172bc80036938ab3\",\n\t  \"list\": [\n\t    {\n\t      \"build\": {\n\t        \"artifacts\": {\n\t          // request name: Display name\n\t          \"pdf\": \"PDF\"\n\t        },\n\t        \"duration\": 203.98801684379578,\n\t        \"errorMsg\": null,\n\t        \"ref\": \"bf3b039261811a106dae03c90341d904378d16dc\",\n\t        \"start\": 1512610571115,\n\t        \"stats\": {\n\t          \"counts\": {\n\t            \"letters\": {}, //...\n\t            \"words\": {} //...\n\t          }\n\t        },\n\t        \"status\": \"success\"\n\t      },\n\t      \"commit\": {\n\t        \"author_name\": \"John Doe\",\n\t        \"date\": 1512611571115,\n\t        \"msg\": \"Changed something\\n\",\n\t        \"parents\": [\n\t          \"0e899e95396a25ea61ed9130e93ec9220b406cd7\"\n\t        ],\n\t        \"ref\": \"bf3b039261811a106dae03c90341d904378d16dc\"\n\t      }\n\t    }\n\t  ]\n\t}\n\n\nThere is also a SSE endpoint at `/api/subscribe` (needs to be authenticated):\n\n\t...\n\n\tid: 12ab8\n\tevent: ....\n\tdata: {...}\n\n\t...\n\n\n| `event` | `data` | Description |\n| ----- | ------ | ----------- |\n| - | - | comment every 15 sec to prevent timeout |\n| `\u003cproj name\u003e` | `{\"event\": \"status\", \"data\": data}` | `data` is the `build` object from above |\n| `\u003cproj name\u003e` | `{\"event\": \"log\", \"ref\": \"abcdef\", \"data\": s}` | `s` is to append to the log |\n\n\u003cbr/\u003e\n\nExample for a badge which links the corresponding build page:\n\n`[![build status](http://ci.example.com/api/Maths/latest/svg)](http://ci.example.com/Maths/latest)`\n\n## As a GitHub webhook\n\nPayload URL: `https://ci.example.com/api/Maths`.\n\nWhen adding the webhook, be sure to set the \"Content type\" to `application/json`. Only the `push` (and `ping` event) event is handled.\n\n## Server configuration\n\nBy default, python-ci listens on `localhost:8000`, meaning that it will only accept connections from the server itself. To reach it you could something like this in your nginx configuration to accept requests from the `ci` subdomain (and serve the React Single-Page App correctly) :\n\n\n\tserver {\n\t\tlisten 80;\n\t\n\t\t# listen 443 ssl;\n\t\t# ssl_certificate ...\n\t\t\n\t\troot \u003c\u003cPath to the react build/ folder\u003e\u003e;\n\t\n\t\tserver_name\tci.example.com;\n\t\t\n\t\tlocation / {\n\t\t\ttry_files $uri /index.html;\n\t\t}\n\n\t\tlocation /api {\n\t\t\trewrite ^/api(.*) $1 break;\n\t\t\tproxy_pass http://localhost:8000;\n\t\t\t\n\t\t\tproxy_set_header Host $host;\n\t\t\tproxy_set_header X-Real-IP $remote_addr;\n\t\t\tproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n\t\t\tproxy_set_header X-Forwarded-Proto $scheme;\n\t\t}\n\t}\n\nTo use nginx to send your build files add the following inside the `server` block and set `NGINX_ACCEL` to any value in your `start.sh` file:\n\n\tlocation /data/ {\n\t\tinternal;\n\t\talias /path/to/python-ci/;\n\t}\n\nIf you only want the api and webhook without the web interface, then you don't need a seperate webserver. In that case, change `'localhost'` in [this](https://github.com/mischnic/python-ci/blob/b5d7e55e94ac528c41a8e30fe6297d768cb244d9/python-ci.py#L323) line to `''`, so that the server will be reachable not only from localhost.\n\n\nIf the server is in your local network and your router doesn't support [NAT loopback](https://en.wikipedia.org/wiki/NAT_loopback) alias [Hairpinning](https://en.wikipedia.org/wiki/Hairpinning) (meaning that trying to access `ci.example.com` in the same network as the server causes a `ERR_CONNECTION_REFUSED`) then you have to add `ci.example.com*` to the `server_name` directive. This enables you to access the server under `ci.example.com.192.168.0.2.nip.io` with `192.168.0.2` being the IP of the server.\n\nAs an alternative (more elegant but more difficult to set up) you could set up an DNS server in your local network on computer, which is always running, and make it respond to `ci.example.com` with the local IP adress of your server.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmischnic%2Fpython-ci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmischnic%2Fpython-ci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmischnic%2Fpython-ci/lists"}