Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mischnic/python-ci

A lightweight CI-server with a web interface
https://github.com/mischnic/python-ci

badge ci ci-server flask github-webhooks jwt latex lightweight python raspberry-pi react

Last synced: about 2 months ago
JSON representation

A lightweight CI-server with a web interface

Awesome Lists containing this project

README 

        
# python-ci



A lightweight CI-server written in python, originally developed for a Raspberry Pi because other existing solutions were to resource-intensive (Jenkins) or cumbersome to use.



- Has a *React*-ive web interface

	+ Count letters and words in LaTeX documents

	+ Show statistics about your build

- GitHub integration

	+ GitHub webhook

	+ Display the build status next to the commit on GitHub



Drawbacks:



- Builds aren't fully isolated, the same cloned repository is `git reset --hard` to the corresponding commit and then used for building.

	+ LaTeX: the git repo should remain untouched, as `latexmk` save the build files elsewhere

	+ npm: `npm build` is run and then the specified build folder is `zip`ped up.



![List view](docs/example_web1.png)



![Details view](docs/example_web2.png)



## Setup



- tested with Python 3.5 and 3.6

- install required libs: `pip3 install -r requirements.txt`



Clone your source folder next to the script (see below), copy `start.sh.in` to `start.sh` and make `start.sh` executable. Enviroment variables in `start.sh` for the python script serve as configuration:



- `OUTPUT_SUFFIX`: the `_build` below; optional (default: `_build`)

- `SECRET`: the secret from the GitHub webhook configuration; optional

- `JWT_SECRET`: the secret for creating a JWT token

- `PASSWORD`: the password (username is hardcoded: `user`)

- `PROJECT`: comma-seperated string of your projects (project folders) (e.g. `Maths` or `Maths,Name`)

- `NGINX_ACCEL`: set to any value to use nginx's `X-Accel-Redirect` for build files

- `CI_PATH`: additional `PATH` entries to set when executing commands (e.g `/Library/TeX/texbin` on macOS)

- Needed to set commit statuses, otherwise optional:

	- `TOKEN`: a GitHub personal access token

	- `URL`: the URL under which the server is accessible (including `http[s]://`)



If you get the error `Permission denied (publickey)` during build, and a line for your private key in `~/.ssh/config`: `IdentityFile ~/.ssh/your_key_name` and uncomment the corresponding section in `start.sh`



To install python-ci as a systemd service, run `./install-service.sh`, this will configure the service and enable it. Then you can use commands like:



- `sudo systemctl start/stop/restart python-ci` to start/stop/restart the server

- `sudo systemctl enable/disable python-ci` to enable/disable the autostart on boot



You need the following file hierarchy: (clone your project like `Maths`)



	python-ci

	 |- build

	   |- Maths

	      |- .git

	      |- .ci.json

	      | - Document.tex

	   |- Maths_build

	      |- Document.pdf

	      |- Document.aux

	     - ...

	 |- README.md

	 |- src

	 |- [TeXcount_3_1]

	 |- ...

(`Maths` and `Document` will serve as example names for the rest of this document)



`.ci.json` is the project's configuration file:



	{

		"language": "latex",

		"main": "Document",

		"stats": ["counts"] // optional

	}



Currently implemented languages:

- `git`: Update repository only

- `latex`: Update repository and run `latexmk` on the `${main}.tex` file

- `npm`: Update repository, run `yarn install` and `yarn build` (with env variables specified in the `env` dict) in the `source` folder, excepts output in the `source`/`output` folder and packages the content into a zip file.



Currently implemented "stats":

- for `latex`:

	- `counts`: Show `main`'s letter count

	

	

Note:

The `counts` stats options needs [TeXcount](http://app.uio.no/ifi/texcount/download.html) to be downloaded to a folder `TeXcount_3_1` inside `python-ci`. To count bibliography, `%TC:subst \printbibliography \bibliography` needs to be the first line of your document and you'll have to patch TeXcount (from [here](https://gist.github.com/mischnic/f8b0433934e046c4e6d0202d99276b82)).



## Usage



To run `python-ci.py` in the background (have it exit when closing the terminal) without using systemd: `nohup ./start.sh &`.



python-ci delivers the following pages: (they accept **only long** commit-hashes)



### Web Interface



With the configuration below, the web interface is served at `ci.example.com`.



### API



(The following links are only correct, if you use a dedicated webserver as a proxy to python-ci with a configuration as seen below. The python-ci server itself responds to requests like `/Maths/1f2a23..`, without `/api`.)



- All API requests except for the last listed here need a JWT token either specified as a GET parameter (`...?token=eyJhbGciOiJIUz...`) or as a  header: `Authentification: Bearer eyJhbGciOiJIUz...`.

- The commit-hashes in URLs can generally be replaced by `latest`



| Desc | URL | Request data | Response data | 

| ---- | --- | ------------ | ------------- |

| Login| `POST /api/login` | json: `{username: "user", password: "pass"}` | text: `jwt token...` |

| Build| `GET /api///build` | - | status code: 200 OK, 503 Busy |

| List projects| `GET /api/` | - | json: `["Maths", "test"]` |

| List builds| `GET /api//` | - | json: see below *|

| PDF build artifact| `GET /api///pdf` | - | `main.pdf` |

| Compile log | `GET /api///log` | - | `.log` |

| Badge | `GET /api///svg` | - | ![badge](docs/example_badge.svg) |

| Badge | `GET /api//latest/svg` | - (no auth. needed) |  |



`/api//`:



	{

	  "id": "user/Maths",

	  "language": "latex",

	  "latest": "947ddfc29b39ab40619e51172bc80036938ab3",

	  "list": [

	    {

	      "build": {

	        "artifacts": {

	          // request name: Display name

	          "pdf": "PDF"

	        },

	        "duration": 203.98801684379578,

	        "errorMsg": null,

	        "ref": "bf3b039261811a106dae03c90341d904378d16dc",

	        "start": 1512610571115,

	        "stats": {

	          "counts": {

	            "letters": {}, //...

	            "words": {} //...

	          }

	        },

	        "status": "success"

	      },

	      "commit": {

	        "author_name": "John Doe",

	        "date": 1512611571115,

	        "msg": "Changed something\n",

	        "parents": [

	          "0e899e95396a25ea61ed9130e93ec9220b406cd7"

	        ],

	        "ref": "bf3b039261811a106dae03c90341d904378d16dc"

	      }

	    }

	  ]

	}



There is also a SSE endpoint at `/api/subscribe` (needs to be authenticated):



	...



	id: 12ab8

	event: ....

	data: {...}



	...



| `event` | `data` | Description |

| ----- | ------ | ----------- |

| - | - | comment every 15 sec to prevent timeout |

| `` | `{"event": "status", "data": data}` | `data` is the `build` object from above |

| `` | `{"event": "log", "ref": "abcdef", "data": s}` | `s` is to append to the log |







Example for a badge which links the corresponding build page:



`[![build status](http://ci.example.com/api/Maths/latest/svg)](http://ci.example.com/Maths/latest)`



## As a GitHub webhook



Payload URL: `https://ci.example.com/api/Maths`.



When adding the webhook, be sure to set the "Content type" to `application/json`. Only the `push` (and `ping` event) event is handled.



## Server configuration



By default, python-ci listens on `localhost:8000`, meaning that it will only accept connections from the server itself. To reach it you could something like this in your nginx configuration to accept requests from the `ci` subdomain (and serve the React Single-Page App correctly) :



	server {

		listen 80;

	

		# listen 443 ssl;

		# ssl_certificate ...

		

		root <>;

	

		server_name	ci.example.com;

		

		location / {

			try_files $uri /index.html;

		}



		location /api {

			rewrite ^/api(.*) $1 break;

			proxy_pass http://localhost:8000;

			

			proxy_set_header Host $host;

			proxy_set_header X-Real-IP $remote_addr;

			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

			proxy_set_header X-Forwarded-Proto $scheme;

		}

	}



To use nginx to send your build files add the following inside the `server` block and set `NGINX_ACCEL` to any value in your `start.sh` file:



	location /data/ {

		internal;

		alias /path/to/python-ci/;

	}



If you only want the api and webhook without the web interface, then you don't need a seperate webserver. In that case, change `'localhost'` in [this](https://github.com/mischnic/python-ci/blob/b5d7e55e94ac528c41a8e30fe6297d768cb244d9/python-ci.py#L323) line to `''`, so that the server will be reachable not only from localhost.



If the server is in your local network and your router doesn't support [NAT loopback](https://en.wikipedia.org/wiki/NAT_loopback) alias [Hairpinning](https://en.wikipedia.org/wiki/Hairpinning) (meaning that trying to access `ci.example.com` in the same network as the server causes a `ERR_CONNECTION_REFUSED`) then you have to add `ci.example.com*` to the `server_name` directive. This enables you to access the server under `ci.example.com.192.168.0.2.nip.io` with `192.168.0.2` being the IP of the server.



As an alternative (more elegant but more difficult to set up) you could set up an DNS server in your local network on computer, which is always running, and make it respond to `ci.example.com` with the local IP adress of your server.