{"id":19000295,"url":"https://github.com/odoo/runbot","last_synced_at":"2025-04-07T10:26:16.902Z","repository":{"id":37742947,"uuid":"124407257","full_name":"odoo/runbot","owner":"odoo","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-10T14:24:15.000Z","size":10549,"stargazers_count":102,"open_issues_count":81,"forks_count":128,"subscribers_count":55,"default_branch":"17.0","last_synced_at":"2024-04-10T15:36:10.518Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/odoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-03-08T15:01:30.000Z","updated_at":"2024-04-15T08:59:25.711Z","dependencies_parsed_at":"2023-12-22T10:24:54.976Z","dependency_job_id":"a03cdd74-26a3-477d-88dc-09dbee76ffd7","html_url":"https://github.com/odoo/runbot","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odoo%2Frunbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odoo%2Frunbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odoo%2Frunbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odoo%2Frunbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/odoo","download_url":"https://codeload.github.com/odoo/runbot/tar.gz/refs/heads/17.0","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247634093,"owners_count":20970457,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T18:06:52.104Z","updated_at":"2025-04-07T10:26:16.880Z","avatar_url":"https://github.com/odoo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Odoo Runbot Repository\n\nThis repository contains the source code of Odoo testing bot [runbot.odoo.com](http://runbot.odoo.com/runbot) and related addons.\n\n------------------\n\n## Warnings\n\n**Runbot will delete folders/ drop databases to free some space during usage.** Even if only elements created by runbot are concerned, don't use runbot on a server with sensitive data.\n\n**Runbot changes some default odoo behaviours** Runbot database may work with other modules, but without any guarantee.\n\n**Runbot is not safe by itsefl** This tutorial describes the minimal way to deploy runbot, without too many security considerations. Only trusted code should be executed with this single machine setup. For more security the builder should be deployed separately with minimal access.\n\n## Glossary/models\nRunbot use a set of concept in order to cover all the use cases we need\n\n- **Project**: regroups a set of repositories that works together. Usually one project is enough and a default *R\u0026D* project exists.\n- **Repository**: A repository name regrouping repo and forks Ex: odoo, enterprise\n- **Remote**: A remote for a repository. Example: odoo/odoo, odoo-dev/odoo\n- **Build**: A test instance, using a set of commits and parameters to run some code and produce a result.\n- **Trigger**: Indicates that a build should be created when a new commit is pushed on a repo. A trigger has both trigger repos, and dependency repo. Ex: new commit on runbot-\u003e build with runbot and a dependency with odoo.\n- **Bundle**: A set or branches that work together: all the branches with the same name and all linked pr in the same project.\n- **Batch**: A container for builds and commits of a bundle. When a new commit is pushed on a branch, if a trigger exists for the repo of that branch, a new batch is created with this commit. After 60 seconds, if no other commit is added to the batch, a build is created by trigger having a new commit in this batch.\n\n## Processes\n\nMainly to allow to distribute runbot on multiple machine and avoid cron worker limitations, the runbot is using 2 process besides the main server.\n\n- **runbot process**: the main runbot process, serving the frontend. This is the odoo-bin process.\n- **leader process**: this process should only be started once, detect new commits and creates builds for builders.\n- **builder process**: this process can run at most once per physical host, will pick unassigned builds and execute them.\n\n## HOW TO\n\nThis section give the basic steps to follow to configure the runbot. The configuration may differ from one use to another, this one will describe how to test addons for odoo, needing to fetch odoo core but without testing vanilla odoo. As an example, the runbot odoo addon will be used as a test case. Runbotception. \n\n### DNS\n\nYou may configure a DNS entry for your runbot domain as well as a CNAME for all subdomain.\n\n```\n*        IN CNAME  runbot.domain.com.\n```\nThis is mainly usefull to access running build but will also give more freedom for future configurations. \nThis is not needed but many features won't work without that.\n\n### nginx\n\nAn exemple of config is given in the example_scripts folder.\n\nThis may be adapted depending on your setup, mainly for domain names. This can be adapted during the install but serving at least the runbot frontend (proxy pass 80 to 8069) is the minimal config needed.\nNote that runbot also has a dynamic nginx config listening on the 8080 port, mainly for running build.\n\nThis config is an ir_ui_view (runbot.nginx_config) and can be edited if needed. The config is applied and updated automatically after some time by the builder process.\n\nIt is also advised to adapt this config to work in https.\n\n### Requirements\n\nRunbot is an addon for odoo, meaning that both odoo and runbot code are needed to run. Some tips to configure odoo are available in [odoo setup documentation](https://www.odoo.com/documentation/15.0/setup/install.html#setup-install-source) (requirements, postgres, ...) This page will mainly focus on runbot specificities.\n\nYou will also need to install docker and other requirements before running runbot.\n\n```bash\nsudo apt-get install docker.io python3-unidiff python3-docker python3-matplotlib\n```\n\n### Setup\n\nChoose a workspace to clone both repositories and checkout the right branch in both of them.\nThe directory used in example scripts is `/home/$USER/odoo/` \n\nNote: It is highly advised to create a user for runbot. This example creates a new user `runbot`\n\n```bash\nsudo adduser runbot\n\n# needed access rights, docker, postgress\nsudo -u postgres createuser -d runbot\nsudo adduser runbot docker\nsudo systemctl restart docker\n\n# no sudo power needed for now\n\nsu runbot\ncd\nmkdir odoo\ncd odoo\n```\n\nYou may [add valid ssh key linked to a github account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)\n to this user in order to clone the different repositories. You could clone in https but this may be a problem latter to access your ptivate repositories. \nIt is important to clone the repo with the runbot user\n\n```bash\ngit clone --depth=1 --branch=15.0 git@github.com:odoo/odoo.git\ngit clone git@github.com:odoo/runbot.git\n\ngit -C odoo checkout 15.0\ngit -C runbot checkout 15.0\n\nmkdir logs\n```\n\nNote: `--depth=1 --branch=15.0 ` is optionnal but will help to reduce the disc usage for the odoo repo.\n\nFinally, check that you have acess to docker, listing the dockers should work without error (but will be empty).\n\n```bash\ndocker ps \n```\nIf it is not working, ensure you have the docker group and logout if needed.\n\n### Install and start runbot\n\nThis parts only consist in configuring and starting the 3 services.\n\nSome example scripts are given in `runbot/runbot/example_scripts`\n\n```bash\nmkdir ~/bin # if not exist\ncp -r ~/odoo/runbot/runbot/example_scripts/runbot ~/bin/runbot\n```\n\nScripts should be adapted, mainly forthe  `--forced-host-name parameter` in builder.sh:\n\n```bash\nsed -i \"s/runbot.domain.com/runbot.my_real_domain.com/\" ~/bin/runbot/builder.sh\n```\n\n*The hostname is initally the machine hostname but it should be different per process, having the same hostname for leader and builder is not ideal. This is why the script is using the forced-host-name parameter.*\n\n*The most important one is the builder hostname since it will be used to define running build, zip download and logs urls. We recommand setting your main domain name on this process. The nginx config given in example should be adapted if not.*\n\n\nCreate the corresponding services. You can copy them from the example scripts and adapt them:\n\n```bash\nexit # go back to a sudoer user\nrunbot_user=\"runbot\"\nsudo bash -c \"cp /home/${runbot_user}/odoo/runbot/runbot/example_scripts/services/* /etc/systemd/system/\"\nsudo sed -i \"s/runbot_user/${runbot_user}/\" /etc/systemd/system/runbot.service\nsudo sed -i \"s/runbot_user/${runbot_user}/\" /etc/systemd/system/leader.service\nsudo sed -i \"s/runbot_user/${runbot_user}/\" /etc/systemd/system/builder.service\n```\n\nEnable all services and start runbot frontend\n\n```bash\nsudo systemctl enable runbot\nsudo systemctl enable leader\nsudo systemctl enable builder\nsudo systemctl daemon-reload\nsudo systemctl start runbot\nsudo systemctl status runbot\n```\n\nRunbot service should be running\n\nYou can now connect to your backend and preconfigure runbot. \n- Install runbot module, if it wasn't done before.\n- Navigate to `/web` to leave the website configurator.\n- Connect as admin (default password: admin).\n\nCheck odoo documentation for other needed security configuration (master password). This is mainly needed for production purpose.\nYou can check that in the `/web/database/manager` page. [More info here](https://www.odoo.com/documentation/15.0/administration/install/deploy.html#security)\nChange your admin user login and password\nYou may want to check the runbot settings (`Runbot \u003e Setting \u003e setting`):\n- Default number of workers should be the max number of parallel build, consider having max `#cpu - 1`\n- Modify `Default odoorc for builds` to change the running build master password to something unique ([idealy a hashed one](https://github.com/odoo/odoo/blob/15.0/odoo/tools/config.py#L722)).\n- Tweak the garbage collection settings if you have limited disk space\n- The `number of running build` is the number of parallel running builds.\n- `Max commit age (in days)` will limt the max age of commit to detect. Increase this limit to detect older branches.\n\nFinally, start the two other services\n\n```bash\nsystemctl start leader\nsystemctl start builder\n```\n\nSeveral log files should have been created in `/home/runbot/odoo/logs/`, one per service.\n\n#### Bootstrap\nOnce launched, the leader process should start to do basic work and bootstrap will start to setup some directories in static.\n\n```bash\nsu runbot\nls ~/odoo/runbot/runbot/static\n```\n\n\u003ebuild  docker  nginx  repo  sources  src\n\n- **repo** contains the bare repositories\n- **sources** contains the exported sources needed for each build\n- **build** contains the different workspaces for dockers, containing logs/ filestore, ...\n- **docker** contains DockerFile and docker build logs\n- **nginx** contains the nginx config used to access running instances\nAll of them are empty for now.\n\nA database defined by *runbot.runbot_db_template* icp will be created. By default, runbot use template0. This database will be used as a template for testing builds. You can change this database for more customisation.\n\nOther cron operations are still disabled for now.\n\n#### DOCKER images\nA default docker image is present in the database and should automatically be build (this may take some time, check builder logs). \nDepending on your version it may not be enough.\nYou can modify it to fit your needs or ask us for the latest version of the Dockerfile waiting for an official link.\n\n#### Add remotes and repositories\nAccess runbot app and go to the `Runbot\u003eSetting\u003eRepositories` menu\n\nCreate a new repo for odoo\n![Odoo repo configuration](runbot/documentation/images/repo_odoo.png \"Odoo repo configuration\")\n\n- **Name**: `odoo` It will be used as the directory name to export the sources\n- **Identityfile** is only usefull if you want to use another ssh key to access a repo\n- **Project**: `R\u0026D` by default.\n- **Modules to install**: `-*` in order to remove them from the default `-i`. This will speed up installation. To install and test all modules, leave this space empty or use `*`. Some modules may be blacklisted individually, by using `*-module,-other_module, l10n_*`.\n- **Server files**: `odoo-bin` will allow runbot to know the possible file to use to launch odoo. odoo-bin is the one to use for the last version, but you may want to add other server files for older versions (comma separated list). The same logic is used for manifest files.\n- **Manifest files**: `__manifest__.py`. This field is only usefull to configure old versions of odoo.  \n- **Addons path**: `addons,odoo/addons`. The paths where addons are stored in this repository.\n- **Mode**: `poll` since github won't hook your runbot instance. Poll mode is limited to one update every 5 minutes. *It is advised to set it in hook mode later and hook it manually of from a cron or automated action to have more control*.\n- **Remotes**: `git@github.com:odoo/odoo.git` A single remote is added, the base odoo repo. Only branches will be fetched to limit disk usage and branches will be created in the backend. It is possible to add multiple remotes for forks.\n\nCreate another project for your repositories `Runbot\u003eSetting\u003eProject`\n\nThis is optionnal you could use the R\u0026D one, but this may be more noisy since every update in odoo/odoo will be displayed on the same page as your own repo one. Splitting by project also allows to manage access rights. \n\nCreate a repo for your custom addons repo\n![Odoo repo configuration](runbot/documentation/images/repo_runbot.png \"Odoo repo configuration\")\n- **Name**: `runbot`\n- **Project**: `runbot`.\n- **Modules to install**: `-*,runbot` ton only install the runbot module.\n- No addons_path given to use repo root as default.\n- (optionnal) For your custom repo, it is advised to configure the repo in `hook` mode if possible, adding a webhook on `/runbot/hook`. Use `/runbot/hook/\u003crepo_id\u003e` to do it manually.\n- **Remotes**: `git@github.com:odoo/runbot.git` \n- The remote *PR* option can be checked if needed to fetch pull request too . Will only work if a github token is given for this repo.\n\nA config file with your remotes should be created for each repo. You can check the content in `/runbot/static/repo/(runbot|odoo)/config`. The repo will be fetched, this operation may take some time too. After that, you should start seeing empty batches in both projects on the frontend (`/` or `/runbot`)\n\n#### Triggers and config\nAt this point, runbot will discover new branches, new commits, create bundle, but no build will be created.\n\nWhen a new commit is discovered, the branch is updated with a new commit. Then this commit is added in a batch, a container for new builds when they arrive, but only if a trigger corresponding to this repo exists. After one minute without a new commit update in the batch, the different triggers will create one build each.\nIn this example, we want to create a new build when a new commit is pushed on runbot, and this build needs a commit in odoo as a dependency.\n\nBy default the basic config will use the step `all` to test all addons. The installed addons will depends on the repo configuration, but all dependencies tests will be executed too.\nThis may not be wanted because some `base` or `web` test may be broken. This is the case with runbot addons. Also, selecting only the test for the addons\nwe are interested in will speedup the build a lot.\n\nEven if it would be better to create new Config and steps, we will modify the curent `all` config step.\n\n`Runbot \u003e Configs \u003e Build Config Steps`\n\nEdit the `all` config step and set `/runbot` as **Test tags**\n\nWe can also check the config were going to use:\n\n`Runbot \u003e Configs \u003e Build Config`\n\nOptionnaly, edit `Default no run` config and remove the `base` step. It will only test the module base.\n\nConfig and steps can be usefull to create custom test behaviour but this is out of the scope of this tutorial.\n\nCreate a new trigger like this:\n\n`Runbot\u003eTriggers`\n\n- *Name*: `Runbot` Just for display \n- *Project id*: `runbot` This is important since you can only chose repo triggering a new build in this project.\n- *Triggers*: `runbot` A new build will be created int the project when pushing on this repo.\n- *Dependencies*: `odoo` Runbot needs odoo to run\n- *Config*: `Default no run` Will start a build but dont make it running at the end. You can still wake up a build.\n\nWhen a branch is pushed, a new batch will be created, and after one minute the new build will be created if no other change is detected.\n\nCI options will only be used to send status on remotes of trigger repositories having a valid token.\n\nYou can either push, or go on the frontend bundle page and use the `Force new batch` button (refresh icon) to test this new trigger.\n\n#### Bundles\n\nBundles can be marked as `no_build`, so that new commit won't create batch creation and the bundle won't be displayed on the main page.\n\n#### Hosts\nRunbot is able to share pending builds across multiple hosts. In the present case, there is only one. A new host will never assign a pending build to himself by default.\nGo in the Build Hosts menu and choose yours. Uncheck *Only accept assigned build*. You can also tweak the number of parallel builds for this host.\n\n### Modules filters\nModules to install can be filtered by repo, and by config step. The first filter to be applied is the repo one, creating the default list for a config step.\nAddon -module on a repo will remove the module from the default, it is advised to reflect the default case on repo. To test only a custom module, adding `-*` on odoo repo will disable all odoo addons. Only dependencies of custom modules will be installed. Some specific modules can also be filtered using `-module1,-module1` or somme specific modules can be kept using `-*,module1,module2`.\nModule can also be filtered on a config step with the same logic as repo filter, except that repo's blacklist can be disabled to allow all modules by starting the list with `*` (all available modules)\nIt is also possible to add test-tags to config step to allow more module to be installed but only testing some specific one. Test tags: `/module1,/module2`\n\n### db template\nDb creation will use template0 by default. It is possible to specify a specific template to use in runbot config *Postgresql template*. It is mainly used to add extensions. This will also avoid having issue if template0 is used when creating a new database.\n\nIt is recommended to generate a `template_runbot`  database based on template0 and set this value in the runbot settings\n\n```\ncreatedb template_runbot -T template0\n```\n\n## Dockerfiles\n\nRunbot is using a Dockerfile Odoo model to define the Dockerfile used for builds and is shipped with a default one. This default Dockerfile is based on Ubuntu Bionic and is intended to build recent supported versions of Odoo.\n\nThe model is using Odoo QWeb views as templates.\n\nA new Dockerfile can be created as needed either by duplicating the default one and adapt parameters in the view. e.g.: changing the key `'from': 'ubuntu:jammy'` to `'from': 'debian:buster'` will create a new Dockerfile based on Debian instead of ubuntu.\nOr by providing a plain Dockerfile in the template.\n\nOnce the Dockerfile is created and the `to_build` field is checked, the Dockerfile will be built (pay attention that no other operations will occur during the build).\n\nA version or a bundle can be assigned a specific Dockerfile.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodoo%2Frunbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fodoo%2Frunbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodoo%2Frunbot/lists"}