{"id":22914136,"url":"https://github.com/suse/pcw","last_synced_at":"2025-05-12T13:43:41.729Z","repository":{"id":37849820,"uuid":"140823511","full_name":"SUSE/pcw","owner":"SUSE","description":"Publiccloud monitor used in automated testing","archived":false,"fork":false,"pushed_at":"2025-05-02T11:22:28.000Z","size":1230,"stargazers_count":10,"open_issues_count":10,"forks_count":17,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-05-02T12:23:34.957Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SUSE.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-07-13T09:00:03.000Z","updated_at":"2025-05-02T11:20:56.000Z","dependencies_parsed_at":"2024-01-29T15:19:25.993Z","dependency_job_id":"93443041-0245-468b-89d0-0071d62e285e","html_url":"https://github.com/SUSE/pcw","commit_stats":null,"previous_names":[],"tags_count":76,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fpcw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fpcw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fpcw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fpcw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SUSE","download_url":"https://codeload.github.com/SUSE/pcw/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253749598,"owners_count":21958148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T05:13:27.841Z","updated_at":"2025-05-12T13:43:41.700Z","avatar_url":"https://github.com/SUSE.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![codecov](https://codecov.io/gh/SUSE/pcw/branch/master/graph/badge.svg)](https://codecov.io/gh/SUSE/pcw)\n\n# OpenQA Public cloud Helper\n\n![PCW project logo](https://repository-images.githubusercontent.com/140823511/394bbeff-cd84-42f2-8a36-b4cd3923e4be)\n\u003e **Jose Lausuch**: Where you thinking about PCW while laying on the beach? :stuck_out_tongue:\n\u003e **Anton Smorodskyi**: YES, constantly! :partygeeko: I see it in every palm on the beach !\n\nPublicCloud-Watcher (PCW) is a web app which monitors, displays and deletes resources on various Cloud Service Providers (CSPs).\nPCW has three main flows :\n\n1. **Update run ( implemented in [ocw/lib/db.py](ocw/lib/db.py) )** Executed every 45 minutes. Concentrates on deleting VMs (in case of Azure Resource Groups).\n    - Each update scans accounts defined in configuration file and writes the obtained results into a local sqlite database. Newly discovered entities get assigned an obligatory time-to-life value (TTL). TTL may be taken from tag `openqa_ttl` if entity is tagged with such tag if not PCW will check `pcw.ini` for `updaterun/default_ttl` setting and if setting is not defined than PCW will use hard-coded value from [webui/settings.py](webui/settings.py). Database has a web UI where you can manually trigger certain entity deletion.\n    - After persisting results into db PCW deciding which entities needs to be deleted. There are two ways to survive for entity:\n        a. Having tag `pcw_ignore` ( with any value)\n        b. Age of entity is lower than TTL defined. Age is calculated as delta of last_seen and first_seen\n    - For entities that survive cleanup PCW will sent notification email to the list defined in config.\n\n2. **Cleanup ( implemented in [ocw/lib/cleanup.py](ocw/lib/cleanup.py) )** Execution via django command. Concentrates on everything except VM deletion. This vary a lot per CSP so let's clarify that on per provider level.\n    - For Azure such entities monitored (check details in [ocw/lib/azure.py](ocw/lib/azure.py)):\n        a. bootdiagnostics\n        b. Blobs in all containers\n        c. Disks assigned to certain resource group defined in pcw.ini ('azure-storage-resourcegroup')\n        d. Images assigned to certain resource group defined in pcw.ini ('azure-storage-resourcegroup')\n        e. Image versions assigned to certain resource group defined in pcw.ini  ('azure-storage-resourcegroup')\n    - For EC2 such entities monitored (check details in [ocw/lib/ec2.py](ocw/lib/ec2.py)):\n        a. Images in all regions defined\n        b. Snapshots in all region defined\n        c. Volumes in all regions defined\n        d. VPC's ( deletion of VPC means deletion of all assigned to VPC entities first ( security groups , networks etc. ))\n    - For GCE deleting disks, images \u0026 network resources (check details in [ocw/lib/gce.py](ocw/lib/gce.py))\n3. **Dump entities quantity ( implemented in [ocw/lib/dumpstate.py](ocw/lib/dumpstate.py) )**. To be able to react fast on possible bugs in PCW and/or unexpected creation of many resources there is ability to dump real time data from each CSP into defined InfluxDB instance. This allow building real-time dashboards and/or setup  notification flow.\n\n\nThe fastest way to run PCW is via the provided containers, as described in the [Running a container](#running-a-container) section.\n\n# Usage\n\n## Python virtualenv\n\n### Requirements files\n\nPCW has 3 sets of virtual env requirements files :\n - [requirements.txt](requirements.txt) common usage for everything except K8S related cleanups\n - [requirements_k8s.txt](requirements_k8s.txt) due to high volume of dependencies needed only in single use case (k8s cleanups) they excluded in independent category\n - [requirements_test.txt](requirements_test.txt) contains dependencies allowing to run pcw's unit tests\n\n### Configuration\nConfiguration of PCW happens via a global config file in `/etc/pcw.ini`. See [templates/pcw.ini](templates/pcw.ini) for a configuration template. To start, copy the template over:\n\n```bash\n    cp templates/pwc.ini /etc/pcw.ini\n```\n\n### CSP credentials\nTo be able to connect to CSP PCW needs Service Principal details. Depending on namespaces defined in `pcw.ini`  PCW will expect some JSON files to be created\nunder `/var/pcw/[namespace name]/[Azure/EC2/GCE].json`. See [templates/var/example_namespace/](templates/var/example_namespace/) for examples.\n\nPCW supports email notifications about left-over instances. See the `notify` section therein and their corresponding comments.\n\n### Build and run\n\n```bash\n# Setup virtual environment\nvirtualenv env\nsource env/bin/activate\npip install -r requirements.txt\n\n\n## Configuration steps, only required once to setup the database and user\n# Setup database\npython manage.py migrate\n# Setup superuser (OPTIONAL)\npython manage.py createsuperuser --email admin@example.com --username admin\npython manage.py collectstatic\n\n\n## Running the webapp server\npython manage.py runserver\n```\n\nBy default, PCW runs on http://127.0.0.1:8000/\n\n## PCW in container\n\n### Available containers\n\nIn [containers](containers/) folder you main find several Dockerfiles to build several different images:\n\n - [Dockerfile](containers/Dockerfile) image based on [bci-python3.11](https://registry.suse.com/categories/bci-devel/repositories/bci-python311) and can be used to run all PCW functionality except k8s cleanup\n - [Dockerfile_k8s](containers/Dockerfile_k8s) image based on [bci-python3.11](https://registry.suse.com/categories/bci-devel/repositories/bci-python311) and can be used to run k8s cleanup\n - [Dockerfile_k8s_dev](containers/Dockerfile_k8s_dev) and [Dockerfile_dev](containers/Dockerfile_dev) images which contains same set of dependencies as [Dockerfile](containers/Dockerfile) and [Dockerfile_k8s](containers/Dockerfile_k8s) and expect PCW source code to be mounted as volumes. Very usefull for development experiments\n\n### Execution\n\nYou can use the already build containers within [this repository](https://github.com/orgs/SUSE/packages?repo_name=pcw):\n\n```bash\npodman pull ghcr.io/suse/pcw_main:latest\npodman pull ghcr.io/suse/pcw_k8s:latest\n```\n\nThe PCW container supports two volumes to be mounted:\n\n* (required) `/etc/pcw.ini` - configuration ini file\n* (optional) `/pcw/db` - volume where the database file is stored\n\nTo create a container using e.g. the data directory `/srv/pcw` for both volumes and expose port 8000, run the following:\n\n```bash\npodman create --hostname pcw --name pcw -v /srv/pcw/pcw.ini:/etc/pcw.ini -v /srv/pcw/db:/pcw/db -v \u003clocal creds storage\u003e:/var/pcw -p 8000:8000/tcp ghcr.io/suse/pcw_main:latest\npodman start pcw\n```\n\nThe `pcw` container runs by default the [/pcw/container-startup](containers/container-startup) startup helper script. You can interact with it by running\n\n```bash\npodman exec pcw /pcw/container-startup help\n\npodman run -ti --rm --hostname pcw --name pcw -v /srv/pcw/pcw.ini:/etc/pcw.ini -v \u003clocal creds storage\u003e:/var/pcw -v /srv/pcw/db:/pcw/db -p 8000:8000/tcp ghcr.io/suse/pcw_main:latest /pcw/container-startup help\n```\n\nTo create an user within the created container named `pcw`, run\n\n```bash\npodman exec pcw /pcw/container-startup createuser admin USE_A_STRONG_PASSWORD\n```\n\n### Devel version\n\nThere is [devel version](containers/Dockerfile_dev) of container file. Main difference is that source files are not copied into image but expected to be mounted via volume. This ease development in environment close as much as possible to production run.\n\nExpected use would be :\n\n```bash\nmake container-devel\npodman run  -v \u003clocal path to ini file\u003e:/etc/pcw.ini -v \u003clocal creds storage\u003e:/var/pcw -v \u003cpath to this folder\u003e:/pcw  -t pcw-devel \"python3 manage.py \u003cany command available\u003e\"\n```\n\n## Test and debug\n\n### Testing\n\n```bash\nvirtualenv .\nsource bin/activate\npip install -r requirements_test.txt\nmake test\n```\n\nThe tests contain a Selenium test for the webUI that uses Podman.  Make sure that you have the latest [geckodriver](https://github.com/mozilla/geckodriver/releases) installed anywhere in your `PATH` and that the `podman.socket` is enabled:\n`systemctl --user enable --now podman.socket`\n\nSet the `SKIP_SELENIUM` environment variable when running `pytest` or `make test` to skip the Selenium test.\n\n### Debug\n\nTo simplify problem investigation pcw has several [django commands](https://docs.djangoproject.com/en/3.1/howto/custom-management-commands/) :\n\n[cleanup](ocw/management/commands/cleanup.py)\n\n[updaterun](ocw/management/commands/updaterun.py)\n\n[dumpstate](ocw/management/commands/dumpstate.py)\n\n[rmclusters](ocw/management/commands/rmclusters.py)\n\nthose allows triggering core functionality without web UI. It is highly recommended to use `dry_run = True` in `pcw.ini` in\nsuch cases.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuse%2Fpcw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuse%2Fpcw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuse%2Fpcw/lists"}