{"id":20996211,"url":"https://github.com/richiejp/jdp","last_synced_at":"2026-01-02T00:05:19.009Z","repository":{"id":144723753,"uuid":"148433711","full_name":"richiejp/jdp","owner":"richiejp","description":"Automatically collect and normalise data, then run algorithms on it.","archived":false,"fork":false,"pushed_at":"2019-09-12T08:29:44.000Z","size":257058,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-06T10:23:06.341Z","etag":null,"topics":["automation-framework","data-analysis","suse-qa"],"latest_commit_sha":null,"homepage":"https://palethorpe.gitlab.io/jdp/","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/richiejp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-12T06:38:48.000Z","updated_at":"2019-09-12T08:29:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"e30a96d8-2b64-4559-bdf7-70b01b94c972","html_url":"https://github.com/richiejp/jdp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richiejp%2Fjdp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richiejp%2Fjdp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richiejp%2Fjdp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richiejp%2Fjdp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/richiejp","download_url":"https://codeload.github.com/richiejp/jdp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243410445,"owners_count":20286395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation-framework","data-analysis","suse-qa"],"created_at":"2024-11-19T07:28:34.400Z","updated_at":"2026-01-02T00:05:18.963Z","avatar_url":"https://github.com/richiejp.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JDP\n\nProcess and view data from many different sources. Regardless of how reliable\nthose sources are or what algorithms and data structures you wish to use.\n\nJDP can fit into a fully automated pipeline, be used interactively or both.\n\nThe broader aim is to *make prototyping arbitrary reporting and inter-tool\nworkflows cheap*. So that we can perform many experiments with a high risk of\nfailure. While incurring a limited cost for failed experiments and an\nunlimited payoff for successful ones.\n\n* JDP processes can be incrementally developed in an interactive, graphical\n  environment (Jupyter).\n* JDP may be used as a library in a larger project.\n* JDP has a distributed, highly-available data cache.\n* JDP is not a polished product for non-technical users, but you can use it to\n  make that.\n* JDP makes data from multiple sources/trackers easily accessible, but it is\n  [not a source of truth](development/index.html#Not-a-source-of-truth-1).\n* JDP can post back to trackers; it can automate workflows other than\n  reporting.\n* JDP is intended to fit *into* a CI/CD pipeline or take over unusual sections\n  of a pipeline, it is not intended as a replacement for Jenkins, GoCD,\n  GitlabCI, etc.\n\nInitially JDP is targeted at SUSE's QA Kernel \u0026 Networking team's\nrequirements, however it is a general purpose tool at its core. It could be\nused with any type of data for most any type of workflow or reporting.\n\n![Video Presentation (internal)](https://w3.suse.cz/~rpalethorpe/jdp-poc-pres.webm)\n[Video Presentation (external)](https://youtu.be/Nzha4itchg8)\n\n!!! tip\n\n    This README is best viewed through the [docs site](https://rpalethorpe.io.suse.de/jdp/) ([public\n    mirror](https://palethorpe.gitlab.io/jdp)).\n    Otherwise *admonition blocks* like this will be misinterpreted as literal\n    blocks.\n\n[^1]:\n\n    In the sense that the JDP project comes bundled with some scripts for\n    using it with Jupyter amongst other things.\n\n# Install\n\nI don't recommend using Docker for development or regular use on your\nworkstation (see installing from Git). However it is very useful for\ndeployment to an automated production environment.\n\n!!! note\n\n    SUSE employees and associates should view this at:\n    [gitlab.suse.de/rpalethorpe/jdp](https://gitlab.suse.de/rpalethorpe/jdp)\n\n## Docker\n\nYou can install using Docker by doing the following from the directory where\nyou cloned this repo.\n\n```sh\ndocker build -t jdp:latest -f install/Dockerfile .\n```\n\nOr you can substitute the build command for the following which will get a\npre-built image from hub.docker.com (it may not be up to date).\n\n```sh\ndocker pull suserichiejp/jdp:latest\n```\n\nThen you can inject the access details for a central data cache server if you\nhave them.\n\n```sh\ndocker build -t jdp:latest -f install/Dockerfile-slave \\\n             --build-arg REDIS_MASTER_HOST=ip-or-name \\\n             --build-arg REDIS_MASTER_AUTH=password .\n```\n\n!!! note\n\n    If you pulled from dockerhub (or wherever) then you will need to change\n    the tag name to suserichiejp/jdp:latest (or whatever).\n\nThen run it\n```sh\ndocker run -it -p 8889:8889 jdp:latest\n```\n\nWith a bit of luck you will see a message from Jupyter describing what to do\nnext.\n\nBy default JDP will create a local Redis instance automatically inside the\ncontainer. Redis will save its data within the Docker volume\n`/home/jdp/data`. Unless you mount this volume your data cache is likely to\nget deleted. It is also possible to configure JDP to connect to an existing\nRedis server; see `conf/data.toml` and `install/Dockerfile-production`.\n\nYou can use the Docker image for developing JDP itself by mounting the\n`/home/jdp/src` volume.\n\n## Git (from source)\n\nInstalling from source should be fairly easy, it just requires a few none\nJulia based dependencies. What you need depends on the trackers you intend to\nuse and whether you want to use Jupyter.\n\n!!! tip\n\n    You can also use `install/Dockerfile(-base)` as a guide. Also check\n    `conf/*.toml`.\n\nYou must install Redis unless you provide a remote Redis address in\n`conf/data.toml`.\n\nGenerally speaking, if you want to use Jupyter 'notebooks', then you should\ninstall Jupyter notebook and client packages. However there are alternatives\nto Jupyter client, which may also load and edit Jupyter notebooks.\n\nAlso install the latest stable release of Julia. JDP currently bundles an\n'upstream' version of Julia in the `install` directory. You will probably have\ndifficulties using your distribution's Julia version.\n\nIf you wish to use the OpenQA integration then install `openQA-client`. Same\ngoes for `mailx`.\n\nFinally, if you installed Jupyter or equivalent, run `julia\ninstall/install.jl` to setup `IJulia`. Otherwise this is not necessary.\n\n# Usage\n\n## With Jupyter\n\nIf you are using the Docker image then just browse to\n[localhost:8889](http://localhost:8889). If not then run `jupyter notebook` in\nthe JDP directory.\n\nOpen either the `notebooks/Report-Status-Diff.ipynb` or `notebooks/Propagate\nBug Tags.ipynb` Jupyter notebooks which are (hopefully) self documenting. I\nhave only tested them with Jupyter itself, but there are fancier alternatives\nsuch as JupyterLab.\n\n## Other\n\nYou can also use the library from a Julia REPL or another project. For example\nin a julia REPL you could run\n\n```julia\ninclude(\"src/init.jl\")\n\nusing JDP: Repository, Trackers.OpenQA, ... etc.\n```\n\nAlso the `run` directory contains scripts which are intended to automate\nvarious tasks. These can be executed with Julia in a similar way to `julia\nrun/all.jl`.\n\n## Automation\n\nJDP is automated using SUSE's internal Gitlab CI instance. Which automates\nbuilding and testing the containers as well as deployment and the execution of\nvarious scripts/services. See `install/gitlab-ci.*`.\n\nThere is also a [public pipeline](https://gitlab.com/Palethorpe/jdp/pipelines)\non GitLab.com, which publishes the public [documentation and\nreports](https://palethorpe.gitlab.io/jdp/).\n\n# Documentation\n\nFurther documentation can be found at\n[palethorpe.gitlab.io/jdp](https://palethorpe.gitlab.io/jdp) or\n[rpalethorpe.io.suse.de/jdp](https://rpalethorpe.io.suse.de/jdp)\n\nYou can also find documentation at the Julia REPL by typing `?` followed by an\nidentifier or in a notebook you can type `@doc identifier` in a code cell.\n\nThe following image may give you some intuition for what JDP is.\n\n![Outer Architecture](outer_arch.svg)\n\n# Contributors\n\nCreated and maintained by Richard Palethorpe (rpalethorpe@suse.com). Sebastian\nChlad (schlad@suse.com) is mainly responsible for it being a serious (I hope)\nproject.\n\nCyril has been asking for a result difference view and matrix for years.\n\n## Ideas and feedback\n\nBecause it is not obvious who has contributed non-code or documentation\nchanges I will try to make a list. Please let me know if I have missed you\nout or want to be removed.\n\n* Sebastian Chlad\n* Cyril Chrubis\n* Yong Sun\n* Anton Smorodskyi\n* Sergio Lindo\n* Petr Vorel\n* Oliver Kurz\n* Clemans Famulla-Conrad\n* Jose Lausuch\n* Petr Cervinka\n\n## Code and documentation\n\nSee the github/lab stats.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichiejp%2Fjdp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frichiejp%2Fjdp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichiejp%2Fjdp/lists"}