{"id":15055504,"url":"https://github.com/change-metrics/monocle","last_synced_at":"2025-04-05T23:12:33.215Z","repository":{"id":37015044,"uuid":"224931762","full_name":"change-metrics/monocle","owner":"change-metrics","description":"Monocle helps teams and individual to better organize daily duties and to detect anomalies in the way changes are produced and reviewed.","archived":false,"fork":false,"pushed_at":"2025-03-28T15:15:35.000Z","size":9168,"stargazers_count":392,"open_issues_count":113,"forks_count":59,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-29T22:11:16.494Z","etag":null,"topics":["analytics","change","changes","gerrit","github","gitlab","graphql","hacktoberfest","metric","pull-requests","reviews","software-analytics","software-engineering","stats"],"latest_commit_sha":null,"homepage":"https://changemetrics.io","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/change-metrics.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-29T22:51:30.000Z","updated_at":"2025-03-27T07:28:33.000Z","dependencies_parsed_at":"2023-12-12T14:29:15.320Z","dependency_job_id":"c7ec5f8c-0aec-4b66-906e-5cca0fa05591","html_url":"https://github.com/change-metrics/monocle","commit_stats":{"total_commits":2353,"total_committers":19,"mean_commits":"123.84210526315789","dds":0.5660858478538037,"last_synced_commit":"d2232bffee1a381c991854dea7abc6784926137c"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/change-metrics%2Fmonocle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/change-metrics%2Fmonocle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/change-metrics%2Fmonocle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/change-metrics%2Fmonocle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/change-metrics","download_url":"https://codeload.github.com/change-metrics/monocle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411239,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","change","changes","gerrit","github","gitlab","graphql","hacktoberfest","metric","pull-requests","reviews","software-analytics","software-engineering","stats"],"created_at":"2024-09-24T21:43:27.775Z","updated_at":"2025-04-05T23:12:33.191Z","avatar_url":"https://github.com/change-metrics.png","language":"Haskell","funding_links":[],"categories":["Haskell"],"sub_categories":[],"readme":"# Monocle\n\nMonocle is designed for development teams to provide:\n\n- analytics on project changes\n- boards to display changes by criteria\n\nIt helps teams and individual to better organize daily duties\nand to detect anomalies in the way changes are produced and reviewed.\n\nMonocle supports GitHub, GitLab and Gerrit.\n\nHow to get started with Monocle:\n\n- Explore the website and blog: [changemetrics.io](https://changemetrics.io)\n- Try on the demo instance: [demo.changemetrics.io](https://demo.changemetrics.io)\n- Chat with us in the project Matrix room: [#monocle:matrix.org](https://matrix.to/#/#monocle:matrix.org)\n- Ask and Discuss about Monocle on the [Discussions page](https://github.com/change-metrics/monocle/discussions)\n- Run your own instance: [Read the installation guide](#installation)\n- Hack on it: [Read the contributing guide](CONTRIBUTING.md)\n- Checkout [who use Monocle](#users-of-monocle)\n\n## Screenshots of Monocle from the demo instance\n\nThe activity view:\n\n\u003cimg src=\"https://raw.githubusercontent.com/change-metrics/monocle/assets/images/monocle-1.3.0/monocle-activity.png\" width=\"70%\" height=\"70%\" /\u003e\n\nThe developer board:\n\n\u003cimg src=\"https://raw.githubusercontent.com/change-metrics/monocle/assets/images/monocle-1.3.0/monocle-board.png\" width=\"70%\" height=\"70%\" /\u003e\n\nThe peers strength view:\n\n\u003cimg src=\"https://raw.githubusercontent.com/change-metrics/monocle/assets/images/monocle-1.3.0/monocle-peers-strength.png\" width=\"70%\" height=\"70%\" /\u003e\n\n## Installation\n\nThe process below describes how to index changes from GitHub repositories\nand then how to start the web UI to browse metrics.\n\nThe deployment is based on Docker via a [Docker compose](https://docs.docker.com/compose/) [definition](./docker-compose.yml).\nThis compose definition can also be used by [Podman compose](https://github.com/containers/podman-compose), then all\n`docker-compose` commands from this documentation could be replaced by `podman-compose`.\n\nAlternatively, the following deployment method are supported:\n\n- [deploy on Kubernetes](k8s/README.md)\n- [deploy from source code](CONTRIBUTING.md#deploy-from-source-code)\n\n### Checkout the code\n\n```Shell\ngit clone https://github.com/change-metrics/monocle.git\ncd monocle\n# Init a .secrets file with a default API key for the crawler process\necho CRAWLERS_API_KEY=$(uuidgen) \u003e .secrets\n```\n\n### Create the config.yaml file\n\nThe `config.yaml` file is used by the crawler and api services.\n\nTo crawl GitHub public repositories, you must generate a personal access\ntoken on GitHub (w/o any specific rights) at https://github.com/settings/tokens.\n\nThen create the config file `etc/config.yaml`. Here is an example your could start with.\nMake sure to write `GITHUB_TOKEN=\u003cgithub_token\u003e` in the `.secrets` file:\n\n```YAML\n---\nworkspaces:\n  - name: monocle\n    crawlers:\n      - name: github-tektoncd\n        provider:\n          github_organization: tektoncd\n          github_repositories:\n            - operator\n            - pipeline\n        update_since: '2021-01-01'\n```\n\nTo crawl the full tektoncd GitHub organization then remove the `github_repositories` entry from the file.\nCheck the section [Workspaces](#workspaces) for a complete description of\nthe configuration.\n\n### Start docker-compose\n\nThe docker-compose file is set to use the container image of the latest stable\nrelease of Monocle (1.11.2). It is adviced to use the latest stable Monocle version.\nHowever, as our CI publishes the latest (devel) container image then it is possible\nto run the very last version. To do so set `COMPOSE_MONOCLE_VERSION` to `latest` in\nthe `.env` file.\n\nPlease refer to the [Environment variables section](#environment-variables).\n\nStart Monocle:\n\n```ShellSession\ndocker-compose up -d\n```\n\nEnsure services are running and healthy:\n\n```ShellSession\ndocker-compose ps\n```\n\nInspect services logs:\n\n```ShellSession\ndocker-compose logs -f\n```\n\nYou should be able to access the web UI at \u003chttp://localhost:8080\u003e.\n\nSee [Troubleshooting](#troubleshooting) section if needed.\n\n\n## Configuration\n\nThe Monocle configuration is splitted between a configuration file and various environment\nvariables.\n\n[Environment variables](#environment-variables) configure system settings and secrets. Any changes require\na restart of the Monocle API.\n\n[The configuration file](#configuration-file) contains non sensitive data and could be exposed in a Git repository.\nA CI/CD flow could be enabled on top of the Git repository to allow Monocle' users to\npropose configuration changes. The Monocle API automatically reload the configuration when it changes.\n\n### Environment variables\n\nFor a local deployment, default settings are fine.\n\nThe following settings are available in the `.env` file (See `.env.sample` for a sample file):\n\n- `COMPOSE_ES_XMS and COMPOSE_ES_XMX` to change the ElasticSearch JVM HEAP SIZE. By default 512m.\n- `COMPOSE_MONOCLE_VERSION` to use a specific version. By default it uses latest stable release version.\n- `COMPOSE_MONOCLE_API_ADDR` to set binding address where the Monocle API is exposed by the container.\n- `COMPOSE_MONOCLE_API_PORT` to set binding port where the Monocle API is exposed by the container.\n- `COMPOSE_MONOCLE_PUBLIC_URL` to configure the public URL to access the UI and API.\n  The URL is required for user redirection during the authentication. By default it is `http://localhost:8080`\n- `COMPOSE_MONOCLE_WEBAPP_TITLE` to change the title of the web application. By default it is `Monocle`.\n\nThe following settings are available in the `.secrets` file:\n\n- `MONOCLE_JWK_GEN_KEY` to set the local JWT issuer key. The key size must be 64 characters minimum.\n  By default the key is automatically generated.\n- `MONOCLE_OIDC_\u003cPROVIDER_NAME\u003e_CLIENT_SECRET` to set the secret used by Monocle to request an ID Token (Unset by default).\n- `MONOCLE_ADMIN_TOKEN` to set the *token* to access admin endpoints.\n  By default not set and endpoints deactivated.\n### Configuration file\n\nThe Monocle configuration file is used by the API and crawlers processes. The format of the file is YAML.\n\nThe file configures the following:\n\n- [Monocle workspaces](#workspaces)\n- The [About](#about) WEB Application endpoint\n- The [Authentication](#authentication) system\n\nYou might want to use Dhall to manage it or to better understand the schema ([dhall-monocle](https://github.com/change-metrics/dhall-monocle)).\n\n#### Workspaces\n\n\nA workspace uses a dedicated ElasticSearch index. A workspace defines:\n\n- crawlers - [details](#crawlers)\n- projects - [details](#projects-definition)\n- identities - [details](#identity-management)\n- groups - [details](#groups-definition)\n- search aliases - [details](#search-aliases-definition)\n\n##### Crawlers\n\nMonocle provides two kinds of crawlers:\n\n- [Change](#change): A crawler to fetch Changes proposed to a repository. Monocle supports Gerrit (Reviews), GitHub (Pull-Requests), GitLab (Merge-Requests).\n- [TaskData](#taskdata): A crawler to fetch task data related to a repository. Monocle supports GitHub (issues),\n  and BugZilla (Bugs).\n\nThe `.secrets` file is used to store credentials and API keys used by crawlers to authenticate on providers.\n\nThe `crawlers` value is a list of crawler. Each crawler is composed of:\n\n- `name`: an abitrary name used to identify the crawler.\n- `update_since`: the crawler will fetch changes that has been created/updated since that date.\n- `provider`: provider settings\n\n```YAML\nworkspaces:\n  - name: demo\n    crawlers:\n      - name: spinnaker\n        update_since: \"2020-01-01\"\n        provider: {}\n```\n\n###### Change\n\nA GitHub provider settings\n\n```YAML\n  provider:\n    github_organization: spinnaker\n    # Optional settings\n    github_repositories:\n      - pipeline\n    github_url: https://github.com/api/graphql\n    github_token: GITHUB_TOKEN\n```\n\n`github_organization` is the only mandatory key. If `github_repositories` is not specified then\nthe crawler will crawl the whole organization repositories. If specified then it will crawl only\nthe listed repositories. To crawl repositories from a personnal GitHub account, you need to set\n`github_organization` to the account name and list repositories under the `github_repositories` key.\n\n`github_url` might be specified in case of an alternate url. Default is \"https://github.com/api/graphql\".\n\n`github_token` might be specified to use an alternate environment variable name to look for the\ntoken value. Default is \"GITHUB_TOKEN\"\n\nRegarding the Github tokens (classic):\n\n- To crawl public repositories, no specific scope are needed (no checkbox selected).\n- To crawl privates repositories, you must set the \"repo\" scope.\n\nRegarding the Github fine grained tokens (new):\n\n- To crawl public repositories, keep the \"Public Repositories (read-only)\" checkbox selected. Nothing else is needed.\n- To crawl privates repositories, select \"All repositories\" or \"Only select repositories\", then in \"Repository permissions\"\n  select \"Pull Requests\", \"Contents\" as \"Read-only\".\n\nThe GitHub provider can also be configured to crawl Pull-Requests created by specific GitHub users.\nFor instance the following crawler's provider will fetch Pull-Requests and related events for\nusers john and jane:\n\n```YAML\n  provider:\n    github_users:\n      - john\n      - jane\n    # Optional settings\n    github_url: https://github.com/api/graphql\n    github_token: GITHUB_TOKEN\n```\n\nA Gerrit provider settings\n\n```YAML\n  provider:\n    gerrit_url: https://review.opendev.org\n    gerrit_repositories:\n      - openstack/nova\n      - openstack/neutron\n    # Optional settings\n    gerrit_login: monocle\n    gerrit_password: GERRIT_PASSWORD\n    gerrit_prefix: opendev/\n```\n\n`gerrit_url` is mandatory and must be the url of the Gerrit provider.\n`gerrit_repositories` is mandatory and is the list of repositories from which the crawler will fetch Reviews from.\n\n`gerrit_login` might be specified to authenticate on the provider API.\n`gerrit_password`might be specified to use an alternate environment variable name to look for the\npassword. Default is \"GERRIT_PASSWORD\"\n\n`gerrit_prefix` might be set to configure the crawler to prepend the repository name with a prefix.\n\n\nA GitLab provider settings\n\n```YAML\n  provider:\n    gitlab_organization: redhat/centos-stream/ci-cd/zuul\n    # Optional settings\n    gitlab_repositories:\n      - jobs-config\n    gitlab_url: https://gitlab.com/api/graphql\n    gitlab_token: GITLAB_TOKEN\n```\n\n`gitlab_organization` is the only mandatory key. If `gitlab_repositories` is not specified then\nthe crawler will crawl the whole organization repositories. If specified then it will crawl only\nthe listed repositories.\n\n`gitlab_url` might be specified in case of an alternate url. Default is \"https://gitlab.com/api/graphql\".\n\n`gitlab_token` might be specified to use an alternate environment variable name to look for the\ntoken value. Default is \"GITLAB_TOKEN\"\n\nTo crawl privates repositories, you must generate a Personal Access Token with the \"read_api\" scope.\n\n###### TaskData\n\nMonocle provides additional crawlers to attach tasks/issues/RFEs to changes based on a\nmatch on `change_url`. Then, Changes are enhanced with information about related\ntasks such as a `priority` or a `score`.\n\nFor GitHub:\n\nThe GitHub TaskData crawler run automatically whenever repositories are specified into a\n[GitHub Changes crawler](#github) definition.\n\nWhen using the \"Fine grained\" token for private repositories, then make sure to select the \"Issues\" permission.\n\nFor Bugzilla:\n\nA BugZilla provider settings\n\n```YAML\n  provider:\n    bugzilla_url: https://redhat.bugzilla.com\n    bugzilla_products:\n      - Awesome product\n    # Optional settings\n    bugzilla_token: BUGZILLA_TOKEN\n```\n\n`bugzilla_product` must be specified. The crawler will crawl listed products for bugs that\ncontain an external bug references `ext_bz_bug_id`. The crawler assumes that the external\nreference is used to link to a change (Pull-Request/Review).\n\n`bugzilla_token` might be specified to use an alternate environment variable name to look for the\ntoken value. Default is \"BUGZILLA_TOKEN\"\n\nNote that this crawler is managed by the `crawler` container.\n\n##### Projects definition\n\nProjects could be defined within a workspace configuration. A project is identified by a name and allows to set the following filter attributes:\n\n- repository_regex\n- branch_regex\n- file_regex\n\nHere is an example of configuration.\n\n```YAML\nworkspaces:\n  - name: example\n    crawlers:\n      - name: openstack\n        provider:\n          gerrit_url: https://review.opendev.org\n          gerrit_repositories:\n            - ^openstack/.*\n        update_since: \"2021-01-01\"\n    projects:\n      - name: compute\n        repository_regex: \".*nova.*\"\n      - name: compute-tests\n        file_regex: \"test[s]/.*\"\n        repository_regex: \".*nova.*\"\n      - name: deployment\n        repository_regex: \".*tripleo.*|.*puppet.*|.*ansible.*\"\n        branch_regex: \"master\"\n```\n\nThe monocle API endpoint `api/1/get_projects` can be queried to\nretrieved the list defined projects for a given workspace. See the\n[Monocle OpenAPI][monocle-openapi].\n\nThe monocle query endpoint handles the query parameter: `project`.\n\n##### Identity Management\n\nMonocle is able to index changes from multiple code review systems. A contributor\nmight get different identities across code review systems. Thus Monocle provides\na configuration section to define aliases for contributors.\n\nLet say a Monocle workspace is configured to fetch changes from github.com and\nreview.opendev.org (Gerrit) and we would like that John's metrics are merged under\nthe `John Doe` identity.\n\n```YAML\nworkspaces:\n  - name: example\n    idents:\n      - ident: John Doe\n        aliases:\n          - github.com/john-doe\n          - review.opendev.org/John Doe/12345\n          - AuthProviderUID:jdoe\n    crawlers:\n      - name: github-containers\n        provider:\n          github_organization: containers\n          github_token: \u003cgithub_token\u003e\n        update_since: '2000-01-01'\n\n      - name: gerrit-opendev\n        provider:\n          gerrit_url: https://review.opendev.org\n          gerrit_repositories:\n            - ^openstack/.*\n        update_since: '2000-01-01'\n```\n\nA contributor id on github.com or a GitHub enterprise instance is formated as `\u003cdomain\u003e/\u003clogin\u003e`.\n\nA contributor id on a Gerrit instance is formated as `\u003cdomain\u003e/\u003cFull Name\u003e/\u003cgerrit-user-id\u003e`.\n\n###### Apply idents configuration\n\nDatabase objects must be updated to reflect the configuration. Once `config.yaml` is updated, run the following commands:\n\n```bash\ndocker-compose run --rm --no-deps api monocle janitor update-idents --elastic elastic:9200 --config /etc/monocle/config.yaml\n```\n\n##### Groups definition\n\nA group in Monocle permits to group authors of Changes and filter them from the web interface.\n\nGroup memberships are defined through the [idents](#identity-management) section of the configuration.\n\nHere is an example:\n\n```YAML\nworkspaces:\n  - name: example\n    idents:\n      - ident: John Doe\n        aliases:\n          - github.com/john-doe\n          - review.opendev.org/John Doe/12345\n        groups:\n          - devs\n      - ident: Jane Doe\n        aliases:\n          - github.com/jane-doe\n        groups:\n          - devs\n          - ptl\n```\n\n\n##### Search aliases definition\n\nThe Monocle configuration file provides a way to define aliases to be used in search queries.\nA use case could be to group `bot` authors to provide an easy way to exclude them from search results.\n\nHere is an example:\n\n```YAML\nworkspaces:\n  - name: example\n    search_aliases:\n      - name: bots\n        alias: '(author:\"github-actions\" or author:\"bedevere-bot\")'\n```\n\nThen the query could be: \"from:now-3weeks and not bot\".\n\n#### About\n\nThis section configures information to be displayed by the WEB Application on the `About` modal.\n\nHere is an example:\n\n```YAML\nabout:\n  links:\n    - category: Community\n      name: Monocle configuration\n      url: https://github.com/change-metrics/demo-node-config\n```\n\n#### Crawlers\n\nThis section configures global crawlers setting.\n\n\u003e Note that, `crawlers` is a config root level parameter.\n\nHere is an example:\n\n```YAML\ncrawlers:\n  # Wait one hour between iteration\n  loop_delay_sec: 3600\n```\n\n\n#### Authentication\n\nMonocle supports user authentication via an OIDC (OpenID Connect) provider.\n\nOnce authenticated, Monocle is able to display personalized content like resolving the\n*self* query value such as in the query: *author: self*.\n\nMonocle supports one provider at a time.\n\nNote that by default and if authentication system is not configured or misconfigured,\nMonocle provides an *unauthentcated login* mechanism which allows a user to enter its\n*author name*.\n\n##### OIDC Provider\n\nHere is an example of configuration:\n\n```YAML\nauth:\n  auth_provider:\n    oidc_client_id: my_client_id\n    oidc_issuer_url: https://accounts.google.com\n    oidc_provider_name: my provider\n    oidc_user_claim: email\n  enforce_auth: false\n```\n\n- oidc_client_id: is the client id that is provided by your OIDC provider.\n- oidc_issuer_url: is the URL used to discover your OIDC's provider\n  configuration (via the *.well-known/openid-configuration* endpoint).\n- oidc_provider_name: is the name displayed on the authentication modal in the WEB Application.\n- oidc_user_claim: (optional) is the JWT claim used as/(or to discover matching) Monocle User ID. (Default: *sub*)\n- enfore_auth: (optional) if set to *True* then Monocle requires a valid JWT to access any API endpoints and the\n  WEB Application requires the user to login to navigate. (Default: *False*)\n\nThe redirect URI to configure on the OIDC provider is: `\u003cMONOCLE_PUBLIC_URL\u003e/api/2/auth/cb`.\n\nFurthermore, two additional settings ([environment variables](#environment-variables)) are required to enable the Authentication provider:\n\n- MONOCLE_PUBLIC_URL: is used to redirect the OIDC provider to the callback endpoint and the user to\n  the Monocle WEB Application after a successful authentication.\n- MONOCLE_OIDC_\u003cPROVIDER_NAME\u003e_CLIENT_SECRET: Monocle looks for this variable to discover the client secret\n  which has been provided by your OIDC provider.\n\nMonocle API generates the following log line when the Authentication provider has been properly set up:\n\n```\n2022-10-11 10:39:11 INFO    Monocle.Main:149: AuthSystemReady {\"provider\":\"my provider\"}\n```\n\nMonocle issues its own JWTs to the WEB Application users. JWTs issued by the OIDC provider are used to get\nthe authenticated user's information such as its unique uid via the *oidc_user_claim*. Monocle uses the UID (from the OIDC provider) to discover through the [idents](#identity-management) settings a matching\nMonocle Ident. A *matching ident* defines an *alias* which is *AuthProviderUID:\u003cuser_unique_id\u003e*. In case of\nno matching ident then the user claim value is used.\n\nA Monocle JWT is valid for 24 hours, once expired the user must login again to the OIDC provider.\n\n###### Service tokens\n\nMonocle can issue service tokens to be used by applications that consume the Monocle API. To request\na token your need to perform the following call:\n\n```Shell\ncurl -XPOST -d '{\"token\": \"\u003cadmin_token\u003e\"}' -H \"Content-type: application/json\" \u003cmonocle_public_url\u003e/auth/get\n{\"jwt\":\"eyJhbGciOiJIUzUxMiJ9.eyJkYXQiOnsiYURlZmF1bHRNdWlkIjoiYm90IiwiYU11aWRNYXAiOnt9fX0.bmj5vcxXxz2LmkrVKxX8jd-aYzHeTng_QBzR_9YZwCb9ToKA59TVlN1wZf6hhPlUX1VU82Y94gVCREDifintZg\"}\n```\n\nThen set the *Authorization* header to access Monocle API:\n\n```Shell\ncurl -XPOST -d '{\"void\": \"\"}' -H \"Content-type: application/json\" -H 'Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJkYXQiOnsiYURlZmF1bHRNdWlkIjoiYm90IiwiYU11aWRNYXAiOnt9fX0.bmj5vcxXxz2LmkrVKxX8jd-aYzHeTng_QBzR_9YZwCb9ToKA59TVlN1wZf6hhPlUX1VU82Y94gVCREDifintZg' \u003cmonocle_public_url\u003e/auth/whoami\n```\n\nThe `MONOCLE_ADMIN_TOKEN` must be set to enable this endpoint.\n\n### Full configuration file example\n\nHere are the expected environment variables that need to be added to the `.secrets` file:\n\n- `CRAWLERS_API_KEY`: an arbitrary api key used by the crawler to index data.\n- `GITHUB_TOKEN`: an API key for GitHub crawler.\n- `GITLAB_TOKEN`: an API key for GitLab crawler.\n\nOpen the sample [config.yaml](test/data/config.yaml).\n\n### Troubleshooting\n\nElasticSearch could need some capabilities to run in container\nmode. Take a look at the logs to see if it started correctly:\n\n```ShellSession\n$ docker-compose logs elastic\n```\n\nFor example, you could need to increase this system parameter:\n\n```ShellSession\n$ sudo sysctl -w vm.max_map_count=262144\n```\n\nor make the data directory writable for other:\n\n```ShellSession\n$ chmod og+w data\n```\n\nTo delete a workspace (a workspace is an elasticsearch index):\n\n```ShellSession\n# List indexes with:\ndocker-compose run --rm --no-deps api curl http://elastic:9200/_aliases?pretty=true\n# Delete an index with\ndocker-compose run --rm --no-deps api curl -XDELETE http://elastic:9200/\u003cindex-name\u003e\n```\n\nElasticSearch sets defaults settings on new indexes. The default setting for queries based\non regex is set to a value that might not fit your usage especially when your project definitions\nuses regex above that limit. However the limit could be increased using the following command:\n\n```ShellSession\ndocker-compose run --rm --no-deps api curl \\\n-XPUT http://localhost:9200/monocle.changes.1.\u003cindex-name\u003e/_settings \\\n-H \"Content-Type: application/json\" -d '{\"index\": {\"max_regex_length\": 50000}}'\n```\n\nCrawler default ciphers can be restrictive and not able to work with some load balancer of remote\nserver advertising an unsupported cipher. If you experience `wrong signature type` errors in the\ncrawler container, you should consider changing ciphers using the `TLS_CIPHER` environment variable\nin your docker-compose configuration file. You can find [additional information](https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2) on Fedora changelog.\n\nTo disable TLS verification, set the `TLS_NO_VERIFY` environment variable to `1`.\n\nIf you're accessing the web service remotely, make sure to set `MONOCLE_PUBLIC_URL` to the right URL in your `.env` file as this will be used to redirect requests from the UI. If not you'll see the UI fail with a \"Network error\" message.\n\n## Wipe crawler data from the database\n\nTo wipe any data related to a workspace's crawler:\n\n```bash\ndocker-compose stop crawler\ndocker-compose run --rm --no-deps api monocle janitor wipe-crawler-data --elastic elastic:9200 --config /etc/monocle/config.yaml --workspace \u003cworkspace\u003e --crawler-name \u003ccrawler-name\u003e\ndocker-compose start crawler\n```\n\n## Reset the crawler commit date\n\nMonocle crawlers keep track of the last date (commit date) when a successful document fetch happened. The command\nbelow can be used to force a crawler to fetch (again) documents since another date.\n\n```bash\ndocker-compose run --rm --no-deps api monocle janitor set-crawler-commit-date --elastic elastic:9200 --config /etc/monocle/config.yaml --workspace \u003cworkspace\u003e --crawler-name \u003ccrawler-name\u003e --commit-date 2023-01-01\n```\n\n## Components\n\n![architecture](./doc/architecture.png)\n\nMonocle is composed of the following services:\n\n1. an Elasticsearch data store.\n2. an API service to serve the Web APP and API requests.\n3. a crawler service to retrieve change from providers.\n\nThe APIs are defined using [protobuf][monocle-protobuf] and served over HTTP through [Monocle OpenAPI][monocle-openapi].\n\n## Consuming metrics via the API\n\nMost of the metrics are exposed via the `api/2/metric/get` endpoint. The `api/2/metric/list` lists the\navailable metrics. The Monocle Web UI leverages those metrics via the `Catalog` page.\n\nThe detail about the query and response format is described in the [Monocle OpenAPI][monocle-openapi] document.\n\nNote that there is an effort in progress to ensure that all metrics displayed by the Monocle Web UI use the\n`api/2/metric` endpoint.\n\nTo list the available metrics:\n\n```Shell\ncurl 'https://demo.changemetrics.io/api/2/metric/list' \\\n  -H 'Content-Type: application/json' \\\n  --data '{\"void\":\"\"}'\n```\n\nTo get the `Changes created` metric as a `trend` with automatic interval:\n\n```Shell\ncurl 'https://demo.changemetrics.io/api/2/metric/get' \\\n  -H 'Content-Type: application/json' \\\n  --data '{\"index\":\"python\",\"username\":\"\",\"query\":\"from:now-3weeks\",\"metric\":\"changes_created\",\"trend\":{\"interval\":\"\"}}'\n```\n\n## Monitoring\n\nTo setup the monitoring:\n\n1. Create the prometheus service by providing the API and CRAWLER location\n\n```ShellSession\nexport API_TARGET=localhost:8080\nexport CRAWLER_TARGET=localhost:9001\nmkdir -p /srv/prometheus\npodman create --network host -v /srv/prometheus:/var/lib/prometheus:Z -e API_TARGET=${API_TARGET} -e CRAWLER_TARGET=${CRAWLER_TARGET} --name monocle-prometheus quay.io/change-metrics/monocle-prometheus:latest\n```\n\n2. Create the grafana service (on the prometheus host)\n\n```ShellSession\nmkdir -p /srv/grafana\npodman create --network host -v /srv/grafana:/var/lib/grafana:Z -e GRAFANA_PASS=secret --name monocle-grafana quay.io/change-metrics/monocle-grafana:latest\n```\n\n3. Starts the services with systemd\n\n```ShellSession\nmkdir -p ~/.config/systemd/user/\nfor service in prometheus grafana; do podman generate systemd -n monocle-$service \u003e ~/.config/systemd/user/monocle-$service.service; done\nsystemctl --user daemon-reload\nfor service in prometheus grafana; do systemctl --user start monocle-$service; done\n```\n\n4. Check metrics with grafana dashboard at http://localhost:19030/ (login with 'admin:secret')\n\n[monocle-protobuf]: ./schemas/monocle\n[monocle-openapi]: ./doc/openapi.yaml\n\n## Users of Monocle\n\n| company name / project name      | main usage (board, metrics, other)      | public instance (url)/private instance        |\n|----------------------------------|-----------------------------------------|-----------------------------------------------|\n| changemetrics.io                 | project demo instance                   | https://demo.changemetrics.io                 |\n\nIf you use Monocle, please open a [Pull Request](https://github.com/change-metrics/monocle/pulls) to get added to this list.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchange-metrics%2Fmonocle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchange-metrics%2Fmonocle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchange-metrics%2Fmonocle/lists"}