{"id":19721970,"url":"https://github.com/datajoint/otumat","last_synced_at":"2025-04-29T22:30:32.559Z","repository":{"id":42679314,"uuid":"237447787","full_name":"datajoint/otumat","owner":"datajoint","description":"A suite of maintainer tools and utilities for pip packages.","archived":false,"fork":false,"pushed_at":"2022-03-31T15:24:49.000Z","size":126,"stargazers_count":1,"open_issues_count":5,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-21T04:05:43.308Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datajoint.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-31T14:34:57.000Z","updated_at":"2022-01-15T01:04:09.000Z","dependencies_parsed_at":"2022-08-30T22:40:25.693Z","dependency_job_id":null,"html_url":"https://github.com/datajoint/otumat","commit_stats":null,"previous_names":["guzman-raphael/setuptools_certificate"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fotumat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fotumat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fotumat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fotumat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datajoint","download_url":"https://codeload.github.com/datajoint/otumat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251592898,"owners_count":21614441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T23:16:05.341Z","updated_at":"2025-04-29T22:30:31.969Z","avatar_url":"https://github.com/datajoint.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Otumat: Maintainer Tools \u0026 Utilities\n\nOtumat (pronounced \"Automate\") is a suite of maintainer tools and utilities for pip packages.\n\nThe following features are currently supported.\n\n## Usage Tracking\n\nHave you ever wondered:\n\n- How many users are actually using my Python package?\n- How often are they using my Python package?\n- Which features/methods are most used and which ones are less interesting to the community?\n- Is there a better alternative to track and measure usage data than using anonymous download data available from PyPi's download logs in Google BigQuery? See [Analyzing PyPI Package Download](https://packaging.python.org/guides/analyzing-pypi-package-downloads/#background) for more details on this\n\nSince a reasonable solution could not be found, I introduced this feature that provides the mechanism and building blocks to have usage tracking data as granular as you need it.\n\nThere are a few pre-requisites or assumptions:\n- User clients will have at least some periodic internet connection to upload usage logs to a centralized, remote usage-data aggregating host\n- 4 HTTP routes need to be implemented on your remote usage-data host\n  1. `GET` GUI-based authenticated route to register package installations with a user. You may use it to collect consent, have your user complete a survey, etc.\n  2. `POST` authenticated API route to accept the form submission of the above GUI route. An `installId` should be returned along with other details to ensure an 'open' connection.\n  3. `POST` authenticated API route that accepts user's event data and will store in an medium of your choice.\n  4. `POST` standard OAuth2.0 route that will allow refreshing `access_token`'s and `refresh_token`'s. [PKCE flow](https://auth0.com/docs/flows/authorization-code-flow-with-proof-key-for-code-exchange-pkce)) currently implemented).\n\nSpecific request/response details for the above 4 routes to follow soon.\n\nOnce your remote server is ready, simply add the following to your package:\n- Include `otumat` as a `requirements` dependency\n- In your `__init__.py`, intantiate an `UsageAgent` that your package can refer to. For example:\n  ```python\n  from otumat.usage import UsageAgent as _UsageAgent\n  usage_agent = _UsageAgent(author='DataJoint',\n                            data_directory='datajoint-python',\n                            package_name=__name__,\n                            host='https://datajoint.io',\n                            install_route='/user/usage-install',\n                            event_route='/api/usage-event',\n                            refresh_route='/auth/token',\n                            response_timeout=300,\n                            upload_frequency='12h')\n  ```\n  Therefore, the first time your package is imported on the client's machine, it will trigger the usage tracking installation enrollment. User's will need to opt-in though the default is **not** to collect usage data.\n- Log any interesting event within your package using the instantiated `UsageAgent`. For example, we can log imports by including the following also in our `__init__.py`:\n  ```python\n  usage_agent.log(event_type='import')\n  ```\n  Events will be buffered locally until the upload interval arrives. Caches are then unloaded. Daemon service runs cross-platform for Windows, MACOS, Linux and activates on startup.\n\nSpecific example of what an implemented flow looks like to follow soon.\n\n### Disable Usage Tracking Registration\n\nThere are some cases where it is undesirable to have the usage tracking flow triggered. For instance, if you'd like to depend on a package (e.g. `datajoint`) which does have the usage tracking flow enabled but would rather not trigger it within your package. For such a case, you could do the following in your package's `__init__.py` before your first import from `datajoint`. It will effectively disable usage tracking checks, flows, and prompts in your package:\n\n```python\nimport otumat as _otumat\n_otumat.DISABLE_USAGE_TRACKING_PACKAGES = (['datajoint'] +\n                                           _otumat.DISABLE_USAGE_TRACKING_PACKAGES)\n# first import from package with usage tracking enabled\nimport datajoint\n```\n\n## File Watching\n\nThis feature allows you to run a given script whenever a selected file is modified.\n\nTo watch a file, install `otumat` using `pip install otumat`, then run the command:\n  \n  `otumat watch [-h] -f WATCH_FILE [-i WATCH_INTERVAL] -s  WATCH_SCRIPT [watch_args ...]`\n\n### Arguments\n\nHelp:\n - `-h, --help`\n\nRequired named arguments:\n - `-f WATCH_FILE`: Path to file to be watched.\n - `-s WATCH_SCRIPT`: Path to script to run on file change.\n\nOptional named arguments:\n - `-i WATCH_INTERVAL`: Interval in seconds between polls.\n    - Defaults to 5 seconds.\n - `watch_args`: Arguments providing state between runs.\n    - Defaults to no arguments.\n\n## Validation of Trusted Plugins\n\nThis package also includes a setuptools extension that provides new keyword arguments `privkey_path` and `pubkey_path`. \n\nBy specifying the `privkey_path`, setuptools will generate the git hash (SHA1) of the module directory and sign the output based on the PEM key path passed in. The resulting signature will be stored as egg metadata `{{module_name}}.sig` accessible via `pkg_resources` module. \n\nIf passing `pubkey_path`, this will simply be copied in as egg metadata `{{module_name}}.pub`.\n\nThis provides a solution to determining the 'trust-worthiness' of plugins or extensions that may be developed by the community for a given pip package if the public key file is available for the RSA keypair. The choice of what to do for failed verification is up to you.\n\n### Use\n\n#### Extensible Package e.g. `base`\n\n``` python\nsetuptools.setup(\n    ...\n    setup_requires=['otumat'],\n    pubkey_path='./pubkey.pem',\n    ...\n```\n\n#### Plugin Package e.g. `plugin1`\n\n``` python\nsetuptools.setup(\n    ...\n    setup_requires=['otumat'],\n    privkey_path='~/keys/privkey.pem',\n    ...\n```\n\n#### Verifying Contents\n\n``` python\nimport pkg_resources\nfrom pathlib import Path\nfrom otumat import hash_pkg, verify\n\nbase_name = 'base'\nplugin_name = 'plugin1'\nbase_meta = pkg_resources.get_distribution(base_name)\nplugin_meta = pkg_resources.get_distribution(plugin_name)\n\ndata = hash_pkg(pkgpath=str(Path(plugin_meta.module_path, plugin_name)))\nsignature = plugin_meta.get_metadata('{}.sig'.format(plugin_name))\npubkey_path = str(Path(base_meta.egg_info, '{}.pub'.format(base_name)))\n\nverify(pubkey_path=pubkey_path, data=data, signature=signature)\n```\n\n\n### Compatibility with `git` and `openssl` CLI\n\nFor reference, certificates may also be generated and verified using `git` and `openssl` by the following process:\n\n#### Generate\n\n``` shell\n$ cd {{/path/to/local/repo/dir}}\n$ git add . --all\n$ GIT_HASH=$(git ls-files -s {{/pip/package/dir}} | git hash-object --stdin)\n$ printf $GIT_HASH | openssl dgst -sha256 -sign {{/path/to/privkey/pem}} -out {{pip_package_name}}.sigbin -sigopt rsa_padding_mode:pss\n$ openssl enc -base64 -in {{pip_package_name}}.sigbin -out {{pip_package_name}}.sig\n$ rm {{pip_package_name}}.sigbin\n$ git reset\n```\n\n#### Verify\n\n``` shell\n$ cd {{/path/to/local/repo/dir}}\n$ git add . --all\n$ GIT_HASH=$(git ls-files -s {{/pip/package/dir}} | git hash-object --stdin)\n$ openssl enc -base64 -d -in {{pip_package_name}}.sig -out {{pip_package_name}}.sigbin\n$ printf $GIT_HASH | openssl dgst -sha256 -verify {{/path/to/pubkey/pem}} -signature {{pip_package_name}}.sigbin -sigopt rsa_padding_mode:pss\n$ rm {{pip_package_name}}.sigbin\n$ git reset\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatajoint%2Fotumat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatajoint%2Fotumat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatajoint%2Fotumat/lists"}