{"id":20358098,"url":"https://github.com/irods/irods_rule_engine_plugin_logical_quotas","last_synced_at":"2025-04-12T03:15:21.190Z","repository":{"id":38274390,"uuid":"212159826","full_name":"irods/irods_rule_engine_plugin_logical_quotas","owner":"irods","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-09T17:15:26.000Z","size":234,"stargazers_count":1,"open_issues_count":6,"forks_count":9,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-12T03:15:15.919Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/irods.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-01T17:39:27.000Z","updated_at":"2025-04-09T17:15:31.000Z","dependencies_parsed_at":"2023-10-11T21:09:58.238Z","dependency_job_id":"d51168e2-e5d1-4a9f-a580-a9a8012d04c6","html_url":"https://github.com/irods/irods_rule_engine_plugin_logical_quotas","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_rule_engine_plugin_logical_quotas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_rule_engine_plugin_logical_quotas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_rule_engine_plugin_logical_quotas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_rule_engine_plugin_logical_quotas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/irods","download_url":"https://codeload.github.com/irods/irods_rule_engine_plugin_logical_quotas/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248510001,"owners_count":21116130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T23:25:22.232Z","updated_at":"2025-04-12T03:15:21.170Z","avatar_url":"https://github.com/irods.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iRODS Rule Engine Plugin - Logical Quotas\n\nAllows administrators to track and enforce limits on the number of bytes and data objects in a collection.\n\nThe following example demonstrates monitoring a collection, setting a quota on the maximum number of data\nobjects, and then violating that quota.\n```bash\n$ ils\n/tempZone/home/rods:\n  foo\n  bar\n$ irule -r irods_rule_engine_plugin-irods_rule_language-instance 'logical_quotas_start_monitoring_collection(\"/tempZone/home/rods\")' null ruleExecOut\n$ imeta ls -C .                                                                                                                                                                \nAVUs defined for collection /tempZone/home/rods:\nattribute: irods::logical_quotas::total_number_of_data_objects\nvalue: 2\nunits: \n----\nattribute: irods::logical_quotas::total_size_in_bytes\nvalue: 1014\nunits: \n$ irule -r irods_rule_engine_plugin-irods_rule_language-instance 'logical_quotas_set_maximum_number_of_data_objects(\"/tempZone/home/rods\", \"2\")' null ruleExecOut              \n$ imeta ls -C .\nAVUs defined for collection /tempZone/home/rods:\nattribute: irods::logical_quotas::maximum_number_of_data_objects\nvalue: 2\nunits: \n----\nattribute: irods::logical_quotas::total_number_of_data_objects\nvalue: 2\nunits: \n----\nattribute: irods::logical_quotas::total_size_in_bytes\nvalue: 1014\nunits: \n$ iput baz\nremote addresses: 152.54.8.75 ERROR: putUtil: put error for /tempZone/home/rods/baz, status = -130000 status = -130000 SYS_INVALID_INPUT_PARAM\nLevel 0: Logical Quotas Policy Violation: Adding object exceeds maximum number of objects limit\n$ ils\n/tempZone/home/rods:\n  foo\n  bar\n```\n\n## Build Dependencies\n\n- iRODS development package\n- iRODS runtime package\n- iRODS externals package for boost\n- iRODS externals package for fmt\n- iRODS externals package for nlohmann-json\n- iRODS externals package for spdlog\n- OpenSSL development package\n\n## Building\n\nTo build, follow the normal CMake steps.\n```bash\nmkdir build # Preferably outside of the repository.\ncd build\ncmake /path/to/repository\nmake package # Pass -j to use more parallelism.\n```\n\n## Configuration\n\nTo enable, prepend the following plugin configuration to the list of rule engines in `/etc/irods/server_config.json`. \n```js\n\"rule_engines\": [\n    {\n        \"instance_name\": \"irods_rule_engine_plugin-logical_quotas-instance\",\n        \"plugin_name\": \"irods_rule_engine_plugin-logical_quotas\",\n        \"plugin_specific_configuration\": {\n            \"namespace\": \"irods::logical_quotas\",\n            \"metadata_attribute_names\": {\n                \"maximum_number_of_data_objects\": \"maximum_number_of_data_objects\",\n                \"maximum_size_in_bytes\": \"maximum_size_in_bytes\",\n                \"total_number_of_data_objects\": \"total_number_of_data_objects\",\n                \"total_size_in_bytes\": \"total_size_in_bytes\"\n            }\n        }\n    },\n    \n    // ... Previously installed rule engine plugin configs ...\n]\n```\n\nThe plugin configuration must be placed ahead of all plugins that define any of the following PEPs:\n- pep_api_data_obj_close_post\n- pep_api_data_obj_close_pre\n- pep_api_data_obj_copy_post\n- pep_api_data_obj_copy_pre\n- pep_api_data_obj_create_and_stat_post\n- pep_api_data_obj_create_and_stat_pre\n- pep_api_data_obj_create_post\n- pep_api_data_obj_create_pre\n- pep_api_data_obj_open_and_stat_pre\n- pep_api_data_obj_open_pre\n- pep_api_data_obj_put_post\n- pep_api_data_obj_put_pre\n- pep_api_data_obj_rename_post\n- pep_api_data_obj_rename_pre\n- pep_api_data_obj_unlink_post\n- pep_api_data_obj_unlink_pre\n- pep_api_mod_avu_metadata_pre\n- pep_api_replica_close_post\n- pep_api_replica_close_pre\n- pep_api_replica_open_pre\n- pep_api_rm_coll_post\n- pep_api_rm_coll_pre\n- pep_api_touch_post\n- pep_api_touch_pre\n\nEven though this plugin will process PEPs first due to its positioning, subsequent Rule Engine Plugins (REP) will \nstill be allowed to process the same PEPs without any issues.\n\nBefore you can start monitoring collections, you'll also need to add the following specific queries to your zone:\n```bash\niadmin asq \"select count(distinct data_id) from R_DATA_MAIN d inner join R_COLL_MAIN c on d.coll_id = c.coll_id where coll_name like ?\" logical_quotas_count_data_objects_recursive\niadmin asq \"select sum(t.data_size) from (select data_id, data_size from R_DATA_MAIN d inner join R_COLL_MAIN c on d.coll_id = c.coll_id where coll_name like ? and data_is_dirty in ('1', '4') group by data_id, data_size) as t\" logical_quotas_sum_data_object_sizes_recursive\n```\nThese queries are required due to a limitation in GenQuery's ability to distinguish between multiple replicas of the same data object.\n\nThe _data_size_ specific query may result in an overcount of bytes on an actively used zone due to write-locked replicas of the same\ndata object having different sizes. For this situation, consider using slightly larger quota limits.\n\n## How To Use\n\n**IMPORTANT NOTE:** To invoke rules provided by the plugin, the only requirement is that the user be a *rodsadmin*. The *rodsadmin* user\ndoes not need permissions set on the target collection.\n\nThe following operations are supported:\n- logical_quotas_count_total_number_of_data_objects\n- logical_quotas_count_total_size_in_bytes\n- logical_quotas_get_collection_status\n- logical_quotas_recalculate_totals\n- logical_quotas_set_maximum_number_of_data_objects\n- logical_quotas_set_maximum_size_in_bytes\n- logical_quotas_start_monitoring_collection\n- logical_quotas_stop_monitoring_collection\n- logical_quotas_unset_maximum_number_of_data_objects\n- logical_quotas_unset_maximum_size_in_bytes\n- logical_quotas_unset_total_number_of_data_objects\n- logical_quotas_unset_total_size_in_bytes\n\n### Invoking operations via the Plugin\n\nTo invoke an operation through the plugin, JSON must be passed using the following structure:\n```javascript\n{\n    // One of the operations listed above.\n    \"operation\": \"\u003cvalue\u003e\",\n\n    // The absolute logical path of an existing collection.\n    \"collection\": \"\u003cvalue\u003e\",\n\n    // This value is only used by \"logical_quotas_set_maximum_number_of_data_objects\" and\n    // \"logical_quotas_set_maximum_size_in_bytes\". This is expected to be an integer\n    // passed in as a string.\n    \"value\": \"\u003cvalue\u003e\"\n}\n```\n\nUse `irule` to execute an operation. For example, we can start monitoring a collection by running the following:\n```bash\nirule -r irods_rule_engine_plugin-logical_quotas-instance '{\"operation\": \"logical_quotas_start_monitoring_collection\", \"collection\": \"/tempZone/home/rods\"}' null ruleExecOut\n```\n\nWe can set a maximum limit on the number of data objects by running the following:\n```bash\nirule -r irods_rule_engine_plugin-logical_quotas-instance '{\"operation\": \"logical_quotas_set_maximum_number_of_data_objects\", \"collection\": \"/tempZone/home/rods\", \"value\": \"100\"}' null ruleExecOut\n```\nIf no errors occurred, then `/tempZone/home/rods` will only be allowed to contain 100 data objects. However, Logical\nQuotas does not guarantee that the numbers produced perfectly reflect the total number of data objects under a collection.\nLogical Quotas only provides a relative value assuming there are many clients accessing the system simultaneously.\n\nTo help with this situation, `logical_quotas_recalculate_totals` is provided. This operation can be scheduled\nto run periodically to keep the numbers as accurate as possible.\n\nYou can also retrieve the quota status for a collection as JSON by invoking `logical_quotas_get_collection_status`, for example:\n```bash\nirule -r irods_rule_engine_plugin-logical_quotas-instance '{\"operation\": \"logical_quotas_get_collection_status\", \"collection\": \"/tempZone/home/rods\"}' null ruleExecOut\n```\nThe JSON output will be printed to the terminal and have the following structure:\n```javascript\n{\n    \u003cmaximum_number_of_data_objects_key\u003e: \"#\",\n    \u003cmaximum_size_in_bytes_key\u003e: \"#\",\n    \u003ctotal_number_of_data_objects_key\u003e: \"#\",\n    \u003ctotal_size_in_bytes_key\u003e: \"#\"\n}\n```\nThe **keys** are derived from the **namespace** and **metadata_attribute_names** defined by the plugin configuration.\n\n### Invoking operations via the Native Rule Language\n\nHere, we demonstrate how to start monitoring a collection just like in the section above.\n```bash\nirule -r irods_rule_engine_plugin-irods_rule_language-instance 'logical_quotas_start_monitoring_collection(*col)' '*col=/tempZone/home/rods' ruleExecOut\n```\n\n## Stream Operations\n\nWith previous iterations of this plugin, changes in data were tracked and checked for violations across all\nstream-based operations in real-time. However, with the introduction of intermediate replicas and logical locking\nin iRODS v4.2.9, maintaining this behavior became complex. Due to the complexity, the handling of quotas has been\nrelaxed. The most important changes are as follows:\n- Quotas are no longer checked, enforced, or updated during write and seek operations.\n- Once a quota has been violated, opening a data object for writing will fail.\n- Only data objects with replicas marked as good in the catalog are counted towards quota totals.\n\nThese changes have the following effects:\n- The plugin allows stream-based writes to violate the maximum bytes quota once.\n- Subsequent stream-based creates and writes will be denied until the quotas are out of violation.\n\n## Questions and Answers\n\n### Sometimes, the total number of bytes for my collection doesn't change when I remove a data object. Why?\n\nWhen it comes to tracking the total number of bytes in use, only **good** replicas are considered. If the data object being removed has no **good** replicas, the plugin will leave the total number of bytes as is. The reason for this is due to there not being a clear path forward for determining which replica's data size should be used for the update. Therefore, the recommendation is for administrators to recalculate the quota totals periodically.\n\nRemember, the plugin is designed to track the totals of **good** replicas only.\n\n### What are the rules around shared monitored nested collections?\n\nAnytime a user performs an operation that results in a quota update, that user MUST have **modify_object** permissions on ALL monitored parent collections. To reduce the management complexity of this, consider the following:\n- Avoid monitoring collections that have parent collections which are already being monitored\n- Use groups to simplify permission management\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_rule_engine_plugin_logical_quotas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firods%2Firods_rule_engine_plugin_logical_quotas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_rule_engine_plugin_logical_quotas/lists"}