{"id":20358173,"url":"https://github.com/irods/irods_pattern_compute_to_data","last_synced_at":"2025-03-04T17:48:09.136Z","repository":{"id":66171959,"uuid":"243310946","full_name":"irods/irods_pattern_compute_to_data","owner":"irods","description":null,"archived":false,"fork":false,"pushed_at":"2020-03-05T19:18:08.000Z","size":15,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-15T01:42:38.992Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/irods.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-26T16:31:05.000Z","updated_at":"2021-04-20T09:16:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"fb41ad12-6add-47ff-b37d-a3a35396f9af","html_url":"https://github.com/irods/irods_pattern_compute_to_data","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_pattern_compute_to_data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_pattern_compute_to_data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_pattern_compute_to_data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_pattern_compute_to_data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/irods","download_url":"https://codeload.github.com/irods/irods_pattern_compute_to_data/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241895342,"owners_count":20038512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T23:25:41.162Z","updated_at":"2025-03-04T17:48:09.112Z","avatar_url":"https://github.com/irods.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# irods_pattern_compute_to_data\n\n---\n\nSetup for Compute to Data Training / Demonstration\n-------------------------------------------------------------------------------------------------------\n\n   * Follow \"Administration - Getting Started\" from UGM 2019 to fullfill prerequisites:\n\n      - set up iRODS 4.2.7 via package install\n      - install Docker-CE\n      - skip the audit plugin setup if desired\n\nClone this repo:\n```\ngit clone https://github.com/irods/irods_pattern_compute_to_data\n```\n\nAdd development packages:\n\n```\n  sudo apt-get -y install \\\n   irods-externals-cmake3.5.2-0 \\\n   irods-externals-clang3.8-0 \\\n   irods-externals-qpid-with-proton0.34-0 \\\n   irods-dev\n```\n\nIn Bash, as ubuntu user:\n\n```\nexport PATH=/opt/irods-externals/cmake3.5.2-0/bin:$PATH\n```\n\nBuild packages for\n   - *compute to data* and\n   - *register as admin* msvc:\n\n```\ncd\nmkdir build_compute_to_data\ncd build_compute_to_data\ncmake ../irods_training/advanced/hpc_compute_to_data\nmake package\nsudo dpkg -i irods-hpc-compute-to-data-example_*.deb\n\ncd\nmkdir build_register_microservice\ncd build_register_microservice\ncmake ../irods_training/advanced/hpc_compute_to_data/msvc__msiregister_as_admin/\nmake package\nsudo dpkg -i irods-microservice-register_as_admin-*.deb\n```\n\nCut and Paste setup :\n---\n\n```\n# -- as user ubuntu :\n\ncd /home/ubuntu/irods_training/advanced/hpc_compute_to_data/jupyter_notebook\ndocker build -t testimages/jupyter-digital-filter .\nsudo apt-get -y install irods-rule-engine-plugin-python\nsudo apt-get -y install python-pip\nsudo usermod -aG docker irods\npip install docker --user\n\n# -- as user irods :\n\n# Create /etc/irods/core.py with the following  import:\n\n# (Follow Native REP stanza with Python REP configuration stanza)\n\n    {\n      \"instance_name\": \"irods_rule_engine_plugin-python-instance\",\n      \"plugin_name\": \"irods_rule_engine_plugin-python\",\n      \"plugin_specific_configuration\": {}\n    }\n\n\niadmin mkuser alice rodsuser\niadmin moduser alice password apass\n\n#=== ubuntu\n\n    iinit ubuntu user as alice/apass\n\n#===\n \nadd to /etc/irods/core.py\n\nfrom compute_to_data import *\n\nsudo su irods -c '~/irodsctl restart'\n\n```\n\nAlso (after installing the compute-to-data and microservice-register_as_admin packages):\n\n\n  1. There's a different docker image to build for this version --\n\n     ```\n     ubuntu $ cd ~/jupyter_container\n     ubuntu $ docker build -t jupyter_sobel:latest -f Dockerfile.jupyter_sobel .\n     ```\n\n  2. add the line 'from compute_container import *' to /etc/irods/core.py. (This is all that is needed\n     in terms of Python rule engine code.)\n\n  3. Instead of the storage resources/AVUs in the UGM2019 presentation, we create the following\n     setup for the data to be routed \"compute-side\" :\n     ```\n     irods $  mkdir /var/lib/irods/compute_Vault\n     irods $  iadmin mkresc compute_resc unixfilesystem `hostname`:/var/lib/irods/compute_Vault\n     irods $  imeta set -R compute_resc irods::resc_compute_role image_processing\n     ```\n\n  4. Create the .json data objects describing C2D applications settings --\n     ```\n     irods $  imkdir  /tempZone/home/configured_applications\n     irods $  ichmod read public /tempZone/home/configured_applications\n     irods $  ichmod inherit /tempZone/home/configured_applications\n     irods $  icd /tempZone/home/configured_applications ; cd ; for x in sobel*.json; do iput -f $x . ;done\n     irods $  imeta set -d sobel_execute.json   irods::compute::application  sobel_auto_run  docker\n     irods $  imeta set -d sobel_notebook.json  irods::compute::application  sobel_jupyter_nb  docker\n     ```\n\n  5. set the native rulebases in the server_config to:\n     ```\n                    \"re_rulebase_set\": [\n                        \"compute_to_data_routing\",\n                        \"container_calls\",\n                        \"core\"\n                    ],\n     ```\n\n\nRunning the Compute To Data Demonstration\n---\n\n```\n\n\tubuntu $ cd ; python generate_csv_input.py \u003esquare.img.csv\n\tubuntu $ icd; imkdir input_data\n\tubuntu $ iput square.img.csv  input_data\n```\nTo show that data object \"square.img.csv\" has landed in the correct storage resource:\n```\n\tubuntu $ ils -l input_data\n        /tempZone/home/alice/input_data:\n          alice             0 compute_resc         2080 2020-01-15.16:00 \u0026 square.img.csv\n```\n\nTo launch the \"sobel_execute.json\" container synchronously:\n```\n\tubuntu $ irule -F launch_container_for_input.r \\\n                   '*outcoll=\"/tempZone/home/alice/output_synchronous\"' \\\n                   '*app=\"*/sobel_execute.json\"'\n```\n\nTo show the resulting data object \"edge_detected_square.img.csv\" was registered,\n```\n\tubuntu $ ils /tempZone/home/alice/output_synchronous\n        /tempZone/home/alice/output_synchronous:\n          edge_detected_square.img.csv\n          sobel_out.ipynb\n```\n\nTo launch the \"sobel_notebook.json\" container interactively:\n```\n\tubuntu $ irule -F launch_container_for_input.r \\\n                   '*outcoll=\"/tempZone/home/alice/output_interactive\"' \\\n                   '*app=\"*/sobel_notebook.json\"'\n```\n   - After a few seconds, the URL of the interactive Jupyter notebook will appear.\n     e.g. http://0.0.0.0:8888/?token=c9e736ae61059798bd37e05cef03f6b2c3fe8d6ae0bb055c\n     (\"0.0.0.0\" can be replaced with the docker host machine's hostname or IP)\n\n\nOpen the URL in a local browser to display the Jupyter Notebook main page.\n\n\nShow the iRODS delay queue watching for the interactive container to exit:\n        ```\n        ubuntu $ iqstat\n        id     name\n        10097  irods_policy_poll_application(*host, *context, *id, *attr_out, *status_out, \"600\")\n        ```\n\n\nHighlight the first Jupyter notebook cell and hit \u003cshift-Return\u003e to execute each cell in succession.\n\n\nShow the interactive results are not yet registered in iRODS:\n\n        ```\n\tubuntu $ ils /tempZone/home/alice/output_interactive\n        remote addresses: 127.0.1.1 ERROR: lsUtil: srcPath /tempZone/home/alice/output_interactive does not exist or user lacks access permission\n        ```\n\n\nSelect the Quit button in the upper right corner of the Jupyter notebook main page.\nThis will exit the notebook and the container will exit.\n\n\nShow the interactive results are now registered in iRODS:\n\n        ```\n        ubuntu $ ils /tempZone/home/alice/output_interactive\n        /tempZone/home/alice/output_interactive:\n          edge_detected_square.img.csv\n        ```\n\n\n#===========================================================================================\n\nIf a VM security (or something else that is running) conflicts with the default http port (8888)\nused to display the  notebook, choose a new port number and then:\n\n    * change all occurrences of the old port \"8888\" in the sobel_notebook.json app configuration file\n      and iput -f the modified file into the collection '/tempZone/home/configured_applications'\n\n#------------------\n\nNotes --\n\nIf by trial and repetition more than one docker app is using the same notebook display port (or the same output\ncollection) as a new app instance about to be spawned, you can do the following to get rid of the old one(s):\n\n```\n $ docker rm -f $(docker ps -aql)\n```\n\n(using  \"-aql\" deletes the last docker app run, using \"-aq\" would terminate all running docker applications).\n\nProbably need to another policy endpoint as a counterpart for for \"docker \u003ccontainerId\u003e stop\" but this\nisn't quite complete yet.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_pattern_compute_to_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firods%2Firods_pattern_compute_to_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_pattern_compute_to_data/lists"}