{"id":19643808,"url":"https://github.com/logicalclocks/hops-util","last_synced_at":"2025-04-28T13:31:15.529Z","repository":{"id":52176227,"uuid":"59962329","full_name":"logicalclocks/hops-util","owner":"logicalclocks","description":"Utility Library for Hopsworks","archived":true,"fork":false,"pushed_at":"2023-04-14T19:03:53.000Z","size":718,"stargazers_count":6,"open_issues_count":0,"forks_count":17,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-14T19:52:23.823Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/logicalclocks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-29T20:24:31.000Z","updated_at":"2023-08-25T07:10:22.000Z","dependencies_parsed_at":"2024-11-11T14:24:17.405Z","dependency_job_id":null,"html_url":"https://github.com/logicalclocks/hops-util","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhops-util","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhops-util/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhops-util/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhops-util/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/logicalclocks","download_url":"https://codeload.github.com/logicalclocks/hops-util/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251319757,"owners_count":21570451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T14:24:08.655Z","updated_at":"2025-04-28T13:31:10.438Z","avatar_url":"https://github.com/logicalclocks.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HopsUtil\n**HopsUtil** is a library facilitating development of Java/Scala programs for Hopsworks. It assists the developer by\n hiding the complexity of having to discover services and setting up security for programs interacting with\n Hopsworks services. Such services include the Hopsworks REST API, Apache Kafka, Apache Spark, Hopsworks Feature Store etc. For detailed\n documentation see [here](https://github.com/logicalclocks/hopsworks/). *For the python version of this library, see\n [here](https://github.com/logicalclocks/hops-util-py)*.\n\nHopsUtil is automatically deployed when users run jobs/notebooks in Hopsworks. If users need to make\nchanges to the library itself, they can build it and provide it as an additional resource to their job/notebook (see\n [doc](https://hopsworks.readthedocs.io/en/latest/user_guide/hopsworks/jupyter.html)).\n\n## Build\nTo build HopsUtil you need to have maven installed. Then simply do,\n\n```\nmvn clean package javadoc:javadoc\n```\nwhich generates under the `target` directory two archives, a thin jar that is deployed on Hops maven repository and a\nfat jar containing all the required dependencies to be used from within Hopsworks .\n\n## Usage\nThe latest version of HopsUtil is available in Hopsworks. When creating and submitting a job or opening a notebook server in\nHopsworks, HopsUtil is automatically distributed on all the nodes managed by YARN on which the job/notebook will run.\n\nIf you want to make changes or append functionality to the library, the new version can be used with the submitted\njob  by providing HopsUtil as a library when creating the job via the job service in HopsWorks. This will override\nthe  default HopsUtil available in the platform.\n\nTo include HopsUtil in your maven project, you should include the following dependency your application's POM file.\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.hops\u003c/groupId\u003e\n  \u003cartifactId\u003ehops-util\u003c/artifactId\u003e\n\u003c/dependency\u003e\n```\n\nand the following repository under your repositories list,\n```\n\u003crepository\u003e\n  \u003cid\u003eHops\u003c/id\u003e\n  \u003cname\u003eHops Repo\u003c/name\u003e\n  \u003curl\u003ehttps://archiva.hops.works/repository/Hops/\u003c/url\u003e\n  \u003creleases\u003e\n    \u003cenabled\u003etrue\u003c/enabled\u003e\n  \u003c/releases\u003e\n  \u003csnapshots\u003e\n    \u003cenabled\u003etrue\u003c/enabled\u003e\n  \u003c/snapshots\u003e\n\u003c/repository\u003e\n```\n\n## API\nHopsUtil API provides default security configurations for Kafka clients in Spark and utility methods to retrieve \nendpoints of various Hopsworks services. Moreover the API provides utility methods for interacting with the Hopsworks\n Feature Store.\n**Javadoc for HopsUtil is available** [here](http://snurran.sics.se/hops/hops-util-javadoc).\n\n\n### Feature Store API\n\nThe HopsUtil API makes it easy to write/read to Hopsworks Feature Store.\n\n#### Read a list of features from the Feature Store\n\n``` scala\nimport io.hops.util.Hops\nimport scala.collection.JavaConversions._\nimport collection.JavaConverters._\n\nval features = List(\"team_budget\", \"average_attendance\", \"average_player_age\")\nval featuresDf = Hops.getFeatures(features).read()\n```\n\n#### SQL Query to the Feature Store\n\n``` scala\nimport io.hops.util.Hops\nHops.queryFeaturestore(\n    \"SELECT team_budget, score \" +\n    \"FROM teams_features_1 JOIN games_features_1 ON \" +\n    \"games_features_1.home_team_id = teams_features_1.team_id\").read().show(5)\n```\n\n#### Write to the Feature Store\n\n``` scala\nimport io.hops.util.Hops\n\nHops.createFeaturegroup(\"teams_features_spanish\").setDataframe(teamsFeaturesDf2).write()      \n```\n\nA complete example of the Scala/Java API for the Feature Store is available [here](https://github.com/logicalclocks/hops-examples).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flogicalclocks%2Fhops-util","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flogicalclocks%2Fhops-util","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flogicalclocks%2Fhops-util/lists"}