{"id":13487313,"url":"https://github.com/logicalclocks/hopsworks","last_synced_at":"2025-05-14T15:07:16.865Z","repository":{"id":37319108,"uuid":"142410331","full_name":"logicalclocks/hopsworks","owner":"logicalclocks","description":"Hopsworks - Data-Intensive AI platform with a Feature Store","archived":false,"fork":false,"pushed_at":"2025-02-10T05:53:39.000Z","size":159798,"stargazers_count":1219,"open_issues_count":15,"forks_count":150,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-04-12T02:56:19.463Z","etag":null,"topics":["aws","azure","data-science","feature-engineering","feature-management","feature-store","gcp","governance","hopsworks","kserve","machine-learning","ml","mlops","model-serving","pyspark","python","serverless"],"latest_commit_sha":null,"homepage":"https://hopsworks.ai","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/logicalclocks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-07-26T08:13:34.000Z","updated_at":"2025-04-03T03:11:24.000Z","dependencies_parsed_at":"2025-04-12T01:50:50.963Z","dependency_job_id":"de9c52f4-2c25-4c7e-9e2d-06e17980937c","html_url":"https://github.com/logicalclocks/hopsworks","commit_stats":{"total_commits":5226,"total_committers":70,"mean_commits":74.65714285714286,"dds":0.8878683505549178,"last_synced_commit":"5a2f32bc18c3743ffcfd78c336f262046f9a3ff1"},"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhopsworks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhopsworks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhopsworks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Fhopsworks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/logicalclocks","download_url":"https://codeload.github.com/logicalclocks/hopsworks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254169581,"owners_count":22026213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","azure","data-science","feature-engineering","feature-management","feature-store","gcp","governance","hopsworks","kserve","machine-learning","ml","mlops","model-serving","pyspark","python","serverless"],"created_at":"2024-07-31T18:00:57.624Z","updated_at":"2025-05-14T15:07:16.817Z","avatar_url":"https://github.com/logicalclocks.png","language":"Java","funding_links":[],"categories":["The Data Science Toolbox","🎯 Tool Categories","Java","Model Training Orchestration","Model Deployment and Orchestration Frameworks","特征工程","Large Scale Deployment","C++","Model Training and Orchestration","Tools","Researchers","数据科学","Machine learning support systems:","Open-source"],"sub_categories":["Miscellaneous Tools","🗄️ Feature Stores","ML Platforms","[Tools](#tools-1)","Feature Stores","Frameworks","Python"],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://hopsworks.ai\"\u003e\n        \u003cimg src=\"https://uploads-ssl.webflow.com/5f6353590bb01cacbcecfbac/6202a13e7cafec5553703f6b_logo.svg\" width=\"55%\" \u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\u003cbr /\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://hopsworks.ai\" alt=\"hopsworks.ai\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/hopsworks-ai-brightgreen\" /\u003e\u003c/a\u003e\n     \u003ca href=\"https://app.hopsworks.ai\" alt=\"app\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Hopsworks-app-green\" /\u003e\u003c/a\u003e    \n    \u003ca href=\"https://docs.hopsworks.ai\" alt=\"docs.hopsworks.ai\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/hopsworks-docs-orange\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://community.hopsworks.ai\" alt=\"community.hopsworks.ai\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/hopsworks-community-blueviolet\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://twitter.com/hopsworks\" alt=\"Hopsworks Twitter\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/hopsworks-twitter-blue\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://bit.ly/publichopsworks\" alt=\"Hopsworks Slack\"\u003e\n        \u003cimg src=\"https://img.shields.io/static/v1?label=Hopsworks\u0026message=Slack\u0026color=36C5F0\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n# Quick Install\nGet up and running with a single command:\n```bash\ncurl -O https://raw.githubusercontent.com/logicalclocks/hopsworks-k8s-installer/master/install-hopsworks.py\npython3 install-hopsworks.py\n```\n\n\n\u003ca name=\"what\"\u003e\u003c/a\u003e\n# What is Hopsworks?\n\nHopsworks is a **Real-Time AI Lakehouse** for ML with a **Python-centric Feature Store** and MLOps capabilities. Hopsworks is a modular platform. You can use it as a standalone Feature Store, you can use it to manage, govern, and serve your models, and you can even use it to develop and operate feature pipelines and training pipelines. Hopsworks brings collaboration for ML teams, providing a secure, governed platform for developing, managing, and sharing ML assets - features, models, training data, batch scoring data, logs, and more.\n\u003cbr /\u003e\n\u003cp align=\"center\" style=\"background-color:white; border-radius:4px;\"\u003e\n\u003cimg src=\"https://uploads-ssl.webflow.com/5f6353590bb01cacbcecfbac/62f21c38bc47b2d313fbf76d_Marchitecture%20-%20readme.svg\" width=\"90%\"\u003e\n\u003c/p\u003e\n\u003cbr /\u003e\n\n\u003ca name=\"quick\"\u003e\u003c/a\u003e\n# 🚀 Quickstart\n\n## **APP - Serverless (beta)**\n### → **Go to [app.hopsworks.ai](https://app.hopsworks.ai)**\nHopsworks is available as a serverless app, simply head to [app.hopsworks.ai](https://app.hopsworks.ai) and register with your **Gmail** or **Github** accounts. You will then be able to run a tutorial or access Hopsworks directly and try yourself. This is the preferred way to first experience the platform before diving into more advanced uses and installation requirements. \n\n## **Azure, AWS \u0026 GCP**\n[Managed Hopsworks](https://managed.hopsworks.ai) is our platform for running Hopsworks and the Feature Store in the cloud and integrates directly with the customer AWS/Azure/GCP environment. It also integrates seamlessly with third party platforms such as Databricks, SageMaker and KubeFlow.\n\nIf you wish to run Hopsworks on your Azure, AWS or GCP environment, follow one of the following guides in our documentation:\n- [AWS Guide](https://docs.hopsworks.ai/latest/setup_installation/aws/getting_started/#step-1-connecting-your-aws-account)\n- [Azure Guide](https://docs.hopsworks.ai/latest/setup_installation/azure/getting_started/#step-1-connecting-your-azure-account)\n- [GCP Guide](https://docs.hopsworks.ai/latest/setup_installation/gcp/getting_started/#step-1-connecting-your-gcp-account)\n\n## **Installer - On-premise**\nIt is possible to use Hopsworks on-premises, which means that companies can run their machine learning workloads on their own hardware and infrastructure, rather than relying on a cloud provider. This can provide greater flexibility, control, and cost savings, as well as enabling companies to meet specific compliance and security requirements.\n\nWorking on-premises with Hopsworks typically involves collaboration with the Hopsworks engineering teams, as each infrastructure is unique and requires a tailored approach to deployment and configuration. The process begins with an assessment of the company's existing infrastructure and requirements, including network topology, security policies, and hardware specifications.\n\nFor further details about on-premise installations: [contact us](https://www.hopsworks.ai/contact).\n\n### **Requirements**\nYou need at least one server or virtual machine on which Hopsworks will be installed with at least the following specification:\n- Centos/RHEL 8.x or Ubuntu 22.04;\n- at least 32GB RAM,\n- at least 8 CPUs,\n- 100 GB of free hard-disk space,\n- a UNIX user account with sudo privileges.\n\u003cbr /\u003e\n\n\u003ca name=\"docs\"\u003e\u003c/a\u003e\n# 🎓 Documentation and API\n### **Documentation**\n[Hopsworks documentation](https://docs.hopsworks.ai) includes user guides, feature store documentation and an administration guide. We also include concepts to help user navigates the abstractions and logics of the feature stores and MLOps in general:\n- **Feature Store:** [https://docs.hopsworks.ai/latest/concepts/fs/](https://docs.hopsworks.ai/latest/concepts/fs/)\n- **Projects:** [https://docs.hopsworks.ai/latest/concepts/projects/governance/](https://docs.hopsworks.ai/latest/concepts/projects/governance/)\n- **MLOps:** [https://docs.hopsworks.ai/latest/concepts/mlops/prediction_services/](https://docs.hopsworks.ai/latest/concepts/mlops/prediction_services/)\n\n### **APIs**\nHopsworks API documentation is divided in 3 categories; Hopsworks API covers project level APIs, Feature Store API covers feature groups, feature views and connectors, and finally MLOps API covers Model Registry, serving and deployment. \n- **Hopsworks API** - [https://docs.hopsworks.ai/hopsworks-api/latest.1/generated/api/connection/](https://docs.hopsworks.ai/hopsworks-api/latest.1/generated/api/connection/)\n- **Feature Store API** - [https://docs.hopsworks.ai/feature-store-api/latest/generated/api/connection_api/](https://docs.hopsworks.ai/feature-store-api/latest/generated/api/connection_api/)\n- **MLOps API** - [https://docs.hopsworks.ai/machine-learning-api/latest/generated/connection_api/](https://docs.hopsworks.ai/machine-learning-api/latest/generated/connection_api/)\n\n### **Tutorials**\nMost of the tutorials require you to have at least an account on [app.hopsworks.ai](https://app.hopsworks.ai). You can explore the dedicated [https://github.com/logicalclocks/hopsworks-tutorials](https://github.com/logicalclocks/hopsworks-tutorials) repository containing our tutorials or jump directly in one of the existing use cases:\n- Fraud (batch): [https://github.com/logicalclocks/hopsworks-tutorials/tree/master/fraud_batch](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/fraud_batch)\n- Fraud (online): [https://github.com/logicalclocks/hopsworks-tutorials/tree/master/fraud_online](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/fraud_online)\n- Churn prediction [https://github.com/logicalclocks/hopsworks-tutorials/tree/master/churn](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/churn)\n\u003cbr /\u003e\n\n\u003ca name=\"features\"\u003e\u003c/a\u003e\n# 📦 Main Features\n\n### **Project-based Multi-Tenancy and Team Collaboration**\nHopsworks provides projects as a secure sandbox in which teams can collaborate and share ML assets. Hopsworks' unique multi-tenant project model even enables sensitive data to be stored in a shared cluster, while still providing fine-grained sharing capabilities for ML assets across project boundaries. Projects can be used to structure teams so that they have end-to-end responsibility from raw data to managed features and models. Projects can also be used to create development, staging, and production environments for data teams. All ML assets support versioning, lineage, and provenance provide all Hopsworks users with a complete view of the MLOps life cycle, from feature engineering through model serving.\n\n### **Development and Operations**\nHopsworks provides development tools for Data Science, including conda environments for Python, Jupyter notebooks, jobs, or even notebooks as jobs. You can build production pipelines with the bundled Airflow, and even run ML training pipelines with GPUs in notebooks on Airflow. You can train models on as many GPUs as are installed in a Hopsworks cluster and easily share them among users. You can also run Spark, Spark Streaming, or Flink programs on Hopsworks, with support for elastic workers in the cloud (add/remove workers dynamically).\n\n### **Available on any Platform**\nHopsworks is available as a both managed platform in the cloud on AWS, Azure, and GCP, and can be installed on any Linux-based virtual machines (Ubuntu/Redhat compatible), even in air-gapped data centers. Hopsworks is also available as a serverless platform that manages and serves both your features and models.\n\u003cbr /\u003e\n\n\u003ca name=\"community\"\u003e\u003c/a\u003e\n# 🧑‍🤝‍🧑 Community\n\n### **Contribute**\nWe are building the most complete and modular ML platform available in the market, and we count on your support to continuously improve Hopsworks. Feel free to give us suggestions, [report bugs](https://github.com/logicalclocks/hopsworks/issues) and [add features to our library](https://github.com/logicalclocks/feature-store-api) anytime.\n\n### **Join the community**\n- Ask questions and give us feedback in the [Hopsworks Community](https://community.hopsworks.ai/)\n- Join our Public [Slack Channel](https://join.slack.com/t/public-hopsworks/shared_invite/zt-24fc3hhyq-VBEiN8UZlKsDrrLvtU4NaA)\n- Follow us on [Twitter](https://twitter.com/hopsworks)\n- Check out all our latest [product releases](https://github.com/logicalclocks/hopsworks/releases)\n\n### **Open-Source**\nHopsworks is available under the **AGPL-V3 license**. In plain English this means that you are free to use Hopsworks and even build paid services on it, but if you modify the source code, you should also release back your changes and any systems built around it as AGPL-V3.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flogicalclocks%2Fhopsworks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flogicalclocks%2Fhopsworks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flogicalclocks%2Fhopsworks/lists"}