{"id":15350506,"url":"https://github.com/takuti/pyhivemall","last_synced_at":"2025-04-15T04:17:47.335Z","repository":{"id":147156652,"uuid":"119583124","full_name":"takuti/PyHivemall","owner":"takuti","description":"Using machine learning model from Apache Hivemall :bee: in Python :snake:","archived":false,"fork":false,"pushed_at":"2018-03-22T03:46:23.000Z","size":157,"stargazers_count":5,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-15T04:17:42.475Z","etag":null,"topics":["data-science","hive","machine-learning","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/takuti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-30T19:22:30.000Z","updated_at":"2019-06-12T07:05:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"70c3828e-91cd-4e79-a135-5d6374449e60","html_url":"https://github.com/takuti/PyHivemall","commit_stats":{"total_commits":54,"total_committers":1,"mean_commits":54.0,"dds":0.0,"last_synced_commit":"c81d42ea0fb180e0dd82ba50e614857ff2bbcc7f"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takuti%2FPyHivemall","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takuti%2FPyHivemall/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takuti%2FPyHivemall/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takuti%2FPyHivemall/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/takuti","download_url":"https://codeload.github.com/takuti/PyHivemall/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249003976,"owners_count":21196793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","hive","machine-learning","python"],"created_at":"2024-10-01T11:58:35.167Z","updated_at":"2025-04-15T04:17:47.316Z","avatar_url":"https://github.com/takuti.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"PyHivemall\n===\n\n![overview](doc/images/overview.png)\n\n**PyHivemall** enables Python code to access and use machine learning model created by [Apache Hivemall](https://github.com/apache/incubator-hivemall), a scalable machine learning library for Apache Hive.\n\n## Installation\n\n```\n$ pip install git+https://github.com/takuti/PyHivemall.git\n```\n\n## Usage\n\n### Connect to [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2)\n\n```\n$ hive --service hiveserver2\n```\n\n```py\nfrom pyhivemall import HiveConnection\n\nconn = HiveConnection(host='localhost', port=10000)\n```\n\n### Connect to [Treasure Data](https://docs.treasuredata.com/)\n\n```py\nimport os\nfrom pyhivemall import TdConnection\n\nconn = TdConnection(apikey=os.environ['TD_API_KEY'],\n                    endpoint=os.environ['TD_API_SERVER'],\n                    database='sample_datasets')\n```\n\n### Load model with vectorizer\n\n```py\nfrom pyhivemall.linear_model import SGDClassifier\nclf, vectorizer = SGDClassifier.load(conn, 'lr_model_table',\n                                     feature_column='feature',\n                                     weight_column='weight',\n                                     bias_feature='bias',\n                                     options='-loss log -opt SGD')\n```\n\nNote that obtained model is basically compatible with corresponding [scikit-learn](http://scikit-learn.org/) model; that is, `clf` has the same parameters and functions as the [`SGDClassifier` model in scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html).\n\n### Vectorize and predict\n\nYour vectorizer may have the following features:\n\n```py\nvectorizer.feature_names_\n# ['categorical1#foo',\n#  'categorical1#bar',\n#  'categorical1#baz',\n#  'categorical2#xxx',\n#  'categorical2#yyy',\n#  'categorical2#zzz',\n#  'quantitative']\n```\n\nIn that case, prediction can be done by:\n\n```py\nd = [{'categorical1': 'foo', 'categorical2': 'xxx', 'quantitative': 2.0},\n     {'categorical1': 'bar', 'categorical2': 'yyy', 'quantitative': 4.0}]\nX = vectorizer.transform(d)\nclf.predict(X)  # yields 0/1 binary label\n```\n\n### Rebuild and update model\n\nOf course, re-fitting model in your local environment and storing the new model is possible:\n\n```py\nclf.fit(X_train, y_train)\nclf.store(conn, 'lr_model_table_sklearn', vectorizer.vocabulary_, bias_feature='bias')\n```\n\nAlso see an [example running on Docker containers](docker/) for the usage.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftakuti%2Fpyhivemall","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftakuti%2Fpyhivemall","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftakuti%2Fpyhivemall/lists"}