{"id":21268979,"url":"https://github.com/qubole/qds-sdk-py","last_synced_at":"2025-04-10T04:49:49.921Z","repository":{"id":8783464,"uuid":"10472606","full_name":"qubole/qds-sdk-py","owner":"qubole","description":"Python SDK for accessing Qubole Data Service","archived":false,"fork":false,"pushed_at":"2025-03-06T09:25:57.000Z","size":1013,"stargazers_count":52,"open_issues_count":23,"forks_count":128,"subscribers_count":19,"default_branch":"unreleased","last_synced_at":"2025-04-03T02:34:02.461Z","etag":null,"topics":["python","qubole","sdk-python"],"latest_commit_sha":null,"homepage":"https://qubole.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qubole.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-06-04T06:49:05.000Z","updated_at":"2025-03-06T09:20:22.000Z","dependencies_parsed_at":"2023-01-13T14:59:59.808Z","dependency_job_id":"3a02cbf8-ee6f-481a-a535-a61690dcb892","html_url":"https://github.com/qubole/qds-sdk-py","commit_stats":{"total_commits":459,"total_committers":78,"mean_commits":5.884615384615385,"dds":0.6557734204793029,"last_synced_commit":"3c6a34ce336561dffe8ab5ed44ddbc151d3f81bf"},"previous_names":[],"tags_count":64,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fqds-sdk-py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fqds-sdk-py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fqds-sdk-py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fqds-sdk-py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qubole","download_url":"https://codeload.github.com/qubole/qds-sdk-py/tar.gz/refs/heads/unreleased","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248161243,"owners_count":21057552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","qubole","sdk-python"],"created_at":"2024-11-21T08:06:58.139Z","updated_at":"2025-04-10T04:49:49.901Z","avatar_url":"https://github.com/qubole.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Qubole Data Service Python SDK\n==============================\n\n.. image:: https://travis-ci.org/qubole/qds-sdk-py.svg?branch=master\n    :target: https://travis-ci.org/qubole/qds-sdk-py\n    :alt: Build Status\n\nA Python module that provides the tools you need to authenticate with,\nand use the Qubole Data Service API.\n\nInstallation\n------------\n\nFrom PyPI\n~~~~~~~~~\nThe SDK is available on `PyPI \u003chttps://pypi.python.org/pypi/qds_sdk\u003e`_.\n\n::\n\n    $ pip install qds-sdk\n\nFrom source\n~~~~~~~~~~~\n* Get the source code:\n\n  - Either clone the project: ``git clone git@github.com:qubole/qds-sdk-py.git`` and checkout latest release tag from `Releases \u003chttps://github.com/qubole/qds-sdk-py/releases\u003e`_.\n  \n  - Or download one of the releases from https://github.com/qubole/qds-sdk-py/releases\n\n* Run the following command (may need to do this as root):\n\n  ::\n\n      $ python setup.py install\n\n* Alternatively, if you use virtualenv, you can do this:\n\n  ::\n\n      $ cd qds-sdk-py\n      $ virtualenv venv\n      $ source venv/bin/activate\n      $ python setup.py install\n\nThis should place a command line utility ``qds.py`` somewhere in your\npath\n\n::\n\n    $ which qds.py\n    /usr/bin/qds.py\n\n\nCLI\n---\n\n``qds.py`` allows running Hive, Hadoop, Pig, Presto and Shell commands\nagainst QDS. Users can run commands synchronously - or submit a command\nand check its status.\n\n::\n\n    $ qds.py -h  # will print detailed usage\n\nExamples:\n\n1. run a hive query and print the results\n\n   ::\n\n       $ qds.py --token 'xxyyzz' hivecmd run --query \"show tables\"\n       $ qds.py --token 'xxyyzz' hivecmd run --script_location /tmp/myquery\n       $ qds.py --token 'xxyyzz' hivecmd run --script_location s3://my-qubole-location/myquery\n\n2. pass in api token from bash environment variable\n\n   ::\n\n       $ export QDS_API_TOKEN=xxyyzz\n\n3. run the example hadoop command\n\n   ::\n\n       $ qds.py hadoopcmd run streaming -files 's3n://paid-qubole/HadoopAPIExamples/WordCountPython/mapper.py,s3n://paid-qubole/HadoopAPIExamples/WordCountPython/reducer.py' -mapper mapper.py -reducer reducer.py -numReduceTasks 1 -input 's3n://paid-qubole/default-datasets/gutenberg' -output 's3n://example.bucket.com/wcout'\n\n4. check the status of command # 12345678\n\n   ::\n\n       $ qds.py hivecmd check 12345678\n       {\"status\": \"done\", ... }\n\n5. If you are hitting api\\_url other than api.qubole.com, then you can pass it in command line as ``--url`` or set in as env variable\n\n   ::\n\n       $ qds.py --token 'xxyyzz' --url https://\u003cenv\u003e.qubole.com/api hivecmd ...\n\n       or\n\n       $ export QDS_API_URL=https://\u003cenv\u003e.qubole.com/api\n\n\nSDK API\n-------\n\nAn example Python application needs to do the following:\n\n1. Set the api\\_token and api\\_url (if api\\_url other than api.qubole.com):\n\n   ::\n\n       from qds_sdk.qubole import Qubole\n\n       Qubole.configure(api_token='ksbdvcwdkjn123423')\n\n       # or\n\n       Qubole.configure(api_token='ksbdvcwdkjn123423', api_url='https://\u003cenv\u003e.qubole.com/api')\n\n2. Use the Command classes defined in commands.py to execute commands.\n   To run Hive Command:\n\n   ::\n\n       from qds_sdk.commands import *\n\n       hc=HiveCommand.create(query='show tables')\n       print \"Id: %s, Status: %s\" % (str(hc.id), hc.status)\n\n``example/mr_1.py`` contains a Hadoop Streaming example\n\n\nReporting Bugs and Contributing Code\n------------------------------------\n\n* Want to report a bug or request a feature? Please open `an issue \u003chttps://github.com/qubole/qds-sdk-py/issues/new\u003e`_.\n* Want to contribute? Fork the project and create a pull request with your changes against ``unreleased`` branch.\n\nWhere are the maintainers ?\n---------------------------\n\nQubole was acquired. All the maintainers of this repo have moved on. Some of the employees founded `ClearFeed \u003chttps://clearfeed.ai\u003e`_. Others are at big data teams in Microsoft, Amazon et al.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubole%2Fqds-sdk-py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqubole%2Fqds-sdk-py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubole%2Fqds-sdk-py/lists"}