{"id":42045863,"url":"https://github.com/vast-data/vastdb_python_beam_sink","last_synced_at":"2026-01-26T06:09:40.021Z","repository":{"id":266672019,"uuid":"898933105","full_name":"vast-data/vastdb_python_beam_sink","owner":"vast-data","description":"itdesk764","archived":false,"fork":false,"pushed_at":"2024-12-05T12:53:07.000Z","size":55,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-05T13:35:43.993Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vast-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-05T10:06:26.000Z","updated_at":"2024-12-05T12:53:11.000Z","dependencies_parsed_at":"2024-12-05T13:35:51.373Z","dependency_job_id":"af75090c-7d8a-48ec-853c-e1d12be675d9","html_url":"https://github.com/vast-data/vastdb_python_beam_sink","commit_stats":null,"previous_names":["vast-data/vastdb_python_beam_sink"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vast-data/vastdb_python_beam_sink","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vast-data%2Fvastdb_python_beam_sink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vast-data%2Fvastdb_python_beam_sink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vast-data%2Fvastdb_python_beam_sink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vast-data%2Fvastdb_python_beam_sink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vast-data","download_url":"https://codeload.github.com/vast-data/vastdb_python_beam_sink/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vast-data%2Fvastdb_python_beam_sink/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28768076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T03:54:34.369Z","status":"ssl_error","status_checked_at":"2026-01-26T03:54:33.031Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-26T06:09:39.939Z","updated_at":"2026-01-26T06:09:40.010Z","avatar_url":"https://github.com/vast-data.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Beam starter for Python with VastDB\n\nIf you want to clone this repository to start your own project,\nyou can choose the license you prefer and feel free to delete anything related to the license you are dropping.\n\nThis project is based on [apache/beam-starter-python](https://github.com/apache/beam-starter-python)\n\n## Before you begin\n\nMake sure you have a [Python 3](https://www.python.org/) development environment ready.\nIf you don't, you can download and install it from the\n[Python downloads page](https://www.python.org/downloads/).\n\nWe recommend using a virtual environment to isolate your project's dependencies.\n\n```sh\n# Create a new Python virtual environment.\npython -m venv env\n\n# Activate the virtual environment.\nsource env/bin/activate\n```\n\nWhile activated, your `python` and `pip` commands will point to the virtual environment,\nso any changes or install dependencies are self-contained.\n\nAs a one time setup, let's install the project's dependencies from the [`requirements.txt`](requirements.txt) file.\n\n```py\n# It's always a good idea to update pip before installing dependencies.\npip install -U pip\n\n# Install the project as a local package, this installs all the dependencies as well.\npip install -e .\n```\n\n\u003e ℹ️ Once you are done, you can run the `deactivate` command to go back to your global Python installation.\n\n### Edit the Vast DB source\n\nCurrently, this MVP creates two hard-coded records.\n\nThe schema is defined in [my_app/app.py](my_app/app.py):\n\n```python\n'pa_schema': pa.schema([\n    ('id', pa.int64()),\n    ('first_name', pa.utf8()),\n    ('last_name', pa.utf8())\n    ])\n```\n\nThe records are created within a pipeline, also in [my_app/app.py](my_app/app.py):\n\n```python\n| \"Create elements\" \u003e\u003e beam.Create([\n    { 'id': 1, 'first_name': 'John', 'last_name': 'Doe' },\n    { 'id': 2, 'first_name': 'Jane', 'last_name': 'Doe' }\n    ])\n```\n\n### Running the pipeline\n\nRunning your pipeline in Python is as easy as running the script file directly.\n\nThe schema (schema-name) and table (table-name) will be created if they don't exist.\n\n```sh\n# You can run the script file directly.\npython main.py \\\n    --vastdb-endpoint=\"http://your_endpoint\" \\\n    --vastdb-access-key-id=\"your_access_key\" \\\n    --vastdb-secret-access-key=\"your_secret_key\" \\\n    --vastdb-bucket-name=\"vastdb\" \\\n    --vastdb-schema-name=\"vastschema\" \\\n    --vastdb-table-name=\"your_table_name\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvast-data%2Fvastdb_python_beam_sink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvast-data%2Fvastdb_python_beam_sink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvast-data%2Fvastdb_python_beam_sink/lists"}