https://github.com/vikramtiwari/dataflow-samples
samples for dataflow
https://github.com/vikramtiwari/dataflow-samples
apache-beam dataflow google-cloud python
Last synced: 8 months ago
JSON representation
samples for dataflow
- Host: GitHub
- URL: https://github.com/vikramtiwari/dataflow-samples
- Owner: VikramTiwari
- Created: 2017-04-14T07:04:28.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2018-02-06T07:19:34.000Z (about 8 years ago)
- Last Synced: 2025-07-15T12:41:17.703Z (8 months ago)
- Topics: apache-beam, dataflow, google-cloud, python
- Language: Python
- Size: 62.5 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Dataflow/Apache Beam Python Samples
## Requirements
- An active Google Cloud project
- Dataflow API activated
- python 2.7
## Setup
Using python 2.7, pip and virtualenv
``` bash
# clone the project
git clone https://github.com/VikramTiwari/dataflow-samples.git dataflow-samples
# switch to the directory
cd dataflow-sample
# make sure it's 2.7
python --version
# this will create a virtual environment without affecting anything on your system
virtualenv dataflow
# activate virual environment
source dataflow/bin/activate
# install necessary packages
pip install --upgrade setuptools apache-beam
pip install --upgrade apache-beam[gcp]
```
## Samples
### To run samples, follow the instructions at the top of each file
- [Hello World](./hello_world.py)
- [Export BigQuery table to Google Storage](./export_bigquery_table_to_google_storage.py)
## Google Cloud - Dataflow Console
