https://github.com/knands42/data-ingestion

Data Ingestion project to evaluate my Kotlin skill using concurrency
https://github.com/knands42/data-ingestion

bigquery golang google-cloud-platform google-storage gradle-kotlin-dsl kotlin kotlin-flow

Last synced: 15 days ago
JSON representation

Data Ingestion project to evaluate my Kotlin skill using concurrency

• Host: GitHub
• URL: https://github.com/knands42/data-ingestion
• Owner: knands42
• Created: 2024-09-12T04:40:32.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-10-09T17:30:22.000Z (about 1 month ago)
• Last Synced: 2024-10-31T09:13:13.126Z (15 days ago)
• Topics: bigquery, golang, google-cloud-platform, google-storage, gradle-kotlin-dsl, kotlin, kotlin-flow
• Language: Kotlin
• Homepage:
• Size: 35.4 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Data Ingestion

## About the project

This project is about data ingestion. It is a simple project that reads data from a csv file and writes it to a

database. It is written in Kotlin, it fetch a csv file from Google Cloud storage and process using Kotlin Flow to

persist into BigQuery using proto api from GCP SDK.

## How to run

1- Generate the sample csv file

```shell

make gen-csv

```

2- Log into Google Cloud

```shell

make gcloud-login

```

3- Upload to Google Cloud Storage

```shell

make gcloud-upload

```

4- Start the application

```shell

make run

```

## Credits

The project was made following the tutorial from [Kotlin by Jetbrains](youtube.com/watch?v=0urxga3q3ty)