https://github.com/denisogr/sentiment-batch-stream-pipeline

https://github.com/denisogr/sentiment-batch-stream-pipeline

gcp-app-engine gcp-cloud-functions gcp-dataproc gcp-storage nlp pyspark sentiment-analysis spark twitter

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/denisogr/sentiment-batch-stream-pipeline
• Owner: DenisOgr
• Created: 2021-01-24T15:02:11.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2021-05-25T16:16:25.000Z (over 3 years ago)
• Last Synced: 2024-11-13T01:34:18.508Z (3 months ago)
• Topics: gcp-app-engine, gcp-cloud-functions, gcp-dataproc, gcp-storage, nlp, pyspark, sentiment-analysis, spark, twitter
• Language: Jupyter Notebook
• Homepage:
• Size: 1.23 MB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
### Spark NLP Twitter sentiment analysis

#### Main page

 ![image info](assets/main_page.png )

 

 #### one job page

 ![image info](assets/job_page.png)

#### Batch approach.

![image info](assets/batch_arch.jpg)

###### GCP Storage architecture

```bash

gs://sentiment-twitter-analys

    |_models

      |_v1

        |_pipeline

        |_scores.txt

      |_v2

        |_pipeline

        |_scores.txt

    |_results

        |_job1

            |_meta.json

            |_twits.csv

            |_results.csv

        |_job2

            |_meta.json

            |_twits.csv

            |_results.csv

gs://sentiment-twitter-analys-scrap-job

    |_scrap_twits_jobs

      |_job1

        |_meta.json

      |_job2

        |_meta.json

gs://sentiment-twitter-analys-compute-job

    |_analytics_jobs

      |_job1

        |_meta.json

        |_twits.csv

      |_job2

        |_meta.json

        |_twits.csv

```

###### meta.json

```

{  

 "job_id": "some name",

 "quert": "some qurty phrase in twitter",

 "created_at": "some dat in ISO format",

}

```

###### twits.csv

```

tweet,created_at,likes_count,replies_count,retweets_count

@so thats my mans, 2021-02-02 09:17:35,10,0,0 

```

##### results.csv

```

2021-02-02 09:23:49, 4.0

2021-02-02 09:23:49, 0.0

2021-02-02 09:23:49, 4.0

2021-02-02 09:23:49, 0.0

2021-02-02 09:23:49, 4.0

```