https://github.com/pdoup/avoulos
Big Data Analytics Project - Fall '21
https://github.com/pdoup/avoulos
big-data-analytics spark
Last synced: about 1 month ago
JSON representation
Big Data Analytics Project - Fall '21
- Host: GitHub
- URL: https://github.com/pdoup/avoulos
- Owner: pdoup
- Created: 2021-12-04T12:17:11.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2022-04-18T09:33:06.000Z (about 4 years ago)
- Last Synced: 2025-10-06T03:45:23.240Z (9 months ago)
- Topics: big-data-analytics, spark
- Language: Scala
- Homepage:
- Size: 3.84 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Analysis of the Greek Parliament Proceedings '89 – '20
Big Data Analytics Project - Fall 2021 - [Link to *iMEdD* GitHub repo](https://github.com/iMEdD-Lab/Greek_Parliament_Proceedings)
---
* [X] Task 1 : Given all speeches (for all years) we need to detect the different topics (i.e., thematic areas), most important keywords and how they change across years
* [X] Task 2 : Given all speeches we need to detect pairwise similarities between parliament members & detect the top-k pairs with the highest degree of similarity
* [X] Task 3 : For each member and also for each party we need to detect how the most important keywords evolve across years.
* [X] Task 4 : Detect any significant deviation (per member, per party or in general) with respect to the speeches before and after the crisis
* [X] Task 5 : Taking into account all speeches, we need to detect if we can group them in meaningful clusters.Check about the participation of each member in each cluster and also the participation of each party in the cluster.
* [X] Task 6 : % of Male/Female positions in the parliament over the years
---
> **How to package and run a spark application**
1. Run ``` package ``` from the ``` sbt shell ``` (IntelliJ)
2. Once the ``` .jar ``` is created in the ``` target ``` folder run this command once inside that folder
```bash
spark-submit \
--class \
--master local[*] \
--executor-memory 8G \
--total-executor-cores 4 \
/path/to/examples.jar
```
---
### **Some useful links**
- [Spark ML Library Documetation](https://spark.apache.org/docs/3.0.1/ml-guide.html)
- [Spark Tutorial](https://www.tutorialspoint.com/apache_spark/index.htm)
- [Spark Tutorial YT](https://www.youtube.com/watch?v=S2MUhGA3lEw)