https://github.com/vikentiosvitalis/advanced_topics_in_database_systems
Data Science Project - for 'Advanced Topics in Database Systems' M.Sc. Course ECE @ntua
https://github.com/vikentiosvitalis/advanced_topics_in_database_systems
apache-hadoop apache-spark big-data data-science pyspark python
Last synced: 2 months ago
JSON representation
Data Science Project - for 'Advanced Topics in Database Systems' M.Sc. Course ECE @ntua
- Host: GitHub
- URL: https://github.com/vikentiosvitalis/advanced_topics_in_database_systems
- Owner: VikentiosVitalis
- License: mit
- Created: 2023-12-23T16:19:19.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-17T15:38:37.000Z (over 2 years ago)
- Last Synced: 2025-01-23T22:46:57.756Z (over 1 year ago)
- Topics: apache-hadoop, apache-spark, big-data, data-science, pyspark, python
- Language: Python
- Homepage:
- Size: 10.6 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## [ECE @NTUA](https://www.ece.ntua.gr/en/undergraduate/info) [Advanced Topics in Database Systems](https://www.ece.ntua.gr/en/undergraduate/courses/3189)
### Data Science Project
This project focuses on data analysis using Apache Hadoop and Apache Spark.
The goal is to familiarize working with distributed systems and modern data science techniques.
The project utilizes large datasets related to crime data in Los Angeles.
### Tools Used
- [Apache Hadoop 3.3.6](https://hadoop.apache.org/)
The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing.
Distributed processing of large datasets across clusters of computers using simple programming models.
- [Apache Spark 3.5.0](https://spark.apache.org/)
Apache Spark is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
Batch/streaming data
SQL analytics
Data science at scale
Machine learning
- [~Okeanos-knossos](https://okeanos-knossos.grnet.gr/home/)
The project uses virtual machines from the public cloud `~Okeanos-knossos`.
A detailed `setup` guide for the installation of the tools used is available in the `files/documents` folder.
### Results
A detailed report with the execution and the interpretation of queries is also available in the `files/documents` folder.
### Contributors
