https://github.com/dankolesnikov/flight-data-analysis
Flight data analysis with Clojure & Cascalog
https://github.com/dankolesnikov/flight-data-analysis
big-data cascalog clojure flight-data
Last synced: about 1 year ago
JSON representation
Flight data analysis with Clojure & Cascalog
- Host: GitHub
- URL: https://github.com/dankolesnikov/flight-data-analysis
- Owner: dankolesnikov
- License: epl-1.0
- Created: 2018-04-03T19:28:46.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2018-09-08T21:51:22.000Z (almost 8 years ago)
- Last Synced: 2025-01-31T10:34:04.502Z (over 1 year ago)
- Topics: big-data, cascalog, clojure, flight-data
- Language: Clojure
- Homepage:
- Size: 32 MB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# CS 152 Project: Cascalog
> Group project for Programming Paradigms class with Professor Cay Horstmann.
## Build
Linux:
Ensure system has Hadoop and can run as a single-node cluster:
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html.
After, build script should work perfectly fine using the following lines:
```
chmod +x build.sh
./build.sh
```
Viola!
## Overview
### Project Goals
* Learn a new functional programming language that isn't Scala - Clojure.
* Pick a framework in that language - Cascalog
* Determine whether it is a suitabe language for the problems that the chosen framework tackles.
### Description
Compare a datasets of flight data over 6 years. Analyze the data by finding the airline carriers that had the biggest delay times in 2012 and compare it to the performance in 2013, 2014, 2015, 2016, 2017. Use Hadoop through Cascalog to reduce the large dataset to a more comprehensible set of 5 csvs seperated by year for data visualization.
Metrics we are looking at:
* Total Number of Flights
* Total Number of Delayed Flights over 15 minutes
* Average Delay
### Dataset Link
[Flight Data](http://stat-computing.org/dataexpo/2009/the-data.html)
## Add-on
* Write Python script to compare performance of each airline
* Plot with bar charts
* Use Jupyter Notebook