https://github.com/arturogonzalezm/transactions
https://github.com/arturogonzalezm/transactions
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/arturogonzalezm/transactions
- Owner: arturogonzalezm
- Created: 2021-10-10T12:06:08.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2021-10-13T00:45:24.000Z (over 3 years ago)
- Last Synced: 2025-01-02T08:14:40.387Z (4 months ago)
- Language: Python
- Size: 937 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## CODING ASSESSMENT – DATA ENGINEERING ##
***
### Instructions:
- Set up PySpark locally.
***
### Specifications:
- PyCharm IDE
- Apache Spark 3.1.2
- Python 3.7
- Notebooks
***
### Diagram:
The implemented approach was a hybrid Delta Lake as the below diagram:
***
### Results:
```
+-------+--------+---------+
|AgentID|PostCode|MaxAmount|
+-------+--------+---------+
|307511 |2081 |5378.03 |
|307507 |2081 |4216.0 |
|307564 |2081 |2770.24 |
|307662 |2722 |1274.57 |
|307509 |2587 |1130.5 |
|307510 |2587 |920.93 |
|307577 |2081 |763.04 |
|307508 |2587 |755.6 |
|307312 |2581 |742.36 |
|307510 |2586 |613.75 |
|307509 |2586 |613.75 |
|307662 |2581 |443.23 |
|307508 |2586 |333.61 |
|307562 |2722 |241.82 |
|306773 |2584 |188.6 |
|307561 |2722 |132.51 |
|307664 |2581 |104.11 |
|307664 |2586 |84.72 |
|307662 |2586 |84.72 |
|307506 |2587 |70.36 |
|307323 |2581 |68.7 |
|307561 |2581 |62.1 |
|307677 |2581 |57.58 |
|307562 |2581 |37.99 |
|307561 |2626 |22.18 |
|307396 |2581 |9.95 |
|307563 |2581 |9.9 |
|307681 |2581 |9.51 |
|307673 |2581 |8.6 |
|307675 |2581 |8.53 |
|307672 |2581 |7.38 |
|307663 |2581 |6.05 |
|307676 |2581 |4.38 |
|307674 |2581 |4.31 |
|307396 |2582 |0.09 |
|307312 |2582 |0.0 |
|307561 |2582 |0.0 |
|307662 |2582 |0.0 |
|307508 |2582 |0.0 |
|307510 |2582 |0.0 |
|307509 |2582 |0.0 |
|307563 |2582 |-0.22 |
|307663 |2582 |-0.48 |
|307562 |2582 |-1.26 |
|307664 |2582 |-1.75 |
|306773 |2582 |-118.05 |
+-------+--------+---------+
```