https://github.com/freddo1503/stock_news_pred

Web application to provide analysis-ready data (Stocks price, News, Tweets) for ML and RNN.
https://github.com/freddo1503/stock_news_pred

apache-kafka data-engineering django docker-compose postgresql python sql

Last synced: 3 months ago
JSON representation

Web application to provide analysis-ready data (Stocks price, News, Tweets) for ML and RNN.

• Host: GitHub
• URL: https://github.com/freddo1503/stock_news_pred
• Owner: freddo1503
• Created: 2022-01-07T17:30:38.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2022-01-18T18:32:17.000Z (over 4 years ago)
• Last Synced: 2024-01-24T14:31:56.809Z (over 2 years ago)
• Topics: apache-kafka, data-engineering, django, docker-compose, postgresql, python, sql
• Language: Jupyter Notebook
• Homepage:
• Size: 37.1 MB
• Stars: 2
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Stock prices prediction with RNN and Stock News Sentiment Analysis.

The main purpose of this personal project it's to setup a entire infrastructure required for the ingestion, processing, and storage of data from third-party APIs.  

Programming language and technologies used:

- Python, SQL, Pandas, Apache Spark, Airflow, Django, FastAPI, Streamlit, Bootstrap

- AWS EC2 (Ubuntu), Docker, PostgreSQL, Kafka, ZooKeeper

From the [project](https://github.com/FrCloers/stock_news_pred) carried out at the end of the Data science bootcamp at Le Wagon, I want to create a data repository and provide for users ready-to-use data (news, tweets and stock prices) for analysis, ML and RNN.





what is being done ?

- Containerize the monolithic application (easier scaling, flexibility to work and create functionality, easier management, improved security by isolating applications)

- Collect data from disparate sources in a postgreSQL database.

    - [Alpha vantage](https://www.alphavantage.co/documentation/)

    - [EOD Historical data](https://eodhistoricaldata.com/financial-apis/financial-news-api/)

    - [Twitter API](https://developer.twitter.com/en/docs/twitter-api)

- Data processing and storage : 

    1. Created pipelines ETL for Data warehouse (Amazon Redshift or Snowflake).

    2. Stream processing for real-time data access (newest) (Apache Spark)

- User interface :

    - backend : Django

    - frontend : Bootstrap, streamlit (https://colorlib.com/polygon/elaadmin/)

## Infrastructure