https://github.com/mahdijamebozorg/cryptonewscrawler

A crawler to receive crypto news from websites
https://github.com/mahdijamebozorg/cryptonewscrawler

crawler crypto cryptocurrency data-mining datamining information-retrieval llm python

Last synced: 1 day ago
JSON representation

A crawler to receive crypto news from websites

• Host: GitHub
• URL: https://github.com/mahdijamebozorg/cryptonewscrawler
• Owner: Mahdijamebozorg
• Created: 2024-09-01T14:48:51.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2024-10-17T20:02:05.000Z (about 1 month ago)
• Last Synced: 2024-10-20T06:26:27.862Z (28 days ago)
• Topics: crawler, crypto, cryptocurrency, data-mining, datamining, information-retrieval, llm, python
• Language: Python
• Homepage:
• Size: 18.6 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Smart Crypto Crawler and Analyzer

An end-to-end AI pipeline that performs technical and fundamental analysis of different cryptocurrency

## 🟢 Phase 1:

#### We have designed two end-to-end crawlers using selenium  for fetching the latest news from Coinmarketcap.com and Cointelegraph.com websites 

## 🟢 Phase 2:design 

#### We did a comprehensive method for technical analysis of different cryptocurrencies prices and volumes also we wrote different deep model for doing text processing task using LLMs for fundemnetal analysis 

###  ✅ fundamental analysis : 

#### We have different modules for each of our tasks :

#### 1- News sentiment analysis ( we perform sentiment analysis using the ensemble method with Fine-Bert and Deberta for each retrieved article, and also perform a weekly and monthly basis analysis too! )

#### 2- News text summarization ( for this pourpose we have used Bart LLM which is one of the best models we have tested)

#### 3- keyword extraction ( we have used KeyBert LLM for doing this)

#### 4-Translation of summary (after searching for an efficient LLM for long input handling, we have used the "mbart-large-50-many-to-many-mmt" model for English to Persian translation)

### ✅ technical analysis : 

#### We have different modules for each of our tasks :

#### 1- we have tried different deep models for time series prediction, finally, we have used Temporal-CNN and Auto-regressive RNN in an ensemble method for price and volume prediction (based on the last 60 days ago) 

## 🟢 Phase 3:

#### We have designed a script for scheduling the crawler and main code of the project and sending results to the telegram bot and telegram channel too!



    



### 👉 this is just a demo,asking to design a more complex and sophisticated version of this project,  please message me: at [email protected]