https://github.com/lmcrean/rosalia
Rosalia is a Data Analyst; with deep knowledge of enterprise codebases; uses qualitative and quantitative data into market opportunities and threats
https://github.com/lmcrean/rosalia
chatbot terminal-based
Last synced: 11 months ago
JSON representation
Rosalia is a Data Analyst; with deep knowledge of enterprise codebases; uses qualitative and quantitative data into market opportunities and threats
- Host: GitHub
- URL: https://github.com/lmcrean/rosalia
- Owner: lmcrean
- Created: 2025-07-28T16:42:41.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-07-28T22:20:03.000Z (11 months ago)
- Last Synced: 2025-07-28T22:27:53.350Z (11 months ago)
- Topics: chatbot, terminal-based
- Language: Jupyter Notebook
- Homepage:
- Size: 54.7 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
```
██████╗ ██████╗ ███████╗ █████╗ ██╗ ██╗ █████╗
██╔══██╗██╔═══██╗██╔════╝██╔══██╗██║ ██║██╔══██╗
██████╔╝██║ ██║███████╗███████║██║ ██║███████║
██╔══██╗██║ ██║╚════██║██╔══██║██║ ██║██╔══██║
██║ ██║╚██████╔╝███████║██║ ██║███████╗██║██║ ██║
╚═╝ ╚═╝ ╚═════╝ ╚══════╝╚═╝ ╚═╝╚══════╝╚═╝╚═╝ ╚═╝
```
# Rosalia -- Research Analyst for IT Markets 📊
[](https://www.python.org/downloads/)
[](LICENSE)
[](https://docs.github.com/en/rest)
[](integration/)
A comprehensive analysis tool for examining GitHub repository statistics across different technology categories including AI/ML, TypeScript, C# ASP.NET, and developer tools.
## Problem Statement:
**Volatility**
Public opinions circulate both of the apparent collapse and growth of certain sectors in IT, such as Web Development and Artificial Intelligence -- with inconsistent sources of data. This can of course be ignored but skipping the need for data-driven research leaves freelancers and IT professionals relying soley on intuition, constantly looking over their shoulder in a volatile/ speculative market conditions.
**Manual Researching data is time consuming.**
Sources such as ITJobsWatch tend to be viewed best individually. Grouping queries together does not always compare trends.
**AI Deep Research shortcomings**
Cutting-edge Deep Research applications such as Gemini, Claud and ChatGPT, however impressive are observed to hallucinate statistics.
## Solution:
```mermaid
---
config:
theme: redux-dark
---
flowchart LR
subgraph s1["Outcome"]
Y["Correct Intuitions"]
n1["Correct Data"]
n2["Data-Driven Decisions"]
n3["Start Over"]
n4["Correct Data"]
n5["Burnout"]
end
subgraph s1["Outcome"]
X["Wrong intuitions"]
W["Wrong Data"]
end
A["Enterprise Repositories (Github)"] --> E["Market Data"]
A2["News"] --> E
F["Office of National Statistics"] --> E
G["ITJobswatch"] --> E
E -- scraped by --> B["Python"]
B -- curated by --> Z["Markdown Reports"]
AI1["AI"] -- consulted by --> H["Human"]
AI2["AI"] -- consulted by --> H2["Human"]
E -- "time-consuming, overwhelming" --> H3["Human"]
E -- hallucinates --> AI2
H3 --> n4
H2 --> W
Z -- Read by --> H & AI1
W --> X
H --> n1
n1 --> Y
X --> n3
Y --> n2
n4 --> n5
n5 --> n3
n4@{ shape: rect}
n5@{ shape: rect}
A@{ shape: cyl}
E@{ shape: rect}
A2@{ shape: cyl}
F@{ shape: cyl}
G@{ shape: cyl}
style n2 fill:#C8E6C9,color:#000000
style n3 fill:#FFCDD2,color:#000000
```
***Diagram (above):*** Using Python Scraping and Markdown Reports in research workflow ensures accurate data while saving burnout of manual research.
Rosalia scrapes data from reputable sources and collates them into a Google Docs.
This can be viewed by a human or synchronised with an AI Project.
## 🎯 Overview
This project collects and analyzes
### Data Collection
Statistics from 55+ popular open-source enterprise codebases, providing insights into:
- Star counts and popularity trends (formatted as 12.3k, 1.2M for readability)
- Fork and contributor statistics
- Pull request activity
- Technology stack distributions
- Repository health metrics
Statistics for emerging and collapsing markets in Technology Sector
- Currently limited to UK-specific, with focus on London
- ITJobswatch.co.uk
- 10-15 year trends
- Keywords: AI, ML, Web Development, Software Engineering, Data Science, Data Analytics, QA Testing
- Languages: Python, C#, Typescript,
- Frameworks/ Technology: ASP.NET, Google Cloud, AWS,
Statistics from Office of National Statistics
- understanding the Technology Sector and UK Economy on 10-15 year trend
## 🚀 Quick Start
### Prerequisites
- Python 3.8 or higher
- GitHub personal access token (for API access)
### Installation
1. Clone the repository:
```bash
git clone https://github.com/yourusername/research-enterprise-codebases.git
cd research-enterprise-codebases
```
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. Create a `.env` file with your GitHub token:
```bash
echo "GITHUB_TOKEN=your_github_token_here" > .env
```
### Usage
#### 1. Verify Setup
Test your configuration:
```bash
python scripts/quick_test.py
```
#### 2. Collect Data
Run the data collection script to fetch fresh statistics:
```bash
python api/github/collector.py
```
This will:
- Fetch repository data from GitHub API
- Handle rate limiting automatically
- Save results to `data/raw/github_repository_stats.csv`
- Create metadata file with collection timestamp
#### 3. Generate Reports
Create markdown reports with formatted data:
```bash
python scripts/generate_reports.py
```
#### 4. View Results
Open the generated markdown reports in `views/`:
- [Overview Report](views/all.md) - Comprehensive summary
- [AI/ML Repositories](views/ai_ml.md) - AI/ML focused analysis
- [TypeScript Projects](views/typescript.md) - TypeScript ecosystem
- [C# ASP.NET Projects](views/csharp.md) - C# ASP.NET analysis
## 📁 Project Structure
```
research-enterprise-codebases/
├── api/
│ ├── github/ # GitHub API client modules
│ │ ├── client.py # API client with rate limiting
│ │ ├── collector.py # Data collection orchestrator
│ │ ├── models.py # Data structures
│ │ └── utils.py # Formatting utilities
│ └── config/
│ └── repositories.yml # Repository configuration
├── data/
│ ├── raw/ # Raw CSV data and metadata
│ └── processed/ # Analysis results
├── views/ # Generated markdown reports
│ ├── all.md # Comprehensive overview
│ ├── ai_ml.md # AI/ML repositories
│ ├── typescript.md # TypeScript projects
│ └── csharp.md # C# ASP.NET projects
├── scripts/
│ ├── generate_reports.py # Report generation
│ └── quick_test.py # Setup verification
├── integration/
│ └── tests/ # Integration tests
├── .github/workflows/ # GitHub Actions automation
```
## 🔧 Configuration & Automation
### Repository Management
Add new repositories to analyze by editing `api/config/repositories.yml`:
```yaml
repositories:
AI/ML:
- tensorflow/tensorflow
- pytorch/pytorch
TypeScript:
- microsoft/TypeScript
- nestjs/nest
```
### GitHub Actions
Automated data collection runs weekly via GitHub Actions:
- Fetches fresh repository statistics
- Generates updated reports
- Commits changes automatically
### API Rate Limits
- With token: 5,000 requests/hour
- Without token: 60 requests/hour
- Automatic rate limit handling and retry logic
## 📊 Repository Categories
### AI/ML Repositories (20 repos)
- TensorFlow, PyTorch, scikit-learn
- Hugging Face ecosystem (Transformers, TRL)
- AutoGPT, ComfyUI, Ollama
- LangChain, Dify, n8n
### TypeScript Projects (21 repos)
- Backend frameworks: NestJS, Fastify, AdonisJS
- ORMs: TypeORM, Prisma, Drizzle
- Utilities: Zod, tRPC, Winston
### C# ASP.NET Projects (13 repos)
- ASP.NET Core, Entity Framework Core
- Microservices: Orleans, Polly, Ocelot
- E-commerce: nopCommerce
### Developer Tools
- Educational resources and development utilities
## 📈 Report Features
### Number Formatting
- Stars displayed as readable format: 12.3k, 1.2M, 147k
- Consistent formatting across all metrics
### Interactive Links
- Repository names link to GitHub (open in new tab)
- Easy navigation between category reports
### Statistics Summary
- Total repositories, stars, and forks per category
- Most popular and most active repositories
- Collection timestamps and metadata
## 🧪 Testing
Run integration tests to verify functionality:
```bash
cd integration
python -m pytest tests/ -v
```
Tests cover:
- API client initialization
- Number formatting utilities
- Configuration loading
- Directory structure validation
## 🤝 Contributing
1. Fork the repository
2. Add new repositories to `api/config/repositories.yml`
3. Run tests to ensure functionality
4. Submit a Pull Request
## 📝 License
This project is licensed under the MIT License.
## 🙏 Acknowledgments
- GitHub API for providing comprehensive repository data
- ITJobswatch.co.uk for UK IT Market Data
- All the open-source projects analyzed in this study
- Automated using GitHub Actions