{"id":29875774,"url":"https://github.com/lmcrean/rosalia","last_synced_at":"2025-07-31T02:43:36.515Z","repository":{"id":306995333,"uuid":"1027864392","full_name":"lmcrean/rosalia","owner":"lmcrean","description":"Rosalia is a Data Analyst; with deep knowledge of enterprise codebases; uses qualitative and quantitative data into market opportunities and threats","archived":false,"fork":false,"pushed_at":"2025-07-28T22:20:03.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-28T22:27:53.350Z","etag":null,"topics":["chatbot","terminal-based"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmcrean.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-28T16:42:41.000Z","updated_at":"2025-07-28T22:20:07.000Z","dependencies_parsed_at":"2025-07-28T22:27:54.716Z","dependency_job_id":"cd8e2eea-aa5c-43de-be21-7b63b86289f9","html_url":"https://github.com/lmcrean/rosalia","commit_stats":null,"previous_names":["lmcrean/research-enterprise-codebases"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/lmcrean/rosalia","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmcrean%2Frosalia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmcrean%2Frosalia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmcrean%2Frosalia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmcrean%2Frosalia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmcrean","download_url":"https://codeload.github.com/lmcrean/rosalia/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmcrean%2Frosalia/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267978384,"owners_count":24175254,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-31T02:00:08.723Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","terminal-based"],"created_at":"2025-07-31T02:43:35.731Z","updated_at":"2025-07-31T02:43:36.484Z","avatar_url":"https://github.com/lmcrean.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\n██████╗  ██████╗ ███████╗ █████╗ ██╗     ██╗ █████╗ \n██╔══██╗██╔═══██╗██╔════╝██╔══██╗██║     ██║██╔══██╗\n██████╔╝██║   ██║███████╗███████║██║     ██║███████║\n██╔══██╗██║   ██║╚════██║██╔══██║██║     ██║██╔══██║\n██║  ██║╚██████╔╝███████║██║  ██║███████╗██║██║  ██║\n╚═╝  ╚═╝ ╚═════╝ ╚══════╝╚═╝  ╚═╝╚══════╝╚═╝╚═╝  ╚═╝\n```\n\n\n# Rosalia -- Research Analyst for IT Markets 📊\n\n[![Python Version](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![GitHub API](https://img.shields.io/badge/GitHub-API%20v3-black.svg)](https://docs.github.com/en/rest)\n[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](integration/)\n\nA comprehensive analysis tool for examining GitHub repository statistics across different technology categories including AI/ML, TypeScript, C# ASP.NET, and developer tools.\n\n## Problem Statement:\n\n**Volatility**\nPublic opinions circulate both of the apparent collapse and growth of certain sectors in IT, such as Web Development and Artificial Intelligence -- with inconsistent sources of data. This can of course be ignored but skipping the need for data-driven research leaves freelancers and IT professionals relying soley on intuition, constantly looking over their shoulder in a volatile/ speculative market conditions.\n\n**Manual Researching data is time consuming.** \nSources such as ITJobsWatch tend to be viewed best individually. Grouping queries together does not always compare trends.\n\n**AI Deep Research shortcomings**\nCutting-edge Deep Research applications such as Gemini, Claud and ChatGPT, however impressive are observed to hallucinate statistics.\n\n## Solution:\n\n\n```mermaid\n---\nconfig:\n  theme: redux-dark\n---\nflowchart LR\n subgraph s1[\"Outcome\"]\n        Y[\"Correct Intuitions\"]\n        n1[\"Correct Data\"]\n        n2[\"Data-Driven Decisions\"]\n        n3[\"Start Over\"]\n        n4[\"Correct Data\"]\n        n5[\"Burnout\"]\n  end\n subgraph s1[\"Outcome\"]\n        X[\"Wrong intuitions\"]\n        W[\"Wrong Data\"]\n  end\n    A[\"Enterprise Repositories (Github)\"] --\u003e E[\"Market Data\"]\n    A2[\"News\"] --\u003e E\n    F[\"Office of National Statistics\"] --\u003e E\n    G[\"ITJobswatch\"] --\u003e E\n    E -- scraped by --\u003e B[\"Python\"]\n    B -- curated by --\u003e Z[\"Markdown Reports\"]\n    AI1[\"AI\"] -- consulted by --\u003e H[\"Human\"]\n    AI2[\"AI\"] -- consulted by --\u003e H2[\"Human\"]\n    E -- \"time-consuming, overwhelming\" --\u003e H3[\"Human\"]\n    E -- hallucinates --\u003e AI2\n    H3 --\u003e n4\n    H2 --\u003e W\n    Z -- Read by --\u003e H \u0026 AI1\n    W --\u003e X\n    H --\u003e n1\n    n1 --\u003e Y\n    X --\u003e n3\n    Y --\u003e n2\n    n4 --\u003e n5\n    n5 --\u003e n3\n    n4@{ shape: rect}\n    n5@{ shape: rect}\n    A@{ shape: cyl}\n    E@{ shape: rect}\n    A2@{ shape: cyl}\n    F@{ shape: cyl}\n    G@{ shape: cyl}\n    style n2 fill:#C8E6C9,color:#000000\n    style n3 fill:#FFCDD2,color:#000000\n\n```\n***Diagram (above):*** Using Python Scraping and Markdown Reports in research workflow ensures accurate data while saving burnout of manual research.\n\n\nRosalia scrapes data from reputable sources and collates them into a Google Docs.\n\nThis can be viewed by a human or synchronised with an AI Project.\n\n## 🎯 Overview\n\nThis project collects and analyzes \n\n### Data Collection\n\nStatistics from 55+ popular open-source enterprise codebases, providing insights into:\n- Star counts and popularity trends (formatted as 12.3k, 1.2M for readability)\n- Fork and contributor statistics\n- Pull request activity\n- Technology stack distributions\n- Repository health metrics\n\nStatistics for emerging and collapsing markets in Technology Sector\n- Currently limited to UK-specific, with focus on London\n- ITJobswatch.co.uk\n- 10-15 year trends\n- Keywords: AI, ML, Web Development, Software Engineering, Data Science, Data Analytics, QA Testing\n- Languages: Python, C#, Typescript, \n- Frameworks/ Technology: ASP.NET, Google Cloud, AWS, \n\nStatistics from Office of National Statistics\n- understanding the Technology Sector and UK Economy on 10-15 year trend\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Python 3.8 or higher\n- GitHub personal access token (for API access)\n\n### Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/yourusername/research-enterprise-codebases.git\ncd research-enterprise-codebases\n```\n\n2. Install dependencies:\n```bash\npip install -r requirements.txt\n```\n\n3. Create a `.env` file with your GitHub token:\n```bash\necho \"GITHUB_TOKEN=your_github_token_here\" \u003e .env\n```\n\n### Usage\n\n#### 1. Verify Setup\nTest your configuration:\n```bash\npython scripts/quick_test.py\n```\n\n#### 2. Collect Data\nRun the data collection script to fetch fresh statistics:\n```bash\npython api/github/collector.py\n```\n\nThis will:\n- Fetch repository data from GitHub API\n- Handle rate limiting automatically\n- Save results to `data/raw/github_repository_stats.csv`\n- Create metadata file with collection timestamp\n\n#### 3. Generate Reports\nCreate markdown reports with formatted data:\n```bash\npython scripts/generate_reports.py\n```\n\n#### 4. View Results\nOpen the generated markdown reports in `views/`:\n- [Overview Report](views/all.md) - Comprehensive summary\n- [AI/ML Repositories](views/ai_ml.md) - AI/ML focused analysis\n- [TypeScript Projects](views/typescript.md) - TypeScript ecosystem\n- [C# ASP.NET Projects](views/csharp.md) - C# ASP.NET analysis\n\n## 📁 Project Structure\n\n```\nresearch-enterprise-codebases/\n├── api/\n│   ├── github/                          # GitHub API client modules\n│   │   ├── client.py                    # API client with rate limiting\n│   │   ├── collector.py                 # Data collection orchestrator\n│   │   ├── models.py                    # Data structures\n│   │   └── utils.py                     # Formatting utilities\n│   └── config/\n│       └── repositories.yml             # Repository configuration\n├── data/\n│   ├── raw/                            # Raw CSV data and metadata\n│   └── processed/                       # Analysis results\n├── views/                              # Generated markdown reports\n│   ├── all.md                          # Comprehensive overview\n│   ├── ai_ml.md                        # AI/ML repositories\n│   ├── typescript.md                   # TypeScript projects\n│   └── csharp.md                       # C# ASP.NET projects\n├── scripts/\n│   ├── generate_reports.py             # Report generation\n│   └── quick_test.py                   # Setup verification\n├── integration/\n│   └── tests/                          # Integration tests\n├── .github/workflows/                  # GitHub Actions automation\n```\n\n## 🔧 Configuration \u0026 Automation\n\n### Repository Management\nAdd new repositories to analyze by editing `api/config/repositories.yml`:\n\n```yaml\nrepositories:\n  AI/ML:\n    - tensorflow/tensorflow\n    - pytorch/pytorch\n  TypeScript:\n    - microsoft/TypeScript\n    - nestjs/nest\n```\n\n### GitHub Actions\nAutomated data collection runs weekly via GitHub Actions:\n- Fetches fresh repository statistics\n- Generates updated reports\n- Commits changes automatically\n\n### API Rate Limits\n- With token: 5,000 requests/hour\n- Without token: 60 requests/hour\n- Automatic rate limit handling and retry logic\n\n## 📊 Repository Categories\n\n### AI/ML Repositories (20 repos)\n- TensorFlow, PyTorch, scikit-learn\n- Hugging Face ecosystem (Transformers, TRL)\n- AutoGPT, ComfyUI, Ollama\n- LangChain, Dify, n8n\n\n### TypeScript Projects (21 repos)\n- Backend frameworks: NestJS, Fastify, AdonisJS\n- ORMs: TypeORM, Prisma, Drizzle\n- Utilities: Zod, tRPC, Winston\n\n### C# ASP.NET Projects (13 repos)\n- ASP.NET Core, Entity Framework Core\n- Microservices: Orleans, Polly, Ocelot\n- E-commerce: nopCommerce\n\n### Developer Tools\n- Educational resources and development utilities\n\n## 📈 Report Features\n\n### Number Formatting\n- Stars displayed as readable format: 12.3k, 1.2M, 147k\n- Consistent formatting across all metrics\n\n### Interactive Links\n- Repository names link to GitHub (open in new tab)\n- Easy navigation between category reports\n\n### Statistics Summary\n- Total repositories, stars, and forks per category\n- Most popular and most active repositories\n- Collection timestamps and metadata\n\n## 🧪 Testing\n\nRun integration tests to verify functionality:\n```bash\ncd integration\npython -m pytest tests/ -v\n```\n\nTests cover:\n- API client initialization\n- Number formatting utilities\n- Configuration loading\n- Directory structure validation\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Add new repositories to `api/config/repositories.yml`\n3. Run tests to ensure functionality\n4. Submit a Pull Request\n\n## 📝 License\n\nThis project is licensed under the MIT License.\n\n## 🙏 Acknowledgments\n\n- GitHub API for providing comprehensive repository data\n- ITJobswatch.co.uk for UK IT Market Data\n- All the open-source projects analyzed in this study\n- Automated using GitHub Actions\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmcrean%2Frosalia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmcrean%2Frosalia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmcrean%2Frosalia/lists"}