{"id":30687615,"url":"https://github.com/andreabozzo/osservatorio","last_synced_at":"2025-09-02T00:04:32.427Z","repository":{"id":304966268,"uuid":"1020212256","full_name":"AndreaBozzo/Osservatorio","owner":"AndreaBozzo","description":"Osservatorio - Open Data Processing Platform","archived":false,"fork":false,"pushed_at":"2025-08-26T18:34:34.000Z","size":44084,"stargazers_count":3,"open_issues_count":68,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-26T19:08:49.640Z","etag":null,"topics":["api-rest","data","data-analysis","data-visualization","database","datamodelling","docker","duckdb","etl","fastapi","jwt-authentication","open-source","pipeline","postgresql","python","react","sqlite"],"latest_commit_sha":null,"homepage":"https://andreabozzo.github.io/Osservatorio/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AndreaBozzo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-15T14:09:38.000Z","updated_at":"2025-08-26T14:04:21.000Z","dependencies_parsed_at":"2025-07-17T17:27:14.810Z","dependency_job_id":"bed75f15-fa80-4fa2-8e65-a0a5e3e40a31","html_url":"https://github.com/AndreaBozzo/Osservatorio","commit_stats":null,"previous_names":["andreabozzo/osservatorio"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AndreaBozzo/Osservatorio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaBozzo%2FOsservatorio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaBozzo%2FOsservatorio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaBozzo%2FOsservatorio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaBozzo%2FOsservatorio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AndreaBozzo","download_url":"https://codeload.github.com/AndreaBozzo/Osservatorio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaBozzo%2FOsservatorio/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273208777,"owners_count":25064204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","data","data-analysis","data-visualization","database","datamodelling","docker","duckdb","etl","fastapi","jwt-authentication","open-source","pipeline","postgresql","python","react","sqlite"],"created_at":"2025-09-02T00:03:34.188Z","updated_at":"2025-09-02T00:04:32.417Z","avatar_url":"https://github.com/AndreaBozzo.png","language":"Python","readme":"# 🇮🇹 Osservatorio - Italian Statistical Data Platform\n\n\u003e **Modern enterprise platform for Italian statistical data processing with production-grade FastAPI backend, comprehensive security, and Docker deployment.**\n\n[![CI](https://img.shields.io/badge/CI-Basic%20Tests-blue.svg)](.github/workflows/ci.yml)\n[![Docker](https://img.shields.io/badge/Docker-Multi--stage-blue.svg)](Dockerfile)\n[![Python](https://img.shields.io/badge/Python-3.9%20|%203.10%20|%203.11%20|%203.12-blue.svg)](pyproject.toml)\n[![FastAPI](https://img.shields.io/badge/FastAPI-Latest-green.svg)](src/api/fastapi_app.py)\n[![Security](https://img.shields.io/badge/Security-Bandit%20|%20OWASP-green.svg)](.pre-commit-config.yaml)\n[![Status](https://img.shields.io/badge/Status-Issue%20%2363%20Complete-brightgreen.svg)](docs/project/PROJECT_STATE.md)\n[![Pipeline](https://img.shields.io/badge/Pipeline-Unified%20Ingestion-blue.svg)](docs/SYSTEM_USAGE_GUIDE.md)\n\n## 🎯 What is Osservatorio?\n\n**Osservatorio** is developing a modern platform for Italian statistical data access. The core REST API and database layer are functional, with ISTAT data integration and Kubernetes infrastructure in development.\n\n### ✨ Current Status\n- **🚀 Core API**: FastAPI backend functional with JWT authentication\n- **⚡ Database**: DuckDB analytics + SQLite metadata hybrid working\n- **🛡️ Security**: JWT auth, rate limiting, security middleware implemented\n- **🐳 Docker**: Multi-stage builds ready (image deployment untested)\n- **☸️ Kubernetes**: Infrastructure manifests complete (cluster deployment untested)\n- **⚠️ BI Integration**: PowerBI/Tableau structure present but needs rework\n\n## 🚀 Quick Start\n\n### Option 1: Docker (Recommended)\n```bash\n# Clone and run with Docker Compose\ngit clone https://github.com/AndreaBozzo/Osservatorio.git\ncd Osservatorio\ndocker-compose up -d\n\n# Access services:\n# - FastAPI: http://localhost:8000\n# - Dashboard: http://localhost:8501\n# - API Docs: http://localhost:8000/docs\n```\n\n### Option 2: Local Development\n```bash\n# 1. Clone and setup\ngit clone https://github.com/AndreaBozzo/Osservatorio.git\ncd Osservatorio\npython -m venv venv \u0026\u0026 source venv/bin/activate  # or venv\\Scripts\\activate on Windows\n\n# 2. Install with modern dependency management\npip install -e .[dev]                           # Development dependencies\n# or pip install -e .[dev,performance,security] # All optional dependencies\n\n# 3. Run services\nmake test-fast                                   # Quick test suite\npython -m uvicorn src.api.fastapi_app:app --reload  # FastAPI server\nstreamlit run dashboard/app.py                  # Dashboard\n```\n\n### Quick Health Check\n```bash\n# Test FastAPI endpoint\ncurl http://localhost:8000/health\n\n# Or use Python client\npython -c \"from src.api.production_istat_client import ProductionIstatClient; client=ProductionIstatClient(); print('✅ System ready:', client.get_status()['status'])\"\n```\n\n## 🏗️ Modern Architecture\n\nOsservatorio uses a **microservices-ready architecture** with production patterns:\n\n```\n🌐 FastAPI REST API (OpenAPI/Swagger docs)\n    ↓ (Security Middleware + JWT Auth)\n🔄 Business Logic Layer (Circuit Breakers + Rate Limiting)\n    ↓ (Async Processing)\n📊 Data Layer: DuckDB Analytics + SQLite Metadata + Redis Cache\n    ↓ (Smart Routing + Fallback)\n🇮🇹 ISTAT SDMX API (509+ datasets)\n```\n\n### Core Components\n- **FastAPI Backend**: Modern async web framework with automatic OpenAPI documentation\n- **Security Layer**: JWT authentication, OWASP-compliant middleware, rate limiting\n- **Data Processing**: DuckDB for analytics, SQLite for metadata, Redis for caching\n- **DevOps Ready**: Docker containers, GitHub Actions CI/CD, comprehensive testing\n- **Monitoring**: Health checks, performance metrics, security scanning\n\n## 📊 What You Can Do\n\n### For Data Analysts\n```python\n# REST API access\nimport requests\nresponse = requests.get(\"http://localhost:8000/api/v1/dataflows?limit=10\")\ndataflows = response.json()\n\n# Or use Python client\nfrom src.api.production_istat_client import ProductionIstatClient\nclient = ProductionIstatClient(enable_cache_fallback=True)\ndataset = client.fetch_dataset(\"DCIS_POPRES1\")  # Population data\n```\n\n### For API Integration\n- **OpenAPI/Swagger**: Interactive API docs at `/docs` endpoint\n- **RESTful Design**: Standard HTTP methods with JSON responses\n- **Authentication**: JWT token-based security with API key support\n- **Rate Limiting**: Fair usage policies with intelligent throttling\n\n### For DevOps Teams\n- **Container Ready**: Multi-stage Docker builds for development and production\n- **Basic CI**: GitHub Actions with unit testing and code quality checks\n- **Security First**: Bandit security scanning, pre-commit hooks\n- **Monitoring**: Health checks, performance metrics, structured logging\n\n## 🛡️ Production Features\n\n### Modern Development\n- **Python 3.9-3.12**: Multi-version compatibility with matrix testing\n- **FastAPI Framework**: High-performance async web framework\n- **Docker Deployment**: Multi-stage builds with development and production targets\n- **Basic CI**: Automated unit testing and code quality checks\n\n### Security \u0026 Compliance\n- **JWT Authentication**: Secure token-based authentication\n- **OWASP Middleware**: Security headers and best practices\n- **Bandit Scanning**: Automated security vulnerability detection\n- **Dependency Monitoring**: Automated vulnerability checking with Safety\n\n### Performance \u0026 Reliability\n- **Async Processing**: Non-blocking operations for better throughput\n- **Circuit Breakers**: Automatic fault detection and recovery\n- **Rate Limiting**: Configurable request throttling\n- **Health Monitoring**: Comprehensive system status and metrics\n- **Redis Caching**: High-performance distributed caching\n\n## 📚 Documentation\n\n- **[Architecture Overview](docs/core/ARCHITECTURE.md)** - System design and patterns\n- **[Production Client API](docs/api/PRODUCTION_ISTAT_CLIENT.md)** - Complete API reference\n- **[Project Status](docs/project/PROJECT_STATE.md)** - Current development state\n- **[Developer Guide](docs/project/CLAUDE.md)** - Commands and development context\n- **[DevOps Guide](docs/core/DEPLOYMENT.md)** - Deployment with Docker Compose\n\n## 🤝 Contributing\n\nWe welcome contributions! Here's how to get started:\n\n1. **Check the [Issues](https://github.com/AndreaBozzo/Osservatorio/issues)** for tasks that need help\n2. **Read our [Contributing Guide](docs/guides/CONTRIBUTING.md)** for development standards\n3. **Join [Discussions](https://github.com/AndreaBozzo/Osservatorio/discussions)** for questions and ideas\n4. **Test First**: Run `pytest tests/ -v` to ensure everything works\n\n### Development Standards\n- **Modern Tooling**: ruff, black, mypy, and pre-commit hooks\n- **Comprehensive Testing**: Unit, integration, and performance tests\n- **Security First**: Bandit scanning, dependency checking, OWASP compliance\n- **Basic CI Integration**: GitHub Actions with unit testing and code quality\n\n## 🎯 Use Cases\n\n- **🏛️ Government Agencies**: Access official Italian statistics for policy making\n- **🏢 Businesses**: Market research and economic analysis with reliable data\n- **🎓 Researchers**: Academic research with comprehensive Italian datasets\n- **📊 Analysts**: BI dashboards and reporting with PowerBI integration\n- **🔍 Journalists**: Data journalism with verified official sources\n\n## 🏆 Development Metrics\n\n**Development Status Achieved:**\n\n- ✅ **FastAPI Backend**: Modern async API with OpenAPI docs functional\n- ✅ **Database Layer**: DuckDB + SQLite hybrid working\n- ✅ **Authentication**: JWT auth, rate limiting, security middleware\n- ✅ **Code Quality**: ruff, black, pre-commit hooks, security scanning\n- ✅ **Docker Build**: Multi-stage Dockerfile (container deployment untested)\n- ⚠️ **Kubernetes**: Complete manifests created (cluster deployment untested)\n- ⚠️ **CI Pipeline**: Basic unit tests (integration/deployment testing limited)\n\n*Core functionality working, infrastructure and BI integrations need validation.*\n\n## 📞 Support\n\n- **🐛 Issues**: [GitHub Issues](https://github.com/AndreaBozzo/Osservatorio/issues) for bug reports\n- **💬 Discussions**: [GitHub Discussions](https://github.com/AndreaBozzo/Osservatorio/discussions) for questions\n- **📚 Documentation**: [docs/](docs/) folder for comprehensive guides\n- **🎯 Live Dashboard**: [osservatorio-dashboard.streamlit.app](https://osservatorio-dashboard.streamlit.app/)\n\n## 📄 License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n---\n\n**🎯 Ready to explore Italian statistical data?**\n**👉 Run `docker-compose up -d` or `make install \u0026\u0026 make test-fast` to get started**\n\n*Built with modern Python, FastAPI, and Docker for the Italian data community*\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreabozzo%2Fosservatorio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreabozzo%2Fosservatorio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreabozzo%2Fosservatorio/lists"}