{"id":31936220,"url":"https://github.com/flowcore-io/speedlocal-streamlit","last_synced_at":"2025-10-14T07:25:18.915Z","repository":{"id":316244384,"uuid":"1062574785","full_name":"flowcore-io/speedlocal-streamlit","owner":"flowcore-io","description":"Speed Local Streamlit dashboard for scientific data visualization and analysis","archived":false,"fork":false,"pushed_at":"2025-09-30T15:26:28.000Z","size":1006,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-30T17:26:50.870Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flowcore-io.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-09-23T12:45:23.000Z","updated_at":"2025-09-30T15:26:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"71b73497-4447-4615-9774-a3579c3dd182","html_url":"https://github.com/flowcore-io/speedlocal-streamlit","commit_stats":null,"previous_names":["flowcore-io/speedlocal-streamlit"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/flowcore-io/speedlocal-streamlit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowcore-io%2Fspeedlocal-streamlit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowcore-io%2Fspeedlocal-streamlit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowcore-io%2Fspeedlocal-streamlit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowcore-io%2Fspeedlocal-streamlit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flowcore-io","download_url":"https://codeload.github.com/flowcore-io/speedlocal-streamlit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowcore-io%2Fspeedlocal-streamlit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018215,"owners_count":26086303,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T07:25:17.628Z","updated_at":"2025-10-14T07:25:18.909Z","avatar_url":"https://github.com/flowcore-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Speed Local Streamlit Analytics\n\n🚀 **A comprehensive multi-page Streamlit application for analyzing TIMES energy system data with advanced visualization capabilities including geospatial mapping, Sankey diagrams, and interactive database tools.**\n\nThis repository contains a sophisticated analytics dashboard for visualizing and analyzing energy system data from the Speed Local project, which focuses on Nordic green transition research through trans-Nordic collaboration.\n\n## 🌟 Features\n\n### 📊 TIMES Data Explorer\n- **Interactive scenario analysis** with sector-based energy and emissions visualization\n- **Dynamic filtering** by scenario, year, and sector\n- **Comparative analysis** between different energy scenarios\n- **Stacked bar charts** for energy inputs by sector\n- **Line charts** for emissions trends over time\n\n### 🌍 Energy Flow Maps  \n- **Interactive geospatial visualization** of energy flows between regions\n- **Folium-based maps** with zoom, pan, and interactive features\n- **Automated geocoding** for region coordinates\n- **Flow magnitude visualization** with proportional line thickness\n- **Color-coded flows**: Blue (exports), Red (imports), Green (bidirectional)\n- **Animated pathways** showing energy flow directions\n\n### 📊 Sankey Diagrams\n- **Comprehensive energy flow analysis** with Plotly Sankey charts\n- **Multi-type flow visualization**: Production, Consumption, Transmission, Import/Export\n- **Node categorization** with color-coding for technologies, demands, and regions\n- **Interactive filtering** with flow threshold controls\n- **Statistical analysis** with flow breakdowns and top flows ranking\n\n### 🛠️ Database Tools\n- **Schema Explorer**: Browse database structure with table and column information\n- **SQL Query Interface**: Execute custom queries with safety controls and templates\n- **Automated Data Analysis**: Quick metrics and statistical summaries\n- **Quick Chart Generator**: Build custom visualizations from SQL results\n- **Data Export**: Download query results and analysis data\n\n## 📁 Repository Structure\n\n```\nspeedlocal-streamlit/\n├── streamlit-app/                        # Streamlit application code\n│   ├── app/\n│   │   ├── main.py                       # Main dashboard and landing page\n│   │   ├── pages/\n│   │   │   ├── 1_📊_TIMES_Data_Explorer.py   # Advanced TIMES data analysis\n│   │   │   ├── 2_🌍_Energy_Flow_Maps.py      # Geospatial energy flow visualization  \n│   │   │   ├── 3_📊_Sankey_Diagrams.py       # Energy flow Sankey charts\n│   │   │   └── 4_🛠️_Database_Tools.py        # Database management utilities\n│   │   └── utils/\n│   │       ├── database.py               # Database connection and utilities\n│   │       └── geo_settings.py           # Geographic constants and settings\n│   ├── assets/                           # Static assets\n│   ├── tests/                            # Application tests\n│   ├── requirements.txt                  # Python dependencies\n│   └── Dockerfile                        # Container image definition\n├── helm-chart/                           # Kubernetes deployment manifests\n│   ├── values.yaml                       # Base Helm values\n│   └── configuration/                    # Environment-specific configs (ArgoCD compatible)\n│       └── production.yaml\n└── README.md                             # This file\n```\n\n## 🎯 Project Overview\n\nThe Speed Local project provides a platform where scientists can:\n- Upload GAMS reports and scientific datasets\n- Process data into reusable flat structures\n- Create shareable DuckDB files by combining datasets\n- Collaborate on Nordic green transition research\n\nThis multi-page Streamlit application provides comprehensive visualization and analysis capabilities for TIMES energy system data, including:\n- **Advanced data exploration** with interactive filtering and comparative analysis\n- **Geospatial visualization** of energy flows between regions\n- **Sankey diagrams** for comprehensive energy flow analysis\n- **Database management tools** for direct data access and custom queries\n\n## 🚀 Quick Start\n\n### Local Development\n\n1. **Clone the repository**:\n   ```bash\n   git clone https://github.com/flowcore-io/speedlocal-streamlit.git\n   cd speedlocal-streamlit\n   ```\n\n2. **Set up Python environment**:\n   ```bash\n   cd streamlit-app\n   python -m venv venv\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\n   pip install -r requirements.txt\n   ```\n\n3. **Run the application**:\n   ```bash\n   streamlit run app/main.py\n   ```\n\n4. **Open in browser**:\n   Navigate to `http://localhost:8501`\n\n### Docker Development\n\n1. **Build the Docker image**:\n   ```bash\n   cd streamlit-app\n   docker build -t speedlocal-streamlit .\n   ```\n\n2. **Run the container**:\n   ```bash\n   docker run -p 8501:8501 speedlocal-streamlit\n   ```\n\n## 🚢 Deployment\n\nThis application is designed to deploy on Flowcore's Kubernetes infrastructure using the `flowcore-microservices` Helm chart.\n\n### Building and Pushing to ECR\n\n```bash\n# Set your AWS configuration\nAWS_ACCOUNT_ID=305363105399\nAWS_REGION=eu-west-1\nECR_REGISTRY=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com\nIMAGE_REPO=speedlocal/streamlit-app\nIMAGE_TAG=v1.0.0\n\n# Login to ECR\naws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}\n\n# Create repository if it doesn't exist\naws ecr describe-repositories --repository-names ${IMAGE_REPO} || aws ecr create-repository --repository-name ${IMAGE_REPO}\n\n# Build and push\ncd streamlit-app\ndocker build -t ${ECR_REGISTRY}/${IMAGE_REPO}:${IMAGE_TAG} .\ndocker push ${ECR_REGISTRY}/${IMAGE_REPO}:${IMAGE_TAG}\n```\n\n### Kubernetes Deployment\n\nThe Helm configuration in `helm-chart/` is designed to be used with the `public-customer-sites-manifests` repository for ArgoCD deployment:\n\n1. Copy the Helm values to the appropriate environment configuration\n2. Update the image tag and hostname as needed\n3. Commit and push to trigger ArgoCD deployment\n\n### Environment URLs\n\n- **Development**: `https://speedlocal-streamlit-dev.flowcore.app`\n- **Production**: `https://speedlocal-streamlit.flowcore.app`\n\n## 🛠 Technical Details\n\n### Features\n\n- **Multi-page Application**: Four specialized analysis pages for different use cases\n- **Advanced Data Visualization**: Interactive charts, maps, and Sankey diagrams\n- **Database Management**: Complete database exploration and query capabilities\n- **Geospatial Analysis**: Interactive maps with energy flow visualization\n- **Export Capabilities**: Download data and visualizations for further analysis\n- **Responsive Design**: Works on desktop and mobile devices\n- **Real-time Analysis**: Dynamic filtering and comparative scenario analysis\n\n### Technology Stack\n\n- **Frontend**: Streamlit 1.32+\n- **Data Processing**: Pandas, NumPy\n- **Visualization**: Plotly, Folium for interactive charts and maps\n- **Database**: DuckDB for efficient analytical queries\n- **Geospatial**: GeoPy for geocoding and coordinate management\n- **Container**: Python 3.11 slim Docker image\n- **Deployment**: Kubernetes with NGINX Ingress\n- **Infrastructure**: Flowcore Kubernetes platform\n\n## 💾 Database Connection\n\nThe application supports two database connection methods:\n\n### Local DuckDB Files\n- Place your `.duckdb` file in an accessible location\n- Use the \"Local File\" connection option\n- Enter the full path to your database file\n\n### Azure Blob Storage\n- Generate a SAS URL for your DuckDB file in Azure Storage\n- Use the \"Azure URL\" connection option\n- Enable local caching for improved performance\n- URLs are automatically validated for expiry\n\n## 📋 Data Requirements\n\nThe application expects a DuckDB database with a `timesreport_facts` table containing:\n\n### Required Columns:\n- `scen`: Scenario identifier\n- `year`: Year of the data point\n- `reg`: Region identifier  \n- `prc`: Process/technology identifier\n- `com`: Commodity/fuel type\n- `attr`: Attribute type (f_in, f_out, etc.)\n- `topic`: Data topic (energy, emissions, etc.)\n- `value`: Numerical value\n- `regfrom`, `regto`: For transmission flows\n\n### Data Types Supported:\n- **Energy flows**: Production, consumption, imports, exports, transmission\n- **Emissions data**: By sector and technology\n- **Regional data**: Multi-region energy systems\n- **Temporal data**: Multi-year scenario analysis\n\n## 🎛️ Usage Guide\n\n### 1. Connect to Database\n- Use the sidebar connection panel on any page\n- Select connection type (Local File or Azure URL)\n- Test connection to verify data access\n\n### 2. TIMES Data Explorer\n- Select scenario, year, and sector filters\n- Generate energy input visualizations by sector\n- Compare emissions trends across scenarios\n- Export data for further analysis\n\n### 3. Energy Flow Maps  \n- Configure scenario, year, and fuel filters\n- Generate interactive maps showing regional energy flows\n- Explore flow statistics and regional connections\n- Use geographic markers and flow animations\n\n### 4. Sankey Diagrams\n- Set scenario, year, and optional region filters\n- Adjust flow threshold to focus on major pathways\n- Analyze flow types and energy transformation chains\n- Export flow summaries and detailed data\n\n### 5. Database Tools\n- **Schema Explorer**: Navigate database structure\n- **SQL Query**: Execute custom analysis queries\n- **Data Analysis**: Get automated insights\n- **Quick Charts**: Build custom visualizations\n\n### Health Checks\n\nStreamlit automatically exposes a health check endpoint at `/_stcore/health` which is used by Kubernetes probes.\n\n## 🔗 Integration\n\nThis dashboard integrates with:\n- **Speed Local Admin** (`speedlocal.flowcore.app`) - Main data management platform\n- **Flowcore Platform** - Event-driven data processing\n- **Azure Blob Storage** - File storage and retrieval\n- **DuckDB Files** - Public dataset access\n\n## 🌱 About Speed Local\n\nSpeed Local is part of the broader initiative to accelerate Nordic green transition through trans-Nordic collaboration. The project enables scientists to:\n\n- Share high-quality scientific datasets\n- Collaborate on research across Nordic countries\n- Provide transparent access to green transition data\n- Support evidence-based policy making\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## 🤝 Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## 📞 Support\n\nFor support and questions, please contact the Flowcore team.\nSpeed Local Streamlit dashboard for scientific data visualization and analysis\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflowcore-io%2Fspeedlocal-streamlit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflowcore-io%2Fspeedlocal-streamlit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflowcore-io%2Fspeedlocal-streamlit/lists"}