{"id":48633128,"url":"https://github.com/amelialouise/shiny-meteorites","last_synced_at":"2026-04-09T06:03:40.728Z","repository":{"id":332233952,"uuid":"1126984022","full_name":"amelialouise/shiny-meteorites","owner":"amelialouise","description":"Uses DuckDB through the R client API to create a database for a Shiny app focused on meteorite landings","archived":false,"fork":false,"pushed_at":"2026-01-13T00:01:37.000Z","size":6653,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-13T04:02:58.267Z","etag":null,"topics":["duckdb","nasa","open-datasets","r","shiny-apps"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amelialouise.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-02T23:58:43.000Z","updated_at":"2026-01-13T00:01:41.000Z","dependencies_parsed_at":"2026-01-13T04:03:16.995Z","dependency_job_id":null,"html_url":"https://github.com/amelialouise/shiny-meteorites","commit_stats":null,"previous_names":["amelialouise/shiny-meteorites"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/amelialouise/shiny-meteorites","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amelialouise%2Fshiny-meteorites","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amelialouise%2Fshiny-meteorites/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amelialouise%2Fshiny-meteorites/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amelialouise%2Fshiny-meteorites/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amelialouise","download_url":"https://codeload.github.com/amelialouise/shiny-meteorites/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amelialouise%2Fshiny-meteorites/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31588039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-09T05:33:47.836Z","status":"ssl_error","status_checked_at":"2026-04-09T05:32:26.579Z","response_time":112,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","nasa","open-datasets","r","shiny-apps"],"created_at":"2026-04-09T06:03:38.448Z","updated_at":"2026-04-09T06:03:40.716Z","avatar_url":"https://github.com/amelialouise.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Meteorite Explorer\n\nAn interactive R Shiny application for exploring meteorite landings data from NASA. Visualize the global distribution of meteorite discoveries and patterns over time from as early as 861 ([Nogata](https://www.lpi.usra.edu/meteor/metbull.php?code=16988))!\n\n![Shiny app screenshot](images/meteorites-explorer-screenshot.png)\n\n[Live on Posit Connect Cloud!](https://doodles-space-rocks.share.connect.posit.cloud)\n\n## App Features\n\n-   **Interactive Global Map**: View 32,050 meteorite landing locations worldwide with size-based markers and color-coded mass categories\n-   **Easy Filtering**: Filter by discovery era (including unknown!), meteorite name, and size \n-   **Quick Stats**: Get numbers on filtered data including total mass, median mass, and heaviest meteorite\n-   **Responsive Design**: Clean, modern interface optimized for data exploration\n\n## Data Sources\n\nThis application uses meteorite landing data from:\n\n\\- **NASA Meteorite Landings Dataset (json)**: \u003chttps://data.nasa.gov/dataset/meteorite-landings\u003e\n\n\\- Links to detailed information from the **Meteoritical Bulletin Database** maintained by the Lunar and Planetary Institute \u003chttps://meteoritical.org/\u003e\n\n## Tech Stack\n\n-   **R Shiny**: Web application framework\n-   **DuckDB**: High-performance analytical database for data processing\n-   **mapgl**: Interactive mapping\n-   **Plotly**: Interactive visualizations\n-   **DataTables**: Enhanced table display with search and sorting\n-   **bslib**: Modern UI components and theming\n\n## Dependencies\n\n-   R (version 4.0 or higher)\n-   Required R packages (see renv)\n\n## Project Structure\n\n```\nShiny implementation          # R Shiny \n│   └── app.R\n├── R/                        # DuckDB via R client\n├── download_meteorites.R     # Download data from NASA portal\n├── build_indexed_db.R        # Build indexed DuckDB database\n└── README.md\n```\n\n## 🔧 Getting Setup\n\n### 1. Install R package dependencies\n```r\ninstall.packages(c(\n  \"arrow\",     \"here\",      \"shiny\",     \"bslib\",     \"mapgl\",     \"dplyr\",     \"stringr\",   \"plotly\",    \"duckdb\",    \"DBI\",       \"DT\",        \"sf\"       ))\n```\n\nOr use *renv*\n\n```r\nrenv::restore()\n```\n\n### 2. Download Data\n```bash\ncd R\nRscript ./download_meteorites.R\nRscript ./build_indexed_db.R\n```\n\n### 3. Run Shiny App\n\n```r\nshiny::runApp(\"Shiny\")\n```\n\n\n## Posit Connect Cloud Architectural Changes\n### Mapping Engine Redesign\n\n**From**: Vector tiles + custom HTTP server + spatial complexity  \n**To**: Direct sf object rendering + circle layers + simplified coordinate handling  \n**Result**: ~500 lines of tile server code removed, zero external dependencies  \n\n### Database Connection Management \n\nImplemented auto-reconnecting database layer via,\n\n```{r}\nensure_connection \u003c- function() {\n  if (is.null(con) || !dbIsValid(con)) {\n    con \u003c\u003c- dbConnect(duckdb(), DB_PATH, read_only = TRUE) # global assignment hack tbh\n  }\n  return(con)\n}\n```\n\nThis handles:\n\n* User sessions timing out  \n* Platform connection recycling  \n* Network hiccups  \n* Database restarts  \n* Concurrent user access  \n\n## Multi-Stage ETL with DuckDB\n\nStage 1: NASA API Data Extraction (download_meteorites.R)\n\nExtract and clean raw meteorite data from NASA's Open Data Portal:\n\n- Direct API Integration: Queries NASA's JSON API using DuckDB's read_json_auto() function  \n- Complex JSON Parsing: Unnests nested arrays and maps 20+ data fields from NASA's schema  \n- Real-time Processing: Transforms data during download (field mapping, type conversion, URL generation)  \n- Data Validation: Filters out records with missing coordinates or mass data  \n- Efficient Storage: Saves to compressed Parquet format with ZSTD compression  \n\n**Output**: meteorites.parquet (~32k meteorites, optimized for analytics)\n\nStage 2: Database Optimization (build_spatial_db.R)\n\nTransform flat data into optimized DuckDB database with computed fields and indexes:\n\n- Pre-computed Categories: Generates size categories and historical eras during build process  \n- Data Quality Fixes: Corrects known data anomalies (e.g., meteorite #57150 year correction)  \n- Unit Conversions: Converts grams to kg/tons for easier visualization  \n- Standardized Coordinates: Ensures lat/lon are in decimal degrees for direct map rendering  \n\n**Output**: meteorites.duckdb \n\nStage 3: Production Application\n\nServe interactive queries against the optimized database:\n\n- Sub-second Queries: Direct coordinate-based filtering across 32k records  \n- Zero Computation: Pre-computed categories eliminate runtime calculations  \n- Direct Map Rendering: Lat/lon coordinates ready for immediate mapGL visualization  \n- Auto-healing Connections: Production-grade connection management for cloud deployment  \n\n**Pipeline Benefits**\n\n| Stage    | Processing Time | Output Size    | Key Optimization            |\n|----------|-----------------|----------------|------------------------------|\n| Download | 5-10 minutes    | ~0.8MB Parquet | Compressed parquet file      |\n| Build    | \u003c1 second       | ~3.5MB DuckDB  | Pre-computed aggregations    |\n| Runtime  | ~5 seconds      | Interactive    | Direct coordinate rendering  |\n\nTotal Build Time: ~5-10 minutes\n\nThe app renders meteorites directly as map markers using their lat/lon coordinates without requiring spatial query operations.\n\n## References\n\n- [Josiah Parry - {duckdb} or {dbplyr}](https://josiahparry.com/posts/2024-05-24-duckdb-and-r)  \n- [Dario Radečić - R Shiny and DuckDB: How to Speed Up Your Shiny Apps When Working With Large Datasets](https://www.appsilon.com/post/r-shiny-duckd)  \n- [Sara Altman - Creating a Shiny app that interacts with a database ](https://posit.co/blog/shiny-with-databases/)  \n- [Federico Tallis - Visualizing Millions of Buildings with duckdb](https://medium.com/@federico.tallis/visualizing-millions-of-buildings-with-duckdb-st-asmvt-a-streamlit-vs-shiny-comparison-5a4c924fe067)  \n\n## Acknowledgments\n\n- [Meteoritical Society](https://meteoritical.org/) - official meteorite database  \n- [NASA Open Data](https://www.nasa.gov/) - meteorite landings dataset  \n- [Shiny Assistant](https://gallery.shinyapps.io/assistant/#) - Shiny app building, tweaks, and troubleshooting!  \n- [DuckDB](https://duckdb.org/) - Python has more support, but the R API works well enough!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famelialouise%2Fshiny-meteorites","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famelialouise%2Fshiny-meteorites","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famelialouise%2Fshiny-meteorites/lists"}