{"id":20845908,"url":"https://github.com/urbanos-public/smartcitiesdata","last_synced_at":"2025-04-06T17:13:56.005Z","repository":{"id":37088193,"uuid":"192407200","full_name":"UrbanOS-Public/smartcitiesdata","owner":"UrbanOS-Public","description":"The core micro services of UrbanOS as an umbrella project with component documentation","archived":false,"fork":false,"pushed_at":"2025-02-17T15:07:07.000Z","size":14587,"stargazers_count":63,"open_issues_count":29,"forks_count":11,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-30T07:32:03.504Z","etag":null,"topics":["data-analytics","data-processing","data-visualization","elixir","elixir-phoenix"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UrbanOS-Public.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-17T19:35:09.000Z","updated_at":"2025-02-14T16:40:46.000Z","dependencies_parsed_at":"2023-02-17T21:46:23.154Z","dependency_job_id":"5c4210b2-d503-4f83-b48e-8d9a3b4581cc","html_url":"https://github.com/UrbanOS-Public/smartcitiesdata","commit_stats":null,"previous_names":[],"tags_count":490,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UrbanOS-Public%2Fsmartcitiesdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UrbanOS-Public%2Fsmartcitiesdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UrbanOS-Public%2Fsmartcitiesdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UrbanOS-Public%2Fsmartcitiesdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UrbanOS-Public","download_url":"https://codeload.github.com/UrbanOS-Public/smartcitiesdata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247517921,"owners_count":20951719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analytics","data-processing","data-visualization","elixir","elixir-phoenix"],"created_at":"2024-11-18T02:14:25.050Z","updated_at":"2025-04-06T17:13:55.980Z","avatar_url":"https://github.com/UrbanOS-Public.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Smart Cities Data Platform\n\n# Project Description\nThe platform is a combination of Elixir micro services custom built to ingest, normalize, transform,\npersist, and stream data from numerous sources, orchestrated via Kubernetes in any cloud provider or\non-prem Kubernetes deployment. The loosely coupled services pass data across the pipeline via Kafka\nmessage queues and persist data to any hyper-scalable object store providing the S3 standard. They\ncoordinate and communicate via a single event bus, also running on top of Kafka. The distributed data\nfiles are persisted and retrieved via SQL queries processed by the PrestoDB engine.\nFinally, user access, discovery, and analysis is facilitated by a ReactJS web application user interface,\na RESTful API, or a web socket API for streaming data feeds.\n\n![scdp architecture diagram](./scdp_arch.png?raw=true \"scdp architecture\")\n\n## Microservices\n| Application       | Short Description | Build Status |\n| ----------------- | ----------------- | ------------ |\n| [Andi](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/andi/README.md)                             | Admin Interface for creating/editing datasets to be ingested  | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/andi.yml/badge.svg)  |\n| [Discovery API](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/discovery_api/README.md)           | API to search for and query datasets                          | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/discovery_api.yml/badge.svg) |\n| [Discovery Streams](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/discovery_streams/README.md)   | Websocket connection to listen to streaming data              | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/discovery_streams.yml/badge.svg)  |\n| [Estuary](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/estuary/README.md)                       | Microservice to persist event stream events                   | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/estuary.yml/badge.svg)  |\n| [Forklift](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/forklift/README.md)                     | Microservice for saving data to Presto DB                     | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/forklift.yml/badge.svg)  |\n| [Reaper](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/reaper/README.md)                         | Microservice to retrieve data                                 | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/reaper.yml/badge.svg)  |\n| [Valkyrie](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/valkyrie/README.md)                     | Microservice to validate data structure during ingestion      | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/valkyrie.yml/badge.svg)  |\n| [Alchemist](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/apps/alchemist/README.md)                   | Microservice to alter data from its original format           | ![](https://github.com/UrbanOS-Public/smartcitiesdata/actions/workflows/alchemist.yml/badge.svg)  |\n\n# Prerequisites\n### General Prerequisites\n* [Elixir](https://elixir-lang.org/) - The primary language that all of the microservices are written in\n* [Docker](https://www.docker.com/) - All microservices are built as docker images\n* [Apache Kafka](https://kafka.apache.org/) -  Communication mechanism between microservices\n* [Redis](https://redis.io/) - General purpose storage and caching\n* [Elasticsearch](https://www.elastic.co/) - Used by Discovery API for search\n* [PostgreSQL](https://www.postgresql.org/) - General purpoase storage\n* [Presto](https://prestodb.io/) - Big Data storage of ingested data\n* [Vault](https://www.vaultproject.io/) - Secure storage of secrets\n\n### Development Enviornment Setup\n\n[Setup guide available on our wiki](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Setup)\n\n# Usage\nThe microservices written in Elixir use [Mix](https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html) as the build tool.\n## Building\nEach microservice under the [apps/](https://github.com/UrbanOS-Public/smartcitiesdata/tree/master/apps) directory has a `Dockerfile` that can be used to build that microservice individually by running the following command:\n```\ndocker build .\n```\n\nAdditional app specific build steps will be in the relative readme at `apps/{app}/readme.md`.\n\n## Testing\n* Unit tests can be executed from the root of this repository or a specific application under the [apps/](https://github.com/UrbanOS-Public/smartcitiesdata/tree/master/apps) directory\n```\nmix test\n```\n* Integration tests can be executed from the root of this repository or a specific application under the [apps/](https://github.com/UrbanOS-Public/smartcitiesdata/tree/master/apps) directory\n```\nmix test.integration\n```\n* End to End (E2E) Tests can be executed from the root of this repository.\n```\nmix test.e2e\n```\n## Execution\n[How to run and use the code](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Run)\n\n# Additional Notes\n* [What is the project and how it works](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/The-What)\n* [What all those application names mean](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Names)\n* [Additional learning resources](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Resources)\n* [A glossary of terms and technologies](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Glossary)\n* [Starting All of the Microservices](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Run)\n# Version History and Retention\nEach microservice is released independently and can be found here in the [Releases](https://github.com/UrbanOS-Public/smartcitiesdata/releases) section.  All releases will be kept indefinitely.\n\nVersioning conforms to the standard versioning pattern of \u003cmajor\u003e.\u003cminor\u003e.\u003cpatch\u003e, for example 3.0.1. 3 being major, 0 being minor, and 1 being patch.\n\nPatch version increments should introduce no breaking changes to the existing public chart. Docker images/Elixir apps are able to be updated in-place with no changes needed.\nMinor version increments may require chart changes to function properly. These changes should be reviewed and charts should be adjusted accordingly before updating.\nMajor version increments likely introduce wide-spread or structural changes that require many configuration changes. \n\n# License\nReleased under [Apache 2 license](https://github.com/UrbanOS-Public/smartcitiesdata/blob/master/LICENSE).\n# Contributions\n[How to contribute](https://github.com/UrbanOS-Public/smartcitiesdata/wiki/Contribute)\n# Contact Information\n# Acknowledgements\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furbanos-public%2Fsmartcitiesdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furbanos-public%2Fsmartcitiesdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furbanos-public%2Fsmartcitiesdata/lists"}