{"id":35815605,"url":"https://github.com/openclimatefix/data-platform","last_synced_at":"2026-04-02T21:28:47.474Z","repository":{"id":312057467,"uuid":"1045500075","full_name":"openclimatefix/data-platform","owner":"openclimatefix","description":"Data API and Storage Platform for OCF's Forecasts","archived":false,"fork":false,"pushed_at":"2026-03-25T08:38:57.000Z","size":18215,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-25T22:58:58.296Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"devsjc/fcfs","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2025-08-27T09:15:21.000Z","updated_at":"2026-03-25T08:38:45.000Z","dependencies_parsed_at":"2026-01-06T07:11:45.436Z","dependency_job_id":null,"html_url":"https://github.com/openclimatefix/data-platform","commit_stats":null,"previous_names":["openclimatefix/data-platform"],"tags_count":53,"template":false,"template_full_name":null,"purl":"pkg:github/openclimatefix/data-platform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdata-platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdata-platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdata-platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdata-platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/data-platform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdata-platform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31316775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-07T16:08:26.943Z","updated_at":"2026-04-02T21:28:47.451Z","avatar_url":"https://github.com/openclimatefix.png","language":"Go","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# Data Platform\n\n**A complete redesign of OCF's forecast data stack for performance and useability.**\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture align=\"center\"\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://github.com/user-attachments/assets/f1b3a35f-4fc5-4a62-b6f3-ed41ddcfb6ed\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://github.com/user-attachments/assets/16e95933-8978-4517-b3d4-57b8e669526b\"\u003e\n    \u003cimg width=\"762\" height=\"285\" alt=\"Shows a bar chart with benchmark results.\" src=\"https://github.com/user-attachments/assets/16e95933-8978-4517-b3d4-57b8e669526b\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\nThe Data Platform is a gRPC API server that provides efficient access to, and storage of, renewable\nenergy forecast data. It has been architected to be performant under the specific workflows and\ndata access patterns required by OCF's applications, in order to enable scaling, and to improve the\ndeveloper experience when integrating with OCF's stack. With this in mind, there is a focus on not\njust the quality of the code, but also of the tooling surrounding the codebase.\n\nThe benefits of the Data Platform, over the current OCF stack (datamodels) include, but aren't\nlimited to:\n\n- **Two orders of magnitude faster** (milliseconds vs seconds)\n- **Performant at scale** (tested to 50x current org scope)\n- **Cheaper deployment** as on-the-fly calculation capability obsoletes analysis microservices\n- **Fully typed** client implementations in Python and Typescript\n- **Simple** to understand due to codegen of boilerplate\n- **Safer architecture** with single, considered source of entry to database\n- Unlocks greater depth of analysis with geometries, capacity limits, history and more\n\n\n## Architecture\n\nThe Data Platform has clear separation boundaries between its components:\n\n```\n                +-------------------------------------------------------------+\n                |                     Data Platform Server                    |\n                +-------------------+                     +-------------------+\n--- Clients --\u003e | External Schema   | \u003c-- Server Impl --- | Database Schema   | \u003c-- Database\n                +-------------------+                     +-------------------+\n                |                                                             |\n                +-------------------------------------------------------------+\n```\n\n### External schema\n\nThe Data Platform defines a strongly typed _data contract_ as its external interface. This is the\nAPI that any external clients have to use to interact with the platform. The schema for this is\ndefined via Protocol Buffers in `proto/ocf/dp`.\n\nBoilerplate code for client and server implementations is generated in the required language from\nthese `.proto` files using the `protoc` compiler.\n\n\u003e [!Note]\n\u003e This is a direct analogue to the Pydantic models used in the old `datamodel` project.\n\nChanges to the schema modifies the data contract, and will require client and server\nimplementations to regenerate their bindings and update their code. As such they should be made\nwith purpose and care.\n\n### Database schema\n\nThe Data Platform can be configured to use different database backends. Each backend has a server\nimplementation that inherits the External Schema. The currently supported backends are:\n\n- PostgreSQL\n- Dummy (a memoryless databse for quick testing)\n\nand are selected according to the relevant environment variables (see the\n[Configration](#configuration) section). \n\nThe schema for the PostgreSQL backend is defined using PostgreSQL's native SQL dialect in the\n`internal/server/postgres/sql/migrations` directory, and access functions to the data are defined\nin `internal/server/postgres/sql/queries`.\n\nBoilerplate code for using these queries is generated using the `sqlc` tool. This generated code\nprovides a strongly typed interface to the database.\n\n\u003e [!Note]\n\u003e This is a direct analogue to the SQLAlchemy models used in the old `datamodel` project.\n\nHaving the queries defined in SQL allows for more efficient interaction with the database,\nas they can be written to take advantage of the design of the database's features and be written\nto be optimal with regards to its indexes.\n\nThese changes can be made without having to update the data contract, and so will not require\nupdates to clients using the Data Platform.\n\n\u003e [!Note]\n\u003e If using PostgreSQL as a backend, it is recommended that you tune your database instance\n\u003e according to the specifications of said instance (available CPU and RAM etc). This will ensure\n\u003e optimal performance for the Data Platform server.\n\n### Server\n\nThe Database Schema is mapped to the External Schema by implementing the server interface generated\nfrom the Data Contract. This is done in `internal/server/\u003cdatabase\u003e/serverimpl.go`. It isn't much\nmore than a conversion layer, with the business logic shared between the implemented functions and\nthe SQL queries.\n\n\n## Usage\n\n### Running the server\n\nThe Data Platform gRPC API server is packaged for portability as a container. This can be run using\na container orchestration tool, e.g. with Docker:\n\n```bash\n$ docker run -p 50051:50051 ghcr.io/openclimatefix/data-platform\n```\n\nAlternatively, it can be run locally using Go. See\n[Local Running](#local-running) in the [Development](#development) section.\n\nOnce running, the server RPCs can be investigated using a gRPC client tool.\n\n### Configuration\n\nTo connect to a backend database and have retention in the platform data, the server must be\nappropriately configured via environment variables. All available options are defined via the\nconfiguration file in `cmd/server.conf`.\n\n\u003e [!Important]\n\u003e Whilst the configuration is held in a file, this is NOT intended to be overwritten or modified in\n\u003e order to configure the Data Platform. Configuration should always be handled via environment\n\u003e variables; the config file is simply provided as a version-controlled single point of reference\n\u003e for what those variables might be.\n\nThe available configuration may differ between versions of the Data Platform. Ensure you check the\ncorrect version of the configuration file for your deployment.\n\n### Running a client\n\nThere is an example Python notebook, written with [Marimo](https://docs.marimo.io/), demonstrating\nhow to use the Python bindings in a client to a Data Platform server. The example runs through a\ndata analysis workflow. To run it, ensure first that the Data Platform Server is running on\n`localhost:50051` (see [Getting Started](#getting-started)); and that the python bindings have been\ngenerated (see [Generating Code](#generating-code)). Then use\n[uvx](https://docs.astral.sh/uv/reference/cli/#uv-tool-run) to run the notebook:\n\n```bash\n$ make gen.proto.python\n$ uvx marimo edit --headless --sandbox examples/python-notebook/example.py \n```\n\nFor ease, the above process is wrapped in a Makefile target:\n\n```bash\n$ make run.notebook\n```\n\n\n## Development\n\n### Getting Started\n\nThis project requires the [Go Toolchain](https://go.dev/doc/install) to be installed.\n\n\u003e [!Note]\n\u003e This project uses Go modules for dependency management. Ensure that your `PATH` environment\n\u003e variable has been updated to include the Go binary installation location, as per the instructions\n\u003e linked above, otherwise you may see errors.\n\nClone the repository, then run\n\n```bash\n$ make init\n```\n\nThis will fetch the dependencies, and install the git hooks required for development.\n\n\u003e [!Important]\n\u003e Since this project is uses lots of generated code, these hooks are vital to keep this generated\n\u003e code up to date, and as such running `make init` is a necessary step towards a smooth development\n\u003e experience.\n\n### Local running\n\nThe server can be run locally with no database connection via a fake database implementation via\na Make target. This is recommended as it will ensure that code generation is up to date and that\nthe running version has been embedded into the built binary.\n\n```bash\n$ make run\n```\n\nThis will start the Data Platform API GRPC's server on `localhost:50051`. The RPCs can then be\ninvestigated using a tool such as [grpcurl](https://github.com/fullstorydev/grpcurl) or \n[grpcui](https://fullstorydev/grpcui). In this testing mode, the data returned by the server is\nentirely generated and has little bearing on the request objects themselves.\n\nThere is also an example Docker compose file in `examples/docker-compose.yml`, which runs the Data\nPlatform API server in a container, backed by Postgres, and which also includes a GRPC UI for\ntesting.\n\n### Testing\n\nUnit tests can be run using `make test`. Benchmarks can be run using `make bench`.\nBoth of these utilise [TestContainers](https://github.com/testcontainers/testcontainers-go),\nso ensure you meet their \n[general system requirements](https://golang.testcontainers.org/system_requirements/).\n\n### Generating Code\n \nIn order to make changes to the *SQL queries*, or add a new *Database migration*, you will need to\nadd or modify the relevant `.sql` files in the `sql` directory. Then, regenerate the Go library\ncode to reflect these changes. This can be done using\n\n```\n$ make gen\n```\n\nThis will populate the `internal/server/postgres/gen` directory with language-specific bindings\nfor implementations of server and client code. Next, update the `serverimpl.go` file for the given\ndatabase to use the newly generated code, and ensure the test suite passes. Since the Data Platform\ncontainer automatically migrates the database on startup, simply re-deploying the container will\npropagate the changes to your deployment environment.\n\nIn order to change the *Data Contract*, you will need to modify the `.proto` files in the `proto`\ndirectory, and regenerate the code. GRPC client/server interfaces - and boilerplate code - gets\ngenerated from these Protocol Buffer definitions. The `make gen` target already handles generating\nthe go code used internall in the application, placing generated code in `internal/gen`.\n\nLanguage-specific client/server bindings for external applications are generated as part of the CI\npipeline, but can also be generated manually, e.g. for python\n\n```bash\n$ make gen.proto.python\n```\n\nThis places the generated code in `gen/python`. See the `Makefile` for more external targets.\n\n## Further Comparisons\n\n\u003cdetails\u003e\u003csummary\u003eComplexity analysis of Data Platform vs old datamodels \u0026 metrics (scc)\u003c/summary\u003e\n\n```\nData Platform:\n───────────────────────────────────────────────────────────────────────────────\nLanguage                 Files     Lines   Blanks  Comments     Code Complexity\n───────────────────────────────────────────────────────────────────────────────\nSQL                          7      1288       75       566      647          6\nGo                           5      2353      222       191     1940        253\nShell                        4       108       11        17       80         11\nYAML                         4       224       34         2      188          0\nProtocol Buffers             2       418       82        79      257          0\nMakefile                     1        88       20         3       65          7\nMarkdown                     1       143       33         0      110          0\n───────────────────────────────────────────────────────────────────────────────\nTotal                       24      4622      477       858     3287        277\n───────────────────────────────────────────────────────────────────────────────\nEstimated Cost to Develop (organic) $94,239\nEstimated Schedule Effort (organic) 5.61 months\nEstimated People Required (organic) 1.49\n───────────────────────────────────────────────────────────────────────────────\nProcessed 11480397 bytes, 11.480 megabytes (SI)\n───────────────────────────────────────────────────────────────────────────────\n\nDatamodels \u0026 Metrics:\n───────────────────────────────────────────────────────────────────────────────\nLanguage                 Files     Lines   Blanks  Comments     Code Complexity\n───────────────────────────────────────────────────────────────────────────────\nPython                     190     23776     3213      3119    17444        508\nYAML                         9       294       30        10      254          0\nMarkdown                     6       825      222         0      603          0\nCSV                          3       978        0         0      978          0\nMako                         3        74       21         0       53          0\nTOML                         3       196       28        20      148          2\nDockerfile                   2        56       18        12       26          2\nINI                          2       213       46        99       68          0\nPlain Text                   2        12        0         0       12          0\nAutoconf                     1         1        0         0        1          0\nLicense                      1        21        4         0       17          0\nMakefile                     1        23        4         5       14          0\n───────────────────────────────────────────────────────────────────────────────\nTotal                      223     26469     3586      3265    19618        512\n───────────────────────────────────────────────────────────────────────────────\nEstimated Cost to Develop (organic) $615,010\nEstimated Schedule Effort (organic) 11.43 months\nEstimated People Required (organic) 4.78\n───────────────────────────────────────────────────────────────────────────────\nProcessed 909596 bytes, 0.910 megabytes (SI)\n───────────────────────────────────────────────────────────────────────────────\n```\n\n(Produced via `$ scc --exclude-dir=\".git,examples,proto/buf,proto/google\"`. Data may be out of date.)\n\n\u003c/details\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fdata-platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fdata-platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fdata-platform/lists"}