{"id":13556466,"url":"https://github.com/datatogether/roadmap","last_synced_at":"2025-04-03T10:30:38.827Z","repository":{"id":95232991,"uuid":"85715262","full_name":"datatogether/roadmap","owner":"datatogether","description":"Coordinating technical work \u0026 roadmapping additional services","archived":true,"fork":false,"pushed_at":"2018-12-20T18:45:38.000Z","size":11285,"stargazers_count":8,"open_issues_count":21,"forks_count":3,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-11-04T06:34:48.899Z","etag":null,"topics":["docs","roadmap"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datatogether.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null}},"created_at":"2017-03-21T14:56:48.000Z","updated_at":"2023-01-28T16:44:47.000Z","dependencies_parsed_at":"2023-04-03T17:32:18.704Z","dependency_job_id":null,"html_url":"https://github.com/datatogether/roadmap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datatogether%2Froadmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datatogether%2Froadmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datatogether%2Froadmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datatogether%2Froadmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datatogether","download_url":"https://codeload.github.com/datatogether/roadmap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246984410,"owners_count":20864434,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docs","roadmap"],"created_at":"2024-08-01T12:03:51.214Z","updated_at":"2025-04-03T10:30:37.576Z","avatar_url":"https://github.com/datatogether.png","language":null,"funding_links":[],"categories":["Others","others"],"sub_categories":[],"readme":"# Data Together Technical Roadmap \u0026 Service Overview\n\n[![GitHub](https://img.shields.io/badge/project-Data_Together-487b57.svg?style=flat-square)](http://github.com/datatogether)\n[![Slack](https://img.shields.io/badge/slack-Archivers-b44e88.svg?style=flat-square)](https://archivers-slack.herokuapp.com/)\n\nWe track project progress using a [Waffle Board](https://waffle.io/datatogether/roadmap) and identify major goals in [`roadmap.md`](./docs/roadmap.md), go check them out! If you are new to the project, check out our [first timer](https://github.com/issues?q=is:open+org:datatogether+label:first-timers-only) issues, which are cross-listed on [up-for-grabs.net](http://up-for-grabs.net/#/names/data%20together).\n\n** **\n\n![services list](./diagrams/services-list.png)\n\n** **\n## What's a service?\n\nWhen we say _service_, we actually mean a _microservice server running in concert with other services_. All of the services that we define end up being run as a cluster of networked containers. Each of our services have a few common characteristics:\n\n### `server.go` defines `main()`\nThe best place to start reading about any service is to open its `server.go` file. All go programs (that aren't packages) define `func main()` as their starting point, and for us that's always defined in `server.go`. Often you'll see a function called `NewServerRoutes` that defines any and all outward-facing http endpoints. These endpoints define what the service can do, and working backward from there is a great way to understand a service.\n\n### `config.go` determines config via ENV variables, using the `config` package\nAll services support at least some degree of configuration via a `config` struct defined in a file called `config.go`. Services use the config package to extract the values of this configuration from environment variables, mapping `config.FieldName` in the service to an all-caps-snake-case `FIELD_NAME` environment variable. More info can be found in the [config package](https://github.com/datatogether/config).\n\n### logging with `logrus` package\n[Logrus](https://github.com/sirupsen/logrus) is our logger of choice. We follow [Dave Cheney's Advice](https://dave.cheney.net/2015/11/05/lets-talk-about-logging) when it comes to logging, using only `log.info` and `log.debug` commands.\n\n### vendored dependencies via `godep`\nWe support the idea of reproducible builds, and as such _vendor_ (that is, write copies) all of our dependencies into version control. This means that it should be possible to rewind git history to any point in time, run `go build` and get a functioning binary (presuming the build process worked at that point in time). [godep](https://github.com/tools/godep) is what we're currently using, but we're keeping a close eye on [dep](https://github.com/golang/dep), as a potential replacement.\n\n### Containerized\nServices are always shipped as docker images.\n\n### Shipped via passing CI tests merged into master\nThe last step of a passing CI test is to build \u0026 push a new docker image to docker hub with the `latest` tag.\n\n### `request` \u0026 `handler` patterns\nTo perform the work of handling a request, all servers register a \"handler\" function that often delegates the majority of their work to a \"request\" function to actually to the thing in question. This pattern is for a few reasons, first \u0026 foremost, each request defines whatever parameters it accepts as a struct, giving clear documentaiton of ways to modify the request. Secondly request functions satisfy the form specified by the `rpc` package. By isolating business logic in request functions, we provide strong garuntees that behaviour will be the same for http handlers \u0026 rpc requests, while also providing a clear abstraction for testing.\n\n### Communicates with other services via RPC\nServices need to talk to each other, they do so via a package called `rpc` that allows us to call go functions on another service without having to worry about\n\n### Use go-datastore interface for storing data\nWhen storing information, we often go to great lengths to accept the ipfs-datastore interface as a point of storage, this is intended as future-proofing for an eventual transition to distributed versions of these same services, and allowing interchangability of the underlying datastore.\n\n\n** ** \n## Production\n\nWe use [kubernetes](https://kubernetes.io) in production to orchestrate containers. Contact [b5](https://github.com/b5) if you'd like to chat production details.\n\n\n** **\n## Running The Data Together platform locally\n\nThe best way to get data together running locally is to clone the [context repo](https://github.com/datatogether/context) and use `docker-compose` to spin up all the necessary backend services. `docker-compose up` will download all the necessary data together images in a single terminal, hook them together via networking, and will spin up a dev version of the webapp to interact with the platform. Many other services come with `docker-compose.yml` files that outline the miniumum number of other services needed to make a sensible working version of the host service.\n\n\n** **\n## Repo Links\nEach repository should carry with it its own roadmap, defined by milestones. Check each repo's `README.md` for details\n\n* [**api**](https://github.com/datatogether/api)\n* [**archive**](https://github.com/datatogether/archive)\n* [**archivertools**](https://github.com/datatogether/archivertools)\n* [**cdxj**](https://github.com/datatogether/cdxj)\n* [**config**](https://github.com/datatogether/config)\n* [**content**](https://github.com/datatogether/content)\n* [**coverage**](https://github.com/datatogether/coverage)\n* [**extract_href**](https://github.com/datatogether/extract_href)\n* [**ffi**](https://github.com/datatogether/ffi)\n* [**identity**](https://github.com/datatogether/identity)\n* [**ipfs**](https://github.com/ipfs/go-ipfs)\n* [**linked_data**](https://github.com/datatogether/linked_data)\n* [**patchbay**](https://github.com/datatogether/patchbay)\n* [**postgres**](https://github.com/postgres/postgres)\n* [**rabbitmq**](https://github.com/rabbitmq/rabbitmq-server)\n* [**redis**](https://github.com/antirez/redis)\n* [**sentry**](https://github.com/datatogether/sentry)\n* [**sql_datastore**](https://github.com/datatogether/sql_datastore)\n* [**sql_util**](https://github.com/datatogether/sql_util)\n* [**task-mgmt**](https://github.com/datatogether/task-mgmt)\n* [**warc**](https://github.com/datatogether/warc)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatatogether%2Froadmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatatogether%2Froadmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatatogether%2Froadmap/lists"}