{"id":15152778,"url":"https://github.com/chop-dbhi/scds","last_synced_at":"2025-10-24T09:30:32.832Z","repository":{"id":141917827,"uuid":"39111260","full_name":"chop-dbhi/scds","owner":"chop-dbhi","description":"Slowly-Changing Dimension Store (prototype)","archived":false,"fork":false,"pushed_at":"2016-09-20T21:10:28.000Z","size":55,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-01-31T00:34:34.138Z","etag":null,"topics":["json-document","json-schema","revision","schema","versioning"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chop-dbhi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-07-15T02:14:41.000Z","updated_at":"2019-02-17T10:30:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"9839da76-684d-45ae-a85a-c5b3457a7535","html_url":"https://github.com/chop-dbhi/scds","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chop-dbhi%2Fscds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chop-dbhi%2Fscds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chop-dbhi%2Fscds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chop-dbhi%2Fscds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chop-dbhi","download_url":"https://codeload.github.com/chop-dbhi/scds/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237944079,"owners_count":19391588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json-document","json-schema","revision","schema","versioning"],"created_at":"2024-09-26T16:22:53.308Z","updated_at":"2025-10-24T09:30:32.495Z","avatar_url":"https://github.com/chop-dbhi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Slowly-Changing Dimension Store (SCDS)\n\n[![Build Status](https://travis-ci.org/chop-dbhi/scds.svg?branch=master)](https://travis-ci.org/chop-dbhi/scds) [![Coverage Status](https://coveralls.io/repos/chop-dbhi/scds/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/chop-dbhi/scds?branch=master) [![GoDoc](https://godoc.org/github.com/chop-dbhi/scds?status.svg)](https://godoc.org/github.com/chop-dbhi/scds)\n\nSCDS is designed to answer a single question, \"has this data changed since the last time I saw it?\" The motivation stems from working on data integration pipelines where it may be unknown when or how downstream data has changed. There are two basic use cases:\n\n- Check the data to fail quickly or apply other logic.\n- Put the data each time it is seen to see how it changes over time.\n\n*Note: albeit functional, this is a prototype implementation for solving this problem.*\n\n## Usage\n\nThere are two interfaces supported, command line and HTTP. They share the same set of operations and work with JSON-encoded data.\n\n### Operations\n\n#### `put`\n\nPut an object in the store where `value` is a valid JSON document. Putting the same object consecutively will not result in duplicate changes.\n\n```\nput \u003ckey\u003e \u003cvalue\u003e\n```\n\n#### `get`\n\nGet the current state of the object. Use the `-version` or `-time` option to get a particular revision.\n\n```\nget \u003ckey\u003e\n```\n\n#### `keys`\n\nGets a list of keys in the store.\n\n```\nkeys\n```\n\n#### `log`\n\nGet the log of changes for an object.\n\n```\nlog \u003ckey\u003e\n```\n\n#### `config`\n\nPrints the configuration options used.\n\n```\nconfig\n```\n\n#### `subscribe`\n\nSubscribes one or more email addresses for notifications.\n\n```\nsubscribe email [emails...]\n```\n\n#### `unsubscribe`\n\nUnsubscribes one or more email addresses from notifications.\n\n```\nunsubscribe email [emails...]\n```\n\n### CLI\n\nSee `scds help` for more information.\n\nInline JSON.\n\n```bash\nscds put bob '{\"name\": \"Bob\"}'\n```\n\n```json\n{\n  \"version\": 1,\n  \"time\": 1436960622,\n  \"additions\": {\n    \"name\": \"Bob\"\n  },\n  \"removals\": null,\n  \"changes\": null\n}\n```\n\nAlternately, if not `value` is supplied, data will be read from stdin.\n\n```bash\nscds put hello \u003c hello.json\n```\n\nRunning the above command again will return nothing since nothing changed. However if we change it a new revision will be created.\n\n```bash\nscds put bob '{\"name\": \"Bob Smith\", \"email\": \"bob@smith.net\"}'\n```\n\n```json\n{\n  \"version\": 2,\n  \"time\": 1436960632,\n  \"additions\": {\n    \"email\": \"bob@smith.net\"\n  },\n  \"removals\": null,\n  \"changes\": {\n    \"name\": {\n      \"before\": \"Bob\",\n      \"after\": \"Bob Smith\"\n  }\n}\n```\n\nTo get the current state of the object use `get`.\n\n```bash\nscds get bob\n```\n\n```json\n{\n  \"key\": \"bob\",\n  \"value\": {\n    \"email\": \"bob@smith.net\",\n    \"name\": \"Bob Smith\"\n  },\n  \"version\": 2,\n  \"time\": 1436960632\n}\n```\n\nTo get the log of changes over time:\n\n```\nscds log bob\n```\n\n```json\n[\n  {\n    \"version\": 1,\n    \"time\": 1436960622,\n    \"additions\": {\n      \"name\": \"Bob\"\n    },\n    \"removals\": null,\n    \"changes\": null\n  },\n  {\n    \"version\": 2,\n    \"time\": 1436960632,\n    \"additions\": {\n      \"email\": \"bob@smith.net\"\n    },\n    \"removals\": null,\n    \"changes\": {\n      \"name\": {\n        \"before\": \"Bob\",\n        \"after\": \"Bob Smith\"\n      }\n    }\n  }\n]\n```\n\n### HTTP\n\nStart the HTTP server.\n\n```bash\nscds http\n* [http] Listening on locahost:5000\n```\n\nThe input and output of the endpoints match the command-line interface.\n\n- `GET /keys`\n- `PUT /objects/\u003ckey\u003e`\n- `GET /objects/\u003ckey\u003e`\n- `GET /objects/\u003ckey\u003e/v/\u003cversion\u003e`\n- `GET /objects/\u003ckey\u003e/t/\u003ctime\u003e`\n- `GET /log/\u003ckey\u003e`\n\n\n## Notifications\n\nSCDS has built-in support for basic email notifications when new objects are added or when objects change. To subscribe, use the `subscribe` command.\n\n```\nscds subscribe \u003cemail\u003e\n```\n\n\n## Dependencies\n\n- MongoDB\n\n\n## Configuration\n\nConfiguration options can be supplied in a file, as environment variables, or command-line arguments (following that precedence). The default configuration options are listed below (in a YAML format).\n\n```yaml\ndebug: false\nconfig: \"\"\nmongo:\n  uri: localhost/scds\nhttp:\n  host: localhost\n  port: 5000\n  tlscert: \"\"\n  tlskey: \"\"\n  cors: false\nsmtp:\n  host: localhost\n  port: 25\n  user: \"\"\n  password: \"\"\n  from: \"\"\n```\n\nEnvironment variables are prefixed with `SCDS_`, are uppercased, and nested options are delimited with an underscore. For example, `SCDS_MONGO_URI` would set the `uri` option in the `mongo` map. Alternately, the command-line flag can be supplied:\n\n```\nscds -mongo.uri dockerhost/scds ...\n```\n\nIf a `scds.yml` file is defined in the working directory, it will be read in automatically. To use an alternate path, the `-config \u003cpath\u003e` (or `SCDS_CONFIG=\u003cpath\u003e`) can be used.\n\n### JSON Schema\n\nSCDS supports document validation against predefined [JSON Schema](http://json-schema.org) documents. The simplest setup is a schema used for all documents.\n\n```yaml\nschemas:\n  default:\n    file: schema.json\n```\n\nTo validate a subset of objects, a `scope` can be specified at the `object` or `value` level. An `object` scope supports the `pattern` key for matching on the `key` of the object.\n\n```yaml\nschemas:\n  users:\n    scope: object\n    pattern: \"users\\..*\"\n    file: user.json\n```\n\nAny object with a key that starts with `users.` will be validated by the `user.json` schema. Thus the following would match:\n\n```http\nPUT /objects/users.1\n\n{ ... }\n```\n\nor via the command line:\n\n```\nscds put users.1 '{}'\n```\n\nThe `value` scope pattern matches against a particular field in the object data.\n\n```yaml\nschemas:\n  users:\n    scope: value\n    field: type\n    pattern: user\n    file: user.json\n```\n\nThe following document would match since it contains the `type` field with a value of `user`.\n\n```http\nPUT /objects/abc123\n\n{\n  \"type\": \"user\",\n  \"firstName\": \"Jon\",\n  \"lastName\": \"Doe\"\n}\n```\n\nAn object that matches multiple schema patterns will be validated against all of them. This allows for composing smaller schemas together to validate various parts of the documents.\n\n## Docker\n\nThe image defaults to running the HTTP interface and looks for a MongoDB server listening on `mongo:27017`.\n\n```\ndocker run -it --link mongo:mongo -p 5000:5000 dbhi/scds\n```\n\n### Compose\n\nA basic Docker Compose file is provided that includes starting a MongoDB container, however it should be changed to mount a volume on the host so the data is persisted.\n\n```\ndocker-compose up -d\n```\n\n## Development\n\nGo 1.7 or later is required for development.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchop-dbhi%2Fscds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchop-dbhi%2Fscds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchop-dbhi%2Fscds/lists"}