{"id":28390836,"url":"https://github.com/proboci/probo-db","last_synced_at":"2026-02-21T06:02:52.696Z","repository":{"id":42343027,"uuid":"90209432","full_name":"ProboCI/probo-db","owner":"ProboCI","description":"A database service that consumes event data from an eventbus stream and stores it in PostgreSQL.","archived":false,"fork":false,"pushed_at":"2025-12-16T01:55:02.000Z","size":356,"stargazers_count":2,"open_issues_count":20,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-12-19T11:41:07.163Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ProboCI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-05-04T01:41:25.000Z","updated_at":"2025-01-21T17:50:31.000Z","dependencies_parsed_at":"2025-01-18T19:26:16.098Z","dependency_job_id":"70d07775-c776-4efc-8e98-23440a24c2ac","html_url":"https://github.com/ProboCI/probo-db","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/ProboCI/probo-db","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProboCI%2Fprobo-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProboCI%2Fprobo-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProboCI%2Fprobo-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProboCI%2Fprobo-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ProboCI","download_url":"https://codeload.github.com/ProboCI/probo-db/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProboCI%2Fprobo-db/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29674931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T05:54:28.202Z","status":"ssl_error","status_checked_at":"2026-02-21T05:53:42.585Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T06:42:53.533Z","updated_at":"2026-02-21T06:02:52.667Z","avatar_url":"https://github.com/ProboCI.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Probo-DB\n\nA database service that consumes event data from an [eventbus](https://github.com/ProboCI/probo-eventbus) stream and stores it in PostgreSQL.\n\nThe database is designed to be able to process and reprocess data from a system such as Apache Kafka. Therefore each step of the data processing and storage pipeline needs to be able to see the same piece of data multiple times without issue.\n\nThe database is abstracted by [knex](http://knexjs.org/) and relies on postgres. Review the appropriate documentation for details on these projects.\n\nThe REST API is built using [Restify](http://restify.com/).\n\n## Configuration\n\nBy default, the knexfile.js is used to run migrations and as the configuration to the proboDb server. You can override this is a few ways, first, you can specify a path to a knexfile anywhere on the filesystem by passing the `-k` flag when starting the server:\n\n```\n./bin/probo-db -k /path/to/knex/file.js\n```\n\nConfiguration is also inherited from yaml config that can be passed in via the `-c` flag:\n\n```\n./bin/probo-db -c /path/to/config/file.yaml\n```\n\nIf both options are used, the knex connection data from the yaml file will be used over that passed in via the knexfile.js.\n\nSince knex and migrations assume a `knexfile.js`, it is recommended to provide this rather than specifying configuration in the yaml file.\n\n## Plugins\n\nThe project supports plugins. Plugins should be a separate project and should be referenced by path in the configuration. The plugin project should expose the plugins as properties on the exported object. Database plugins should be called `dbPlugins` and should be in the order they should be run. Api plugins will add routes to the REST API server. They must be added as an array to a property called `apiPlugins`. Order is not important as these will be routes to access data out of band.\n\n```javascript\nconst myPlugin = require('./myDbPlugin');\nconst myApiPlugin = require('./myApiPlugin');\n\nmodule.exports = {\n  dbPlugins: [\n    myPlugin\n  ],\n  apiPlugins: [\n    myApiPlugin\n  ]\n};\n```\n\nEach database plugin must export a class that must implements a `process` method. The `process` method will be given a build object which it can then act on. Additionally, the constructor for this class will be handed options that include a logger and a database connection via knex. A valid database plugin might look like the following:\n\n```javascript\nclass myDbPlugin {\n\n  constructor(options) {\n    this.knex = options.knex;\n    this.logger = options.logger;\n  }\n\n  process(build) {\n    let record = this.prepare(build);\n\n    return this.knex('some_table')\n      .insert(record);\n  }\n\n  prepare(build) {\n\t  // extract data and return object.\n    self.logger.info('Return some feedback');\n  }\n}\n\nmodule.exports = myDbPlugin;\n```\n\nAn API plugin will also receive an object that includes a knex connection and a logger. The API plugin must implement an `addRoutes` method that will be handed a restify `server` object and will append new routes to it. A valid api plugin might look like the following:\n\n```javascript\nclass myApiPlugin {\n\n  constructor(options) {\n    this.knex = options.knex;\n    this.logger = options.logger;\n  }\n\n  addRoutes(server) {\n    let self = this;\n    server.addRoute('/some/route', function(req, res, next) {\n      self.knex\n        .select('*')\n        .from('mytable')\n        .then(function(result) {\n          self.logger.info('Return some feedback');\n          res.send(result);\n          next()\n        })\n        .catch(function(error) {\n          self.logger.error('Bad stuff happened...');\n          next(error);\n        });\n    });\n  }\n\n}\n\nmodule.exports = myApiPlugin;\n```\n\n## Migrations\n\nMigrations are stored in the `baseMigrations` directory. Create these using knex.\n\nPlugins can also have migrations in order to add their own tables. Migrations are combined into a single folder and run together using `./bin/migrate`. To have your plugin’s migrations run, add them to a directory called `migrations` in the plugin's folder. See the knex migration documentation for more information on creating migrations.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproboci%2Fprobo-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproboci%2Fprobo-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproboci%2Fprobo-db/lists"}