{"id":20030936,"url":"https://github.com/mountain-pass/ryvr","last_synced_at":"2025-03-02T05:21:23.482Z","repository":{"id":95088548,"uuid":"84367352","full_name":"mountain-pass/ryvr","owner":"mountain-pass","description":"Let Your Data Flow","archived":false,"fork":false,"pushed_at":"2019-05-22T22:01:16.000Z","size":5878,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-01-12T17:47:30.171Z","etag":null,"topics":["database-access","java","spring-boot"],"latest_commit_sha":null,"homepage":"https://mountain-pass.github.io/ryvr/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mountain-pass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-08T21:21:59.000Z","updated_at":"2020-09-14T15:53:01.000Z","dependencies_parsed_at":"2023-03-11T11:16:15.634Z","dependency_job_id":null,"html_url":"https://github.com/mountain-pass/ryvr","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mountain-pass%2Fryvr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mountain-pass%2Fryvr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mountain-pass%2Fryvr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mountain-pass%2Fryvr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mountain-pass","download_url":"https://codeload.github.com/mountain-pass/ryvr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241461915,"owners_count":19966775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-access","java","spring-boot"],"created_at":"2024-11-13T09:29:31.147Z","updated_at":"2025-03-02T05:21:23.465Z","avatar_url":"https://github.com/mountain-pass.png","language":"Java","readme":"## Ryvr\n\n_**- Let Your Data Flow**_\n\n[![CircleCI](https://img.shields.io/circleci/project/github/mountain-pass/ryvr.svg)](https://circleci.com/gh/mountain-pass/workflows/ryvr)\n[![Dependency Status](https://www.versioneye.com/user/projects/58ee953c0f9f35004e5c4bf2/badge.svg?style=flat-round)](https://www.versioneye.com/user/projects/58ee953c0f9f35004e5c4bf2)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/7785f1049bd045dda89fcfff65bff3da)](https://www.codacy.com/app/mountain-pass/ryvr?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=mountain-pass/ryvr\u0026utm_campaign=Badge_Grade)\n[![Codacy Badge](https://api.codacy.com/project/badge/Coverage/7785f1049bd045dda89fcfff65bff3da)](https://www.codacy.com/app/mountain-pass/ryvr?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=mountain-pass/ryvr\u0026utm_campaign=Badge_Coverage)\n\n[![GitHub release](https://img.shields.io/github/release/mountain-pass/ryvr.svg)](https://github.com/mountain-pass/ryvr/releases/latest)\n[![license](https://img.shields.io/github/license/mountain-pass/ryvr.svg)](https://github.com/mountain-pass/ryvr/blob/master/LICENSE)\n[![Github All Releases](https://img.shields.io/github/downloads/mountain-pass/ryvr/total.svg)](https://github.com/mountain-pass/ryvr/releases)\n\n[![Sauce Test Status](https://saucelabs.com/browser-matrix/tompahoward.svg)](https://saucelabs.com/u/tompahoward)\n\nRyvr provides highly optimised access to your data, when it's stored as a series of immutable events.\nThink bank transaction lists, trade histories, log records, etc.\n\n## Performance\u003csup id=\"myfootnotelink-*\"\u003e[\\*](#myfootnote-*)\u003c/sup\u003e\n\n### First Read\n\nWhen reading a ryvr for the first time (i.e. none of the data is in cache), clients can pull down over **100,000 records per second** at a throughput of **4.7MB/s**. That's an average of **less than 7µs per record**.\n\n### Subsequent Reads\n\nOn subsequent reads (i.e. previously read data is in cache), clients can pull down over **20 million records per second** at a throughput of **900MB/s**. That's an average of **less than 0.049µs per record**.\n\n### Multiple Consumers\n\nRyvrs have very good economies-of-scale when there are multiple consumers. With 1000 clients all consuming the same ryvr, the average client can pull down over 4 million records per second, with the fastest client pulling down **55 million records per second**. This provides a throughput of over **160MB/s** for the average client and over **2.1GB/s** for the fastest client.\n\n\u003ca id=\"myfootnote-*\" href=\"#myfootnotelink-*\"\u003e\\*\u003c/a\u003e All performance results measured on:\n\n- [Shippable's Default Dedicated Dynamic Nodes](http://docs.shippable.com/platform/tutorial/runtime/dynamic-nodes/)'s, which are running Ubuntu 14.04.5 LTS on 2 Cores and 3.675 GB of Memory (AWS's C4.Large)\n- MacBook Pro 2.8 GHz Intel Core i7, with 16 GB 1600 MHz DDR3 and 1TB SSD, running OS X 10.11.6, using the REST API and a local running MySQL v5.7.18 data source.\n\nWe often see performance test runs with significantly better results than the above, however the results\nabove represent the performance results that are consistently achieved.\n\nIn the future, these performance metrics will be updated with AWS based results.\n\nYou can reproduce the performance tests results yourself, by running `RyvrTests_Integration_Performance_Rest_MySqlLocal.java` as a JUnit test, or by running the `testRyvrTests_Integration_Performance_Rest_MySqlLocal` gradle task. This will require a local MySql database called `test_db` with a `dbuser` user with the password `dbpass`.\n\n### Latency\n\nThe write-read latency isn't that good at the moment. The average latency between adding a record to a data source and reading it from a ryvr is less than 0.72s. The 95th percentile is less than 1.25s and the maximum is 1.5s.\n\nThis is because we set the TTL on the current/last page to 1 second, so that when there is a large\nnumber of Ryvr clients, they don't smash the data source when polling for new records.\ni.e. even if you had 10,000 clients for a Ryvr, polling every 100ms, the database will\nstill only see 1 query per second, rather than 1,000,000 queries per second.\nIdeally in the short term, we would reduce the time to live (TTL) to 100ms, but the HTTP spec (RFC2616) doesn't\nallow sub-second TTLs.\n\nIf you know that you will only have a small number of clients for a Ryvr, you can\ngreatly improve the write-read latency by setting `au.com.mountainpass.ryvr.cache.current-page-max-age` to `0`\n\nLonger term, there are a number of approaches we intend to use to improve this\n\n#### [Hystrix Request Collapsing](https://github.com/Netflix/Hystrix/wiki/How-it-Works#RequestCollapsing)\n\nThis would allow us combine requests received within a certain time period. For instance if we\ncollapsed all the requests within 100ms, then in the above example of 10,000 clients polling\nevery 100ms, with a HTTP TTL of 0, the database would still only see 10 queries per second.\nHowever, we expect this will result in a mean latency of just over 50ms and a max latency\nof just over 100ms\n\n#### Notifications\n\nSome event sources provide a mechanism for alerting a registered subscriber when there are new events\nFor instance, for MySQL event sources, the [mysql-binlog-connector-java](https://github.com/shyiko/mysql-binlog-connector-java) library allows a\nclient to subsribe to and receive committed change events, such as inserts, on the database\nA similar capability is provided by [CouchDB's Continuous Changes](http://guide.couchdb.org/draft/notifications.html#continuous) if we\nhad a CouchDB event source, and the [WatchService API](http://docs.oracle.com/javase/tutorial/essential/io/notification.html) for file system\nchanges if we had File bases event sources.\nFor these sorts of event sources, we can trigger a refresh only when there is an actual change. This\nwould allow us to remove the TTL on the current/last page without increasing the query load on the\ndata source.\n\nAt the same time we would look to implement a change notification service to advise ryvr clients when there\nare new events.\n\nWith both of these in place, ryvr clients can subscribe to changes, when there is a new event in the\ndata source they ryvr will perform and refresh and then clients will get notified that they can refresh\nthe current/last page, allowing to receive new events with a very small latency.\n\nWith a large number of ryvr clients, this can result in a large deluge of requests for the current/last page\neach time there are new events, however since the ryvr has already been refreshed, no additional load\nwould be applied to the data source. At the same time, we would need to make sure that Ryvr is capable of\nhandling the large deluge of requests and if there is an issue, we'll be investigating it at that time, however\nload balancing across multiple ryvr instances would be the most likely solution.\n\nFor data sources that don't have a change notifaction mechansim, the polling can be moved to Ryvr, which would\nstill allow us to provide a change notifications to ryvr clients (albeit less efficently).\n\n## Running\n\n    java -jar ryvr-\u003cVERSION\u003e.jar --spring.config.location=application.yml\n\n### Configuration\n\nRyvrs are configured using application properties.\n\n#### Database Ryvrs\n\nDatabase Ryvrs are configured as [spring data sources|https://docs.spring.io/spring-boot/docs/current/reference/html/howto-data-access.html] under the property prefix `au.com.mountainpass.ryvr.data-sources`. This property expects an\narray of data sources. For example\n\n    au.com.mountainpass.ryvr:\n      data-sources:\n        - url: jdbc:mysql://db_host_1\n          username: dbuser1\n          password: dbpass1\n        - url: jdbc:postgresql://localhost/postgres\n          username: dbuser2\n          password: dbpass2\n\nFor each data source, you can configure one or more ryvrs as a map under the\n`au.com.mountainpass.ryvr.data-sources[*].ryvrs` property prefix.\n\nThe key of the map specifies the name of the ryvr, which must be unique.\n\n| Property  | Description                                                                                             |\n| --------- | ------------------------------------------------------------------------------------------------------- |\n| page-size | Specifies how many records to include in each page. You will need to tune this. Try 1024 to start with. |\n| query     | Specifies the SQL for querying the database. The records _MUST_ be order from oldest to newest          |\n\nThese properties can be set using an `application.yml` file within the same directory as Ryvr.\n\n_NOTE:_ Only string, numeric and boolean datatypes are supported at this time.\n\n##### Example\n\n    au.com.mountainpass.ryvr:\n      data-sources:\n        - url: jdbc:mysql://localhost\n          username: dbuser\n          password: dbpass\n          ryvrs:\n            transactions:\n              page-size: 10\n              query: select `id`, `account`, `description`, `amount` from `transactions` ORDER BY `id` ASC\n\n**NOTE:** At this time, only MySQL, PostGresSQL and H2 Ryvrs are supported.\n\nSee the [Externalized Configuration section of the Spring Boot documentation](https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html) for other ways to set application properties.\n\n#### HTTPS\n\nRyvr is configured to enable HTTPS and disable HTTP, in order to ensure that all traffic is encrypted.\n\nHTTPS is configured using the following standard Spring Boot application properties:\n\n    server.ssl.key-store\n    server.ssl.key-alias\n    server.ssl.key-password\n    server.ssl.key-store\n    server.ssl.key-store-password\n\nSee https://docs.spring.io/spring-boot/docs/current/reference/html/howto-embedded-servlet-containers.html#howto-configure-ssl for more details regarding these properties\n\n##### Automatic Certificate Generation\n\nBy default, if Ryvr is unable to find the certificate specified by `server.ssl.key-alias`, it will generate one for you. The type of certificate generated is controlled via the `au.com.mountainpass.ryvr.ssl.genCert` application property, which defaults to `selfSigned`. Currently, `selfSigned` is the only type of certificate generation implemented. `au.com.mountainpass.ryvr.ssl.genCert` can be set to false, which will disable certificate generation. Ryvr will shutdown on startup if `au.com.mountainpass.ryvr.ssl.genCert` is false and it is unable to find the certificate specified by `server.ssl.key-alias`.\n\nWhen generating a certificate, Ryvr will set the hostname to the value of the `au.com.mountainpass.ryvr.ssl.hostname` application property. `au.com.mountainpass.ryvr.ssl.hostname` defaults to `localhost`.\n\n## Building\n\nThe project uses [Gradle](https://gradle.org/) for its build system and you can build the project by running:\n\n    ./gradlew build\n\nYou can also run the app using the [Spring Boot Gradle Plugin](http://docs.spring.io/spring-boot/docs/current/reference/html/build-tool-plugins-gradle-plugin.html) like so:\n\n    ./gradlew bootRun\n\nThe swagger definition will be available at the following URI:\n\n- [https://localhost:8443/api-docs/](http://localhost:8443/api-docs/)\n\nThe Spring Boot Actuator end-points are available here:\n\n- [https://localhost:8443/info](http://localhost:8443/info)\n\n## Testing\n\nRyvr uses JUnit with layered Cucumber testing in order to execute the same tests against the Java, REST and UI layers, which\nare execute as separate test runs. Each run shares the same Cucumber scenarios and test steps, which calls a test\nclient to execute the step against the appropriate layer. Spring profiles are used to enable the appropriate client\nfor each layer.\n\nTests for the three layers can be executed by running `./gradlew test`\n\n### Unit Tests\n\nRyvr does not have unit test in the traditional sense. Unit test will verify if the unit behaves as we\nexpected, but it does not verify if the unit behaves as we need in the context of the full application.\n\n### Integration Tests\n\nThe integration tests verify that Ryvr behaves as we expect as an integrated application.\n\nThe integration tests are run with the `integrationTest` Spring profile, which uses @SpringBootTest to run launch\nRyvr in the same JVM as our tests.\n\n### System Tests\n\nThe system tests verify that Ryvr behaves as we expect when it's run as a seperate application.\n\nThe System tests are run with the `systemTest` Spring profile, which launches\nRyvr in a separate.\n\nAt this time, Ryvr can be launched using the `bootRun` and `distZipRun` gradle tasks. Other profiles will be added\nin the future\n\n### Profiles\n\nThe `javaApi` spring profile is used to verify the behaviour of Ryvr's internal Java API. This profile is only used (and only makes sense) when running integration tests. It cannot be used during system tests as Ryvr is\nrunning in a separate JVM.\n\nThe `restApi` spring profile is used to verify the behaviour of Ryvr's REST API.\n\nThe `ui` spring profile is used to verify the behaviour of Ryvr's User Interface.\n\nThe `h2` spring profile is used to verify the behaviour of Ryvr's using a H2 embedded database. This profile is only used (and only makes sense) when running integration tests. It cannot be used during system tests as Ryvr is\nrunning in a separate JVM and therefore the database is running in a separate JVM, preventing us from setting up the\ntest data. We could configure H2 to accept connections from seperate processes, but we see no point.\n\nThe `mysql` spring profile is used to verify the behaviour of Ryvr's using a MySQL database\n\n## Road Map\n\n- [x] Add tests using MySQL instead of H2\n- [x] SSL Config\n- [x] Add support for configuring ryvrs from config file\n- [x] Rename test phases to correctly specify their nature\n- [x] Add tests for various ways of starting Ryvr\n- [x] release\n- [x] Switch to [Link headers](https://tools.ietf.org/html/rfc5988#page-6) rather than HAL links, which would allow\n      navigation without having to parse the body and URL rewriting without having to parse the body\n- [x] Examine faster options than deserialisation with Jackson\n- [x] Change rest of ryvr response to use StringBuilder or OutputStreams\n- [x] Add caching headers\n- [x] Fix test clients and UI after performance tuning improvements\n- [x] move serialiser out of datesource ryvr\n- [x] Performance test with multiple clients\n- [x] Add performance test on write -\u003e read latency\n- [x] Add rest ryvr\n- [x] Add vary response header\n- [x] Add integration with SourceClear for security scans\n- [x] Use a different ryvr name for each test, so as to avoid the need to clear cache between test scenarios\n- [x] Add test for incorrect Ryvr name (404)\n- [x] Add test for deleted Ryvr (404)\n- [x] Add test for getting a RyvrsCollection as HTML\n- [x] Add test with negative page number (404)\n- [x] Add authentication\n- [ ] Decompose UI tests so they can be run on CircleCI. Currently they intermittently fail because the connection to saucelabs drops out.\n- [ ] Swtich to using a proper load generation framework for perf testing\n- [ ] Add test with different SQL types\n- [ ] Add test with different characters that require JSON escaping\n- [ ] Test with non-self-signed certificates, because [Chrome doesn't cache when using self-signed certs](https://www.sitepoint.com/solve-caching-conundrums/)\n- [ ] refactor remaining endpoints to use link headers and remove hal\n- [ ] remove dead code\n- [ ] Stablise API\n- [ ] Fix API Docs\n- [ ] Performance test with real world dataset (e.g., https://support.spatialkey.com/spatialkey-sample-csv-data/ and/or http://lisp.vse.cz/pkdd99/berka.htm, http://sorry.vse.cz/~berka/challenge/pkdd1999/data_berka.zip, https://www.kaggle.com/dalpozz/creditcardfraud, https://www.dunnhumby.com/sourcefiles)\n- [ ] release as zip/tarball with example properties\n- [ ] Modify Perf test to use latency per event rather than latency per page\n- [ ] Compare performance to [Kafka](https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines)\n- [ ] Add Circuit Breaker\n- [ ] Add logic to calculate optimal page size\n- [ ] Switch to async io\n- [ ] Add support for more databases\n- [ ] Add logic to create triggers or use [change data capture|https://github.com/shyiko/mysql-binlog-connector-java] to allow ryvrs from non-event based tables\n- [ ] Add support for more non-DB datasources\n- [ ] Add client library\n- [ ] decouple rest ryvr page size and UI page size\n- [ ] Add change notification service\n- [ ] Add tests for unauthenticated\n- [ ] Add tests for logout\n\n## Model\n\nApplication\n-\u003e API Docs\n-\u003e Rvyrs Map\n---\u003e name, Ryvr\n\nRyvr can be\n-\u003e MySQL Ryvr\n-\u003e Rest Ryvr\n-\u003e HTML Ryvr (for testing)\n\n?? Do we need Ryvr and Source to be different types? or can we just have the Sources being instances of Ryvrs?\n\n## Stream\n\nThoughts\n\n- Get a stream from the mysql query\n- Use a csv transformer to convert each row\n- Use a custom writeable stream, which only write's out the page. When \\_write() stops calling callback(), it stops signalling to the incoming stream that it ready to receive.\n\nSee https://blog.developer.bazaarvoice.com/2017/11/01/telling-your-data-to-back-off-or-how-to-effectively-use-streams/\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmountain-pass%2Fryvr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmountain-pass%2Fryvr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmountain-pass%2Fryvr/lists"}