https://github.com/wh1isper/redis-canal
Proxy redis stream from one to another through global queue service
https://github.com/wh1isper/redis-canal
proxy python redis
Last synced: about 1 month ago
JSON representation
Proxy redis stream from one to another through global queue service
- Host: GitHub
- URL: https://github.com/wh1isper/redis-canal
- Owner: Wh1isper
- License: bsd-3-clause
- Created: 2024-06-07T13:51:17.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-03-04T02:58:18.000Z (3 months ago)
- Last Synced: 2025-04-06T03:58:36.653Z (about 2 months ago)
- Topics: proxy, python, redis
- Language: Python
- Homepage:
- Size: 88.9 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
[](https://codecov.io/gh/Wh1isper/redis-canal)
# redis-canal
Proxy redis stream from one to another through global queue service.
Motivation: Redis provided on the cloud is usually only available within a VPC and does not provide external access. We want to provide a way to synchronize messages across clouds without going through a VPN.
## Supported queue service
- [x] AWS SQS
- [x] Google PubSub
Welcome to contribute more queue service, see [adapter/impl](./redis_canal/adapter/impl/) for more details.
We also support the plugin system, the adapter can also be imported as a third-party library. See [example](./example/extension/custom-adapter/) for more details.
## Install
`pip install redis-canal[all]` for all components.
Or use docker image
`docker pull wh1isper/redis-canal`
## Usage
Use CLI
- `redis-canal stream-to-queue` to start a daemon to consume redis stream and push to global queue
- `redis-canal queue-to-stream` to pull from global queue and push to redis stream
CLI args support `envvar`, which you can refer to [envs.py](./redis_canal/envs.py).
## TODO
- More adapter for Google PubSub, Kafka, etc
- Allows users to more easily configure the adapter(Now only support environment variables, within `Adapter`)
## Develop
Install pre-commit before commit
```
pip install pre-commit
pre-commit install
```
Install package locally
```
pip install -e .[test]
```
Run unit-test before PR
```
pytest -v tests
```
### Develop sqs adapter
You need to configure your AWS account first. See [AWS CLI docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) for more details.
Make sure your account has `sqs:*` permission. Then you can run `pytest -v tests/adapter/test_sqs_adapter.py` to test it.