Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/41north/besu-exflo
A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores and processing pipelines.
https://github.com/41north/besu-exflo
besu crypto ethereum hyperledger hyperledger-besu kafka kotlin postgres
Last synced: 3 days ago
JSON representation
A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores and processing pipelines.
- Host: GitHub
- URL: https://github.com/41north/besu-exflo
- Owner: 41north
- License: apache-2.0
- Archived: true
- Created: 2020-01-30T19:05:08.000Z (almost 5 years ago)
- Default Branch: develop
- Last Pushed: 2022-06-09T12:12:04.000Z (over 2 years ago)
- Last Synced: 2024-11-02T16:05:53.560Z (15 days ago)
- Topics: besu, crypto, ethereum, hyperledger, hyperledger-besu, kafka, kotlin, postgres
- Language: Kotlin
- Homepage:
- Size: 3.67 MB
- Stars: 17
- Watchers: 3
- Forks: 7
- Open Issues: 16
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Codeowners: CODEOWNERS
Awesome Lists containing this project
- awesome-besu - Besu Exflo - A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores (Postgres) and processing pipelines (Kafka). (Plugins)
README
A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores and processing pipelines.
Written with ❤️ in Kotlin.
⚠️ Warning: This project is **DEPRECATED**, use it at your own risk!
## 🗒️ Features
Exflo can extract the following information from a Besu archive instance into either a [Postgres](https://www.postgresql.org/) database or a [Kafka](https://kafka.apache.org/) topic:
- Block headers.
- Transactions.
- Transaction traces.
- Log events for standards-compliant [ERC20](https://eips.ethereum.org/EIPS/eip-20), [ERC721](https://eips.ethereum.org/EIPS/eip-721), [ERC777](https://eips.ethereum.org/EIPS/eip-777) and [ERC1155](https://eips.ethereum.org/EIPS/eip-1155) tokens.
- A detailed breakdown of Ether movements e.g. block rewards, tx fees, simple ether transfers and so on.
- Contract creations and self-destructs.
- Per block account state changes.
Some screenshots of captured data:
| Plugin | Screenshots |
| :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Postgres |
|
| Kafka | 
|
## ⏲️ Try Exflo in 10 seconds
We offer two `docker-compose` files [ready to launch Exflo](docker/exflo/) configured for Ethereum's Ropsten network (and easily configurable to
other networks).
Postgres
```bash
docker-compose -f docker/exflo/docker-compose.postgres.yml up
```
Kafka
```bash
docker-compose -f docker/exflo/docker-compose.kafka.yml up
```
Wait for docker to properly initialize each service. Once everything is ready navigate to
[`http://localhost:8082`](http://localhost:8082), you will be greeted with either [`pgweb`](https://sosedoff.github.io/pgweb/) or
[`kafkahq`](https://akhq.io/).
## 🚆 Development quick start
Ensure you have the following programs installed in your system:
- Java 11 or higher ([AdoptOpenJDK](https://adoptopenjdk.net/), [Zulu Community](https://www.azul.com/products/zulu-community/) or [OpenJDK](https://openjdk.java.net/))
- [Docker](https://docs.docker.com/install/)
- [Docker Compose](https://docs.docker.com/compose/install/)
- [direnv](https://github.com/direnv/direnv/blob/master/docs/installation.md)
- (Optional but highly recommendable) [IntelliJ IDEA Community or Ultimate](https://www.jetbrains.com/)
Clone the repository:
```bash
git clone [email protected]:41north/exflo.git
```
Generate [Intellij's run configurations](https://www.jetbrains.com/help/idea/running-applications.html) (it will read from the file [`intellij-run-configs.yml`](intellij-run-configs.yml) and auto create XML configurations to ease common operations inside the IDE):
```bash
./gradlew generateIntellijRunConfigs
```
Decide which data store you want to run:
Postgres
Terminal:
```bash
docker-compose -f docker-compose.exflo-postgres.yml up
```
Intellij:
Kafka
Terminal:
```bash
docker-compose -f docker-compose.exflo-kafka.yml up
```
Intellij:
Wait for each docker service to initialize.
After that, you can start processing with Exflo:
Postgres
Terminal:
```bash
./gradlew runPostgres
```
Intellij:
Kafka
Terminal:
```bash
./gradlew runKafka
```
Intellij:
You can now check the logs or open your browser and navigate to [`http://localhost:8082`](http://localhost:8082)
and you will be greeted with either [`pgweb`](https://sosedoff.github.io/pgweb/) or [`kafkahq`](https://akhq.io/) respectively.
Exflo will start processing immediately!
## Usage with Besu
There are two possible ways of running Exflo with Besu.
### Bundled docker images
We have provide a docker image that contains a pre-packaged version of Besu to make your life easier. This is the recommended way of running Exflo as you don't need to worry about placing the plugin in the correct folder path. Below is an an example using `docker-compose`:
```yaml
version: "3.7"
services:
besu:
image: 41north/exflo:latest
environment:
BESU_LOGGING: INFO
BESU_NETWORK: ROPSTEN
BESU_SYNC_MODE: FULL
BESU_DATA_PATH: /opt/besu/data
BESU_PLUGIN_EXFLO_KAFKA_ENABLED: "false"
BESU_PLUGIN_EXFLO_POSTGRES_ENABLED: "true"
BESU_PLUGIN_EXFLO_POSTGRES_JDBC_URL: jdbc:postgresql://postgres/exflo?user=exflo&password=exflo
```
Have a look at the [usage section](.github/USAGE.md) for a full list of parameters.
### JAR
If you are already using Besu with other plugins it is possible to include Exflo as a jar. Below are instructions assuming you are running Besu within docker:
1. Go to [releases](https://github.com/41North/exflo/releases) and download the `tar` or `zip` file and extract it.
2. [Besu docker image](https://hub.docker.com/r/hyperledger/besu) relies upon a `/etc/besu/plugins` folder where it plugin jars. Ensure the Exflo jar is in this folder.
Here's an example of a possible docker configuration using `docker-compose` syntax and a bind mount for the plugin folder:
```yaml
version: "3.7"
services:
besu:
image: hyperledger/besu:1.4.4
volumes:
- ./path/to/exflo-jar/:/etc/besu/plugins
environment:
BESU_LOGGING: INFO
BESU_NETWORK: ROPSTEN
BESU_SYNC_MODE: FULL
BESU_DATA_PATH: /opt/besu/data
BESU_PLUGIN_EXFLO_KAFKA_ENABLED: "false"
BESU_PLUGIN_EXFLO_POSTGRES_ENABLED: "true"
BESU_PLUGIN_EXFLO_POSTGRES_JDBC_URL: jdbc:postgresql://postgres/exflo?user=exflo&password=exflo
```
Have a look at the [usage section](.github/USAGE.md) for more information.
## 💻 Contribute
We welcome any kind of contribution or support to this project but before to do so:
- Read our [development guide](/.github/DEVELOPMENT.md) to understand how to properly develop on the codebase.
- Make sure you have read the [contribution guide](/.github/CONTRIBUTING.md) for more details on how to submit a good PR (pull request).
In addition you can always:
- Add a [GitHub Star 🌟](https://github.com/41north/exflo/stargazers) to the project.
- ETH donations to [this address](https://etherscan.io/address/0xcee9ad6d00237e25A945D7ac2f7532C602d265Df)!
- Tweet about this project.
- Write a review or tutorial:
- [Exflo: A plugin for Hyperledger Besu](https://41north.dev/blog/open-source/exflo-a-plugin-for-hyperledger-besu/)
## ❔ FAQ
### Why Besu?
We chose Besu for several reasons:
1. It is enterprise ready.
2. It is written in Java, which allows us to work in Kotlin
3. It has a modern codebase with good patterns and documentation.
4. There is a plugin system allowing for customisation and integration without the need to maintain a fork.
5. Their [community is open and welcoming](https://chat.hyperledger.org/channel/besu) (we recommend you join!).
### Why not use the Web3 interface that every Ethereum client has?
If you have ever tried this, you will quickly realise that extracting even just the basic information from an Ethereum client via the Web3
interface requires a lot of requests, and some non-trivial logic to do well. On top of that, depending on the client
(we won't name anyone in particular), you may find that under heavy load, such as when syncing for the first time, your client may become
unstable and periodically core dump. Maximising throughput, whilst keeping the client happy, quickly becomes a tedious exercise.
Put simply it has been our experience that pulling via the Web3 interface is sub-optimal for a variety reasons which are better explored
in a blog post.
## 📬 Get in touch
`Exflo` has been developed initially by [°41North](https://41north.dev).
If you think this project would be useful for your use case and want to talk more about it you can reach out to us via our contact form
or by sending an email to `[email protected]`. We try to respond within 48 hours and look forward to hearing from you.
## ✍️ License
This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.