{"id":13687502,"url":"https://github.com/deepeth/mars","last_synced_at":"2025-05-01T12:34:18.884Z","repository":{"id":42328361,"uuid":"510565121","full_name":"deepeth/mars","owner":"deepeth","description":"The powerful analysis platform to explore and visualize data from blockchain.","archived":false,"fork":false,"pushed_at":"2023-06-20T16:15:31.000Z","size":5309,"stargazers_count":114,"open_issues_count":6,"forks_count":18,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-08-03T15:06:33.027Z","etag":null,"topics":["bitcoin","blockchain","ethereum","etl","rust","schema","web3"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepeth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-07-05T02:44:10.000Z","updated_at":"2024-07-22T03:18:04.000Z","dependencies_parsed_at":"2024-01-14T15:23:27.153Z","dependency_job_id":"bdec0111-448c-43db-be36-482f9b57c394","html_url":"https://github.com/deepeth/mars","commit_stats":null,"previous_names":[],"tags_count":323,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepeth%2Fmars","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepeth%2Fmars/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepeth%2Fmars/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepeth%2Fmars/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepeth","download_url":"https://codeload.github.com/deepeth/mars/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224257734,"owners_count":17281767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","blockchain","ethereum","etl","rust","schema","web3"],"created_at":"2024-08-02T15:00:55.701Z","updated_at":"2025-05-01T12:34:18.861Z","avatar_url":"https://github.com/deepeth.png","language":"Rust","funding_links":[],"categories":["Rust","ethereum"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eMars: The Powerful Analysis Platform to Explore and Visualize Data From Web3\u003c/h1\u003e\n\n## Features\n\n- **Blazing Fast:** Mars is built with Rust, which makes it incredibly fast and efficient, ensuring that you can work with your data in real-time.\n\n- **Pipeline Processor:** Mars offers a powerful pipeline processor that allows you to export Ethereum chain data to structured data quickly and easily.\n\n- **Low Cost:** You can store structured data to AWS S3, Azure Blob, ensuring low cost and high availability.\n\n- **Easy to Use:** Web3 visualization and analysis at your fingertips.\n\n\n## How to Use\n\n### 1. Download the `ethetl` binary\n\nDownload the `ethetl` binary from [GitHub releases](https://github.com/deepeth/mars/tags).\n\n### 2. Configuration\n\nCopy the [sample config](https://github.com/deepeth/mars/blob/main/scripts/deploy/ethetl_config_spec.toml) to `mars.toml`:\n\n```toml\n[log]\nlevel = \"ERROR\"\ndir = \"_logs\"\n\n[export]\n# Exporter directory.\noutput_dir = \"pub\"\n\n# Storage config.\n[storage]\n# Fs| S3 | Azblob\ntype = \"S3\"\n\n# To use S3-compatible object storage, uncomment this block and set your values.\n[storage.s3]\n bucket = \"\u003cyour-bucket-name\u003e\"\n access_key_id = \"\u003cyour-key-id\u003e\"\n secret_access_key = \"\u003cyour-account-key\u003e\"\n```\nNote that the data will be stored to `/\u003cyour-bucket-name\u003e/pub` in your S3 location.\n\n### 3. Export Data from the Ethereum Chain by Mars\n\nOnce you have configured Mars, you can start exporting data from the Ethereum chain:\n```shell\n./ethetl  -p \u003cyour-eth-node-endpoint-url\u003e -s 16600001 -e 16600002 -c ./mars.toml\n\n... ...\n\n[2023-02-27T08:54:07Z INFO ] block 2 processed/2, latest block 16600002, 292 transactions processed, 292 receipts processed, 658 logs processed, 329 token_transfers processed, 1 ens processed. Progress is 100%\n```\n\nHere, we recommend you use a SaaS like [GetBlock](https://getblock.io/) for your `your-eth-node-endpoint-url`.\n\n### 4. Deploy Databend\n\nDatabend is the only warehouse supported by Mars, which has blazing performance and stores data to cloud-based object storage. \n\nThere are two choice for you:\n* Self-Deploy. See [How to deploy Databend](https://databend.rs/doc/deploy/deploying-databend)\n* Cloud. Use https://app.databend.com\n\n### 5. Create Table\n\nYou can find the schema files for Databend in the [schemas/databend](schemas/databend/1_schema.sql).\n\n### 6. Ingest Data into Databend\n\nIngesting data from S3 into Databend is straightforward. You can use Databend [COPY INTO](https://databend.rs/doc/sql-commands/dml/dml-copy-into-table) command to do that:\n\n```sql\n-- Create a external stage\n-- https://databend.rs/doc/sql-commands/ddl/stage/ddl-create-stage#externalstageparams\nCREATE STAGE eth_stage URL='s3://\u003cyour-s3-bucket\u003e/pub' CONNECTION = (ACCESS_KEY_ID = '\u003cyour-access-key-ID\u003e' SECRET_ACCESS_KEY = '\u003cyour-secret-access-key\u003e');\n\n-- Databend provides idempotency by keeping track of files that have already been processed for a default period of 7 days\n-- https://databend.rs/doc/sql-commands/dml/dml-copy-into-table#externalstage\nCOPY INTO blocks FROM @eth_stage/blocks/ PATTERN = '*.*parquet' FILE_FORMAT = (type = 'PARQUET');\nCOPY INTO transactions FROM @eth_stage/transactions/ PATTERN = '*.*parquet' FILE_FORMAT = (type = 'PARQUET');\nCOPY INTO receipts FROM @eth_stage/receipts/ PATTERN = '*.*parquet' FILE_FORMAT = (type = 'PARQUET');\nCOPY INTO token_transfers FROM @eth_stage/token_transfers/ PATTERN = '*.*parquet' FILE_FORMAT = (type = 'PARQUET');\nCOPY INTO logs FROM @eth_stage/logs/ PATTERN = '*.*parquet' FILE_FORMAT = (type = 'PARQUET');\nCOPY INTO ens FROM @eth_stage/ens/ PATTERN = '*.*parquet' FILE_FORMAT = (type = 'PARQUET');\n```\n\n## License\n\nMars is licensed under [Apache 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepeth%2Fmars","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepeth%2Fmars","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepeth%2Fmars/lists"}