{"id":28380699,"url":"https://github.com/0xsequence/ethwal","last_synced_at":"2026-02-12T15:31:34.489Z","repository":{"id":244251433,"uuid":"810342744","full_name":"0xsequence/ethwal","owner":"0xsequence","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-11T14:17:17.000Z","size":12969,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":11,"default_branch":"master","last_synced_at":"2026-02-11T22:23:36.968Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0xsequence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-06-04T14:04:23.000Z","updated_at":"2026-02-11T14:17:20.000Z","dependencies_parsed_at":"2024-06-27T17:30:02.179Z","dependency_job_id":"1f9f2758-b6b1-4233-b395-bf17ecd4356c","html_url":"https://github.com/0xsequence/ethwal","commit_stats":null,"previous_names":["0xsequence/ethwal"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/0xsequence/ethwal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xsequence%2Fethwal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xsequence%2Fethwal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xsequence%2Fethwal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xsequence%2Fethwal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0xsequence","download_url":"https://codeload.github.com/0xsequence/ethwal/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xsequence%2Fethwal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29370546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T08:51:36.827Z","status":"ssl_error","status_checked_at":"2026-02-12T08:51:26.849Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-30T03:09:03.891Z","updated_at":"2026-02-12T15:31:34.473Z","avatar_url":"https://github.com/0xsequence.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ethwal\n\nA simple library for creating Ethereum based flat file datasets.\n\n## Usage\n\nThe library consists of two main components, `Writer` and `Reader`. Each reader and writer needs to specify dataset name and path. The name is used to identify the dataset, and the path is used to specify the directory where the dataset is stored.\nThey may use ``json`` or ``cbor`` encoding for the data as well as ``zstd`` compression. The other custom encoders and compressors can be added by implementing the `ethwal.Encoder` and `ethwal.Compressor` interfaces.\n\nThe writer supports file roll over strategies such as: rolling over every Nth block, or after writing a certain amount of data.\n\n### Writer\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/0xsequence/ethwal\"\n\t\"github.com/ethereum/go-ethereum/common\"\n\t\"github.com/ethereum/go-ethereum/core/types\"\n)\n\nfunc main() {\n\tw, err := ethwal.NewWriter[[]types.Log](ethwal.Options{\n\t\tDataset: ethwal.Dataset{\n\t\t\tName: \"event-logs\",\n\t\t\tPath: \"data\",\n\t\t},\n\t\tFileRollPolicy: ethwal.NewFileSizeRollPolicy(128 \u003c\u003c 10), /* 128 KB */\n\t})\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\tfor i := 0; i \u003c 1000; i++ {\n\t\terr = w.Write(ethwal.Block[[]types.Log]{\n\t\t\tNumber: uint64(i),\n\t\t\tHash:   \"0x123\",\n\t\t\tData: []types.Log{\n\t\t\t\t{\n\t\t\t\t\tAddress: common.HexToAddress(\"0x123\"),\n\t\t\t\t\tTopics: []common.Hash{\n\t\t\t\t\t\tcommon.HexToHash(\"0x123\"),\n\t\t\t\t\t},\n\t\t\t\t\tData: []byte(\"0x123\"),\n\t\t\t\t},\n\t\t\t},\n\t\t})\n\t\tif err != nil {\n\t\t\tfmt.Println(err)\n\t\t\treturn\n\t\t}\n\t}\n\n\t// Close the writer\n\tw.Close()\n}\n}\n```\n\n### Reader\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\n\t\"github.com/0xsequence/ethwal\"\n\t\"github.com/ethereum/go-ethereum/core/types\"\n)\n\nfunc main() {\n\tr, err := ethwal.NewReader[[]types.Log](ethwal.Options{\n\t\tDataset: ethwal.Dataset{\n\t\t\tName: \"event-logs\",\n\t\t\tPath: \"data\",\n\t\t},\n\t})\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\t// Read all the blocks\n\tvar b ethwal.Block[[]types.Log]\n\tfor b, err = r.Read(); err == nil; b, err = r.Read() {\n\t\tfmt.Println(b.Number)\n\t\tfmt.Println(b.Hash)\n\t\tfmt.Println(b.Data)\n\t}\n\n\tif err != nil \u0026\u0026 err != io.EOF {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\t// Close the reader\n\tr.Close()\n}\n```\n\n## CLI examples\n\n### Read ethwal from local fs\n```bash\n$ ./ethwalcat --mode=read --path=./../indexer-data/db-logwal-new/137/v3/ --from=20000001 --to=20000005 --decompressor=zstd\n{\"blockHash\":\"0x90220f1f2d13248bef5ed31739b3625cb3696061ce891070ee7768dd6f94474f\",\"blockNum\":20000001,\"blockTS\":1633732467,\"blockData\":null}\n{\"blockHash\":\"0xb1db1d89202cee5cbc99f537ee0107e0d175d7f458e5c566657b644a3b205843\",\"blockNum\":20000002,\"blockTS\":1633732469,\"blockData\":null}\n{\"blockHash\":\"0x1c475400153bcfb8c1558cadc4fa94ed00ba28282fd63f13e0193e7a5e651d20\",\"blockNum\":20000003,\"blockTS\":1633732471,\"blockData\":null}\n{\"blockHash\":\"0xed17a5cfa53dffe8489c2f0dbea7d64cf732b3ef1d235ba3438a41154935b110\",\"blockNum\":20000004,\"blockTS\":1633732473,\"blockData\":null}\n```\n\n### Transcode ethwal from local cbor zstd to local json not compressed\n```bash\n./ethwalcat --mode=read --path=./../indexer-data/db-logwal-new/137/v3/ --from=20000001 --to=20000005 --decompressor=zstd | ./ethwalcat --mode=write --path=./ --encoder=json --compressor=none\n```\n\n### Read transcoded ethwal\n```bash\n$ ./ethwalcat --mode=read --path=./ --decoder=json --decompressor=none\n{\"blockHash\":\"0x90220f1f2d13248bef5ed31739b3625cb3696061ce891070ee7768dd6f94474f\",\"blockNum\":20000001,\"blockTS\":1633732467,\"blockData\":null}\n{\"blockHash\":\"0xb1db1d89202cee5cbc99f537ee0107e0d175d7f458e5c566657b644a3b205843\",\"blockNum\":20000002,\"blockTS\":1633732469,\"blockData\":null}\n{\"blockHash\":\"0x1c475400153bcfb8c1558cadc4fa94ed00ba28282fd63f13e0193e7a5e651d20\",\"blockNum\":20000003,\"blockTS\":1633732471,\"blockData\":null}\n{\"blockHash\":\"0xed17a5cfa53dffe8489c2f0dbea7d64cf732b3ef1d235ba3438a41154935b110\",\"blockNum\":20000004,\"blockTS\":1633732473,\"blockData\":null}\n```\n\n### Read ethwal from Google Cloud Bucket\n```bash\n$ ./ethwalcat --mode=read --google-cloud-bucket=sequence-dev-cluster-indexer-wal --path=./polygon-db-logwal/137/v2 --decompressor=zstd --from=1455120 --to=1455130\n{\"blockHash\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"blockNum\":1455120,\"blockTS\":0,\"blockData\":null}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xsequence%2Fethwal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0xsequence%2Fethwal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xsequence%2Fethwal/lists"}