{"id":13503466,"url":"https://github.com/blockwatch-cc/tzindex","last_synced_at":"2025-03-29T18:31:19.788Z","repository":{"id":41061969,"uuid":"215060548","full_name":"blockwatch-cc/tzindex","owner":"blockwatch-cc","description":"Tezos Blockchain Indexer","archived":true,"fork":false,"pushed_at":"2024-03-29T12:14:59.000Z","size":2595,"stargazers_count":71,"open_issues_count":4,"forks_count":12,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-01T00:31:41.035Z","etag":null,"topics":["blockchain","indexer","tezos","web3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blockwatch-cc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-14T14:07:02.000Z","updated_at":"2024-07-06T19:45:52.000Z","dependencies_parsed_at":"2024-02-07T08:28:08.735Z","dependency_job_id":"c67f5dfd-a458-4df8-b5be-3a87214c55a8","html_url":"https://github.com/blockwatch-cc/tzindex","commit_stats":null,"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockwatch-cc%2Ftzindex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockwatch-cc%2Ftzindex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockwatch-cc%2Ftzindex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blockwatch-cc%2Ftzindex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blockwatch-cc","download_url":"https://codeload.github.com/blockwatch-cc/tzindex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246226970,"owners_count":20743862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain","indexer","tezos","web3"],"created_at":"2024-07-31T23:00:36.373Z","updated_at":"2025-03-29T18:31:19.112Z","avatar_url":"https://github.com/blockwatch-cc.png","language":"Go","funding_links":[],"categories":["Go","Projects"],"sub_categories":["Indexer"],"readme":"# Blockwatch Tezos Indexer\n\n© 2020-2024 Blockwatch Data Inc., All rights reserved.\n\nTzIndex is [Blockwatch](https://blockwatch.cc)'s all-in-one zero-conf blockchain indexer for Tezos. A fast, convenient and resource-friendly way to gain tactical insights and build dapps on top of Tezos. Pro version available on request, see [License](https://github.com/blockwatch-cc/tzindex#license).\n\nFor support, talk to us on [Discord](https://discord.gg/D5e98Hw) or [Twitter](https://twitter.com/blockwatch_data).\n\n**Core Features**\n\n- supports protocols up to Oxford (v018)\n- indexes and cross-checks full on-chain state\n- feature-rich [REST API](https://docs.tzpro.io/docs/api/index) with objects, bulk tables and time-series\n- auto-detects and locks Tezos network (never mixes data from different networks)\n- indexes all accounts and smart-contracts (including genesis data)\n- follows chain reorgs as they are resolved\n- can passively monitor for new blocks\n- self-heals broken node connections (retries until node RPC comes back up)\n- API supports CORS and HTTP caching\n- high-performance embedded data-store\n- flexible in-memory caching for fast queries\n- automatic database backups/snapshots\n- configurable HTTP request rate-limiter\n- flexible metadata support\n- pruning of unused historic snapshots\n- configurable indexing delay to avoid reorgs and serve finalized data only\n- decodes on-chain (tezos domains reverse records) and off-chain (tezos profiles) account metadata\n- identifies and decodes mint/burn/transfer of a broad range of FA tokens\n\n**Supported indexes and data tables**\n\n- **blocks**: all blocks including orphans, indexed by hash and height\n- **operations**: all on-chain operations including contract call data, indexed by hash ( endorsements are split into separate table, only available in full mode)\n- **accounts**: running account details like most recent balances, indexed by address\n- **balances**: end-of-block balance history for all addresses\n- **contracts**: running smart contract details, code and initial storage\n- **flows**: complete list of balance, freezer and delegation balance updates\n- **chain**: running blockchain totals\n- **supply**: running supply totals\n- **bigmap**: bigmap smart contract storage index\n- **rights**: compact representations of assigned and used baking and endorsing rights\n- **elections**, **votes**, **proposals** and **ballots** capturing all on-chain governance activities\n- **snapshots**: balances of active delegates \u0026 delegators at all snapshot blocks\n- baker **income**: per-cycle statistics on baker income, efficiency, etc\n- **metadata**: standardized and custom account metadata (tezos domains reverse record and spruce tzprofile support)\n- **constants**: global constants (e.g. smart contract code/type macros to lower contract size and reuse common features)\n- **storage**: separate smart contract storage updates to decrease operation table cache pressure\n- **event**: emitted smart contract events\n- **tickets**: ticket index including updates, events, types, owners and statistics\n- **cycle**: per-cycle statistics\n- **token**: FA token index including events, identity, metadata, owners and statistics\n\n**Experimental Features**\n\nSome tzindex features are considered experimental such as\n\n* tezos domains \u0026 tzprofiles metadata indexing\n* FA \u0026 NFT token indexing\n\nThese features are generally stable, but generate extra load during indexing because data is fetched from off-chain APIs. If you really need this data, run tzindex with the `-experimental` flag. Read also the comments on off-chain data below.\n\n**Operation modes**\n\n- **Light** (default) light-weight mode without consensus and governance indexes (CLI: `-light`)\n- **Full** regular operation mode that builds all indexes (CLI: `-full`)\n- **Validate** state validation mode for checking accounts and balances each block/cycle (CLI: `-validate`)\n- **Experimental** enable experimental features (CLI: `-experimental`)\n\n**Light mode** dramatically reduces our maintenance costs for TzIndex and is best suited for dapps where access to baking-related data is not necessary. Light mode saves roughly \\~50% storage costs and \\~50% indexing time while still keeping all data required for Dapps.\n\n**Validate mode** works in combination with full and light mode. At each block it checks balances and states of all touched accounts against a Tezos archive node before any change is written to the database. At the end of each cycle, all known accounts in the indexer database are checked as well. This ensures 100% consistency although at the cost of a reduction in indexing speed.\n\n\n### Requirements\n\n- Storage: 43GB (full Mainnet index, May 2023), 29G (light mode)\n- RAM:  8-64GB (configurable, use more memory for better query latency)\n- CPU:  2+ cores (configurable, use more for better query parallelism)\n- Tezos node in archive mode\n\nRuns against any Tezos Archive Node (also full nodes when cycle 0 history is not yet pruned). This can be a local node or one of the public Tezos RPC nodes on the Internet. Note that syncing from public nodes over the Internet works but may be slow.\n\n**IMPORTANT: WHEN USING OCTEZ V12+ YOU MUST RUN YOUR ARCHIVE NODE WITH `--metadata-size-limit unlimited`**\n\nRequires access to the following Tezos RPC calls\n\n```\n/chains/main/blocks/{blockid}\n/chains/main/blocks/{blockid}/helpers/baking_rights (full mode only)\n/chains/main/blocks/{blockid}/helpers/endorsing_rights (full mode only)\n/chains/main/blocks/{blockid}/context/selected_snapshot (Ithaca+ full mode only)\n/chains/main/blocks/{blockid}/context/raw/json/cycle/{cycle} (full mode only)\n/chains/main/blocks/{blockid}/context/constants\n/chains/main/blocks/head/header\n/monitor/heads/main (optional)\n/chains/main/blocks/{blockid}/context/contracts/{address}/script\n/chains/main/blocks/{blockid}/context/contracts/{address} (validate mode only)\n/chains/main/blocks/{blockid}/context/delegates/{address} (validate mode only)\n```\n\n### Off-chain Data\n\n\u003e With default settings you need a TzPro API key and a MAX subscription to be able to send 9M+ API calls for a full reindex.\n\nStarting with v18 TzIndex decodes FA tokens and Tezos Profiles. Both rely on off-chain data that cannot be fetched from a Tezos archive node. Tezos Profiles uses the Kepler protocol from Spruce Inc which hosts profile claims on a dedicated server which is rate limited. Some tokens store their metadata on-chain (many currency tokens do that) but most NFTs just store a url link on-chain. Metadata resolution and download are complex and often fragile because calls have to be made to reliable IPFS nodes but also many hosted servers which can go offline at any time. For best performance and user experience TzIndex pulls all token and profile metadata from the TzPro API as a default. You can change the source of metadata downloads in configuration.\n\n```\n# TzProfiles original server\n-meta.kepler.url=https://kepler.tzprofiles.com\n\n# TzProfiles Mirror on TzPro\n-meta.kepler.url=https://api.tzpro.io/v1/profiles/claim\n\n# Token Metadata on TzPro\n-meta.token.url=https://api.tzpro.io/v1/tokens/{addr}/meta\n\n# Set your TzPro API key via env TZPRO_API_KEY or a CLI\n-meta.http.api_key=your_key_here\n```\n\nYouy can also control how retries and timeouts are handled\n\n```\n-meta.max_tasks=128                 # max concurrent tasks\n-meta.http.rate_limit=50            # max API calls per second\n-meta.http.max_retries=10           # max number of retries when offline or 404\n-meta.http.retry_delay=1m           # time between retries\n-meta.http.retry_interval=1s        # extra delay to reschedule more tasks\n```\n\nNote that there are currently 8M+ NFTs and FA tokens with metadata and 56k+ profiles with 150k+ claims. It takes a substantial amount of additional time to sync this metadata depending on your settings.\n\n### How to build\n\nThe contained Makefile supports local and Docker builds. For local builds you need a recent Golang distribution installed on your system.\n\n```\n# build a binary for your OS\nmake build\n\n# or directly with Go\ngo build ./cmd/tzindex.go\n\n# build docker images\nmake image\n```\n\n### How to run\n\ntzindex aims to be zero-conf and comes with sane defaults. All you need to do is point it to the RPC endpoint of a running Tezos node.\n\n```\ntzindex -rpc.url tezos-node\n```\n\nIf you prefer running from docker, check out the docker directory. Official images are available for the [indexer](https://hub.docker.com/r/blockwatch/tzindex) and [frontend](https://hub.docker.com/r/blockwatch/tzstats) (note the frontend may not support advanced features of new protocols). You can run both, the indexer and the frontend in local Docker containers and have them connect to your Tezos node in a third container. Make sure all containers are connected to the same Docker network or if you choose different networks that they are known. Docker port forwarding on Linux usually works, on OSX its broken.\n\n\n### Configuration\n\n**Config file**\n\nOn start-up tzindex tries loading its config from the `config.json` file in the current directory. You may override this name either on the command line using `-c myconf.json` or by setting the environment variable `TZ_CONFIG_FILE=/some/path/myconf.json`\n\nConfig file sections\n```\nrpc      - configures RPC connection to the Tezos node\ncrawler  - configures the blockchain crawl logic\ndb       - configures the embedded database\nserver   - configures the built-in HTTP API server\nlog      - configures logging for all subsystems\n```\n\nSee the default `config.json` in the `docker` subfolder for a detailed list of all settings.\n\n**Environment variables**\n\nEnv variables allow you to override settings from the config file or even specify all configuration settings in the process environment. This makes it easy to manage configuration in Docker and friends. Env variables are all uppercase, start with `TZ` and use an underscore `_` as separator between sub-topics.\n\n```\n# in config.json\n{ \"rpc\": { \"url\" : \"http://127.0.0.1:8732\" }}\n\n# same as env variable\nTZ_RPC_URL=http://127.0.0.1:8732\n```\n\n**Command line arguments**\n\nAll TzIndex config options can be controlled via cli arguments.\n\nGlobal options:\n\n```\nUsage: tzindex [flags]\n\nFlags\n  -c file\n      read config from file (default \"config.json\")\n  -config file\n      read config from file (default \"config.json\")\n  -enable-cors\n      enable API CORS support\n  -full\n      full mode (including baker and gov data)\n  -insecure\n      disable RPC TLS certificate checks (not recommended)\n  -light\n      light mode (use to skip baker and gov data) (default true)\n  -noapi\n      disable API server\n  -noindex\n      disable indexing\n  -nomonitor\n      disable block monitor\n  -norpc\n      disable RPC client\n  -notls\n      disable RPC TLS support (use http)\n  -stop height\n      stop indexing after height\n  -v  be verbose\n  -validate\n      validate account balances\n  -version\n      show version\n  -vv\n      debug mode\n  -vvv\n      trace mode\n```\n\nSubsystem Options\n\n```\nDatabase\n  -db.path=./db             path for database storage\n  -db.log_slow_queries=1s   warn when DB queries take longer than this\n  -db.max_storage_entry_size=131072  max size limit for storing contract storage updates\n\nGo runtime\n  -go.cpu=0            max number of CPU cores to use (0 = all)\n  -go.gc=20            trigger GC when used mem grows by N percent\n  -go.sample_rate=0    block and mutex profiling sample rate (0 = off)\n\nCrawler\n  -crawler.cache_size_log2=15                max number of cached accounts when crawling\n  -crawler.queue=100                         max number of blocks to prefetch\n  -crawler.delay=1                           offset from chain head (use 1 or 2 for reorg safe indexing)\n  -crawler.snapshot.path=./db/snapshot       target path for indexer database snapshots\n  -crawler.snapshot.blocks=height1,height2   target blocks to create snapshots\n  -crawler.snapshot.interval=0               interval between blocks to create snapshots\n\nServer\n  -server.addr=127.0.0.1            server listen address\n  -server.port=8000                 server listen port\n  -server.workers=64                number of Goroutines for executing API queries\n  -server.queue=128                 number of open requests to queue for execution\n  -server.read_timeout=5s           max timeout for receiving complete requests\n  -server.header_timeout=2s         max timeout for receiving request headers\n  -server.write_timeout=90s         max timeout for sending replies\n  -server.keepalive=90s             connection keep alive time\n  -server.shutdown_timeout=60s      delay to wait for draining open requests on shutdown\n  -server.max_list_count=500000     max number of result rows for table queries\n  -server.default_list_count=500    default number of result rows for table queries\n  -server.max_series_duration=0     max time-series duration per request\n  -server.max_explore_count=100     max number or explorer API results in lists\n  -server.default_explore_count=20  default number of results in explorer API lists\n  -server.cors_enable=false         add CORS response headers\n  -server.cors_origin=*             CORS origin header contents\n  -server.cors_allow_headers=       CORS allow header contents\n  -server.cors_expose_headers=      CORS expose header contents\n  -server.cors_methods=             CORS methods header contents\n  -server.cors_maxage=              CORS maxage header contents\n  -server.cors_credentials=         CORS credentials header contents\n  -server.cache_control=public      cache control header contents\n  -server.cache_expires=30s         default cache expiry time for mutable API responses\n  -server.cache_max=24h             max cache expiry time for immutable API responses\n\nRPC\n  -rpc.url=http://127.0.0.1:8732    Tezos RPC host\n  -rpc.disable_tls=true             use HTTP by default\n  -rpc.insecure_tls=false           disable TLS certificate checks\n  -rpc.proxy=                       set HTTP proxy\n  -rpc.proxy_user=                  optional HTTP proxy username\n  -rpc.proxy_pass=                  optional HTTP proxy password\n  -rpc.dial_timeout=10s             max connection delay before failing\n  -rpc.keepalive=30m                connection keep alive\n  -rpc.idle_timeout=30m             close connections when idle for longer\n  -rpc.response_timeout=60m         max delay waiting for RPC responses\n  -rpc.continue_timeout=60s         max delay waiting for HTTP chunks\n  -rpc.idle_conns=16                max server connections\n\nLogging\n  -log.progress=10s                 interval for progress logs\n  -log.backend=stdout               log backend (stdout, stderr, syslog, file)\n  -log.flags= date,time,micro,utc   log flags (see Golang log package)\n  -log.level=info                   default global log level\n  -log.etl=info                     log level for chain crawler\n  -log.db=info                      log level for database layer\n  -log.rpc=info                     log level for RPC layer\n  -log.api=info                     log level for API server\n  -log.micheline=info               log level for TzGo micheline package\n```\n\n### License\n\nThis Software is available under two different licenses, the open-source **MIT** license with limited support / best-effort updates and a **PRO** license with professional support and scheduled updates. The professional license is meant for businesses such as dapps, marketplaces, staking services, wallet providers, exchanges, asset issuers, and auditors who would like to use this software for their internal operations or bundle it with their commercial services.\n\nThe PRO licenses helps us pay for software updates and maintenance and operate the free community services on [TzStats](https://tzstats.com).\n\nThe following table may help you pick the right license for your intended use-case. If in doubt, send an email to license@blockwatch.cc and we'll get back to you.\n\n\n| | MIT | PRO |\n|-|---------------|----------------|\n| Costs | Free | Subscription |\n| Type | Perpetual | Perpetual |\n| Use | Any | Commercial Use |\n| Mode | All | All |\n| Limitations | - | See Agreement |\n| Support | Best effort | Commercial Support Available |\n| Upgrades | Best-effort | Early |\n| Health check | No | Available |\n| API statistics | No | Available |\n| ZMQ | No | Available |\n| RPC Proxy | No | Available |\n| QA Tools | No | Available |\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockwatch-cc%2Ftzindex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockwatch-cc%2Ftzindex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockwatch-cc%2Ftzindex/lists"}