{"id":13504505,"url":"https://github.com/Deliquified/lsp-token-indexer","last_synced_at":"2025-03-29T22:30:51.063Z","repository":{"id":236075066,"uuid":"791506665","full_name":"Deliquified/lsp-token-indexer","owner":"Deliquified","description":"Indexer for retrieving LSP7 Token, LSP7 NFT and LSP8 NFT token metadata on the Lukso chain.","archived":false,"fork":false,"pushed_at":"2024-05-03T20:28:52.000Z","size":62,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-11-01T02:33:54.116Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deliquified.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-24T20:55:38.000Z","updated_at":"2024-08-28T20:06:59.000Z","dependencies_parsed_at":"2024-11-01T02:30:22.838Z","dependency_job_id":"b73c1665-d7a8-4546-ad09-614e491c4594","html_url":"https://github.com/Deliquified/lsp-token-indexer","commit_stats":null,"previous_names":["deliquified/lsp-token-indexer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deliquified%2Flsp-token-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deliquified%2Flsp-token-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deliquified%2Flsp-token-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deliquified%2Flsp-token-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deliquified","download_url":"https://codeload.github.com/Deliquified/lsp-token-indexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246254077,"owners_count":20747946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T00:00:44.243Z","updated_at":"2025-03-29T22:30:50.806Z","avatar_url":"https://github.com/Deliquified.png","language":"JavaScript","funding_links":[],"categories":["💻 GitHub Repositories from the community"],"sub_categories":["Others (dApps, Backend, etc...)"],"readme":"# LSP Token \u0026 NFT Indexer Powered by Envio's Hypersync\n\n## Overview\nLSP Token Indexer was built by **Deliquified Labs** to index all LSP7 tokens, LSP7 NFTs and LSP8 NFTs on the Lukso blockchain.\nThe indexer utilizes Envio's Hypersync solution to retrieve on-chain data blazing fast (100x improvement compared to the Graph).\n\nAlong with Envio, we use:\n\n**Lukso API:** To fetch token contract metadata - name, symbol, supply, links etc. \u003cbr /\u003e\n\n**Pinata:** Fetch off-chain metadata like NFT attributes, image etc. based on token ID \u003cbr /\u003e\n\n**Prisma:** For storing the above in a database for later use\n\nResources: \u003cbr /\u003e\n\n[Envio](https://envio.dev/) \u003cbr /\u003e\n\n[Lukso API](https://docs.lukso.tech/tools/indexer) \u003cbr /\u003e\n\n[Pinata](https://www.pinata.cloud/) \u003cbr /\u003e\n\n[Prisma](https://www.prisma.io/)\n\n## Disclaimer\n\nThis is a great resource for anyone new to Web3 development or blockchain in general. Whether you're a project building on Lukso and you need to display token metadata or you're learning about how to index blockchain events and extract on-chain/off-chain metadata - this indexer is for you. \u003cbr /\u003e\n\nWe'll be improving the performance over time, adding new features, optimizing existing code, adding support for LSP23, EOA contract deployment etc. Other than that, if you're looking to iterate on the indexer and adjust it to your needs, feel free to do so.\n\n## Getting Started\n\n### Prerequisites\n\nBefore you begin, ensure you have Node.js installed on your system. You can download it from [Node.js official website](https://nodejs.org/en).\n\n### Cloning the Repository\n\nTo get started with the LSP Token \u0026 NFT Indexer, clone the repository to your local machine:\n\n```\ngit clone https://github.com/Deliquified/lsp-token-indexer\ncd hypersync-token-indexer\n```\n### Installation\n\nAfter cloning the repository, install the necessary dependencies by running:\n\n```\nnpm install\n```\n\n### Configuration\n\nCreate a .env file in the root directory of the project and populate it with the necessary environment variables:\n\n```\nNOWNODES_KEY=\nLUKSO_API_KEY=\nPINATA_PRIVATE_GATEWAY=\nPINATA_GATEWAY_KEY=\nDATABASE_API_ENDPOINT=\nDATABASE_API_ENDPOINT_KEY=\n```\n\nYou can get the API keys \u0026 gateway here:\n\n[NowNodes](https://nownodes.io/)\u003cbr /\u003e\n\n[Lukso API](https://docs.lukso.tech/tools/indexer)\u003cbr /\u003e\n\n[Pinata](https://www.pinata.cloud/)\n\nFor the database - we use our own API solution to store the data in our database and cache the data in our API for further use. If you plan on using your own solution for storing the indexed data, you can leave it empty and adjust the following function:\n\n```\n// Send indexed token metadata to backend\nasync function sendMetadataToAPI(metadata, idCidPairs, type, standard) {\n  console.log(`📡 Sending metadata and ID-CID pairs to the backend API...`);\n\n  const apiEndpoint = process.env.DATABASE_API_ENDPOINT;\n  const apiKey = process.env.DATABASE_API_ENDPOINT_KEY;\n  const payload = {\n      metadata,\n      idCidPairs,\n      type,\n      standard\n  };\n\n  try {\n    console.log(\"Payload\", payload)\n    await axios.post(apiEndpoint, payload, {\n      headers: {\n          'Content-Type': 'application/json',\n          'x-api-key': apiKey\n      }\n    });\n    \n\n    console.log('🎉 Metadata and ID-CID pairs sent to API successfully');\n  } catch (error) {\n      console.error('Failed to send metadata to API for following asset', payload.metadata, error);\n  }\n}\n```\n### Running the Indexer\n\nTo start the indexer, execute the following command:\n\n```\nnode index.js\n```\n \u003cbr /\u003e\nThis will start the process of indexing from block 0 to the latest block. The indexer will continuously listen for new blocks and index all the deployed LSP tokens. \u003cbr /\u003e\n\u003cbr /\u003e\nCurrently indexer doesn't support EOA contract deployment i.e. contracts deployed through EOA are not tracked \u0026 indexed. As a temporary work around we've added manual indexing via *processContractsManually()* function. We've added *BurntPix* and *Universal Page Name* contracts as default since they're both deployed via EOA.\n\n## Technical Deep Dive\n\nThe indexer is retrieving all transactions that have the Keymanager (LSP6) \u0026 Universal Factory (LSP16) topics:\n\n```\nconst LSP6EventTopic = '0xa1fb700aaee2ae4a2ff6f91ce7eba292f89c2f5488b8ec4c5c5c8150692595c3';\nconst LSP16EventTopic = '0x8872a323d65599f01bf90dc61c94b4e0cc8e2347d6af4122fccc3e112ee34a84';\n```\n\n*These topics represent events emitted when a contract is deployed through Universal Profile or Universal Factory*\n\nOn Lukso, usually tokens are deployed through Universal Profiles. New projects launching on Lukso create their on-chain brand identity first - by deploying a UP - and through the UP are the tokens then created.\nSometimes brands utilize Universal Factory (LSP16) to deploy the token. This is to ensure that their tokens can be redeployed to Lukso L2s with the same address.\n\n\u003eTo learn more about how LSP16 works visit [the Lukso docs](https://docs.lukso.tech/standards/generic-standards/lsp16-universal-factory)\n\nLinked Contracts Factory (LSP23) can also be used to deploy a more complex token as it allows developers to deploy two inter-dependent contracts at the same time. But we don't track this yet as it's mostly used to deploy Universal Profiles, though it'll be added in the future.\n\n\u003eTo learn more about how LSP23 works visit [the Lukso docs](https://docs.lukso.tech/standards/generic-standards/lsp16-universal-factory)\n\nEnvio's Hypersync returns all the transactions with the above topics in batches from block X to block Y. We then process the contracts by:\n\n1. Checking if the deployed contract supports old LSP7 interface, new LSP7 interface or LSP8 interface. If one of them returns true, it means the deployed contract is an LSP token.\n2. After we've identified LSP token contracts within the given batch, we then fetch the metadata from the Lukso's API endpoint, if there's no metadata we create a file named after the contract address in ./NoMetadata folder, otherwise we continue processing the metadata.\n3. Once we've the metadata of the token, we determine its LSP standard and type. There are 3 token types on Lukso:\u003cbr /\u003e\n   \n   a) LSP7 token (divisible/fungible)\u003cbr /\u003e\n   \n   b) LSP7 NFT (non-divisible/non-fungible)\u003cbr /\u003e\n   \n   c) LSP8 NFT (non-divisible/non-fungible)\u003cbr /\u003e\n   \n   Since there can be LSP7 NFT and LSP8 NFT, we need to differentiate the two. For this, we check the appropriate metadata fields (tokenType, token, LSP4Standard...) to determine the standard (LSP7 or LSP8) and token type (token or NFT)\n4. If it is LSP7 token or NFT, we create a folder under \"./LSP7Divisible\" || \"./LSP7NonDivisible\" with contract address as name and save the metadata as a json file. If it is an LSP8 NFT, then we also want to get token ID \u003e CID mapping as well.\n5. In case of LSP8, we query Pinata with CID hash of the directory that holds references to all the token IDs with their CID hashes, extract and save it to a .json file. End result looks something like this (array of objects): \u003cbr /\u003e\n\n```\n[{\"id\":1,\"cid\":\"bafkreicpm5eeu6egsvnd3q3dl5m5nne25opnhuthjvna3r5aomcqsgwkky\"},\n{\"id\":2,\"cid\":\"bafkreievz477n6435xm7bwc5rgh6jo22ifg2dbbumfxuto4tbt4brypa7u\"},\n{\"id\":3,\"cid\":\"bafkreicsvapifvepqfquk5c2w3ipo4obcvsxby2zaui7j6oxv4eob5ubc4\"},\n{\"id\":4,\"cid\":\"bafkreighnerq42bel37isymal7npzwak7gqilkuahxpdof5jza336rtu4y\"},\n{\"id\":5,\"cid\":\"bafkreiea7nxo5avgshdhlqqkycvexhtrlp5upfbyn7wxbentmm6co2ox4u\"}]\n...\n```\n\nThe tokenID \u003e CID hash can be further processed with the following function:\n\n```\n// Fetches Metadata for individual Token IDs\n// Extracts IPFS Image hash and downloads it\nasync function fetchNFTTokenMetadata(id, cid, directory) {\n  try {\n      const url = `${process.env.PINATA_PRIVATE_GATEWAY}${cid}?${process.env.PINATA_GATEWAY_KEY}`;\n      const response = await fetch(url);\n      const jsonData = await response.json(); // Assuming the data is JSON\n\n      console.log(`Data for CID ${cid}:`, jsonData);\n\n      // Extracting metadata\n      const metadata = jsonData.LSP4Metadata;\n      console.log(`Metadata for CID ${cid}:`, metadata);\n\n      // Ensure the directory exists\n      if (!fs.existsSync(directory)) {\n          fs.mkdirSync(directory, { recursive: true });\n      }\n\n      // Saving the metadata to a JSON file within the specified directory\n      const filePath = path.join(directory, `${id}.json`);\n      fs.writeFileSync(filePath, JSON.stringify(metadata, null, 2));\n      console.log(`Metadata for CID ${cid} saved to ${filePath}`);\n      \n      // Handle image download if present\n      if (metadata.images \u0026\u0026 metadata.images.length \u003e 0 \u0026\u0026 metadata.images[0].length \u003e 0) {\n        const imageInfo = metadata.images[0][0];\n        if (imageInfo \u0026\u0026 imageInfo.url) {\n            const imagePath = path.join(directory, `${id}.png`);\n            await downloadImage(imageInfo.url, imagePath);\n        }\n      }\n\n      return metadata;\n  } catch (error) {\n      console.error(`Error fetching data for CID ${cid}:`, error);\n  }\n}\n```\n\nThe above function is commented out as you can see:\n\n```\n/****************** Process only the first 5 ID and CID pairs for testing purposes ******************/ \n\n    // Here we call fetchNFTTokenMetadata which fetches metadata for each tokenID and saves them under /nfts folder\n    /*for (let i = 0; i \u003c Math.min(5, idCidPairs.length); i++) {\n        const { id, cid } = idCidPairs[i];\n        console.log(`Fetching metadata for token ID ${id} with CID ${cid}...`);\n        await fetchNFTTokenMetadata(id, cid, nftDirectory);\n    }*/\n```\n\nIf you wish to go through the directory of an LSP8 NFT and fetch metadata \u0026 corresponding image of the token ID, you can uncomment the above and remove the limit. This will index not only the token metadata, the tokenID \u003e CID pairs but also individual token ID metadata and image.\n\n## Wrap up\n\nBig thanks to Envio! They've been assisting us with setting up the indexer throughout the entire process from start to finish. Majority of the time the indexer is fetching and processing off-chain metadata as opposed to on-chain transactions. This speaks volumes on how fast Envio's Hypersync is!\n\nIf you've any questions regarding the indexer or anything else, feel free to reach out to us on [Commonground](https://app.cg/c/JErL2vNVPh/channel/~13NTr3pydFawzCj45UmXGD/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeliquified%2Flsp-token-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDeliquified%2Flsp-token-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeliquified%2Flsp-token-indexer/lists"}