{"id":15060684,"url":"https://github.com/cata-network/cata_database","last_synced_at":"2025-10-10T23:36:42.193Z","repository":{"id":147544911,"uuid":"396218498","full_name":"cata-network/cata_database","owner":"cata-network","description":"CATA.Search. Blockchain database, cata metadata query","archived":false,"fork":false,"pushed_at":"2021-08-19T09:44:25.000Z","size":59,"stargazers_count":107,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-10T06:15:15.493Z","etag":null,"topics":["bigquery","blockchain","database","drill"],"latest_commit_sha":null,"homepage":"https://cata-network.gitbook.io/cata-docs/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cata-network.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-15T04:47:50.000Z","updated_at":"2025-01-17T16:07:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"471536cb-5dd4-4b8d-8881-55b9dbce9b46","html_url":"https://github.com/cata-network/cata_database","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cata-network/cata_database","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cata-network%2Fcata_database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cata-network%2Fcata_database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cata-network%2Fcata_database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cata-network%2Fcata_database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cata-network","download_url":"https://codeload.github.com/cata-network/cata_database/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cata-network%2Fcata_database/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279005572,"owners_count":26083919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","blockchain","database","drill"],"created_at":"2024-09-24T23:02:49.406Z","updated_at":"2025-10-10T23:36:42.167Z","avatar_url":"https://github.com/cata-network.png","language":"Java","funding_links":[],"categories":["数据库"],"sub_categories":["Spring Cloud框架"],"readme":"# Drill Storage Plugin for IPFS\n\n[中文](README.zh.md)\n\n## Contents\n\n0. [Introduction](#Introduction)\n1. [Compile](#Compile)\n2. [Install](#Install)\n2. [Configuration](#Configuration)\n3. [Run](#Run)\n\n## Introduction\n\nMinerva is a storage plugin of Drill that connects IPFS's decentralized storage and Drill's flexible query engine. Any data file stored on IPFS can be easily accessed from Drill's query interface, just like a file stored on a local disk. Moreover, with Drill's capability of distributed execution, other instances who are also running Minerva can help accelerate the execution: the data stays where it was, and the queries go to the most suitable nodes which stores the data locally and from there the operations can be performed most efficiently. \n\nSlides that explain our ideas and the technical details of Minerva: \u003chttps://www.slideshare.net/BowenDing4/minerva-ipfs-storage-plugin-for-ipfs\u003e\n\nA live demo: \u003chttp://www.datahub.pub/\u003e hosted on a private cluster of Minerva.\n\nNote that it's still in early stages of development and the overall stability and performance is not satisfactory. PRs are very much welcome!\n\n## Compile\n\n### Dependencies\n\nThis project depends on forks of the following projects:\n\n* IPFS Java API： [java-ipfs-api](https://github.com/bdchain/java-ipfs-api)\n\n* Drill 1.16.0：[Drill-fork](https://github.com/bdchain/Drill-fork) （`1.16.0-fork` branch）\n\nPlease clone and build these projects locally, or the compiler will complain about unknown symbols when you compile this project.\n\n### Compile under the Drill source tree\n\nClone to the `contrib` directory in Drill source tree, e.g. `contrib/storage-ipfs`:\n```\ncd drill/contrib/\ngit clone https://github.com/bdchain/Minerva.git storage-ipfs\n```\n\nEdit the parent POM of Drill contrib module (contrib/pom.xml), add this plugin under `\u003cmodules\u003e` section:\n\n```\n\u003cmodules\u003e\n    \u003cmodule\u003estorage-hbase\u003c/module\u003e\n    \u003cmodule\u003eformat-maprdb\u003c/module\u003e\n    .....\n    \u003cmodule\u003estorage-ipfs\u003c/module\u003e\n\u003c/modules\u003e\n```\n\nBuild from the root directory of Drill source tree:\n\n```\nmvn -T 2C clean install -DskipTests　-Dcheckstyle.skip=true\n```\n\nThe jars are in the `storage-ipfs/target` directory.\n\n## Install\n\nThe executables and configurations are in `distribution/target/apache-drill-1.16.0`. Copy the entire directory to somewhere outside the source tree, and name it `drill-run` e.g., for testing later.\n\nCopy the `drill-ipfs-storage-{version}.jar` generated jar file to `drill-run/jars`.\n\nCopy `java-api-ipfs-v1.2.2.jar` which is IPFS's Java API, along with its dependencies provided as jar files:\n\n```\ncid.jar\njunit-4.12.jar\nmultiaddr.jar\nmultibase.jar\nmultihash.jar\nhamcrest-core-1.3.jar\n```\n\nto `drill-run/jars/3rdparty`.\n\nOptionally, copy the configuration override file `storage-plugin-override.conf` to `drill-run/conf`, if you want Drill to auto configure and enable IPFS storage plugin at every (re)start.\n\n## Configuration\n\n1. Set Drill hostname to the IP address of the node to run Drill:\n    \n    Edit file `conf/drill-env.sh` and change the environment variable `DRILL_HOST_NAME` to the IP address of the node. Use private or global addresses, depending on whether you plan to run it on a cluster or the open Internet.\n\n2. Configure the IPFS storage plugin:\n    \n    If you are not using the configuration override file, you will have to manually configure and enable the plugin.\n    \n    Run Drill according to [Section Run](#Run) and go to the webui of Drill (can be found at \u003chttp://localhost:8047\u003e). Under the Storage tab, create a new storage plugin named `ipfs` and click the Create button.\n    \n    Copy and paste the default configuration of the IPFS storage plugin located at `storage-ipfs/src/resources/bootstrap-storage-plugins.json`:\n    \n    ```\n    ipfs : {\n        \"type\":\"ipfs\",\n        \"host\": \"127.0.0.1\",\n        \"port\": 5001,\n        \"max-nodes-per-leaf\": 3,\n        \"ipfs-timeouts\": {\n          \"find-provider\": 4,\n          \"find-peer-info\": 4,\n          \"fetch-data\": 5\n        },\n        \"groupscan-worker-threads\": 50,\n        \"formats\": null,\n        \"enabled\": true\n    }\n    ```\n    \n    where \n    \n    `host` and `port` are the host and API port where your IPFS daemon will be listening. Change it so that it matches the configuration of your IPFS instance.\n\n    `max-nodes-per-leaf` controls how many provider nodes will be considered when the query is being planned. A larger value increases the parallelization width but typically takes longer to find enough providers from DHT resolution. A smaller value does the opposite.\n    \n    `ipfs-timeouts` set the maximum amount of time in seconds for various time consuming operations: `find-provider` is the time allowed to do DHT queries to find providers, `find-peer-info` is the time allowed to resolve the network addresses of the providers and `fetch-data` is the time the actual transmission is allowed to take. \n    \n    `groupscan-worker-threads` limits the number of worker threads when the planner communicate with IPFS daemon to resolve providers and peer info.\n    \n    `formats` specifies the formats of the files. It is unimplemented for now and does nothing.\n    \n    Click the Update button after finishing editing. You should see the IPFS storage plugin is registered with Drill and you can enable it with the Enable button.\n    \n3. Configure IPFS\n\n    Start the IPFS daemon first. \n    \n    Set a Drill-ready flag to the node:\n    \n    ```\n    ipfs name publish $(\\\n      ipfs object patch add-link $(ipfs object new) \"drill-ready\" $(\\\n        printf \"1\" | ipfs object patch set-data $(ipfs object new)\\\n      )\\\n    )\n    ```\n    \n    This flag indicates that an IPFS node is also capable of handling Drill quries and the planner will consider it when scheduling a query to execute distributedly. A node without this flag will be ignored.\n    \n\n## Run\n\n### Embedded mode\n\nStart IPFS daemon：\n\n```\nipfs daemon \u0026\u003e/dev/null \u0026\n```\n\nstart drill-embedded：\n\n```\ndrill-run/bin/drill-embedded\n```\n\nYou can now execute queries via the command line as well as the web interface.\n\n### As a background service\n\nYou can run drill-embedded as a background process without controlling a terminal. This is done with the help of tmux, which is available in many distributions of Linux.\n\nEdit the systemd service file `drill-embedded.service`, so that the environment variable `DRILL_HOME` pointes to where Drill is installed：\n```\nEnvironment=\"DRILL_HOME=/home/drill/apache-drill-1.16.0\"\n```\nCopy the service file to systemd's configuration directory, e.g. `/usr/lib/systemd/system`：\n```\ncp drill-embedded.service /usr/lib/systemd/system\n```\nReload the systemd daemon：\n```\nsystemd daemon-reload\n```\nStart the service:\n```\nsystemd start drill-embedded.service\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcata-network%2Fcata_database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcata-network%2Fcata_database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcata-network%2Fcata_database/lists"}