{"id":15461472,"url":"https://github.com/ibnesayeed/ipfsapi-concurrency-test","last_synced_at":"2025-10-26T13:11:27.946Z","repository":{"id":152153065,"uuid":"141085791","full_name":"ibnesayeed/ipfsapi-concurrency-test","owner":"ibnesayeed","description":"IPFS API concurrency profiling script with a reproducible environment","archived":false,"fork":false,"pushed_at":"2018-07-16T13:15:33.000Z","size":26,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-12T12:26:23.134Z","etag":null,"topics":["api","docker","ipfs","ipfsapi","ipwb","profiling","python","testing"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ibnesayeed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-16T04:19:22.000Z","updated_at":"2024-06-12T16:17:37.000Z","dependencies_parsed_at":"2023-07-03T05:35:13.084Z","dependency_job_id":null,"html_url":"https://github.com/ibnesayeed/ipfsapi-concurrency-test","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"ec0181c677cd01e0d7ae23a108cf3c7605f3606a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibnesayeed%2Fipfsapi-concurrency-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibnesayeed%2Fipfsapi-concurrency-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibnesayeed%2Fipfsapi-concurrency-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibnesayeed%2Fipfsapi-concurrency-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ibnesayeed","download_url":"https://codeload.github.com/ibnesayeed/ipfsapi-concurrency-test/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246634929,"owners_count":20809324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","docker","ipfs","ipfsapi","ipwb","profiling","python","testing"],"created_at":"2024-10-01T23:42:26.607Z","updated_at":"2025-10-26T13:11:27.851Z","avatar_url":"https://github.com/ibnesayeed.png","language":"Python","readme":"# IPFS API Concurrency Test\n\nThis repository provides a reproducible test environment to measure concurrency performance of the [IPFS API](https://pypi.org/project/ipfsapi/).\n[IPWB](https://github.com/oduwsdl/ipwb/), an [IPFS](https://ipfs.io/)-based web archival replay system, fetches two objects from the IPFS for each replay request it receives, then combines them to prepare the response.\nIn its initial implementation we used to fetch the two objects sequentially, but planned to fetch them concurrently later.\nWhen [we tried concurrency using threads](https://github.com/oduwsdl/ipwb/pull/425) and benchmarked our replay response time we were surprised to see that it it was taking twice as long.\nThis was counter intuitive, so we started profiling our code to investigate the culprit.\nWe found that the API's `cat()` call was taking longer when invoked in concurrent threads.\nSo, we decided to create this isolated repository for further investigation.\n\n## Run Test\n\nAssuming that [ipfs](https://ipfs.io/docs/install/) and [ipfsapi](https://pypi.org/project/ipfsapi/) are installed and this repository is cloned, run the following commands from the repository directory:\n\n```\n$ ipfs daemon --init \u0026\n$ ./main.py -h\nUsage:   ./main.py [\u003cmax-items\u003e [\u003cattempts\u003e]]\n\u003cmax-items\u003e : Number of items from the data to be used (Default: 5)\n\u003cattempts\u003e  : Number of iterations to perform fetches  (Default: 3)\n```\n\nThe first command will Initialize IPFS and run its daemon in the background.\nThe second command shows the usage which suggests it can be run without any arguments as the two arguments are optional and have associated default values.\nThe script will produce timing data for both individual interactions as well as an aggregated summary.\n\n## Run in Docker\n\nRunning this test in a Docker container is recommended as it is more reproducible and contains all the necessary requirements.\nAssuming that [Docker is installed](https://docs.docker.com/install/), run the following commands:\n\n```\n$ git clone https://github.com/ibnesayeed/ipfsapi-concurrency-test\n$ cd ipfsapi-concurrency-test\n$ docker image build -t ipfsapitest .\n$ docker container run --rm -it ipfsapitest\n```\n\nThe last command will run the script with the default arguments.\nAn instance of the IPFS daemon will be started inside the container itself without interfering with any other instances on the host.\nEvery time a new container is run, it will start with a clean slate.\nTo run the test with only three items from the included data and repeat the task 10 times, run the following command:\n\n```\n$ docker container run --rm -it ipfsapitest ./main.py 3 10\n```\n\nThe Dockerfile provides three [build-args](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg) `PYTHON_TAG`, `IPFS_VERSION`, and `IPFSAPI_VERSION` to customize the environment.\nTheir default values are `latest`, `v0.4.15`, and `\u003e=0` respectively, but one or more of these can be modified as necessary.\nTo test the behavior of the API version `0.4.2` in Python `2.7` with IPFS version `v0.4.14`, build an image as following:\n\n```\n$ docker image build --build-arg PYTHON_TAG=2.7 \\\n                     --build-arg IPFS_VERSION=\"v0.4.14\" \\\n                     --build-arg IPFSAPI_VERSION=\"==0.4.2\" \\\n                     -t ipfsapitest:custom .\n```\n\nThen run the test in a container using this custom image as following:\n\n```\n$ docker container run --rm -it ipfsapitest:custom ./main.py 3 10\n```\n\n## Sample Output\n\n```\n$ ./main.py 3 2\n\n======= Sequential Push (Just Once) =======\nIndex  Start         End           Elapsed\n0      0.0000391006  0.1890790462  0.1890399456\n1      0.1891169548  0.3722760677  0.1831591129\n2      0.3723781109  0.5568559170  0.1844778061\nTotal 3 items pushed to IPFS sequentially in 0.556885957718 seconds\n\n======= Sequential Fetch (Attempt #0) =======\nIndex  Start         End           Elapsed\n0      0.0000429153  0.0053079128  0.0052649975\n1      0.0053539276  0.0090999603  0.0037460327\n2      0.0091428757  0.0153949261  0.0062520504\nTotal 3 items fetched from IPFS sequentially in 0.0154371261597 seconds\n\n======= Threaded Fetch (Attempt #0) =======\nIndex  Start         End           Elapsed\n0      0.0007300377  0.0119800568  0.0112500191\n2      0.0054049492  0.0198419094  0.0144369602\n1      0.0045778751  0.0214118958  0.0168340206\n\nTotal 3 items fetched from IPFS concurrently in 0.0217299461365 seconds\n\n======= Sequential Fetch (Attempt #1) =======\nIndex  Start         End           Elapsed\n0      0.0000441074  0.0050020218  0.0049579144\n1      0.0050911903  0.0096251965  0.0045340061\n2      0.0096790791  0.0137782097  0.0040991306\nTotal 3 items fetched from IPFS sequentially in 0.0138120651245 seconds\n\n======= Threaded Fetch (Attempt #1) =======\nIndex  Start         End           Elapsed\n1      0.0013508797  0.0101108551  0.0087599754\n0      0.0006079674  0.0165028572  0.0158948898\n2      0.0054647923  0.0182588100  0.0127940178\nTotal 3 items fetched from IPFS concurrently in 0.0185489654541 seconds\n\n======= SUMMARY =======\nData Items:      3\nFetch Attempts:  2\nMean Fetch Time (Sequential):  0.0048748652 seconds/item\nMean Fetch Time (Threaded):    0.0067131519 seconds/item\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibnesayeed%2Fipfsapi-concurrency-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibnesayeed%2Fipfsapi-concurrency-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibnesayeed%2Fipfsapi-concurrency-test/lists"}