https://github.com/insightsoftwareconsortium/cmake-w3-externaldata-upload
CMake Web3 ExternalData Upload UI
https://github.com/insightsoftwareconsortium/cmake-w3-externaldata-upload
Last synced: 10 months ago
JSON representation
CMake Web3 ExternalData Upload UI
- Host: GitHub
- URL: https://github.com/insightsoftwareconsortium/cmake-w3-externaldata-upload
- Owner: InsightSoftwareConsortium
- License: apache-2.0
- Created: 2022-11-10T01:56:12.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-05-13T19:40:49.000Z (about 1 year ago)
- Last Synced: 2025-08-28T06:52:57.674Z (10 months ago)
- Language: JavaScript
- Homepage: https://content-link-upload.netlify.app
- Size: 354 KB
- Stars: 1
- Watchers: 15
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# cmake-w3-externaldata-upload
[](https://content-link-upload.netlify.app)
[CMake Web3 ExternalData Upload UI](https://content-link-upload.netlify.app)
Based on [the web3.storage browser client](https://web3.storage/docs/reference/js-client-library/#store-files) and [w3ui](https://github.com/web3-storage/w3ui).
## About
Since every local [Git](https://git-scm.com/) repository contains a copy
of the entire project history, it is important to avoid adding large
binary files directly to the repository. Large binary files added and
removed throughout a project\'s history will cause the repository to
become bloated, take up too much disk space, require excessive time and
bandwidth to download, etc.
A [solution to this
problem](https://blog.kitware.com/cmake-externaldata-using-large-files-with-distributed-version-control/)
which has been adopted by this project, is to store binary files such as
images in a separate location outside the Git repository. Then, download
the files at build time with [CMake](https://cmake.org/).
A \"content link\" file contains an identifying [Content Identifier
(CID)](https://proto.school/anatomy-of-a-cid). The content link is
stored in the [Git](https://git-scm.com/) repository at the path where
the file would exist, but with a \".cid\" extension appended to the file
name. CMake will find these content link files at *build* time, download
them from a list of HTTP server resources, and create symlinks or copies of
the original files at the corresponding location in the *build tree*.
The [Content Identifier (CID)](https://proto.school/anatomy-of-a-cid) is
self-describing hash following the [multiformats
](https://multiformats.io/) standard created by the Interplanetary
Filesystem ([IPFS](https://ipfs.io/)) community. A file with a CID for
its filename is content verifable. Locating files according to their CID
makes content-addressed, as opposed to location-addressed, data exchange
possible. This practice is the foundation of the decentralized web, also
known as the dWeb or Web3. By adopting Web3, we gain:
- Permissionless data uploads
- Robust, redundant storage
- Local and peer-to-peer storage
- Scalability
- Sustainability
Contributors to upload their data through an easy-to-use,
permissionless, free service, [web3.storage](https://web3.storage/).
Data used in the Git repository is periodically tracked in a
dedicated [Datalad
repository](https://datalad.org)
and stored across redundant locations so it can be retrieved from any of
the following:
- Local [IPFS](https://ipfs.io/) nodes
- Peer [IPFS](https://ipfs.io/) nodes
- [web3.storage](https://web3.storage/)
- [pinata.cloud](https://pinata.cloud)
*Note: This currently requires an extended version of the ExternalData.cmake module developed in the [CMakeIPFSExternalData repository](https://github.com/InsightSoftwareConsortium/CMakeIPFSExternalData). This has not been integrated into upstream CMake due to the availability of C++ CID verification code to complete the feature set in a
portable way.*
## Development
### Clone, install dependencies:
```sh
git clone https://github.com/InsightSoftwareConsortium/cmake-w3-externaldata-upload
cd cmake-w3-externaldata-upload
```
Install [pixi](https://pixi.sh/) if not already installed:
```sh
curl -fsSL https://pixi.sh/install.sh | sh
```
Start with an empty database:
```sh
cp data.duckdb.empty data.duckdb
```
### Populate .env
Populate a *.env* file in the repository with environmental variables used by application services in the format `=`.
#### web3.storage
Populate `DID`, `KEY`, and `PROOF` as [described in the web3.storage documentation on creation of delegations](https://web3.storage/docs/how-to/upload/#bring-your-own-delegations).
### mailjet
For monitor emails, set the [MailJet](https://mailjet.com) keys `MJ_APIKEY_PUBLIC`, `MJ_APIKEY_PRIVATE`. Also `SENDER_EMAIL` and `RECIPIENT_EMAIL`. Note that DNS records should be set for the sender and the sender configured in MailJet.
### Run the dev server
```sh
pixi run start
```
This starts, a [hypha server](https://ha.amun.ai/#/), a hypha Python service, and the client side build, and serves them locally.
Visit [http://localhost:9000/app/index.html](http://localhost:9000/app/index.html) to view the application.
### Production service
The production server currently runs on a cloud VM with
```sh
pixi run start-hypha-service
```
Running under [supervisord](https://supervisord.org/).
The content-link-upload.itk.org is served on
https://content-link-upload.netlify.app and built via `pnpm run build`.