{"id":17494187,"url":"https://github.com/bears-r-us/arkouda-njit","last_synced_at":"2025-04-15T11:54:45.089Z","repository":{"id":38794759,"uuid":"457047564","full_name":"Bears-R-Us/arkouda-njit","owner":"Bears-R-Us","description":"Home of Arachne and other Arkouda functionality provided by researchers at NJIT","archived":false,"fork":false,"pushed_at":"2025-04-13T22:04:32.000Z","size":256908,"stargazers_count":21,"open_issues_count":8,"forks_count":13,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-13T23:21:08.638Z","etag":null,"topics":["arkouda","chapel","chapel-lang","chapel-language","distributed-computing","graph-algorithms","parallel-programming"],"latest_commit_sha":null,"homepage":"","language":"Chapel","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bears-R-Us.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-08T18:05:37.000Z","updated_at":"2025-04-07T21:18:13.000Z","dependencies_parsed_at":"2023-11-18T16:30:52.886Z","dependency_job_id":"63ec77ba-9006-42c2-a2b7-f0c4f6edae58","html_url":"https://github.com/Bears-R-Us/arkouda-njit","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bears-R-Us%2Farkouda-njit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bears-R-Us%2Farkouda-njit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bears-R-Us%2Farkouda-njit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bears-R-Us%2Farkouda-njit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bears-R-Us","download_url":"https://codeload.github.com/Bears-R-Us/arkouda-njit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249067774,"owners_count":21207395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arkouda","chapel","chapel-lang","chapel-language","distributed-computing","graph-algorithms","parallel-programming"],"created_at":"2024-10-19T13:05:29.770Z","updated_at":"2025-04-15T11:54:45.082Z","avatar_url":"https://github.com/Bears-R-Us.png","language":"Chapel","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Arkouda-NJIT\nThis is an external repository to build functionality for [Arkouda](https://github.com/Bears-R-Us/Arkouda) with a focus on advanced graph processing. It is built with the same structure as [arkouda-contrib](https://github.com/Bears-R-Us/arkouda-contrib) to manage modules and easily swap between the production (`arachne`) and development (`arachne_development`) directories.\n\n## Configuring Prerequisites\nTo install the prerequisites below, the following libraries must be installed on your system. This can be done via any package handler depending on your distribution, or if on a cluster HPC system, they can be loaded in as modules. At the time of writing, the following versions were confirmed to work.\n1. GCC 11.2.0 or later.\n2. CMake 3.26.3.\n3. OpenMPI GNU 4.1.4 (needed by CMake)\n4. Anaconda 2023.09-0\n5. `jq` 1.6 [command-line JSON processor](https://jqlang.github.io/jq/).\n\n### Prerequisite Installation Steps\nWe recommend following the installation instructions provided by the Arkouda development team. Most specifically, follow the [Prerequisites](https://github.com/Bears-R-Us/arkouda?tab=readme-ov-file#prerequisites-toc) section in its entirety, and only the [Dependency Configuration](https://github.com/Bears-R-Us/arkouda/blob/master/pydoc/setup/BUILD.md#building-the-server) section of the build instructions. The installation steps usually involve the following:\n1. Download [Chapel](https://chapel-lang.org/download.html) from the Chapel downloads page. **Use Chapel version 2.1.0.**\n    * Alternatively, you may clone [Chapel](https://github.com/chapel-lang/chapel) and switch to a given tagged version. The commands for these should look something like:\n        ```bash\n        git clone https://github.com/chapel-lang/chapel.git\n        cd chapel\n        git fetch --tags origin\n        git checkout tags/2.1.0 --force\n        ```\n2. Build Chapel by executing the commands below. This assumes you have installed all [Chapel prerequisites](https://chapel-lang.org/docs/usingchapel/prereqs.html#chapel-prerequisites). **Note:** We recommend using `gcc/11.2.0` or later due to dependencies with [VieCut](https://github.com/MinhyukPark/VieCut).\n    ```bash\n    cd /path/to/chapel/\n    source ./util/setchplenv.bash\n    export CHPL_GMP=bundled\n    export CHPL_HWLOC=bundled\n    export CHPL_RE2=bundled\n    export CHPL_LLVM=bundled\n    make -j 4 # This value can be increased dependent on your device's number of processors\n    ```\n    * **Note:** This installs single locale (shared-memory) Chapel. For multilocale (distributed-memory) Chapel please follow the documentation guide on [Multilocale Chapel Execution](https://chapel-lang.org/docs/usingchapel/multilocale.html#multilocale-chapel-execution). Arachne has its best performance on shared-memory. However, kernels such as breadth-first search and property graph querying have multilocale-optimized versions that require multilocale Chapel to be installed.\n3. Download, **but do not build**, [Arkouda](https://github.com/Bears-R-Us/arkouda). **Use Arkouda version v2024.06.21.** A specified version can be selected for download by clicking on `Releases` in the main GitHub page for [Arkouda](https://github.com/Bears-R-Us/arkouda).\n    * Alternatively, you may clone Arkouda and switch to a given tagged version.\n        ```bash\n        git clone https://github.com/Bears-R-Us/arkouda.git\n        cd arkouda\n        git fetch --tags origin\n        git checkout tags/v2024.06.21 --force\n        ```\n4. Install Arkouda dependencies with `Anaconda`. An environment containing all dependencies can be installed from `arkouda-env.yml` within your Arkouda home directory.\n    * This can be done by executing the following command within your Arkouda directory:\n        ```bash\n        conda env create -f arkouda-env.yml\n        ```\n5. [Configure your Arkouda dependencies](https://github.com/Bears-R-Us/arkouda/blob/master/pydoc/setup/BUILD.md#dependency-configuration). This involves creating (or modifying) the `Makefile.paths` within your Arkouda home directory.\n6. Install [constrained-clustering](https://github.com/MinhyukPark/constrained-clustering) and compile the C++ object files required by Arachne by following the commands below. Constrained-clustering requires a C++ compiler that supports `c++-20`, such as `clang++11`, and `cmake`. These and other prerequisites should be covered by the prerequisites in items 1-5 above.\n    ```bash\n    cd /path/to/arkouda-njit/arachne/server/external_libs\n    git clone https://github.com/MinhyukPark/constrained-clustering.git\n    cd constrained-clustering\n    ./setup.sh\n    ./easy_build_and_compile.sh\n    cd ../../../../arachne/server/viecut_helpers/\n    source compileLogger.sh -f logger.cpp -o logger.cpp.o\n    gcc -c -fPIC -I../external_libs/constrained-clustering/external_libs/VieCut/lib/ -I../external_libs/constrained-clustering/external_libs/VieCut/extlib/tlx/ computeMinCut.cpp -o computeMinCut.o\n    cd ../../../\n    ```\n\n## Building Arachne\nBuilding Arachne is performed through executing the `module_configuration.py` file. The complete path to the location of `arkouda` must be specified through `ak_loc` and the complete path to the location of `arachne` should be specified through `pkg_path`.\n\n```bash\npython module_configuration.py --ak_loc=/complete/path/to/arkouda/ --pkg_path=/complete/path/to/arkouda-njit/arachne/ | bash\n```\n\nThe above command will pipe the following three commands to terminal that installs Arachne using pip, copies the Arkouda server modules to a temporary file, and combines them with the Arachne server modules to build the enhanced `arkouda_server`.\n```bash\npip install -U /complete/path/to/arkouda-njit/arachne/client\ncp /complete/path/to/arkouda/ServerModules.cfg ~/TmpServerModules.cfg.1683320760\nARKOUDA_SERVER_USER_MODULES=\" /complete/path/to/arkouda-njit/arachne/server/BuildGraphMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/PropertyGraphMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/GraphInfoMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/BFSMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/TriCtrMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/TriCntMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/TrussMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/CCMsg.chpl\" ARKOUDA_CONFIG_FILE=~/TmpServerModules.cfg.1683320760 ARKOUDA_SKIP_CHECK_DEPS=true make -C /Users/alvaradoo/Research/arkouda\n```\n\nFor usage instructions of `module_configuration.py` please execute the the following.\n```bash\npython module_configuration.py --help\n```\n\n### Building Development Arachne\nIf you are interested in installing the development version of Arachne, please follow the same instructions as above, but for `pkg_path` include `/complete/path/to/arkouda-njit/arachne_development/`.\n\n## Starting the Arachne and Arkouda Server\nThe server can be started as specified in the [Arkouda documentation](https://github.com/Bears-R-Us/arkouda#running-arkouda_server-toc). Simply put, navigate to your Arkouda directory, and an executable named `arkouda_server` should exist. Execute it with the command below to start a server instance.\n```bash\n./arkouda_server # - nl X\n```\n\nThe output should be something that looks like below.\n```bash\n********************************************************************************************************\n********************************************************************************************************\n*                                                                                                      *\n*                          server listening on tcp://n118.cluster.local:5555                           *\n*                                 arkouda server version = v2024.06.21                                 *\n*                                    built with chapel version2.1.0                                    *\n*                                     memory limit = 973796998348                                      *\n*                                       bytes of memory used = 0                                       *\n*                                                                                                      *\n********************************************************************************************************\n********************************************************************************************************\n```\n\nTo run a simple test file as well as pytests please proceed to the [Arachne](arachne/) directory for those instructions.\n\n## Usage Notes\n```python\nimport arkouda as ak\nimport arachne as ar\n# code using arachne and arkouda below\n```\n\n## Common Issues\n* **Issue**: Unrecognized HDF5, Apache Arrow, etc. installations. \n  **Fix**: Ensure `Makefile.paths` was properly added to the base Arkouda directory. More information can be found in the [Arkouda build instructions](https://github.com/Bears-R-Us/arkouda#building-arkouda-toc).\n* **Issue**: Arkouda or Arachne functions are not recognized when executing scripts.\n  **Fix**: Make sure to run `pip3 install -e .` at both `/complete/path/to/arkouda-njit/arachne/client/.` and `/complete/path/to/arkouda/.`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbears-r-us%2Farkouda-njit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbears-r-us%2Farkouda-njit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbears-r-us%2Farkouda-njit/lists"}