{"id":13439100,"url":"https://github.com/weld-project/weld","last_synced_at":"2025-05-14T07:08:29.970Z","repository":{"id":37734238,"uuid":"61008232","full_name":"weld-project/weld","owner":"weld-project","description":"High-performance runtime for data analytics applications","archived":false,"fork":false,"pushed_at":"2022-06-22T01:07:30.000Z","size":3015,"stargazers_count":3000,"open_issues_count":75,"forks_count":256,"subscribers_count":109,"default_branch":"master","last_synced_at":"2025-05-11T23:35:11.262Z","etag":null,"topics":["analytics","code-generation","data","llvm","machine-learning","pandas","performance","rust","stanford"],"latest_commit_sha":null,"homepage":"https://www.weld.rs","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weld-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-13T05:00:24.000Z","updated_at":"2025-05-08T10:44:39.000Z","dependencies_parsed_at":"2022-07-10T18:00:54.142Z","dependency_job_id":null,"html_url":"https://github.com/weld-project/weld","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weld-project","download_url":"https://codeload.github.com/weld-project/weld/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254092776,"owners_count":22013290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","code-generation","data","llvm","machine-learning","pandas","performance","rust","stanford"],"created_at":"2024-07-31T03:01:11.140Z","updated_at":"2025-05-14T07:08:29.946Z","avatar_url":"https://github.com/weld-project.png","language":"Rust","readme":"# Weld\n\n[![Build Status](https://travis-ci.org/weld-project/weld.svg?branch=master)](https://travis-ci.org/weld-project/weld)\n\n[Documentation](https://www.weld.rs/docs/latest/weld/)\n\nWeld is a language and runtime for improving the performance of data-intensive applications. It optimizes across libraries and functions by expressing the core computations in libraries using a common intermediate representation, and optimizing across each framework.\n\nModern analytics applications combine multiple functions from different libraries and frameworks to build complex workflows. Even though individual functions can achieve high performance in isolation, the performance of the combined workflow is often an order of magnitude below hardware limits due to extensive data movement across the functions. Weld’s take on solving this problem is to lazily build up a computation for the entire workflow, and then optimizing and evaluating it only when a result is needed.\n\nYou can join the discussion on Weld on our [Google Group](https://groups.google.com/forum/#!forum/weld-users) or post on the Weld mailing list at [weld-group@lists.stanford.edu](mailto:weld-group@lists.stanford.edu).\n\n## Contents\n\n  * [Building](#building)\n      - [MacOS LLVM Installation](#macos-llvm-installation)\n      - [Ubuntu LLVM Installation](#ubuntu-llvm-installation)\n      - [Building Weld](#building-weld)\n  * [Documentation](#documentation)\n  * [Grizzly (Pandas on Weld)](#grizzly)\n  * [Tools](#tools)\n\n## Building\n\nTo build Weld, you need [the latest stable version of Rust](http://rust-lang.org) and [LLVM/Clang++](http://llvm.org) 6.0.\n\nTo install Rust, follow the steps [here](https://rustup.rs). You can verify that Rust was installed correctly on your system by typing `rustc` into your shell. If you already have Rust and  `rustup` installed, you can upgrade to the latest stable version with:\n\n```bash\nrustup update stable\n```\n\n#### MacOS LLVM Installation\n\nTo install LLVM on macOS, first install [Homebrew](https://brew.sh/). Then:\n\n```bash\nbrew install llvm@6\n```\n\nWeld's dependencies require `llvm-config` on `$PATH`, so you may need to create a symbolic link so the correct `llvm-config` is picked up (note that you might need to add `sudo` at the start of this command):\n\n```bash\nln -sf `brew --prefix llvm@6`/bin/llvm-config /usr/local/bin/llvm-config\n```\n\nTo make sure this worked correctly, run `llvm-config --version`. You should see `6.0.x`.\n\n#### Ubuntu LLVM Installation\n\nTo install LLVM on Ubuntu, get the LLVM 6.0 sources and then `apt-get`:\n\nOn Ubuntu 16.04 (Xenial):\n```bash\nwget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -\nsudo apt-add-repository \"deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main\"\nsudo apt-get update\nsudo apt-get install llvm-6.0-dev clang-6.0\n```\nOn Ubuntu 14.04 (Trusty):\n```bash\nwget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -\nsudo apt-add-repository \"deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-6.0 main\"\n\n# gcc backport is required on 14.04, for libstdc++. See https://apt.llvm.org/\nsudo apt-add-repository \"deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu trusty main\"\nsudo apt-get update\nsudo apt-get install llvm-6.0-dev clang-6.0\n```\n\nWeld's dependencies require `llvm-config`, so you may need to create a symbolic link so the correct `llvm-config` is picked up. `sudo` may be required:\n\n```bash\nln -s /usr/bin/llvm-config-6.0 /usr/local/bin/llvm-config\n```\n\nTo make sure this worked correctly, run `llvm-config --version`. You should see `6.0.x` or newer.\n\nYou will also need `zlib`:\n\n```bash\nsudo apt-get install zlib1g-dev\n```\n\n#### Building Weld\n\nWith LLVM and Rust installed, you can build Weld. Clone this repository, set the `WELD_HOME` environment variable, and build using `cargo`:\n\n```bash\ngit clone https://www.github.com/weld-project/weld\ncd weld/\nexport WELD_HOME=`pwd`\ncargo build --release\n```\nWeld builds two dynamically linked libraries (`.so` files on Linux and `.dylib` files on Mac): `libweld` and `libweldrt`.\n\nFinally, run the unit and integration tests:\n\n```bash\ncargo test\n```\n\n## Documentation\n\nThe [Rust Weld crate](https://crates.io/crates/weld) is documented [here](https://www.weld.rs/docs/latest/weld/).\n\nThe `docs/` directory contains documentation for the different components of Weld.\n\n* [language.md](https://github.com/weld-project/weld/blob/master/docs/language.md) describes the syntax of the Weld IR.\n* [api.md](https://github.com/weld-project/weld/blob/master/docs/api.md) describes the low-level C API for interfacing with Weld.\n* [python.md](https://github.com/weld-project/weld/blob/master/docs/python.md) gives an overview of the Python API.\n* [tutorial.md](https://github.com/weld-project/weld/blob/master/docs/tutorial.md) contains a tutorial for how to build a small vector library using Weld.\n\n## Python Bindings\n\nWeld's Python bindings are in [`python`](https://github.com/weld-project/weld/tree/master/python), with examples in [`examples/python`](https://github.com/weld-project/weld/tree/master/examples/python).\n\n## Grizzly\n\n**Grizzly** is a subset of [Pandas](http://pandas.pydata.org/) integrated with Weld. Details on how to use Grizzly are in\n[`python/grizzly`](https://github.com/weld-project/weld/tree/master/python/grizzly).\nSome example workloads that make use of Grizzly are in [`examples/python/grizzly`](https://github.com/weld-project/weld/tree/master/examples/python/grizzly).\nTo run Grizzly, you will also need the `WELD_HOME` environment variable to be set, because Grizzly needs to find its own native library through this variable.\n\n## Testing\n\n`cargo test` runs unit and integration tests. A test name substring filter can be used to run a subset of the tests:\n\n   ```\n   cargo test \u003csubstring to match in test name\u003e\n   ```\n\n## Tools\n\nThis repository contains a number of useful command line tools which are built\nautomatically with the main Weld repository, including an interactive REPL for\ninspecting and debugging programs.  More information on those tools can be\nfound under [docs/tools.md](https://github.com/weld-project/weld/tree/master/docs/tools.md).\n","funding_links":[],"categories":["Libraries","Rust","库 Libraries","Optimized Computation","其他_机器学习与深度学习","库"],"sub_categories":["Data processing","数据处理 Data processing","数据处理"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweld-project%2Fweld","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweld-project%2Fweld","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweld-project%2Fweld/lists"}