{"id":13906257,"url":"https://github.com/ekzhang/percival","last_synced_at":"2025-04-05T04:14:49.305Z","repository":{"id":40449697,"uuid":"421529672","full_name":"ekzhang/percival","owner":"ekzhang","description":"📝 Web-based, reactive Datalog notebooks for data analysis and visualization","archived":false,"fork":false,"pushed_at":"2023-02-16T20:20:00.000Z","size":662,"stargazers_count":600,"open_issues_count":8,"forks_count":25,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-10-13T11:24:57.233Z","etag":null,"topics":["charts","data-visualization","datalog","dsl","logic-programming","notebook","query-language","reactive","visualization"],"latest_commit_sha":null,"homepage":"https://percival.ink","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ekzhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-10-26T17:53:30.000Z","updated_at":"2024-10-08T09:28:39.000Z","dependencies_parsed_at":"2023-01-29T20:00:51.240Z","dependency_job_id":"84c1f7c1-c45f-403c-8498-c5ff4dc32efa","html_url":"https://github.com/ekzhang/percival","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhang%2Fpercival","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhang%2Fpercival/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhang%2Fpercival/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhang%2Fpercival/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ekzhang","download_url":"https://codeload.github.com/ekzhang/percival/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247284954,"owners_count":20913704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["charts","data-visualization","datalog","dsl","logic-programming","notebook","query-language","reactive","visualization"],"created_at":"2024-08-06T23:01:32.132Z","updated_at":"2025-04-05T04:14:49.282Z","avatar_url":"https://github.com/ekzhang.png","language":"Rust","funding_links":[],"categories":["Rust","visualization"],"sub_categories":[],"readme":"# Percival\n\n[Percival](https://percival.ink/) is a **declarative data query and\nvisualization language**. It provides a reactive, web-based notebook environment\nfor exploring complex datasets, producing interactive graphics, and sharing\nresults.\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://percival.ink/\"\u003e\n    \u003cimg src=\"https://i.imgur.com/zW5cuBH.png\" width=\"720\"\u003e\u003cbr\u003e\n    \u003cstrong\u003epercival.ink\u003c/strong\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nPercival combines the flexibility of\n[_Datalog_](https://en.wikipedia.org/wiki/Datalog) as a query language for\nrelational data with the beauty of\n[_exploratory visualization grammars_](https://observablehq.com/@observablehq/plot).\nThese declarative components interact through a reactive dataflow system.\nBecause Percival uses web technologies (including Web Workers for multithreaded,\nsandboxed execution), fully-interactive notebooks can be shared with anyone on\nthe Internet, making data analyses more tangible to others.\n\nAt the core of Percival is a custom Datalog compiler, built with Rust and\nWebAssembly, which integrates with its notebook runtime. This compiles the query\nlanguage to JavaScript through a staged evaluation process that also allows\nusers to embed their own JavaScript code. The interface aims to be lightweight,\nfriendly, and accessible, and there is no hidden workspace state.\n\nThis is an early-stage research project, and we welcome your feedback, so please\nfeel free to say hello at our\n[discussions page](https://github.com/ekzhang/percival/discussions)!\n\n## Getting Started\n\nIf you've gotten to this point in the README, please first try out the web\napplication and demo notebook at [percival.ink](https://percival.ink/)! The\ninformation below is technical documentation intended for contributors.\n\nBuilding Percival from scratch requires [Node v16+](https://nodejs.org/en/),\n[NPM v8+](https://www.npmjs.com/), [Rust 1.56+](https://www.rust-lang.org/),\n[Cargo](https://crates.io/), and\n[Wasm-Pack](https://rustwasm.github.io/wasm-pack/) installed on your machine. To\nbuild the Rust/WebAssembly portion of the project, use the command:\n\n```shell\nwasm-pack build --target web crates/percival-wasm\n```\n\nNext, run `npm install` to install JavaScript dependencies, then run the\nfollowing command to start the development server:\n\n```shell\nnpm run dev\n```\n\nThis should open a Percival notebook in your browser, with live reloading.\n\n## Architecture\n\nThis section outlines the high-level technical design of Percival.\n\n### User Interface\n\nPercival is a client-side web application running fully in the user's browser.\nThe notebook interface is built with [Svelte](https://svelte.dev/) and styled\nwith [Tailwind CSS](https://tailwindcss.com/). It relies on numerous other open\nsource libraries, including [CodeMirror 6](https://codemirror.net/6/) for live\ncode editing and syntax highlighting,\n[Remark](https://github.com/remarkjs/remark) and [KaTeX](https://katex.org/) for\nMarkdown rendering, and [Vite](https://vitejs.dev/) for frontend bundling.\n\nThe code for the web frontend is located in `src/`, which contains a mix of\nSvelte (in `src/components/`) and TypeScript (in `src/lib/`). These modules are\nbundled into a static website at build time, and there is no dynamic server-side\nrendering.\n\n### JIT Compiler\n\nUsers write code cells in a custom dialect of Datalog, and they are translated\nto JavaScript by a Rust compiler, which itself is compiled to WebAssembly using\n[wasm-bindgen](https://github.com/rustwasm/wasm-bindgen). The Percival\ncompiler's code is located in the `crates/` folder. For ergonomic parsing with\nhuman-readable error messages, the compiler relies on\n[chumsky](https://github.com/zesterer/chumsky), a parser combinator library.\n\nAfter the `percival-wasm` crate is compiled to WebAssembly, it can be used by\nclient-side code. The compiler processes code cells, then sends the resulting\nJavaScript to separate\n[web workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)\nthat sandbox the code and execute it just-in-time. As the user writes queries,\ntheir notebook automatically tracks inter-cell dependencies and evaluates cells\nin topological order, spawning / terminating worker threads on demand.\n\n### Data Visualization\n\nPlotting is done using a specialized web worker that runs JavaScript code with\naccess to the [Observable Plot](https://observablehq.com/@observablehq/plot)\nlibrary. In order for this library (and D3) to run in a worker context, we patch\nthe global document with a lightweight virtual DOM implementation ported from\n[Domino](https://github.com/fgnass/domino).\n\n### Deployment\n\nIn production, the `main` branch of this repository is continuously deployed to\n[percival.ink](https://percival.ink/) via [Vercel](https://vercel.com/), which\nhosts the static website. It also runs a serverless function (see\n`api/index.go`) that allows users to share notebooks through the GitHub Gist\nAPI.\n\n## Development\n\nTo build, lint, and format the Svelte project, use the corresponding scripts:\n\n```shell\nnpm run build\nnpm run check\nnpm run format\n```\n\nFor the Rust crates, you can run unit tests for the core functionality with:\n\n```shell\ncargo test\n```\n\nYou can also run tests for the WebAssembly component using a headless Chrome or\nFirefox browser:\n\n```shell\nwasm-pack test --chrome --headless crates/percival-wasm\n```\n\nSince Percival uses a Rust-based compiler but outputs JavaScript, the easiest\nway to test code generation functionality is within the browser. We use Mocha\nand Puppeteer for this, and tests can be run with:\n\n```shell\nnpm test\n```\n\n## Acknowledgements\n\nCreated by Eric Zhang ([@ekzhang1](https://twitter.com/ekzhang1)). Licensed\nunder the [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekzhang%2Fpercival","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fekzhang%2Fpercival","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekzhang%2Fpercival/lists"}