{"id":27881860,"url":"https://github.com/src-d/engine-deprecated","last_synced_at":"2025-08-02T11:33:33.457Z","repository":{"id":79453990,"uuid":"144944188","full_name":"src-d/engine-deprecated","owner":"src-d","description":"[DISCONTINUED] Go to https://github.com/src-d/sourced-ce/","archived":false,"fork":false,"pushed_at":"2019-10-09T11:35:19.000Z","size":7095,"stargazers_count":217,"open_issues_count":49,"forks_count":26,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-05-05T05:05:53.788Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://docs.sourced.tech/engine","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/src-d.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-08-16T06:07:24.000Z","updated_at":"2024-02-26T10:47:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"d6003b07-4089-491c-b3e2-93495ca607ce","html_url":"https://github.com/src-d/engine-deprecated","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/src-d/engine-deprecated","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fengine-deprecated","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fengine-deprecated/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fengine-deprecated/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fengine-deprecated/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/src-d","download_url":"https://codeload.github.com/src-d/engine-deprecated/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fengine-deprecated/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268378965,"owners_count":24240907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-05T05:05:51.066Z","updated_at":"2025-08-02T11:33:33.442Z","avatar_url":"https://github.com/src-d.png","language":"Go","readme":"\u003ca href=\"https://www.sourced.tech/engine\"\u003e\n  \u003cimg src=\"docs/sourced-engine.png\" alt=\"source{d} Engine\" height=\"120px\" /\u003e\n\u003c/a\u003e\n\n**Powerful language-agnostic analysis of your source code and git history.**\n\n[![GitHub version](https://badge.fury.io/gh/src-d%2Fengine.svg)](https://github.com/src-d/engine/releases)\n[![Build Status](https://travis-ci.com/src-d/engine.svg?branch=master)](https://travis-ci.com/src-d/engine)\n[![Go Report Card](https://goreportcard.com/badge/github.com/src-d/engine)](https://goreportcard.com/report/github.com/src-d/engine)\n[![GoDoc](https://godoc.org/github.com/src-d/engine?status.svg)](https://godoc.org/github.com/src-d/engine)\n[![source{d} design document](https://img.shields.io/badge/source%7Bd%7D-design%20document-blue.svg)](https://drive.google.com/open?id=1Fw-qE8mC3lZUP5hoH1Jo5bYcGV0Z018uAq7er3oAMLw)\n\n[Website](https://www.sourced.tech/engine) •\n[Documentation](https://docs.sourced.tech/engine) •\n[Blog](https://blog.sourced.tech) •\n[Slack](http://bit.ly/src-d-community) •\n[Twitter](https://twitter.com/sourcedtech)\n\n## Introduction\n\nsource{d} Engine exposes powerful [Universal ASTs](#babelfish-uast) to analyze your code and a SQL engine to analyze your git history:\n\n- **Code Processing**: use git repositories as a dataset.\n- **Language-Agnostic Code Analysis**: automatically identify languages, parse source code, and extract the pieces that matter with language-independent queries.\n- **Git Analysis**: powerful SQL based analysis on top of your git repositories.\n- **Querying With Familiar APIs**: analyze your code through powerful friendly APIs, such as SQL, gRPC, and various client libraries.\n\nYou can access a rendered version of this documentation at [docs.sourced.tech/engine](https://docs.sourced.tech/engine).\n\n## Contents\n\n- [Quick Start](#quick-start)\n  - [Requirements](#1-install-docker)\n  - [Installation](#2-install-source-d-engine)\n  - [Initialization](#3-start-source-d-engine-with-your-local-repositories)\n  - [Commands](#4-explore-source-d-engine)\n  - [Examples](#5-start-executing-queries)\n- [Other Guides \u0026 Examples](#other-guides-and-examples)\n- [Architecture](#architecture)\n- [Babelfish UAST](#babelfish-uast)\n- [Clients \u0026 Connectors](#clients-and-connectors)\n- [Community](#community)\n- [Contributing](#contributing)\n- [Credits](#credits)\n- [License](#license)\n\n## Quick Start\n\nFollow the steps below to get started with source{d} Engine.\n\n### 1. Install Docker\n\nFollow these instructions based on your OS:\n\n#### Docker on macOS\n\nFollow instructions at [Docker for Mac](https://store.docker.com/editions/community/docker-ce-desktop-mac). You may also use [Homebrew](https://brew.sh/):\n\n  ```bash\n  brew cask install docker\n  ```\n\n#### Docker on Ubuntu Linux\n\nFollow instructions at [Docker for Ubuntu Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1):\n\n  ```bash\n  sudo apt-get update\n  sudo apt-get install docker-ce\n  ```\n\n#### Docker on Arch Linux\n\nFollow instructions at [Docker for Arch Linux](https://wiki.archlinux.org/index.php/Docker#Installation):\n\n  ```bash\n  sudo pacman -S docker\n  ```\n\n#### Docker on Windows\n\n[Docker Desktop for Windows](https://hub.docker.com/editions/community/docker-ce-desktop-windows). Make sure to read the system requirements [here](https://docs.docker.com/docker-for-windows/install/).\n\n### 2. Install source{d} Engine\n\n*Please note that Docker Toolbox is not supported neither for Windows nor for macOS. In case that you're running Docker Toolbox, please consider updating to newer Docker Desktop for Mac or Docker Desktop for Windows. See [previous section](#1-install-docker)*\n\nDownload the **[latest release](https://github.com/src-d/engine/releases/latest)** for MacOS (Darwin), Linux or Windows.\n\n#### Engine on macOS\n\nDouble-click on the tar file to extract it.\n\nOpen your terminal and move the binary to your local bin folder to make it executable from any directory:\n\n```bash\nsudo mv ~/replace/path/to/engine_darwin_amd64/srcd /usr/local/bin/\n```\n\nIf you want to setup bash completion, add the following to your `$HOME/.bash_profile` file:\n\n```bash\n# source{d} Engine auto completion\nsource \u003c(srcd completion)\n```\n\n#### Engine on Linux\n\nExtract the contents of the tar file from your terminal:\n\n```bash\ntar -xvf ~/replace/path/to/engine_REPLACEVERSION_linux_amd64.tar.gz\n```\n\nMove the binary to your local bin folder to be executable from any directory:\n\n```bash\nsudo mv engine_linux_amd64/srcd /usr/local/bin/\n```\n\nIf you want to setup bash completion, add the following to your `$HOME/.bashrc` file:\n\n```bash\n# source{d} Engine auto completion\nsource \u003c(srcd completion)\n```\n\n#### Engine on Windows\n\n*Please note that from now on we assume that the commands are executed in `powershell` and not in `cmd`. Running them in `cmd` is not guaranteed to work. Proper support may be added in future releases.*\n\nTo run the following preparatory steps you need to run powershell as administrator.\n\n```powershell\nmkdir 'C:\\Program Files\\srcd'\n# Add the directory to the `%path%` to make it available from anywhere\nsetx /M PATH \"$($env:path);C:\\Program Files\\srcd\"\n# Now open a new powershell to apply the changes\n```\n\nExtract the tar file with the tool you prefer. Copy `srcd.exe` into the directory you created:\n```powershell\nmv engine_windows_amd64\\srcd.exe 'C:\\Program Files\\srcd'\n```\n\n### 3. Start source{d} Engine With Your Local Repositories\n\nNow it's time to initialize source{d} Engine and provide it with some repositories to analyze:\n\n```bash\n# Without a path Engine operates on the local directory,\n# it works with nested or sub-directories.\nsrcd init\n\n# You can also provide a path\nsrcd init \u003cpath\u003e\n```\n\nYou can also optionally provide a configration file through the `--config` flag as follows:\n```bash\nsrcd init --config \u003cconfig file path\u003e\n# Or\nsrcd init \u003cpath\u003e --config \u003cconfig file path\u003e\n```\n\nSee the documentation for the `--config` flag [here](docs/commands.md#srcd).\n\n**Note:**\nOnce Engine is initialized with a working dir, it does not watch for new repository creation, so if you want to add (or delete) repositories, you need to `init` again.\nAlso, database indexes are not updated automatically when its contents change, so in that cases, the index must be manually recreated.\nIf you want to inspect different datasets, it needs to be done separatelly: you can `init` one working directory, perform your queries, and then change to the other dataset performing an `init` with the working directory containing it. In such situations, indexes are kept, so you can change from one dataset to the other, without having to recreate them.\n\n**Note for MacOS:**\nDocker for Mac [requires enabling file sharing](https://docs.docker.com/docker-for-mac/troubleshoot/#volume-mounting-requires-file-sharing-for-any-project-directories-outside-of-users) for any path outside of `/Users`.\n\n### 4. Explore source{d} Engine\n\n_For the full list of the commands supported by `srcd` and those\nthat have been planned, please read [commands.md](docs/commands.md)._\n\n**Note for Windows:** Docker for Windows [requires shared drives](https://docs.docker.com/docker-for-windows/#shared-drives). Other than that, it's important to use a workdir that doesn't include any sub-directory whose access is not readable by the user running `srcd`. As an example using `C:\\Users` as workdir will most probably not work. For more details see [this issue](https://github.com/src-d/engine/issues/250).\n\nsource{d} Engine provides interfaces to [query code repositories](#querying-code) and to [parse code](#parsing-code) into [Universal Abstract Syntax Trees](#babelfish-uast).\n\nIn this section we will cover a mix of some commands and interfaces available.\n\n**Note:**\nsource{d} Engine will download and install Docker images on demand. Therefore, the first time you run some of these commands, they might take a bit of time to start up. Subsequent runs will be faster.\n\n#### Querying Code\n\n##### Query Web Interface\n\nTo launch the [web client for the SQL interface](https://github.com/src-d/gitbase-web), run the following command and start executing queries:\n\n```bash\n# Launch the query web client\nsrcd web sql\n```\n\nThis should open the [web interface](https://github.com/src-d/gitbase-web) in your browser.\nYou can also access it directly at [http://localhost:8080](http://localhost:8080).\n\n##### Query Command Line Interface (CLI)\n\nIf you prefer to work within your terminal via command line, you can open a SQL REPL\nthat allows you to execute queries against your repositories by executing:\n\n```bash\n# Launch the query CLI REPL\nsrcd sql\n```\n\nIf you want to run a query directly, you can also execute it as such:\n\n```bash\n# Run query via CLI\nsrcd sql \"SHOW tables;\"\n```\n\n**Note:**\nEngine's SQL supports a [UAST](#babelfish-uast) function that returns a Universal AST for the selected source text. UAST values are returned as binary blobs, and are best visualized in the `web sql` interface rather than the CLI where are seen as binary data.\n\n##### From Any MySQL Client\n\nYou may also connect directly to gitbase using any MySQL compatible client. Use the login user **root**, no password, and database name **gitbase**.\n\n```bash\n# Start the component if needed\nsrcd start srcd-cli-gitbase\n\nmysql --user=root --host=127.0.0.1 --port=3306 gitbase\n```\n\n#### Parsing Code\n\nSometimes you may want to parse files directly as [UASTs](#babelfish-uast).\n\nTo see which languages are available, check the table of [supported languages](#babelfish-uast).\n\n##### Parse Web Client\n\nIf you want a playground to see examples of the UAST, or run your own, you can launch the [parse web client](https://github.com/bblfsh/web).\n\n\n```bash\n# Launch the parse web client\nsrcd web parse\n```\n\nThis should open the [web interface](https://github.com/bblfsh/web) in your browser.\nYou can also access it directly at [http://localhost:8081](http://localhost:8081).\n\n##### Parse Command Line Interface (CLI)\n\nAlternatively, you can also start parsing files on the command line:\n\n```bash\n# Parse file via CLI\nsrcd parse uast /path/to/file.java\n```\n\nTo parse a file specifying the programming language:\n\n```bash\nsrcd parse uast --lang=LANGUAGE /path/to/file\n```\n\nTo see the installed language drivers:\n\n```bash\nsrcd parse drivers\n```\n\n### 5. Start Executing Queries\n\n**Understand which tables are available for you to query:**\n\n```bash\ngitbase\u003e show tables;\n+--------------+\n|    TABLE     |\n+--------------+\n| blobs        |\n| commit_blobs |\n| commit_files |\n| commit_trees |\n| commits      |\n| files        |\n| ref_commits  |\n| refs         |\n| remotes      |\n| repositories |\n| tree_entries |\n+--------------+\n```\n\n```bash\ngitbase\u003e DESCRIBE TABLE commits;\n+---------------------+-----------+\n|        NAME         |   TYPE    |\n+---------------------+-----------+\n| repository_id       | TEXT      |\n| commit_hash         | TEXT      |\n| commit_author_name  | TEXT      |\n| commit_author_email | TEXT      |\n| commit_author_when  | TIMESTAMP |\n| committer_name      | TEXT      |\n| committer_email     | TEXT      |\n| committer_when      | TIMESTAMP |\n| commit_message      | TEXT      |\n| tree_hash           | TEXT      |\n| commit_parents      | JSON      |\n+---------------------+-----------+\n```\n\n**Show me the repositories I am analyzing:**\n\n```sql\nSELECT * FROM repositories;\n```\n\n**Top 10 repositories by commit count in [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):**\n\n```sql\nSELECT repository_id,commit_count\nFROM (\n    SELECT r.repository_id, COUNT(*) AS commit_count\n    FROM ref_commits r\n    WHERE r.ref_name = 'HEAD'\n    GROUP BY r.repository_id\n) AS q\nORDER BY commit_count\nDESC\nLIMIT 10;\n```\n\n**Query all files from [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):**\n\n```sql\nSELECT cf.file_path, f.blob_content\nFROM ref_commits r\nNATURAL JOIN commit_files cf\nNATURAL JOIN files f\nWHERE r.ref_name = 'HEAD'\nAND r.history_index = 0;\n```\n\n**Retrieve the UAST for all files from [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):**\n\n```sql\nSELECT * FROM (\n    SELECT cf.file_path,\n           UAST(f.blob_content, LANGUAGE(f.file_path,  f.blob_content)) as uast\n    FROM ref_commits r\n    NATURAL JOIN commit_files cf\n    NATURAL JOIN files f\n    WHERE r.ref_name = 'HEAD'\n    AND r.history_index = 0\n) t WHERE uast != '';\n```\n\n**Query for all LICENSE \u0026 README files across history:**\n\n```sql\nSELECT repository_id, blob_content\nFROM files\nWHERE file_path = 'LICENSE'\nOR file_path = 'README.md';\n```\n\nYou can find further sample queries in the [examples](examples/README.md) folder.\n\n\u003c!---\n### 6. Next steps\n\nYou can now run source{d} Engine, choose what you would like to do next:\n\n- [**Analyze your git repositories**](#)\n- [**Understand how your code has evolved**](#)\n- [**Write your own static analysis rules**](#)\n- [**Build a data pipeline for MLonCode**](#)\n---\u003e\n\n## Other Guides and Examples\n\nHere is a collection of documentation, guides, and examples of the components exposed by source{d} Engine:\n\n- [gitbase documentation](https://docs.sourced.tech/gitbase/): table schemas, syntax, functions, examples.\n- [Babelfish documentation](https://docs.sourced.tech/babelfish/): specifications, usage, examples.\n\n## Architecture\n\nsource{d} Engine functions as a command-line interface tool that provides easy access to components of source{d} stack for Code As Data.\n\nIt consists of a daemon managing all of the services, which are packaged as Docker containers:\n\n- [enry](https://github.com/src-d/enry): language classifier\n- [babelfish](https://doc.bblf.sh): universal code parser\n  - [daemon](https://github.com/bblfsh/bblfshd): Babelfish server\n  - [language drivers](https://github.com/search?q=topic%3Adriver+org%3Abblfsh\u0026type=Repositories): parsers + normalizers for programming languages\n  - [babelfish-web](https://github.com/bblfsh/web): web client for Babelfish server\n- [gitbase](https://github.com/src-d/gitbase): SQL database interface to Git repositories\n  - [gitbase-web](https://github.com/src-d/gitbase-web): web client for gitbase\n\nFor more details on the architecture of this project, read [docs/architecture.md](docs/architecture.md).\n\n![diagram of source{d} Engine](docs/architecture-diagram.png)\n\n\n## Babelfish UAST\n\n### Definition\n\nOne of the most important components of source{d} Engine is the UAST, which stands for:\n[Universal Abstract Syntax Tree](https://docs.sourced.tech/babelfish/uast/uast-specification).\n\nUASTs are a normalized form of a programming language's AST, annotated with language-agnostic roles and transformed with language-agnostic concepts (e.g. Functions, Imports etc.).\n\nThese enable advanced static analysis of code and easy feature extraction for statistics or [Machine Learning on Code](https://github.com/src-d/awesome-machine-learning-on-source-code).\n\n### UAST Usage\n\nTo parse a file for a UAST using source{d} Engine, head to the [Parsing Code section](#parsing-code) of this document.\n\n### Supported Languages\n\nTo see which languages are available, check the table of [Babelfish supported languages](https://docs.sourced.tech/babelfish/languages).\n\n## Clients and Connectors\n\nFor connecting to the language parsing server (Babelfish) and analyzing the UAST, there are several language clients currently supported and maintained:\n\n- [Babelfish Go Client](https://github.com/bblfsh/go-client)\n- [Babelfish Python Client](https://github.com/bblfsh/client-python)\n- [Babelfish Scala Client](https://github.com/bblfsh/client-scala)\n\nThe Gitbase Spark connector is under development, which aims to allow for an easy integration with Spark \u0026 PySpark:\n\n- [Gitbase Spark Connector](https://github.com/src-d/gitbase-spark-connector) - coming soon!\n\n## Community\n\nsource{d} has an amazing community of developers and contributors who are interested in Code As Data and/or Machine Learning on Code. Please join us! 👋\n\n- [Community](https://sourced.tech/community/)\n- [Slack](http://bit.ly/src-d-community)\n- [Twitter](https://twitter.com/sourcedtech)\n- [Email](mailto:hello@sourced.tech)\n\n## Contributing\n\nContributions are **welcome and very much appreciated** 🙌\nPlease refer [to our Contribution Guide](CONTRIBUTING.md) for more details.\n\n## Credits\n\nThis software uses code from open source packages. We'd like to thank the contributors for all their efforts:\n\n- [Cobra](https://github.com/spf13/cobra)\n\n## License\n\n[Apache 2.0 License](LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fengine-deprecated","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrc-d%2Fengine-deprecated","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fengine-deprecated/lists"}