{"id":14988131,"url":"https://github.com/apache/incubator-livy","last_synced_at":"2025-05-12T15:25:28.846Z","repository":{"id":22115722,"uuid":"95343930","full_name":"apache/incubator-livy","owner":"apache","description":"Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.","archived":false,"fork":false,"pushed_at":"2025-04-23T13:45:58.000Z","size":3719,"stargazers_count":911,"open_issues_count":107,"forks_count":608,"subscribers_count":57,"default_branch":"master","last_synced_at":"2025-05-10T17:16:32.790Z","etag":null,"topics":["apachelivy","bigdata","livy","spark"],"latest_commit_sha":null,"homepage":"https://livy.apache.org/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-06-25T07:00:09.000Z","updated_at":"2025-05-07T22:05:34.000Z","dependencies_parsed_at":"2024-06-10T09:58:31.060Z","dependency_job_id":"5ce2b6e1-8dee-4435-8c49-06f8576fdbed","html_url":"https://github.com/apache/incubator-livy","commit_stats":{"total_commits":519,"total_committers":100,"mean_commits":5.19,"dds":0.7822736030828517,"last_synced_commit":"1f6bd7ab55db3e5b08ff133d554b32db83a23890"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-livy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-livy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-livy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-livy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/incubator-livy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253764510,"owners_count":21960589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apachelivy","bigdata","livy","spark"],"created_at":"2024-09-24T14:16:08.301Z","updated_at":"2025-05-12T15:25:28.821Z","avatar_url":"https://github.com/apache.png","language":"Scala","funding_links":[],"categories":["大数据","Packages"],"sub_categories":["Middleware"],"readme":"# Apache Livy\n\n[![Build Status](https://travis-ci.org/apache/incubator-livy.svg?branch=master)](https://travis-ci.org/apache/incubator-livy)\n\nApache Livy is an open source REST interface for interacting with\n[Apache Spark](http://spark.apache.org) from anywhere. It supports executing snippets of code or\nprograms in a Spark context that runs locally or in\n[Apache Hadoop YARN](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html).\n\n* Interactive Scala, Python and R shells\n* Batch submissions in Scala, Java, Python\n* Multiple users can share the same server (impersonation support)\n* Can be used for submitting jobs from anywhere with REST\n* Does not require any code change to your programs\n\n[Pull requests](https://github.com/apache/incubator-livy/pulls) are welcomed! But before you begin,\nplease check out the [Contributing](http://livy.incubator.apache.org/community#Contributing)\nsection on the [Community](http://livy.incubator.apache.org/community) page of our website.\n\n## Online Documentation\n\nGuides and documentation on getting started using Livy, example code snippets, and Livy API\ndocumentation can be found at [livy.incubator.apache.org](http://livy.incubator.apache.org).\n\n## Before Building Livy\n\nTo build Livy, you will need:\n\nDebian/Ubuntu:\n  * mvn (from ``maven`` package or maven3 tarball)\n  * openjdk-8-jdk (or Oracle JDK 8)\n  * Python 2.7+\n  * R 3.x\n\nRedhat/CentOS:\n  * mvn (from ``maven`` package or maven3 tarball)\n  * java-1.8.0-openjdk (or Oracle JDK 8)\n  * Python 2.7+\n  * R 3.x\n\nMacOS:\n  * Xcode command line tools\n  * Oracle's JDK 1.8\n  * Maven (Homebrew)\n  * Python 2.7+\n  * R 3.x\n\nRequired python packages for building Livy:\n  * cloudpickle\n  * requests\n  * requests-kerberos\n  * flake8\n  * flaky\n  * pytest\n\n\nTo run Livy, you will also need a Spark installation. You can get Spark releases at\nhttps://spark.apache.org/downloads.html.\n\nLivy requires Spark 2.4+. You can switch to a different version of Spark by setting the\n``SPARK_HOME`` environment variable in the Livy server process, without needing to rebuild Livy.\n\n\n## Building Livy\n\nLivy is built using [Apache Maven](http://maven.apache.org). To check out and build Livy, run:\n\n```\ngit clone https://github.com/apache/incubator-livy.git\ncd incubator-livy\nmvn package\n```\n\nYou can also use the provided [Dockerfile](./dev/docker/livy-dev-base/Dockerfile):\n\n```\ngit clone https://github.com/apache/incubator-livy.git\ncd incubator-livy\ndocker build -t livy-ci dev/docker/livy-dev-base/\ndocker run --rm -it -v $(pwd):/workspace -v $HOME/.m2:/root/.m2 livy-ci mvn package\n```\n\n\u003e **Note**: The `docker run` command maps the maven repository to your host machine's maven cache so subsequent runs will not need to download dependencies.\n\nBy default Livy is built against Apache Spark 2.4.5, but the version of Spark used when running\nLivy does not need to match the version used to build Livy. Livy internally handles the differences\nbetween different Spark versions.\n\nThe Livy package itself does not contain a Spark distribution. It will work with any supported\nversion of Spark without needing to rebuild.\n\n### Build Profiles\n\n| Flag         | Purpose                                                            |\n|--------------|--------------------------------------------------------------------|\n| -Phadoop2    | Choose Hadoop2 based build dependencies (default configuration)    |\n| -Pspark2     | Choose Spark 2.x based build dependencies (default configuration)  |\n| -Pspark3     | Choose Spark 3.x based build dependencies                          |\n| -Pscala-2.11 | Choose Scala 2.11 based build dependencies (default configuration) |        \n| -Pscala-2.12 | Choose scala 2.12 based build dependencies                         |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-livy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fincubator-livy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-livy/lists"}