{"id":23479500,"url":"https://github.com/eleflow/sparknotebook","last_synced_at":"2025-04-14T21:45:10.998Z","repository":{"id":28278338,"uuid":"31788871","full_name":"eleflow/sparknotebook","owner":"eleflow","description":"A fast way of getting a Spark cluster up and running on AWS with the friendly IPython interface.","archived":false,"fork":false,"pushed_at":"2015-05-08T18:36:14.000Z","size":1555,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-28T09:51:31.445Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eleflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-06T21:01:21.000Z","updated_at":"2017-03-01T23:46:40.000Z","dependencies_parsed_at":"2022-08-26T15:00:22.443Z","dependency_job_id":null,"html_url":"https://github.com/eleflow/sparknotebook","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eleflow%2Fsparknotebook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eleflow%2Fsparknotebook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eleflow%2Fsparknotebook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eleflow%2Fsparknotebook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eleflow","download_url":"https://codeload.github.com/eleflow/sparknotebook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248966773,"owners_count":21190841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-24T19:29:53.698Z","updated_at":"2025-04-14T21:45:10.969Z","avatar_url":"https://github.com/eleflow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Notebook\n\n[![Build Status][build-badge]][build-url]\n[![License][license-badge]][license-url]\n\nThe Spark Notebook project is fast way of getting a [Spark](http://spark.apache.org/) cluster up and running on [AWS](http://aws.amazon.com) with the friendly [IPython](http://ipython.org) interface.\n\n## Before you start\nYou'll need \n\n1. to have [Docker installed](https://docs.docker.com/installation/) (recommended) or [no docker setup](nodocker.md)\n1. [AWS access keys](http://aws.amazon.com/developers/access-keys) \n1. One [AWS keypair](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair)\n\n \n## Setup\n1. git clone https://github.com/eleflow/sparknotebook.git\n1. cd sparknotebook\n1. create a aws.deploy.env file with these:\n\n  ```sh\n  AWS_ACCESS_KEY_ID=\u003cYOUR AWS ACCESS KEY\u003e\n  AWS_SECRET_ACCESS_KEY=\u003cYOUR AWS SEECRET ACCESS KEY\u003e\n  AWS_KEY_PAIR=\u003cYOUR AWS KEY PAIR NAME\u003e\n  ```\n1. Run \n\n``` $ docker build --rm -f=aws.deploy.Dockerfile -t=aws.deploy .```\n\n## Running the Notebook on AWS\n\n1. Run `sudo docker run -it --env-file ./aws.deploy.env --volume $PWD:/sparknotebook --volume $HOME/.ssh:/.ssh aws.deploy` and if all goes well you will see the ip of your sparknotebook server in a line like this\n  ```sh \n  ...\n\n  PLAY RECAP ******************************************************************** \n  52.10.183.42               : ok=21   changed=3    unreachable=0    failed=0   \n  ```\n1. Where 52.10.183.42 will be replaced with another ip address. Put that ip address on your browser to get access to the notebook\n\n## Spark Notebook\n\nSpark Notebook kernel is deployed into your server, and you can access it through the port 80, using an HTTP browser.\nThe initial notebook state is showed in the picture below:\n\n![Alt text](/../images/images/EmptyNotebook.png?raw=true \"Initial state of a Spark Notebook\")\n\nTo start a new notebook, just click in the New Notebook button, and you will be redirected to a new tab, containing an empty notebook.\nThe notebook is a code container that contains multiple TextArea components, where you can insert any kind of Scala code, including multi lines scripts. To execute the desired code, put the focus into the code TextArea component and hit Shift + ENTER or click in the play button (positioned at the notebook Header). Each time that you submit a code to the notebook, it will be compiled and if it compiles, it will be executed.\n\n## Cluster Settings\n\nOne of the cluster settings you are likely to change is the number of slaves. To change it to 30, you can run this code on the Spark Noteook\n```scala\n  ClusterSettings.coreInstanceCount = 30 // Number of workers available in your cluster - default to 3\n```\nTo see other settings see  [ClusterSettings](src/main/scala/eleflow/sparknotebook/SparkNotebookContext.scala)\n## SparkContext\nA SparkContext can be accessed with:\n```scala\n  sparkContext\n```\nThis is a method of SparkNotebookContext and it provisions the machines and sets up the cluster the first time it runs. An example of output of this method is showed below:\n\n![Alt text](/../images/images/ClusterInstantiation.png?raw=true \"Sample output of a cluster instantiation\")\n\n## Shutdown\n\nTo shutdown the cluster and terminate the cluster master and slaves run:\n```scala\n    terminate\n```\n\n## Monitoring\n### Ganglia\n\nThe master instance of your cluster also has a monitoring tool named Ganglia installed and it's address is displayed when you create the SparkContext.\nGanglia is a useful tool that help you to monitor the CPU, memory and disk usage, displaying graphs of this components. JVM data like, gc executions. It's very useful to help you to setup the correct cluster size, for your tasks.\nThe ganglia address is printed in the screen during the cluster instantiation. It's always deployed to the masterhost:5080/ganglia address.\nIt's important to note that the information showed at ganglia has a little delay.\n\n# Local build\n\nTo build and run locally go [here](buildlocal.md)\n\n# License\n\nThis project is distributed under Apache License Version 2.0\n\n[build-badge]: https://travis-ci.org/eleflow/sparknotebook.svg?branch=master\n[build-url]: https://travis-ci.org/eleflow/sparknotebook\n[license-badge]: https://img.shields.io/badge/License-Apache%202-blue.svg?style=flat\n[license-url]: LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feleflow%2Fsparknotebook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feleflow%2Fsparknotebook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feleflow%2Fsparknotebook/lists"}