https://github.com/zonca/comet-pyspark

https://github.com/zonca/comet-pyspark

Last synced: 3 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/zonca/comet-pyspark
• Owner: zonca
• Created: 2015-07-19T01:55:26.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2015-08-07T20:40:16.000Z (almost 10 years ago)
• Last Synced: 2025-01-13T10:30:46.897Z (5 months ago)
• Language: Shell
• Size: 129 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
Interactive pyspark shell on Comet

=================================

* Set `module load python scipy` in `.bashrc`

* Submit `spark` job to SLURM with `qsub spark.cmd`

* Load environment with `source slurm-env.sh` (created after `spark` starts)

* Login to the head node with `ssh $SLURMD_NODENAME`

* Configure the environment with:

        source slurm-env.sh

        module load spark

        source ~/mycluster.conf/spark/spark-env.sh

        

* Start the interactive console with `pyspark`