{"id":31183748,"url":"https://github.com/monajemi-arman/sparkling","last_synced_at":"2026-04-12T00:03:56.848Z","repository":{"id":311733906,"uuid":"1043616507","full_name":"monajemi-arman/sparkling","owner":"monajemi-arman","description":"Easy to use Spark cluster management panel with GPU support","archived":false,"fork":false,"pushed_at":"2025-09-14T06:55:19.000Z","size":305,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-14T08:36:42.927Z","etag":null,"topics":["apache-spark","csharp","cuda","distributed-computing","distributed-learning","docker","gpu","javascript","nextjs","torch","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monajemi-arman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-24T08:45:46.000Z","updated_at":"2025-09-14T06:55:22.000Z","dependencies_parsed_at":"2025-08-26T09:25:28.746Z","dependency_job_id":"d8179b59-0933-4c6a-953e-3049a1754974","html_url":"https://github.com/monajemi-arman/sparkling","commit_stats":null,"previous_names":["monajemi-arman/sparkling"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/monajemi-arman/sparkling","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monajemi-arman%2Fsparkling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monajemi-arman%2Fsparkling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monajemi-arman%2Fsparkling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monajemi-arman%2Fsparkling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monajemi-arman","download_url":"https://codeload.github.com/monajemi-arman/sparkling/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monajemi-arman%2Fsparkling/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275981935,"owners_count":25564143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-19T02:00:09.700Z","response_time":108,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","csharp","cuda","distributed-computing","distributed-learning","docker","gpu","javascript","nextjs","torch","typescript"],"created_at":"2025-09-19T18:10:18.315Z","updated_at":"2026-04-12T00:03:56.841Z","avatar_url":"https://github.com/monajemi-arman.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ✨ Sparkling ✨\nSparkling is an easy-to-use Apache Spark setup and management platform. Written in Typescript and C#, its main features are as following:\n*   🚀 Setup Spark nodes automatically\n*   👥 Manage users working with Spark nodes\n*   💰 User account balance by hours\n*   📚 Jupyter notebook included\n*   🐳 Powered by Docker under-the-hood\n\n![screenshot](dashboard.png)\n\n---\n## 🛠️ Requirements\n* **Nvidia Drivers** and **Cuda Toolkit 12.9 (or higher)** installed on all GPU-enabled systems.\n* This software is meant for **Linux** based operating systems only!  \n* **Docker** and **SSH** are required. Install Docker from the official repository:\n```bash\ncurl -fsSL https://get.docker.com | sh\n```\n* Nvidia container toolkit must be installed using official guide at: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html\n* Finally, install and enable the SSH service:\n```bash\nsudo apt update\nsudo apt install -y openssh-server\nsudo systemctl enable ssh\nsudo systemctl start ssh\n```\n\nOnly for **development**, you would need **dotnet-9** and **node.js** installed.\n\n---\n\n## 🚀 Usage\nFollow these steps to start the project:\n*   ⬇️ **Clone** this repository\n*   🛠️ **Build our docker images**, in the repository directory, run `./docker-images/build`\n*   🔐 **Generate self-signed certificates**, in the repository directory, run `./sparkling-frontend/generate-ssl-certificates.sh`\n*   ⬆️ **To start the program**, in the repository directory, run `docker compose up`\n*   🌐 The web-based panel should now be up at `http://localhost`\n*   🔑 **Login** with default admin credentials: `info@sparklean.io` and `123456Aa!@#`\n*   ➕ Under the node list, **add a local node** (required) and any other desired nodes.\n    * For each new node, click **Manage**, then **Setup Script** to download and run the script on the target node.\n    * Once the setup script is complete, return to **Manage** and click **Activate**. You can now create work on your nodes.\n*   ▶️ Go to work list and **start your first work**\n*   📊 Click on Jupyter to **open your work session**\n*   ❓ Use the help for PySpark commands\n*   🗑️ When done, **delete the work** (hours will be diminished from your account if you are not admin)\n\n## Maintating the Project\nThe following are notes to take into consideration for collaborating or maintaining the project:\n* Python version of **spark-cuda** and **jupyter-custom** docker must be the same.\n\n## Known Issues\n* After creating a master node for the first time, if you want to create it a second time, you get an error unless you manually stop and remove the spark-master container on the master node before re-activating:\n```bash\ndocker stop spark-master \u0026\u0026 docker rm spark-master\n```\n\n## Security Bugs\nIf you care to run this infrastructure in production environment, you are highly advised against doing so.  \nFor starters, master address is vulnerable to **remote command execution**, but you would need admin access to the panel.  \nWhen running this software, you will have to open up your docker API on a tcp port which exposes it to whoever has localhost access, possibly causing **escalation** exploits.\nIf in the future I fix these bugs, I will modify this README. I may even re-write the backend with Rust as a project.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonajemi-arman%2Fsparkling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonajemi-arman%2Fsparkling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonajemi-arman%2Fsparkling/lists"}