{"id":16514785,"url":"https://github.com/andgineer/spark-aws-rdkit","last_synced_at":"2025-10-28T04:32:37.456Z","repository":{"id":145706799,"uuid":"346337792","full_name":"andgineer/spark-aws-rdkit","owner":"andgineer","description":"Docker image with Apache Spark / Hadoop3 (compatible with AWS services like S3) and with RDKit installed in anaconda environment","archived":false,"fork":false,"pushed_at":"2024-09-20T04:24:35.000Z","size":67,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-01T11:35:01.810Z","etag":null,"topics":["anaconda","aws","pyspark","rdkit","spark"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andgineer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-03-10T11:49:05.000Z","updated_at":"2024-09-20T04:24:38.000Z","dependencies_parsed_at":"2024-03-17T08:33:02.154Z","dependency_job_id":"26ad110c-43b4-426e-89e7-9fded8db37a3","html_url":"https://github.com/andgineer/spark-aws-rdkit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fspark-aws-rdkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fspark-aws-rdkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fspark-aws-rdkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fspark-aws-rdkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andgineer","download_url":"https://codeload.github.com/andgineer/spark-aws-rdkit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238597388,"owners_count":19498396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anaconda","aws","pyspark","rdkit","spark"],"created_at":"2024-10-11T16:13:48.195Z","updated_at":"2025-10-28T04:32:37.439Z","avatar_url":"https://github.com/andgineer.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Docker-based PySpark cluster with AWS services and RDKit\n\nDocker-based Apache Spark standalone cluster with \nAWS services integration (S3, etc.) and comprehensive Data Science environment \nincluding PySpark, Pandas, and RDKit for cheminformatics.\n\n## Features\n\n- Apache Spark cluster in standalone mode\n- Full AWS services compatibility (S3, etc.)\n- Conda environment with data science tools:\n  - PySpark\n  - Pandas\n  - RDKit for cheminformatics (https://www.rdkit.org) \n- Deployment options:\n  - Local with `docker compose`\n  - Cloud with AWS ECS\n\n## Quick Start\n\nLaunch locally with `docker compose`:\n\n```bash\n./compose.sh up --build\n```\n\nThis starts:\n- Spark Master\n- Two Spark Workers\n- Example job container (`submit`)\n\nAccess points:\n- Spark Web UI: http://localhost:8080\n- Spark Driver: `spark://localhost:7077`\n  - For PySpark, use `setMaster('spark://localhost:7077')`\n\n\u003e Note: On Linux, change `docker.for.mac.localhost` to `localhost` in `.env` file.\n\n## Example PySpark Application\n\nThe `submit` container demonstrates how to:\n- Connect to the Spark cluster\n- Submit Spark jobs\n- Process data with PySpark\n\nCheck the `src/` directory for implementation details.\n\n## AWS ECS Deployment\n\nFor production deployment on AWS Elastic Container Service (ECS):\n\n1. Navigate to `ecs/` directory\n2. Configure your deployment in `config.sh`\n3. Run the automated deployment scripts\n\nDetailed instructions available in `ecs/README.md`.\n\n## Docker Images\n\nPublic Docker images are available on Docker Hub—no need to build locally:\n\n1. **[andgineer/spark-aws](https://hub.docker.com/r/andgineer/spark-aws)**\n   - Base image with Spark 3 and Hadoop 3\n   - AWS services integration\n\n2. **[andgineer/spark-aws-conda](https://hub.docker.com/r/andgineer/spark-aws-conda)**\n   - Extends base image\n   - Adds Anaconda with Pandas\n\n3. **[andgineer/spark-aws-rdkit](https://hub.docker.com/r/andgineer/spark-aws-rdkit)**\n   - Adds RDKit for cheminformatics\n   - Complete data science environment\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandgineer%2Fspark-aws-rdkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandgineer%2Fspark-aws-rdkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandgineer%2Fspark-aws-rdkit/lists"}