{"id":19614634,"url":"https://github.com/activestate/aws-emr-spark-env","last_synced_at":"2026-06-10T02:31:10.122Z","repository":{"id":244290688,"uuid":"814814744","full_name":"ActiveState/aws-emr-spark-env","owner":"ActiveState","description":"A script to generate a runtime environment tarball from an ActiveState project","archived":false,"fork":false,"pushed_at":"2024-06-14T07:45:58.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-26T17:48:18.843Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ActiveState.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-13T19:02:01.000Z","updated_at":"2024-06-14T07:46:01.000Z","dependencies_parsed_at":"2024-11-11T10:53:17.752Z","dependency_job_id":"231d4468-1165-4199-a13a-d55b9ccbac4a","html_url":"https://github.com/ActiveState/aws-emr-spark-env","commit_stats":null,"previous_names":["activestate/aws-emr-spark-env"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ActiveState/aws-emr-spark-env","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveState%2Faws-emr-spark-env","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveState%2Faws-emr-spark-env/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveState%2Faws-emr-spark-env/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveState%2Faws-emr-spark-env/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ActiveState","download_url":"https://codeload.github.com/ActiveState/aws-emr-spark-env/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveState%2Faws-emr-spark-env/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34134633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T10:53:11.683Z","updated_at":"2026-06-10T02:31:10.081Z","avatar_url":"https://github.com/ActiveState.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repository provides a script that checks out an ActiveState project and copies the artifacts in a\nspecific runtime directory.\n\nThe runtime directory is /home/hadoop/environment/runtime which is a place where\nthe runtime will be un-packed on the EMR images later. (Refer to [1] to see how\nthis should look like without the State Tool)\n\nAs the runtime will be executed by potentially hundreds of hosts in parallel\nand thousands of time on each host, we do not want to use executors!\nInstead we want to use the binaries directly which is an undocumented\nuse case.\n\nThe writes a tarball with the artifacts in the `/output` directory.\nThe Path to the python executable can then be set as:\n\n/home/hadoop/environment/usr/bin/python\n\nHow to use this script inside your project:\n\nCreate a directory `bin` in your project workspace and copy the script file in it:\n\n```sh\nmkdir -p bin\ncurl https://raw.githubusercontent.com/ActiveState/aws-emr-spark-env/main/bin/create_spark_state_env.sh -O bin/create_spark_state_env.sh\n```\n\nNow you can create a `spark_env.tar.gz` file:\n\n```sh\nPROJECT=myorg/myproject\n\nstate auth\nACTIVESTATE_API_KEY=$(state export new-api-key state_env_key)\ndocker run -it -v $PWD/bin:/output --entrypoint=/bin/bash amazonlinux:2 /output /create_spark_state_env.sh $ACTIVESTATE_API_KEY $PROJECT\n```\n\nupload it to S3 \n\n```sh\nexport S3_UPLOAD_PREFIX=s3://your-bucket/your-prefix\n\n# bundle up your own source code (optional)\ngit archive --format=zip HEAD:src \u003e project_archive.zip\n\ns3_upload() {\n    S3_PATH=$1; shift\n    FILE=$1; shift\n\n    BASE_NAME=`basename $FILE`\n    S3_URL=$S3_PATH/$BASE_NAME\n    aws --profile sso s3 ls $S3_URL \u0026\u0026 return\n\n    echo \"Uploading $FILE to $S3_URL ...\"\n    aws --profile sso s3 cp $FILE $S3_URL\n}\n\n# upload to the s3\ns3_upload $S3_UPLOAD_PREFIX bin/state_env.tar.gz\ns3_upload $S3_UPLOAD_PREFIX project_archive.zip \ns3_upload $S3_UPLOAD_PREFIX migration-script.py\n```\n\nand finally schedule the job like this:\n\n```sh\nexport APP_ID='...'\n\nexport EXECUTION_ROLE='...'\nexport EXECUTION_ROLE_ARN=$(aws iam get-role --role-name $EXECUTION_ROLE | jq -r .Role.Arn )\n\nJOB_DRIVER=$(jq -n \\\n     --arg cs \"$CORES\" \\\n     --arg mem \"$MEMORY\" \\\n     --arg execs \"$MAX_EXECUTORS\" \\\n     --arg migration_file \"$S3_UPLOAD_PATH/migration-script.py\" \\\n     --arg state_env_file \"$S3_UPLOAD_PATH/state_env.tar.gz\" \\\n     --arg project_archive_file \"$S3_PATH/project_archive.zip\" \\\n     --arg python_path \"./environment/usr/bin/python\" \\\n     --argjson args \"[$LIMIT_ARGS\\\"--output_suffix=$SUFFIX\\\", \\\"$SOURCE\\\"]\" \\\n     '{\n        sparkSubmit: {\n          entryPoint: $migration_file,\n          entryPointArguments: $args,\n          sparkSubmitParameters: (\n            \"--conf spark.executor.cores=\"+$cs+\n            \" --conf spark.executor.memory=\"+$mem+\n            \" --conf spark.driver.cores=\"+$cs+\n            \" --conf spark.driver.memory=\"+$mem+\n            \" --conf spark.archives=\"+$state_env_file+\"#environment,\"+\n            \" --conf spark.emr-serverless.driverEnv.PYSPARK_DRIVER_PYTHON=\"+$python_path+\n            \" --conf spark.emr-serverless.driverEnv.PYSPARK_PYTHON=\"+$python_path+\n            \" --conf spark.executorEnv.PYSPARK_PYTHON=\"+$python_path+ \n            \" --conf spark.submit.pyFiles=\"+$project_archive+\n            \" --conf spark.dynamicAllocation.maxExecutors=\"+$execs+\n            \" --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory\"\n            )\n        }\n      }')\n\nCONFIG_OVERRIDES=$(jq -n \\\n    --arg log_uri \"$S3_PATH/logs\" \\\n    '{\n        monitoringConfiguration: {\n            s3MonitoringConfiguration: {\n                logUri: $log_uri\n            }\n        }\n    }'\n)\n\nJOB_ID=$(aws emr-serverless start-job-run --application-id $APP_ID \\\n   --execution-role-arn $EXECUTION_ROLE_ARN \\\n   --name $APP_NAME \\\n   --job-driver \"$JOB_DRIVER\" \\\n   --configuration-overrides \"$CONFIG_OVERRIDES\" | jq -r .jobRunId )\n\necho $JOB_ID\n\nwatch \"aws emr-serverless get-job-run --application-id $APP_ID --job-run-id $JOB_ID | jq '.jobRun | {state: .state, details: .stateDetails}'\"\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Factivestate%2Faws-emr-spark-env","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Factivestate%2Faws-emr-spark-env","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Factivestate%2Faws-emr-spark-env/lists"}