{"id":15710297,"url":"https://github.com/aws/sagemaker-training-toolkit","last_synced_at":"2025-05-14T04:07:50.699Z","repository":{"id":37243395,"uuid":"212439434","full_name":"aws/sagemaker-training-toolkit","owner":"aws","description":"Train machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.","archived":false,"fork":false,"pushed_at":"2025-02-11T16:56:57.000Z","size":1084,"stargazers_count":512,"open_issues_count":59,"forks_count":134,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-05-08T00:08:05.110Z","etag":null,"topics":["aws","deep-learning","docker","machine-learning","python","sagemaker","training"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-02T20:54:32.000Z","updated_at":"2025-05-01T15:29:44.000Z","dependencies_parsed_at":"2023-02-13T18:01:27.256Z","dependency_job_id":"4e900d3e-d80e-46b6-87fa-ee9bd2f20230","html_url":"https://github.com/aws/sagemaker-training-toolkit","commit_stats":{"total_commits":482,"total_committers":58,"mean_commits":8.310344827586206,"dds":0.5809128630705394,"last_synced_commit":"54721512597e77530a68e14c35a857be2f5c5687"},"previous_names":[],"tags_count":151,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fsagemaker-training-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fsagemaker-training-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fsagemaker-training-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fsagemaker-training-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws","download_url":"https://codeload.github.com/aws/sagemaker-training-toolkit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254069453,"owners_count":22009557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","deep-learning","docker","machine-learning","python","sagemaker","training"],"created_at":"2024-10-03T21:05:45.304Z","updated_at":"2025-05-14T04:07:50.639Z","avatar_url":"https://github.com/aws.png","language":"Python","readme":"![SageMaker](https://github.com/aws/sagemaker-training-toolkit/raw/master/branding/icon/sagemaker-banner.png)\n\n# SageMaker Training Toolkit\n\n[![Latest Version](https://img.shields.io/pypi/v/sagemaker-training.svg)](https://pypi.python.org/pypi/sagemaker-training) [![Supported Python Versions](https://img.shields.io/pypi/pyversions/sagemaker-training.svg)](https://pypi.python.org/pypi/sagemaker-training) [![Code Style: Black](https://img.shields.io/badge/code_style-black-000000.svg)](https://github.com/python/black)\n\nTrain machine learning models within a Docker container using Amazon SageMaker.\n\n\n## :books: Background\n\n[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a fully managed service for data science and machine learning (ML) workflows.\nYou can use Amazon SageMaker to simplify the process of building, training, and deploying ML models.\n\nTo train a model, you can include your training script and dependencies in a [Docker container](https://www.docker.com/resources/what-container) that runs your training code.\nA container provides an effectively isolated environment, ensuring a consistent runtime and reliable training process. \n\nThe **SageMaker Training Toolkit** can be easily added to any Docker container, making it compatible with SageMaker for [training models](https://aws.amazon.com/sagemaker/train/).\nIf you use a [prebuilt SageMaker Docker image for training](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html), this library may already be included.\n\nFor more information, see the Amazon SageMaker Developer Guide sections on [using Docker containers for training](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms.html).\n\n## :hammer_and_wrench: Installation\n\nTo install this library in your Docker image, add the following line to your [Dockerfile](https://docs.docker.com/engine/reference/builder/):\n\n``` dockerfile\nRUN pip3 install sagemaker-training\n```\n\n## :computer: Usage\n\nThe following are brief how-to guides.\nFor complete, working examples of custom training containers built with the SageMaker Training Toolkit, please see [the example notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/custom-training-containers).\n\n### Create a Docker image and train a model\n\n1. Write a training script (eg. `train.py`).\n\n2. [Define a container with a Dockerfile](https://docs.docker.com/get-started/part2/#define-a-container-with-dockerfile) that includes the training script and any dependencies.\n\n    The training script must be located in the `/opt/ml/code` directory.\n    The environment variable `SAGEMAKER_PROGRAM` defines which file inside the `/opt/ml/code` directory to use as the training entry point.\n    When training starts, the interpreter executes the entry point defined by `SAGEMAKER_PROGRAM`.\n    Python and shell scripts are both supported.\n    \n    ``` docker\n    FROM yourbaseimage:tag\n  \n    # install the SageMaker Training Toolkit \n    RUN pip3 install sagemaker-training\n\n    # copy the training script inside the container\n    COPY train.py /opt/ml/code/train.py\n\n    # define train.py as the script entry point\n    ENV SAGEMAKER_PROGRAM train.py\n    ```\n\n3. Build and tag the Docker image.\n\n    ``` shell\n    docker build -t custom-training-container .\n    ```\n\n4. Use the Docker image to start a training job using the [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk).\n\n    ``` python\n    from sagemaker.estimator import Estimator\n\n    estimator = Estimator(image_name=\"custom-training-container\",\n                          role=\"SageMakerRole\",\n                          train_instance_count=1,\n                          train_instance_type=\"local\")\n\n    estimator.fit()\n    ```\n    \n    To train a model using the image on SageMaker, [push the image to ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html) and start a SageMaker training job with the image URI.\n    \n\n### Pass arguments to the entry point using hyperparameters\n\nAny hyperparameters provided by the training job are passed to the entry point as script arguments.\nThe SageMaker Python SDK uses this feature to pass special hyperparameters to the training job, including `sagemaker_program` and `sagemaker_submit_directory`.\nThe complete list of SageMaker hyperparameters is available [here](https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/params.py).\n\n1. Implement an argument parser in the entry point script. For example, in a Python script:\n\n    ``` python\n    import argparse\n\n    if __name__ == \"__main__\":\n      parser = argparse.ArgumentParser()\n\n      parser.add_argument(\"--learning-rate\", type=int, default=1)\n      parser.add_argument(\"--batch-size\", type=int, default=64)\n      parser.add_argument(\"--communicator\", type=str)\n      parser.add_argument(\"--frequency\", type=int, default=20)\n\n      args = parser.parse_args()\n      ...\n    ```\n\n2. Start a training job with hyperparameters.\n\n    ``` python\n    {\"HyperParameters\": {\"batch-size\": 256, \"learning-rate\": 0.0001, \"communicator\": \"pure_nccl\"}}\n    ```\n\n### Read additional information using environment variables\n\nAn entry point often needs additional information not available in `hyperparameters`.\nThe SageMaker Training Toolkit writes this information as environment variables that are available from within the script.\nFor example, this training job includes the channels `training` and `testing`:\n\n``` python\nfrom sagemaker.pytorch import PyTorch\n\nestimator = PyTorch(entry_point=\"train.py\", ...)\n\nestimator.fit({\"training\": \"s3://bucket/path/to/training/data\", \n               \"testing\": \"s3://bucket/path/to/testing/data\"})\n```\n\nThe environment variables `SM_CHANNEL_TRAINING` and `SM_CHANNEL_TESTING` provide the paths to the channels:\n\n``` python\nimport argparse\nimport os\n\nif __name__ == \"__main__\":\n  parser = argparse.ArgumentParser()\n\n  ...\n\n  # reads input channels training and testing from the environment variables\n  parser.add_argument(\"--training\", type=str, default=os.environ[\"SM_CHANNEL_TRAINING\"])\n  parser.add_argument(\"--testing\", type=str, default=os.environ[\"SM_CHANNEL_TESTING\"])\n\n  args = parser.parse_args()\n\n  ...\n```\n\nWhen training starts, SageMaker Training Toolkit will print all available environment variables. Please see the [reference on environment variables](https://github.com/aws/sagemaker-training-toolkit/blob/master/ENVIRONMENT_VARIABLES.md) for a full list of provided environment variables.\n\n### Get information about the container environment\n\nTo get information about the container environment, initialize an `Environment` object.\n`Environment` provides access to aspects of the environment relevant to training jobs, including hyperparameters, system characteristics, filesystem locations, environment variables and configuration settings.\nIt is a read-only snapshot of the container environment during training, and it doesn't contain any form of state.\n\n``` python\nfrom sagemaker_training import environment\n\nenv = environment.Environment()\n\n# get the path of the channel \"training\" from the `inputdataconfig.json` file\ntraining_dir = env.channel_input_dirs[\"training\"]\n\n# get a the hyperparameter \"training_data_file\" from `hyperparameters.json` file\nfile_name = env.hyperparameters[\"training_data_file\"]\n\n# get the folder where the model should be saved\nmodel_dir = env.model_dir\n\n# train the model\ndata = np.load(os.path.join(training_dir, file_name))\nx_train, y_train = data[\"features\"], keras.utils.to_categorical(data[\"labels\"])\nmodel = ResNet50(weights=\"imagenet\")\n...\nmodel.fit(x_train, y_train)\n\n#save the model to the model_dir at the end of training\nmodel.save(os.path.join(model_dir, \"saved_model\"))\n```\n\n### Execute the entry point\n\nTo execute the entry point, call `entry_point.run()`.\n\n``` python\nfrom sagemaker_training import entry_point, environment\n\nenv = environment.Environment()\n\n# read hyperparameters as script arguments\nargs = env.to_cmd_args()\n\n# get the environment variables\nenv_vars = env.to_env_vars()\n\n# execute the entry point\nentry_point.run(uri=env.module_dir,\n                user_entry_point=env.user_entry_point,\n                args=args,\n                env_vars=env_vars)\n\n```\n\nIf the entry point execution fails, `trainer.train()` will write the error message to `/opt/ml/output/failure`. Otherwise, it will write to the file `/opt/ml/success`.\n\n## :scroll: License\n\nThis library is licensed under the [Apache 2.0 License](http://aws.amazon.com/apache2.0/).\nFor more details, please take a look at the [LICENSE](https://github.com/aws/sagemaker-training-toolkit/blob/master/LICENSE) file.\n\n## :handshake: Contributing\n\nContributions are welcome!\nPlease read our [contributing guidelines](https://github.com/aws/sagemaker-training-toolkit/blob/master/CONTRIBUTING.md)\nif you'd like to open an issue or submit a pull request.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Fsagemaker-training-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws%2Fsagemaker-training-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Fsagemaker-training-toolkit/lists"}