{"id":13528179,"url":"https://github.com/lyft/metadataproxy","last_synced_at":"2025-05-16T13:06:37.578Z","repository":{"id":47021281,"uuid":"52999353","full_name":"lyft/metadataproxy","owner":"lyft","description":"A proxy for AWS's metadata service that gives out scoped IAM credentials from STS","archived":false,"fork":false,"pushed_at":"2023-06-23T21:01:47.000Z","size":141,"stargazers_count":461,"open_issues_count":3,"forks_count":68,"subscribers_count":300,"default_branch":"master","last_synced_at":"2025-04-03T10:11:17.319Z","etag":null,"topics":["lyft"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lyft.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-03-02T21:56:23.000Z","updated_at":"2025-03-30T23:00:57.000Z","dependencies_parsed_at":"2022-08-23T12:10:29.589Z","dependency_job_id":"73f6f6ef-b235-4bf6-9fea-0ec6e085afdd","html_url":"https://github.com/lyft/metadataproxy","commit_stats":{"total_commits":138,"total_committers":43,"mean_commits":"3.2093023255813953","dds":0.6666666666666667,"last_synced_commit":"57045cb730a60ba7f3ee8ec2b7f953dcc3175f10"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyft%2Fmetadataproxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyft%2Fmetadataproxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyft%2Fmetadataproxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyft%2Fmetadataproxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lyft","download_url":"https://codeload.github.com/lyft/metadataproxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525564,"owners_count":21118699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lyft"],"created_at":"2024-08-01T06:02:16.619Z","updated_at":"2025-04-12T06:20:18.157Z","avatar_url":"https://github.com/lyft.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"1a9934198e37d6d06b881705b863afc8\"\u003e\u003c/a\u003e通信\u0026\u0026代理\u0026\u0026反向代理\u0026\u0026隧道","Python","\u003ca id=\"d03d494700077f6a65092985c06bf8e8\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"56acb7c49c828d4715dce57410d490d1\"\u003e\u003c/a\u003e未分类-Proxy","\u003ca id=\"0ff94312f3ab4898f5996725133ea9d1\"\u003e\u003c/a\u003e未分类"],"readme":"# metadataproxy\n\nThe metadataproxy is used to allow containers to acquire IAM roles. By metadata we mean [EC2 instance meta data](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) which is normally available to EC2 instances. This proxy exposes the meta data to containers inside or outside of EC2 hosts, allowing you to provide scoped IAM roles to individual containers, rather than giving them the full IAM permissions of an IAM role or IAM user.\n\n## Installation\n\nFrom inside of the repo run the following commands:\n\n```bash\nmkdir -p /srv/metadataproxy\ncd /srv/metadataproxy\nvirtualenv venv\nsource venv/bin/activate\npip install metadataproxy\ndeactivate\n```\n\n## Configuration\n\n### Modes of operation\n\nSee [the settings file](https://github.com/lyft/metadataproxy/blob/master/metadataproxy/settings.py)\nfor specific configuration options.\n\nThe metadataproxy has two basic modes of operation:\n\n1. Running in AWS where it simply proxies most routes to the real metadata\n   service.\n2. Running outside of AWS where it mocks out most routes.\n\nTo enable mocking, use the environment variable:\n\n```\nexport MOCK_API=true\n```\n\n### AWS credentials\n\nmetadataproxy relies on boto configuration for its AWS credentials. If metadata\nIAM credentials are available, it will use this. Otherwise, you'll need to use\n.aws/credentials, .boto, or environment variables to specify the IAM\ncredentials before the service is started.\n\n### Role assumption\n\nFor IAM routes, the metadataproxy will use STS to assume roles for containers.\nTo do so it takes the incoming IP address of metadata requests and finds the\nrunning docker container associated with the IP address. It uses the value of\nthe container's `IAM_ROLE` environment variable as the role it will assume. It\nthen assumes the role and gives back STS credentials in the metadata response.\n\nSTS-attained credentials are cached and automatically rotated as they expire.\n\n#### Container-specific roles\n\nTo specify the role of a container, simply launch it with the `IAM_ROLE`\nenvironment variable set to the IAM role you wish the container to run with.\n\nIf the trust policy for the role requires an ExternalId, you can set this\nusing the `IAM_EXTERNAL_ID` environment variable. This is most frequently\nused with cross-account role access scenarios. For more information on\nwhen you should use an External ID for your roles, see:\n\nhttp://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html\n\n```shell\ndocker run -e IAM_ROLE=my-role ubuntu:14.04\ndocker run -e IAM_ROLE=their-role@another-account -e IAM_EXTERNAL_ID=random-unique-string ubuntu:14.04\n```\n\n#### Configurable Behavior\n\nThere are a number of environment variables that can be set to tune\nmetadata proxy's behavior. They can either be exported by the start\nscript, or set via docker environment variables.\n\n| Variable | Type | Default | Description |\n| -------- | ---- | ------- | ----------- |\n| **DEFAULT\\_ROLE** | String | | Role to use if IAM\\_ROLE is not set in a container's environment. If unset the container will get no IAM credentials. |\n| **DEFAULT\\_ACCOUNT\\_ID** | String | | The default account ID to assume roles in, if IAM\\_ROLE does not contain account information. If unset, metadataproxy will attempt to lookup role ARNs using iam:GetRole. |\n| **ROLE\\_SESSION\\_KEY** | String | | Optional key in container labels or environment variables to use for role session name. Prefix with `Labels:` or `Env:` respectively to indicate where key should be found. Useful to pass through metadata such as a CI job ID or launching user for audit purposes, as the role session name is included in the ARN that appears in access logs. |\n| DEBUG | Boolean | False | Enable debug mode. You should not do this in production as it will leak IAM credentials into your logs |\n| DOCKER\\_URL | String | unix://var/run/docker.sock | Url of the docker daemon. The default is to access docker via its socket. |\n| METADATA\\_URL | String | http://169.254.169.254 | URL of the metadata service. Default is the normal location of the metadata service in AWS. |\n| MOCK\\_API | Boolean | False | Whether or not to mock all metadata endpoints. If True, mocked data will be returned to callers. If False, all endpoints except for IAM endpoints will be proxied through to the real metadata service. |\n| MOCKED\\_INSTANCE\\_ID | String | mockedid | When mocking the API, use the following instance id in returned data. |\n| AWS\\_ACCOUNT\\_MAP | JSON String | `{}` | A mapping of account names to account IDs. This allows you to use user-friendly names instead of account IDs in IAM\\_ROLE environment variable values. |\n| AWS\\_REGION | String |  | AWS Region for the STS endpoint allow you to call region based endpoint instead of global one. [AWS STS region endpoints.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html#id_credentials_region-endpoints) |\n| ROLE\\_EXPIRATION\\_THRESHOLD | Integer | 15 | The threshold before credentials expire in minutes at which metadataproxy will attempt to load new credentials. |\n| ROLE\\_MAPPING\\_FILE | Path String | | A json file that has a dict mapping of IP addresses to role names. Can be used if docker networking has been disabled and you are managing IP addressing for containers through another process. |\n| ROLE\\_REVERSE\\_LOOKUP | Boolean | False | Enable performing a reverse lookup of incoming IP addresses to match containers by hostname. Useful if you've disabled networking in docker, but set hostnames for containers in /etc/hosts or DNS. |\n| HOSTNAME\\_MATCH\\_REGEX | Regex String | `^.*$` | Limit reverse lookup container matching to hostnames that match the specified pattern. |\n| PATCH_ECS_ALLOWED_HOSTS | String | | Patch botocore's allowed hosts for ContainerMetadataFetcher to support aws-vault's --ecs-server option. This will inject the provided host into the allowed addresses botocore will allow for the AWS_CONTAINER_CREDENTIALS_FULL_URI environment. |\n\n#### Default Roles\n\nWhen no role is matched, `metadataproxy` will use the role specified in the \n`DEFAULT_ROLE` `metadataproxy` environment variable. If no DEFAULT\\_ROLE is\nspecified as a fallback, then your docker container without an `IAM_ROLE`\nenvironment variable will fail to retrieve credentials.\n\n#### Role Formats\n\nThe following are all supported formats for specifying roles:\n\n- By Role:\n\n    ```shell\n    IAM_ROLE=my-role\n    ```\n\n- By Role@AccountId\n\n    ```shell\n    IAM_ROLE=my-role@012345678910\n    ```\n\n- By ARN:\n\n    ```shell\n    IAM_ROLE=arn:aws:iam::012345678910:role/my-role\n    ```\n\n### Role structure\n\nA useful way to deploy this metadataproxy is with a two-tier role\nstructure:\n\n1.  The first tier is the EC2 service role for the instances running\n    your containers.  Call it `DockerHostRole`.  Your instances must\n    be launched with a policy that assigns this role.\n\n2.  The second tier is the role that each container will use.  These\n    roles must trust your own account (\"Role for Cross-Account\n    Access\" in AWS terms).  Call it `ContainerRole1`.\n\n3.  metadataproxy needs to query and assume the container role.  So\n    the `DockerHostRole` policy must permit this for each container\n    role.  For example:\n    ```\n    \"Statement\": [ {\n        \"Effect\": \"Allow\",\n        \"Action\": [\n            \"iam:GetRole\",\n            \"sts:AssumeRole\"\n        ],\n        \"Resource\": [\n            \"arn:aws:iam::012345678901:role/ContainerRole1\",\n            \"arn:aws:iam::012345678901:role/ContainerRole2\"\n        ]\n    } ]\n    ```\n\n4. Now customize `ContainerRole1` \u0026 friends as you like\n\nNote: The `ContainerRole1` role should have a trust relationship that allows it to be assumed by the `user` which is associated to the host machine running the `sts:AssumeRole` command.  An example trust relationship for `ContainRole1` may look like:\n\n```\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::012345678901:root\",\n        \"Service\": \"ec2.amazonaws.com\"\n      },\n      \"Action\": \"sts:AssumeRole\"\n    }\n  ]\n}\n```\n\n### Routing container traffic to metadataproxy\n\nUsing iptables, we can forward traffic meant to 169.254.169.254 from docker0 to\nthe metadataproxy. The following example assumes the metadataproxy is run on\nthe host, and not in a container:\n\n```\n/sbin/iptables \\\n  --append PREROUTING \\\n  --destination 169.254.169.254 \\\n  --protocol tcp \\\n  --dport 80 \\\n  --in-interface docker0 \\\n  --jump DNAT \\\n  --table nat \\\n  --to-destination 127.0.0.1:8000 \\\n  --wait\n```\n\nIf you'd like to start the metadataproxy in a container, it's recommended to\nuse host-only networking. Also, it's necessary to volume mount in the docker\nsocket, as metadataproxy must be able to interact with docker.\n\nBe aware that non-host-mode containers will not be able to contact\n127.0.0.1 in the host network stack.  As an alternative, you can use\nthe meta-data service to find the local address.  In this case, you\nprobably want to restrict proxy access to the docker0 interface!\n\n```\nLOCAL_IPV4=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)\n\n/sbin/iptables \\\n  --append PREROUTING \\\n  --destination 169.254.169.254 \\\n  --protocol tcp \\\n  --dport 80 \\\n  --in-interface docker0 \\\n  --jump DNAT \\\n  --table nat \\\n  --to-destination $LOCAL_IPV4:8000 \\\n  --wait\n\n/sbin/iptables \\\n  --wait \\\n  --insert INPUT 1 \\\n  --protocol tcp \\\n  --dport 80 \\\n  \\! \\\n  --in-interface docker0 \\\n  --jump DROP\n```\n\n## Run metadataproxy without docker\n\nIn the following we assume \\_my\\_config\\_ is a bash file with exports for all of\nthe necessary settings discussed in the configuration section.\n\n```\nsource my_config\ncd /srv/metadataproxy\nsource venv/bin/activate\ngunicorn metadataproxy:app --workers=2 -k gevent\n```\n\n## Run metadataproxy with docker\n\nFor production purposes, you'll want to kick up a container to run.\nYou can build one with the included Dockerfile.  To run, do something like:\n```bash\ndocker run --net=host \\\n    -v /var/run/docker.sock:/var/run/docker.sock \\\n    lyft/metadataproxy\n```\n\n### gunicorn settings\n\nThe following environment variables can be set to configure gunicorn (defaults\nare set in the examples):\n\n```\n# Change the IP address the gunicorn worker is listening on. You likely want to\n# leave this as the default\nHOST=0.0.0.0\n\n# Change the port the gunicorn worker is listening on.\nPORT=8000\n\n# Change the number of worker processes gunicorn will run with. The default is\n# 1, which is likely enough since metadataproxy is using gevent and its work is\n# completely IO bound. Increasing the number of workers will likely make your\n# in-memory cache less efficient\nWORKERS=1\n\n# Enable debug mode (you should not do this in production as it will leak IAM\n# credentials into your logs)\nDEBUG=False\n```\n\n## Contributing\n\n### Code of conduct\n\nThis project is governed by [Lyft's code of\nconduct](https://github.com/lyft/code-of-conduct).\nAll contributors and participants agree to abide by its terms.\n\n### Sign the Contributor License Agreement (CLA)\n\nWe require a CLA for code contributions, so before we can accept a pull request\nwe need to have a signed CLA. Please [visit our CLA\nservice](https://oss.lyft.com/cla)\nfollow the instructions to sign the CLA.\n\n### File issues in Github\n\nIn general all enhancements or bugs should be tracked via github issues before\nPRs are submitted. We don't require them, but it'll help us plan and track.\n\nWhen submitting bugs through issues, please try to be as descriptive as\npossible. It'll make it easier and quicker for everyone if the developers can\neasily reproduce your bug.\n\n### Submit pull requests\n\nOur only method of accepting code changes is through github pull requests.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyft%2Fmetadataproxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flyft%2Fmetadataproxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyft%2Fmetadataproxy/lists"}