{"id":19852469,"url":"https://github.com/someengineering/cloud2sql","last_synced_at":"2025-05-02T00:31:17.556Z","repository":{"id":64308437,"uuid":"565862878","full_name":"someengineering/cloud2sql","owner":"someengineering","description":"Read infrastructure data from your cloud ☁️ and export it to a SQL database 📋.","archived":false,"fork":false,"pushed_at":"2023-11-09T22:37:28.000Z","size":712,"stargazers_count":33,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-26T06:41:08.504Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://cloud2sql.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/someengineering.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null}},"created_at":"2022-11-14T13:41:41.000Z","updated_at":"2025-03-14T02:45:32.000Z","dependencies_parsed_at":"2023-01-15T11:15:39.206Z","dependency_job_id":"8085bbaf-2a32-4aa5-a5ed-0fcc1d0f12f8","html_url":"https://github.com/someengineering/cloud2sql","commit_stats":{"total_commits":43,"total_committers":6,"mean_commits":7.166666666666667,"dds":0.6046511627906976,"last_synced_commit":"9d1a97731b5685fe2810a7b0cd3ca527e1f98a40"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/someengineering%2Fcloud2sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/someengineering%2Fcloud2sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/someengineering%2Fcloud2sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/someengineering%2Fcloud2sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/someengineering","download_url":"https://codeload.github.com/someengineering/cloud2sql/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251966431,"owners_count":21672666,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T14:02:50.075Z","updated_at":"2025-05-02T00:31:12.544Z","avatar_url":"https://github.com/someengineering.png","language":"Python","readme":"# Cloud2SQL 🤩\nRead infrastructure data from your cloud ☁️ and export it to a SQL database 📋.\n\n![Cloud2SQL](misc/cloud2sql.gif)\n\n## Installation\n\n### Install via homebrew\n\nThis is the easiest way to install Cloud2SQL. \nPlease note, that the installation process will take a couple of minutes.\n\n```bash\nbrew install someengineering/tap/cloud2sql\n```\n\n### Install via Python pip\n\nAlternatively you can install Cloud2SQL as Python package, where Python 3.9 or higher is required.\n\nIf you only need support for a specific database, instead of `cloud2sql[all]` you can choose between `cloud2sql[snowflake]`, `cloud2sql[parquet]`, `cloud2sql[postgresql]`, `cloud2sql[mysql]`.\n\n```bash\npip3 install --user \"cloud2sql[all]\"\n```\n\nThis will install the executable to the user install directory of your platform. Please make sure this installation directory is listed in `PATH`.\n\n\n## Usage\n\nThe sources and destinations for `cloud2sql` are configured via a configuration file. Create your own configuration by adjusting the [config template file](config-template.yaml).\n\nYou can safely delete the sections that are not relevant to you (e.g. if you do not use AWS, you can delete the `aws` section).\nAll sections refer to cloud providers and are enabled if a configuration section is provided.\n\nIn the next section you will create a YAML configuration file. Once you have created your configuration file, you can run `cloud2sql` with the following command:\n\n```bash\ncloud2sql --config myconfig.yaml\n```\n\n## Configuration\n\nCloud2SQL uses a YAML configuration file to define the `sources` and `destinations`.\n\n### Sources\n\n#### AWS\n\n```yaml\nsources:\n  aws:\n    # AWS Access Key ID (null to load from env - recommended)\n    access_key_id: null\n    # AWS Secret Access Key (null to load from env - recommended)\n    secret_access_key: null\n    # IAM role name to assume\n    role: null\n    # List of AWS profiles to collect\n    profiles: null\n    # List of AWS Regions to collect (null for all)\n    region: null\n    # Scrape the entire AWS organization\n    scrape_org: false\n    # Assume given role in current account\n    assume_current: false\n    # Do not scrape current account\n    do_not_scrape_current: false\n```\n\n#### Google Cloud\n    \n```yaml\nsources:\n  gcp:\n    # GCP service account file(s)\n    service_account: []\n    # GCP project(s)\n    project: []\n```\n\n#### Kubernetes\n\n```yaml\nsources:\n  k8s:\n    # Configure access via kubeconfig files.\n    # Structure:\n    #   - path: \"/path/to/kubeconfig\"\n    #     all_contexts: false\n    #     contexts: [\"context1\", \"context2\"]\n    config_files: []\n    # Alternative: configure access to k8s clusters directly in the config.\n    # Structure:\n    #   - name: 'k8s-cluster-name'\n    #     certificate_authority_data: 'CERT'\n    #     server: 'https://k8s-cluster-server.example.com'\n    #     token: 'TOKEN'\n    configs: []\n```\n\n#### DigitalOcean\n\n```yaml\nsources:\n  digitalocean:\n    # DigitalOcean API tokens for the teams to be collected\n    api_tokens: []\n    # DigitalOcean Spaces access keys for the teams to be collected, separated by colons\n    spaces_access_keys: []\n```\n\n### Destinations\n\n#### SQLite\n\n```yaml\ndestinations:\n  sqlite:\n    database: /path/to/database.db\n```\n\n#### PostgreSQL\n\n```yaml\ndestinations:\n  postgresql:\n    host: 127.0.0.1\n    port: 5432\n    user: cloud2sql\n    password: changeme\n    database: cloud2sql\n    args:\n      key: value\n```\n\n#### MySQL\n\n```yaml\ndestinations:\n  mysql:\n    host: 127.0.0.1\n    port: 3306\n    user: cloud2sql\n    password: changeme\n    database: cloud2sql\n    args:\n      key: value\n```\n\n#### MariaDB\n\n```yaml\ndestinations:\n  mariadb:\n    host: 127.0.0.1\n    port: 3306\n    user: cloud2sql\n    password: changeme\n    database: cloud2sql\n    args:\n      key: value\n```\n\n#### Snowflake\n\n```yaml\ndestinations:\n  snowflake:\n    host: myorg-myaccount\n    user: cloud2sql\n    password: changeme\n    database: cloud2sql/public\n    args:\n      warehouse: compute_wh\n      role: accountadmin\n```\n\n#### Apache Parquet\n\n```yaml\ndestinations:\n  file:\n    path: /where/to/write/parquet/files/\n    format: parquet\n    batch_size: 100_000\n```\n\n#### CSV\n\n```yaml\ndestinations:\n  file:\n    path: /where/to/write/to/csv/files/\n    format: csv\n    batch_size: 100_000\n```\n\n#### Upload to S3\n\n```yaml\ndestinations:\n  s3:\n    uri: s3://bucket_name/\n    region: eu-central-1\n    format: csv\n    batch_size: 100_000\n```\n\n### Upload to Google Cloud Storage\n\n```yaml\ndestinations:\n  gcs:\n    uri: gs://bucket_name/\n    format: parquet\n    batch_size: 100_000\n```\n\n#### My database is not listed here\n\nCloud2SQL uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported in [SQLAlchemy Dialects](https://docs.sqlalchemy.org/en/20/dialects/index.html).\nInstall the relevant driver and use the connection string from the documentation.\n\n#### Example\n\nWe use a minimal configuration [example](config-example.yaml) and export the data to a SQLite database.\nThe example uses our AWS default credentials and the default kubernetes config.\n\n```bash\ncloud2sql --config config-example.yaml\n```\n\nFor a more in-depth example, check out our [blog post](https://resoto.com/blog/2022/12/21/installing-cloud2sql).\n\n## Local Development\n\nCreate a local development environment with the following command:\n\n```bash\nmake setup\nsource venv/bin/activate\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsomeengineering%2Fcloud2sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsomeengineering%2Fcloud2sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsomeengineering%2Fcloud2sql/lists"}