{"id":20710438,"url":"https://github.com/mobiletelesystems/onetl","last_synced_at":"2025-04-23T06:03:47.744Z","repository":{"id":153628979,"uuid":"629993050","full_name":"MobileTeleSystems/onetl","owner":"MobileTeleSystems","description":"One ETL tool to rule them all","archived":false,"fork":false,"pushed_at":"2025-04-21T20:22:56.000Z","size":5419,"stargazers_count":74,"open_issues_count":2,"forks_count":6,"subscribers_count":8,"default_branch":"develop","last_synced_at":"2025-04-23T06:03:36.467Z","etag":null,"topics":["etl","etl-components","etl-pipeline","hwm","spark"],"latest_commit_sha":null,"homepage":"https://onetl.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MobileTeleSystems.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.rst","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-19T12:53:20.000Z","updated_at":"2025-04-21T07:39:09.000Z","dependencies_parsed_at":"2023-10-17T04:46:17.608Z","dependency_job_id":"9041200f-0a42-497e-a9a8-4e2674585bd8","html_url":"https://github.com/MobileTeleSystems/onetl","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MobileTeleSystems%2Fonetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MobileTeleSystems%2Fonetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MobileTeleSystems%2Fonetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MobileTeleSystems%2Fonetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MobileTeleSystems","download_url":"https://codeload.github.com/MobileTeleSystems/onetl/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250379798,"owners_count":21420842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","etl-components","etl-pipeline","hwm","spark"],"created_at":"2024-11-17T02:11:58.913Z","updated_at":"2025-04-23T06:03:47.731Z","avatar_url":"https://github.com/MobileTeleSystems.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. _readme:\n\nonETL\n=====\n\n|Repo Status| |PyPI Latest Release| |PyPI License| |PyPI Python Version| |PyPI Downloads|\n|Documentation| |CI Status| |Test Coverage| |pre-commit.ci Status|\n\n.. |Repo Status| image:: https://www.repostatus.org/badges/latest/active.svg\n    :alt: Repo status - Active\n    :target: https://github.com/MobileTeleSystems/onetl\n.. |PyPI Latest Release| image:: https://img.shields.io/pypi/v/onetl\n    :alt: PyPI - Latest Release\n    :target: https://pypi.org/project/onetl/\n.. |PyPI License| image:: https://img.shields.io/pypi/l/onetl.svg\n    :alt: PyPI - License\n    :target: https://github.com/MobileTeleSystems/onetl/blob/develop/LICENSE.txt\n.. |PyPI Python Version| image:: https://img.shields.io/pypi/pyversions/onetl.svg\n    :alt: PyPI - Python Version\n    :target: https://pypi.org/project/onetl/\n.. |PyPI Downloads| image:: https://img.shields.io/pypi/dm/onetl\n    :alt: PyPI - Downloads\n    :target: https://pypi.org/project/onetl/\n.. |Documentation| image:: https://readthedocs.org/projects/onetl/badge/?version=stable\n    :alt: Documentation - ReadTheDocs\n    :target: https://onetl.readthedocs.io/\n.. |CI Status| image:: https://github.com/MobileTeleSystems/onetl/workflows/Tests/badge.svg\n    :alt: Github Actions - latest CI build status\n    :target: https://github.com/MobileTeleSystems/onetl/actions\n.. |Test Coverage| image:: https://codecov.io/gh/MobileTeleSystems/onetl/branch/develop/graph/badge.svg?token=RIO8URKNZJ\n    :alt: Test coverage - percent\n    :target: https://codecov.io/gh/MobileTeleSystems/onetl\n.. |pre-commit.ci Status| image:: https://results.pre-commit.ci/badge/github/MobileTeleSystems/onetl/develop.svg\n    :alt: pre-commit.ci - status\n    :target: https://results.pre-commit.ci/latest/github/MobileTeleSystems/onetl/develop\n\n|Logo|\n\n.. |Logo| image:: docs/_static/logo_wide.svg\n    :alt: onETL logo\n    :target: https://github.com/MobileTeleSystems/onetl\n\nWhat is onETL?\n--------------\n\nPython ETL/ELT library powered by `Apache Spark \u003chttps://spark.apache.org/\u003e`_ \u0026 other open-source tools.\n\nGoals\n-----\n\n* Provide unified classes to extract data from (**E**) \u0026 load data to (**L**) various stores.\n* Provides `Spark DataFrame API \u003chttps://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html\u003e`_ for performing transformations (**T**) in terms of *ETL*.\n* Provide direct assess to database, allowing to execute SQL queries, as well as DDL, DML, and call functions/procedures. This can be used for building up *ELT* pipelines.\n* Support different `read strategies \u003chttps://onetl.readthedocs.io/en/stable/strategy/index.html\u003e`_ for incremental and batch data fetching.\n* Provide `hooks \u003chttps://onetl.readthedocs.io/en/stable/hooks/index.html\u003e`_ \u0026 `plugins \u003chttps://onetl.readthedocs.io/en/stable/plugins.html\u003e`_ mechanism for altering behavior of internal classes.\n\nNon-goals\n---------\n\n* onETL is not a Spark replacement. It just provides additional functionality that Spark does not have, and improves UX for end users.\n* onETL is not a framework, as it does not have requirements to project structure, naming, the way of running ETL/ELT processes, configuration, etc. All of that should be implemented in some other tool.\n* onETL is deliberately developed without any integration with scheduling software like Apache Airflow. All integrations should be implemented as separated tools.\n* Only batch operations, no streaming. For streaming prefer `Apache Flink \u003chttps://flink.apache.org/\u003e`_.\n\nRequirements\n------------\n\n* **Python 3.7 - 3.13**\n* PySpark 2.3.x - 3.5.x (depends on used connector)\n* Java 8+ (required by Spark, see below)\n* Kerberos libs \u0026 GCC (required by ``Hive``, ``HDFS`` and ``SparkHDFS`` connectors)\n\nSupported storages\n------------------\n\n+--------------------+--------------+-------------------------------------------------------------------------------------------------------------------------+\n| Type               | Storage      | Powered by                                                                                                              |\n+====================+==============+=========================================================================================================================+\n| Database           | Clickhouse   | Apache Spark `JDBC Data Source \u003chttps://spark.apache.org/docs/latest/sql-data-sources-jdbc.html\u003e`_                      |\n+                    +--------------+                                                                                                                         +\n|                    | MSSQL        |                                                                                                                         |\n+                    +--------------+                                                                                                                         +\n|                    | MySQL        |                                                                                                                         |\n+                    +--------------+                                                                                                                         +\n|                    | Postgres     |                                                                                                                         |\n+                    +--------------+                                                                                                                         +\n|                    | Oracle       |                                                                                                                         |\n+                    +--------------+                                                                                                                         +\n|                    | Teradata     |                                                                                                                         |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | Hive         | Apache Spark `Hive integration \u003chttps://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html\u003e`_               |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | Kafka        | Apache Spark `Kafka integration \u003chttps://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html\u003e`_    |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | Greenplum    | VMware `Greenplum Spark connector \u003chttps://docs.vmware.com/en/VMware-Greenplum-Connector-for-Apache-Spark/index.html\u003e`_ |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | MongoDB      | `MongoDB Spark connector \u003chttps://www.mongodb.com/docs/spark-connector/current\u003e`_                                       |\n+--------------------+--------------+-------------------------------------------------------------------------------------------------------------------------+\n| File               | HDFS         | `HDFS Python client \u003chttps://pypi.org/project/hdfs/\u003e`_                                                                  |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | S3           | `minio-py client \u003chttps://pypi.org/project/minio/\u003e`_                                                                    |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | SFTP         | `Paramiko library \u003chttps://pypi.org/project/paramiko/\u003e`_                                                                |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | FTP          | `FTPUtil library \u003chttps://pypi.org/project/ftputil/\u003e`_                                                                  |\n+                    +--------------+                                                                                                                         +\n|                    | FTPS         |                                                                                                                         |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | WebDAV       | `WebdavClient3 library \u003chttps://pypi.org/project/webdavclient3/\u003e`_                                                      |\n+                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | Samba        | `pysmb library \u003chttps://pypi.org/project/pysmb/\u003e`_                                                                      |\n+--------------------+--------------+-------------------------------------------------------------------------------------------------------------------------+\n| Files as DataFrame | SparkLocalFS | Apache Spark `File Data Source \u003chttps://spark.apache.org/docs/latest/sql-data-sources-generic-options.html\u003e`_           |\n|                    +--------------+                                                                                                                         +\n|                    | SparkHDFS    |                                                                                                                         |\n|                    +--------------+-------------------------------------------------------------------------------------------------------------------------+\n|                    | SparkS3      | `Hadoop AWS \u003chttps://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html\u003e`_ library                  |\n+--------------------+--------------+-------------------------------------------------------------------------------------------------------------------------+\n\n.. documentation\n\nDocumentation\n-------------\n\nSee https://onetl.readthedocs.io/\n\nHow to install\n---------------\n\n.. _install:\n\nMinimal installation\n~~~~~~~~~~~~~~~~~~~~\n\n.. _minimal-install:\n\nBase ``onetl`` package contains:\n\n* ``DBReader``, ``DBWriter`` and related classes\n* ``FileDownloader``, ``FileUploader``, ``FileMover`` and related classes, like file filters \u0026 limits\n* ``FileDFReader``, ``FileDFWriter`` and related classes, like file formats\n* Read Strategies \u0026 HWM classes\n* Plugins support\n\nIt can be installed via:\n\n.. code:: bash\n\n    pip install onetl\n\n.. warning::\n\n    This method does NOT include any connections.\n\n    This method is recommended for use in third-party libraries which require for ``onetl`` to be installed,\n    but do not use its connection classes.\n\nWith DB and FileDF connections\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. _spark-install:\n\nAll DB connection classes (``Clickhouse``, ``Greenplum``, ``Hive`` and others)\nand all FileDF connection classes (``SparkHDFS``, ``SparkLocalFS``, ``SparkS3``)\nrequire Spark to be installed.\n\n.. _java-install:\n\nFirstly, you should install JDK. The exact installation instruction depends on your OS, here are some examples:\n\n.. code:: bash\n\n    yum install java-1.8.0-openjdk-devel  # CentOS 7 + Spark 2\n    dnf install java-11-openjdk-devel  # CentOS 8 + Spark 3\n    apt-get install openjdk-11-jdk  # Debian-based + Spark 3\n\n.. _spark-compatibility-matrix:\n\nCompatibility matrix\n^^^^^^^^^^^^^^^^^^^^\n\n+--------------------------------------------------------------+-------------+-------------+-------+\n| Spark                                                        | Python      | Java        | Scala |\n+==============================================================+=============+=============+=======+\n| `2.3.x \u003chttps://spark.apache.org/docs/2.3.1/#downloading\u003e`_  | 3.7 only    | 8 only      | 2.11  |\n+--------------------------------------------------------------+-------------+-------------+-------+\n| `2.4.x \u003chttps://spark.apache.org/docs/2.4.8/#downloading\u003e`_  | 3.7 only    | 8 only      | 2.11  |\n+--------------------------------------------------------------+-------------+-------------+-------+\n| `3.2.x \u003chttps://spark.apache.org/docs/3.2.4/#downloading\u003e`_  | 3.7 - 3.10  | 8u201 - 11  | 2.12  |\n+--------------------------------------------------------------+-------------+-------------+-------+\n| `3.3.x \u003chttps://spark.apache.org/docs/3.3.4/#downloading\u003e`_  | 3.7 - 3.12  | 8u201 - 17  | 2.12  |\n+--------------------------------------------------------------+-------------+-------------+-------+\n| `3.4.x \u003chttps://spark.apache.org/docs/3.4.4/#downloading\u003e`_  | 3.7 - 3.12  | 8u362 - 20  | 2.12  |\n+--------------------------------------------------------------+-------------+-------------+-------+\n| `3.5.x \u003chttps://spark.apache.org/docs/3.5.5/#downloading\u003e`_  | 3.8 - 3.13  | 8u371 - 20  | 2.12  |\n+--------------------------------------------------------------+-------------+-------------+-------+\n\n.. _pyspark-install:\n\nThen you should install PySpark via passing ``spark`` to ``extras``:\n\n.. code:: bash\n\n    pip install \"onetl[spark]\"  # install latest PySpark\n\nor install PySpark explicitly:\n\n.. code:: bash\n\n    pip install onetl pyspark==3.5.5  # install a specific PySpark version\n\nor inject PySpark to ``sys.path`` in some other way BEFORE creating a class instance.\n**Otherwise connection object cannot be created.**\n\nWith File connections\n~~~~~~~~~~~~~~~~~~~~~\n\n.. _files-install:\n\nAll File (but not *FileDF*) connection classes (``FTP``,  ``SFTP``, ``HDFS`` and so on) requires specific Python clients to be installed.\n\nEach client can be installed explicitly by passing connector name (in lowercase) to ``extras``:\n\n.. code:: bash\n\n    pip install \"onetl[ftp]\"  # specific connector\n    pip install \"onetl[ftp,ftps,sftp,hdfs,s3,webdav,samba]\"  # multiple connectors\n\nTo install all file connectors at once you can pass ``files`` to ``extras``:\n\n.. code:: bash\n\n    pip install \"onetl[files]\"\n\n**Otherwise class import will fail.**\n\nWith Kerberos support\n~~~~~~~~~~~~~~~~~~~~~\n\n.. _kerberos-install:\n\nMost of Hadoop instances set up with Kerberos support,\nso some connections require additional setup to work properly.\n\n* ``HDFS``\n  Uses `requests-kerberos \u003chttps://pypi.org/project/requests-kerberos/\u003e`_ and\n  `GSSApi \u003chttps://pypi.org/project/gssapi/\u003e`_ for authentication.\n  It also uses ``kinit`` executable to generate Kerberos ticket.\n\n* ``Hive`` and ``SparkHDFS``\n  require Kerberos ticket to exist before creating Spark session.\n\nSo you need to install OS packages with:\n\n* ``krb5`` libs\n* Headers for ``krb5``\n* ``gcc`` or other compiler for C sources\n\nThe exact installation instruction depends on your OS, here are some examples:\n\n.. code:: bash\n\n    apt install libkrb5-dev krb5-user gcc  # Debian-based\n    dnf install krb5-devel krb5-libs krb5-workstation gcc  # CentOS, OracleLinux\n\nAlso you should pass ``kerberos`` to ``extras`` to install required Python packages:\n\n.. code:: bash\n\n    pip install \"onetl[kerberos]\"\n\nFull bundle\n~~~~~~~~~~~\n\n.. _full-bundle:\n\nTo install all connectors and dependencies, you can pass ``all`` into ``extras``:\n\n.. code:: bash\n\n    pip install \"onetl[all]\"\n\n    # this is just the same as\n    pip install \"onetl[spark,files,kerberos]\"\n\n.. warning::\n\n    This method consumes a lot of disk space, and requires for Java \u0026 Kerberos libraries to be installed into your OS.\n\n.. _quick-start:\n\nQuick start\n------------\n\nMSSQL → Hive\n~~~~~~~~~~~~\n\nRead data from MSSQL, transform \u0026 write to Hive.\n\n.. code:: bash\n\n    # install onETL and PySpark\n    pip install \"onetl[spark]\"\n\n.. code:: python\n\n    # Import pyspark to initialize the SparkSession\n    from pyspark.sql import SparkSession\n\n    # import function to setup onETL logging\n    from onetl.log import setup_logging\n\n    # Import required connections\n    from onetl.connection import MSSQL, Hive\n\n    # Import onETL classes to read \u0026 write data\n    from onetl.db import DBReader, DBWriter\n\n    # change logging level to INFO, and set up default logging format and handler\n    setup_logging()\n\n    # Initialize new SparkSession with MSSQL driver loaded\n    maven_packages = MSSQL.get_packages()\n    spark = (\n        SparkSession.builder.appName(\"spark_app_onetl_demo\")\n        .config(\"spark.jars.packages\", \",\".join(maven_packages))\n        .enableHiveSupport()  # for Hive\n        .getOrCreate()\n    )\n\n    # Initialize MSSQL connection and check if database is accessible\n    mssql = MSSQL(\n        host=\"mssqldb.demo.com\",\n        user=\"onetl\",\n        password=\"onetl\",\n        database=\"Telecom\",\n        spark=spark,\n        # These options are passed to MSSQL JDBC Driver:\n        extra={\"applicationIntent\": \"ReadOnly\"},\n    ).check()\n\n    # \u003e\u003e\u003e INFO:|MSSQL| Connection is available\n\n    # Initialize DBReader\n    reader = DBReader(\n        connection=mssql,\n        source=\"dbo.demo_table\",\n        columns=[\"on\", \"etl\"],\n        # Set some MSSQL read options:\n        options=MSSQL.ReadOptions(fetchsize=10000),\n    )\n\n    # checks that there is data in the table, otherwise raises exception\n    reader.raise_if_no_data()\n\n    # Read data to DataFrame\n    df = reader.run()\n    df.printSchema()\n    # root\n    #  |-- id: integer (nullable = true)\n    #  |-- phone_number: string (nullable = true)\n    #  |-- region: string (nullable = true)\n    #  |-- birth_date: date (nullable = true)\n    #  |-- registered_at: timestamp (nullable = true)\n    #  |-- account_balance: double (nullable = true)\n\n    # Apply any PySpark transformations\n    from pyspark.sql.functions import lit\n\n    df_to_write = df.withColumn(\"engine\", lit(\"onetl\"))\n    df_to_write.printSchema()\n    # root\n    #  |-- id: integer (nullable = true)\n    #  |-- phone_number: string (nullable = true)\n    #  |-- region: string (nullable = true)\n    #  |-- birth_date: date (nullable = true)\n    #  |-- registered_at: timestamp (nullable = true)\n    #  |-- account_balance: double (nullable = true)\n    #  |-- engine: string (nullable = false)\n\n    # Initialize Hive connection\n    hive = Hive(cluster=\"rnd-dwh\", spark=spark)\n\n    # Initialize DBWriter\n    db_writer = DBWriter(\n        connection=hive,\n        target=\"dl_sb.demo_table\",\n        # Set some Hive write options:\n        options=Hive.WriteOptions(if_exists=\"replace_entire_table\"),\n    )\n\n    # Write data from DataFrame to Hive\n    db_writer.run(df_to_write)\n\n    # Success!\n\nSFTP → HDFS\n~~~~~~~~~~~\n\nDownload files from SFTP \u0026 upload them to HDFS.\n\n.. code:: bash\n\n    # install onETL with SFTP and HDFS clients, and Kerberos support\n    pip install \"onetl[hdfs,sftp,kerberos]\"\n\n.. code:: python\n\n    # import function to setup onETL logging\n    from onetl.log import setup_logging\n\n    # Import required connections\n    from onetl.connection import SFTP, HDFS\n\n    # Import onETL classes to download \u0026 upload files\n    from onetl.file import FileDownloader, FileUploader\n\n    # import filter \u0026 limit classes\n    from onetl.file.filter import Glob, ExcludeDir\n    from onetl.file.limit import MaxFilesCount\n\n    # change logging level to INFO, and set up default logging format and handler\n    setup_logging()\n\n    # Initialize SFTP connection and check it\n    sftp = SFTP(\n        host=\"sftp.test.com\",\n        user=\"someuser\",\n        password=\"somepassword\",\n    ).check()\n\n    # \u003e\u003e\u003e INFO:|SFTP| Connection is available\n\n    # Initialize downloader\n    file_downloader = FileDownloader(\n        connection=sftp,\n        source_path=\"/remote/tests/Report\",  # path on SFTP\n        local_path=\"/local/onetl/Report\",  # local fs path\n        filters=[\n            # download only files matching the glob\n            Glob(\"*.csv\"),\n            # exclude files from this directory\n            ExcludeDir(\"/remote/tests/Report/exclude_dir/\"),\n        ],\n        limits=[\n            # download max 1000 files per run\n            MaxFilesCount(1000),\n        ],\n        options=FileDownloader.Options(\n            # delete files from SFTP after successful download\n            delete_source=True,\n            # mark file as failed if it already exist in local_path\n            if_exists=\"error\",\n        ),\n    )\n\n    # Download files to local filesystem\n    download_result = downloader.run()\n\n    # Method run returns a DownloadResult object,\n    # which contains collection of downloaded files, divided to 4 categories\n    download_result\n\n    #  DownloadResult(\n    #      successful=[\n    #          LocalPath('/local/onetl/Report/file_1.json'),\n    #          LocalPath('/local/onetl/Report/file_2.json'),\n    #      ],\n    #      failed=[FailedRemoteFile('/remote/onetl/Report/file_3.json')],\n    #      ignored=[RemoteFile('/remote/onetl/Report/file_4.json')],\n    #      missing=[],\n    #  )\n\n    # Raise exception if there are failed files, or there were no files in the remote filesystem\n    download_result.raise_if_failed() or download_result.raise_if_empty()\n\n    # Do any kind of magic with files: rename files, remove header for csv files, ...\n    renamed_files = my_rename_function(download_result.success)\n\n    # function removed \"_\" from file names\n    # [\n    #    LocalPath('/home/onetl/Report/file1.json'),\n    #    LocalPath('/home/onetl/Report/file2.json'),\n    # ]\n\n    # Initialize HDFS connection\n    hdfs = HDFS(\n        host=\"my.name.node\",\n        user=\"someuser\",\n        password=\"somepassword\",  # or keytab\n    )\n\n    # Initialize uploader\n    file_uploader = FileUploader(\n        connection=hdfs,\n        target_path=\"/user/onetl/Report/\",  # hdfs path\n    )\n\n    # Upload files from local fs to HDFS\n    upload_result = file_uploader.run(renamed_files)\n\n    # Method run returns a UploadResult object,\n    # which contains collection of uploaded files, divided to 4 categories\n    upload_result\n\n    #  UploadResult(\n    #      successful=[RemoteFile('/user/onetl/Report/file1.json')],\n    #      failed=[FailedLocalFile('/local/onetl/Report/file2.json')],\n    #      ignored=[],\n    #      missing=[],\n    #  )\n\n    # Raise exception if there are failed files, or there were no files in the local filesystem, or some input file is missing\n    upload_result.raise_if_failed() or upload_result.raise_if_empty() or upload_result.raise_if_missing()\n\n    # Success!\n\n\nS3 → Postgres\n~~~~~~~~~~~~~~~~\n\nRead files directly from S3 path, convert them to dataframe, transform it and then write to a database.\n\n.. code:: bash\n\n    # install onETL and PySpark\n    pip install \"onetl[spark]\"\n\n.. code:: python\n\n    # Import pyspark to initialize the SparkSession\n    from pyspark.sql import SparkSession\n\n    # import function to setup onETL logging\n    from onetl.log import setup_logging\n\n    # Import required connections\n    from onetl.connection import Postgres, SparkS3\n\n    # Import onETL classes to read files\n    from onetl.file import FileDFReader\n    from onetl.file.format import CSV\n\n    # Import onETL classes to write data\n    from onetl.db import DBWriter\n\n    # change logging level to INFO, and set up default logging format and handler\n    setup_logging()\n\n    # Initialize new SparkSession with Hadoop AWS libraries and Postgres driver loaded\n    maven_packages = SparkS3.get_packages(spark_version=\"3.5.5\") + Postgres.get_packages()\n    exclude_packages = SparkS3.get_exclude_packages()\n    spark = (\n        SparkSession.builder.appName(\"spark_app_onetl_demo\")\n        .config(\"spark.jars.packages\", \",\".join(maven_packages))\n        .config(\"spark.jars.excludes\", \",\".join(exclude_packages))\n        .getOrCreate()\n    )\n\n    # Initialize S3 connection and check it\n    spark_s3 = SparkS3(\n        host=\"s3.test.com\",\n        protocol=\"https\",\n        bucket=\"my-bucket\",\n        access_key=\"somekey\",\n        secret_key=\"somesecret\",\n        # Access bucket as s3.test.com/my-bucket\n        extra={\"path.style.access\": True},\n        spark=spark,\n    ).check()\n\n    # \u003e\u003e\u003e INFO:|SparkS3| Connection is available\n\n    # Describe file format and parsing options\n    csv = CSV(\n        delimiter=\";\",\n        header=True,\n        encoding=\"utf-8\",\n    )\n\n    # Describe DataFrame schema of files\n    from pyspark.sql.types import (\n        DateType,\n        DoubleType,\n        IntegerType,\n        StringType,\n        StructField,\n        StructType,\n        TimestampType,\n    )\n\n    df_schema = StructType(\n        [\n            StructField(\"id\", IntegerType()),\n            StructField(\"phone_number\", StringType()),\n            StructField(\"region\", StringType()),\n            StructField(\"birth_date\", DateType()),\n            StructField(\"registered_at\", TimestampType()),\n            StructField(\"account_balance\", DoubleType()),\n        ],\n    )\n\n    # Initialize file df reader\n    reader = FileDFReader(\n        connection=spark_s3,\n        source_path=\"/remote/tests/Report\",  # path on S3 there *.csv files are located\n        format=csv,  # file format with specific parsing options\n        df_schema=df_schema,  # columns \u0026 types\n    )\n\n    # Read files directly from S3 as Spark DataFrame\n    df = reader.run()\n\n    # Check that DataFrame schema is same as expected\n    df.printSchema()\n    # root\n    #  |-- id: integer (nullable = true)\n    #  |-- phone_number: string (nullable = true)\n    #  |-- region: string (nullable = true)\n    #  |-- birth_date: date (nullable = true)\n    #  |-- registered_at: timestamp (nullable = true)\n    #  |-- account_balance: double (nullable = true)\n\n    # Apply any PySpark transformations\n    from pyspark.sql.functions import lit\n\n    df_to_write = df.withColumn(\"engine\", lit(\"onetl\"))\n    df_to_write.printSchema()\n    # root\n    #  |-- id: integer (nullable = true)\n    #  |-- phone_number: string (nullable = true)\n    #  |-- region: string (nullable = true)\n    #  |-- birth_date: date (nullable = true)\n    #  |-- registered_at: timestamp (nullable = true)\n    #  |-- account_balance: double (nullable = true)\n    #  |-- engine: string (nullable = false)\n\n    # Initialize Postgres connection\n    postgres = Postgres(\n        host=\"192.169.11.23\",\n        user=\"onetl\",\n        password=\"somepassword\",\n        database=\"mydb\",\n        spark=spark,\n    )\n\n    # Initialize DBWriter\n    db_writer = DBWriter(\n        connection=postgres,\n        # write to specific table\n        target=\"public.my_table\",\n        # with some writing options\n        options=Postgres.WriteOptions(if_exists=\"append\"),\n    )\n\n    # Write DataFrame to Postgres table\n    db_writer.run(df_to_write)\n\n    # Success!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmobiletelesystems%2Fonetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmobiletelesystems%2Fonetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmobiletelesystems%2Fonetl/lists"}