{"id":15287733,"url":"https://github.com/osahp/pyspark_db_utils","last_synced_at":"2025-07-30T07:44:01.727Z","repository":{"id":62583625,"uuid":"107673818","full_name":"osahp/pyspark_db_utils","owner":"osahp","description":"The easy to use database connector that allows one-command operations between PySpark and PostgreSQL or ClickHouse databases.","archived":false,"fork":false,"pushed_at":"2018-05-24T10:27:49.000Z","size":6159,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-01T15:36:15.279Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/osahp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-20T12:10:55.000Z","updated_at":"2022-05-10T03:24:33.000Z","dependencies_parsed_at":"2022-11-03T21:57:55.773Z","dependency_job_id":null,"html_url":"https://github.com/osahp/pyspark_db_utils","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osahp%2Fpyspark_db_utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osahp%2Fpyspark_db_utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osahp%2Fpyspark_db_utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osahp%2Fpyspark_db_utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/osahp","download_url":"https://codeload.github.com/osahp/pyspark_db_utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219846951,"owners_count":16556418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-30T15:36:10.489Z","updated_at":"2024-10-14T18:40:32.208Z","avatar_url":"https://github.com/osahp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pyspark_db_utils  \n\nIt helps you with your DB deals in Spark\n\n## Documentation\n\nhttp://pyspark-db-utils.readthedocs.io/en/latest/\n\n## Example of using\n\nYou need jdbc drivers for using this lib!\nJust get drivers from\nhttps://jdbc.postgresql.org/download.html\nhttps://github.com/yandex/clickhouse-jdbc\nand put it in jars/ directory in your project\n\n### Example settings:\n```\nsettings = {\n  \"PG_PROPERTIES\": {\n    \"user\": \"user\",\n    \"password\": \"pass\",\n    \"driver\": \"org.postgresql.Driver\"\n  },\n  \"PG_DRIVER_PATH\": \"jars/postgresql-42.1.4.jar\",\n  \"PG_URL\": \"jdbc:postgresql://db.olabs.com/dbname\",\n}\n```\n\n### Example of code\n\nsee example.py\n\n### Example of run\n```\nvsmelov@vsmelov:~/PycharmProjects/pyspark_db_utils$ mkdir jars\nvsmelov@vsmelov:~/PycharmProjects/pyspark_db_utils$ cp /var/bigdata/spark-2.2.0-bin-hadoop2.7/jars/postgresql-42.1.4.jar ./jars/\nvsmelov@vsmelov:~/PycharmProjects/pyspark_db_utils$ python3 pyspark_db_utils/example.py \nhost: ***SECRET***\ndb: ***SECRET***\nuser: ***SECRET***\npassword: ***SECRET***\n\nUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\nSetting default log level to \"WARN\".\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n18/03/05 11:43:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n18/03/05 11:43:29 WARN Utils: Your hostname, vsmelov resolves to a loopback address: 127.0.1.1; using 192.168.43.26 instead (on interface wlp2s0)\n18/03/05 11:43:29 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\nTRY: create df\nOK: create df\n+---+-----------+\n| id|    mono_id|\n+---+-----------+\n|  1|          0|\n|  2|          1|\n|  3|          2|\n|  4|          3|\n|  5| 8589934592|\n|  6| 8589934593|\n|  7| 8589934594|\n|  8| 8589934595|\n|  9| 8589934596|\n| 10|17179869184|\n| 11|17179869185|\n| 12|17179869186|\n| 13|17179869187|\n| 14|17179869188|\n| 15|25769803776|\n| 16|25769803777|\n| 17|25769803778|\n| 18|25769803779|\n| 19|25769803780|\n+---+-----------+\n\n\nTRY: write_to_pg\nOK: write_to_pg                                                                 \n\nTRY: read_from_pg\nOK: read_from_pg\n+---+-----------+\n| id|    mono_id|\n+---+-----------+\n| 10|17179869184|\n| 11|17179869185|\n| 12|17179869186|\n| 13|17179869187|\n| 14|17179869188|\n|  1|          0|\n|  2|          1|\n|  3|          2|\n|  4|          3|\n|  5| 8589934592|\n|  6| 8589934593|\n|  7| 8589934594|\n|  8| 8589934595|\n|  9| 8589934596|\n| 15|25769803776|\n| 16|25769803777|\n| 17|25769803778|\n| 18|25769803779|\n| 19|25769803780|\n|  1|          0|\n+---+-----------+\nonly showing top 20 rows\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosahp%2Fpyspark_db_utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosahp%2Fpyspark_db_utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosahp%2Fpyspark_db_utils/lists"}