{"id":22572758,"url":"https://github.com/martinitus/databums","last_synced_at":"2025-08-12T17:43:41.557Z","repository":{"id":266869360,"uuid":"899604652","full_name":"martinitus/databums","owner":"martinitus","description":"Testing optional conflicting dependencies with poetry: databricks-connect \u0026 vanilla pyspark","archived":false,"fork":false,"pushed_at":"2024-12-23T12:08:14.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-13T19:50:37.580Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/martinitus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-06T16:01:10.000Z","updated_at":"2024-12-06T16:03:19.000Z","dependencies_parsed_at":"2024-12-06T17:22:48.884Z","dependency_job_id":"eb7ccad1-1ad3-48a9-926c-07176e758b7e","html_url":"https://github.com/martinitus/databums","commit_stats":null,"previous_names":["martinitus/databums"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/martinitus/databums","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinitus%2Fdatabums","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinitus%2Fdatabums/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinitus%2Fdatabums/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinitus%2Fdatabums/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/martinitus","download_url":"https://codeload.github.com/martinitus/databums/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinitus%2Fdatabums/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270108674,"owners_count":24528764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-08T02:11:45.840Z","updated_at":"2025-08-12T17:43:41.519Z","avatar_url":"https://github.com/martinitus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Testing conflicting depenencies with poetry - Thanks Databricks...\n\nNormally, we cannot install both `pyspark` and `databricks-connect`, because Databricks, in their infinite wisdom,\ndecided to ship their own pyspark python modules, which happen to cause name collisions with the vanilla pyspark\npackage.\n\nSo the following code would not know whether to import SparkSession from databricks-connect of vanilla pyspark - which\nobviously is a problem.\n\n```python\nfrom pyspark.sql import SparkSession\n```\n\nSo given that we cannot install them at the same time, what about having two exclusive optional dependencies?\nFortunately, in Poetry we can finally specify this after [#9553](https://github.com/python-poetry/poetry/pull/9553) has\nbeen merged. Thanks guys :heart:\n\nSo lets get to work. This repo contains the above mentioned two optional exclusive dependencies. Simply installing the\npackage and running the test entrypoint with `poetry install \u0026\u0026 databums` yields:\n\n```\nTraceback (most recent call last):\n  File \"/path/to/clone/spark-databums-conflict/.venv/bin/databums\", line 6, in \u003cmodule\u003e\n    sys.exit(main())\n             ^^^^^^\n  File \"/path/to/clone/spark-databums-conflict/src/databums/main.py\", line 4, in main\n    import pyspark\nModuleNotFoundError: No module named 'pyspark'\n```\n\nGood :) - Thats what we expected. We can only import pyspark if we either add the `vanilla` (vanilla pyspark) or the\n`databums` extra (databricks-connect).\n\nRunning `poetry install --extra=vanilla \u0026\u0026 databums` yields:\n\n```\nHey there, i imported \u003cclass 'pyspark.sql.session.SparkSession'\u003e from \u003cmodule 'pyspark' from '/path/to/clone/spark-databums-conflict/.venv/lib/python3.12/site-packages/pyspark/__init__.py'\u003e\n```\n\nAlso Good :-)\n\nRunning `poetry install --extra=databums \u0026\u0026 databums` first uninstalls vanilla pyspark. Yes poetry actually seems to\nactively uninstall the non-specified extra dependency, AWESOME! Then it attempts to install `databricks-connect`, and\nTADA: `Bus error`. Who would have expected something to just work, when using Databricks.\n\nLet me know if it works for you... \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartinitus%2Fdatabums","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmartinitus%2Fdatabums","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartinitus%2Fdatabums/lists"}