{"id":39778521,"url":"https://github.com/sinzlab/datajoint-link","last_synced_at":"2026-01-18T12:01:07.431Z","repository":{"id":53542572,"uuid":"285852124","full_name":"sinzlab/datajoint-link","owner":"sinzlab","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-18T16:02:40.000Z","size":1709,"stargazers_count":4,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-10-02T10:42:40.866Z","etag":null,"topics":["clean-architecture","database","datajoint","domain-driven-design"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sinzlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-08-07T14:43:56.000Z","updated_at":"2025-01-07T10:33:57.000Z","dependencies_parsed_at":"2022-08-24T19:41:28.619Z","dependency_job_id":"b82afa97-f5e6-421b-9785-da9aa920cf1d","html_url":"https://github.com/sinzlab/datajoint-link","commit_stats":null,"previous_names":["sinzlab/link"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/sinzlab/datajoint-link","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinzlab%2Fdatajoint-link","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinzlab%2Fdatajoint-link/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinzlab%2Fdatajoint-link/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinzlab%2Fdatajoint-link/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sinzlab","download_url":"https://codeload.github.com/sinzlab/datajoint-link/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinzlab%2Fdatajoint-link/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28535270,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T10:13:46.436Z","status":"ssl_error","status_checked_at":"2026-01-18T10:13:11.045Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clean-architecture","database","datajoint","domain-driven-design"],"created_at":"2026-01-18T12:00:31.743Z","updated_at":"2026-01-18T12:01:07.401Z","avatar_url":"https://github.com/sinzlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# :link: datajoint-link\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"datajoint_link.png\" alt=\"datajoint-link logo\" style=\"display: block; margin-left: auto; margin-right: auto; width: 20%\"\u003e\n\u003cbr\u003e\n\u003ca href=\"https://github.com/sinzlab/datajoint-link/actions/workflows/ci.yml\"\u003e\u003cimg alt=\"Actions Status\" src=\"https://github.com/sinzlab/datajoint-link/actions/workflows/ci.yml/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://codecov.io/gh/sinzlab/datajoint-link/\"\u003e\u003cimg alt=\"Coverage\" src=\"https://img.shields.io/codecov/c/github/sinzlab/datajoint-link\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/datajoint-link/\"\u003e\u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/datajoint-link\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e A tool for convenient and integrity-preserving data sharing between database servers. \u003c/p\u003e\n\n## :floppy_disk: Installation \n\nOnly users interacting with the destination of the data need to install the datajoint-link package:\n\n```bash\npip install datajoint-link\n```\n\n## :wrench: Setup\n\n### Source\n\nDatajoint-link requires access to the database server from which data will be pulled. It is recommended to create a new user for this purpose:\n\n```sql\nCREATE USER 'djlink'@'%' IDENTIFIED BY 'secret-password';\n```\n\nThe user needs to have certain privileges on the table from which data will be pulled:\n\n```sql\nGRANT SELECT, REFERENCES ON `source\\_schema`.`source\\_table` TO 'djlink'@'%';\n```\n\nEach table from which data will be pulled also needs an additional helper table:\n\n```sql\nGRANT ALL PRIVILEGES ON `helper\\_schema`.`helper\\_table` TO 'djlink'@'%';\n```\n\nIn order to preserve data integrity across the link regular users must not have any privileges on this helper table. \n\n### Destination\n\nDatajoint-link needs to be configured with the username and password of the user created in the previous section. This is accomplished via environment variables:\n\n```bash\nLINK_USER=djlink\nLINK_PASS=secret-password\n```\n\n## :computer: Usage\n\nThe destination table is created by passing information about where to find the source table to the `link` decorator:\n\n```python\nfrom link import link\n\n@link(\n    \"databaseserver.com\", \n    \"source_schema\", \n    \"helper_schema\", \n    \"helper_table\", \n    \"destination_schema\"\n)\nclass Table:\n    \"\"\"Some table present in the source schema on the source database server.\"\"\"\n```\n\nNote that the name of the declared class must match the name of the table from which the data will be pulled.\n\nThe class returned by the decorator behaves like a regular table with some added functionality. For one it allows the browsing of rows present in the source:\n\n```python\nTable().source\n```\n\nAll the rows can be pulled like so:\n\n```python\nTable().source.pull()  # Hint: Pass display_progress=True to get a progress bar\n```\n\nThat said usually we only want to pull rows that match a certain criteria:\n\n```python\n(Table().source \u0026 \"foo = 1\").pull()\n```\n\nThe deletion of already pulled rows works the same as for any other table:\n\n```python\n(Table() \u0026 \"foo = 1\").delete()\n```\n\nThe deletion of certain rows from the destination can also be requested by flagging them in the corresponding helper table:\n\n```python\nrow = (Helper() \u0026 \"foo = 1\").fetch1()\n(Helper() \u0026 row).delete()\nrow[\"is_flagged\"] = \"TRUE\"\nHelper().insert1(row)\n```\n\nThe `flagged` attribute makes the deletion of flagged rows from the destination table convenient:\n\n```python\n(Table() \u0026 Table().source.flagged).delete()\n```\n\nDeleting a flagged row automatically updates its corresponding row in the helper table:\n\n```python\nassert (Helper() \u0026 \"foo = 1\").fetch1(\"is_deprecated\") == \"TRUE\" # No error!\n```\n\nNow it is save to delete the row from the source table as well!\n\n## :package: External Storage\n\nData stored in a source table that refers to one (or more) external stores can be stored in different store(s) after pulling:\n\n```python\n@link(\n    ...,\n    stores={\"source_store\": \"destination_store\"}\n)\nclass Table:\n    ...\n```\n\nNote that all stores mentioned in the dictionary need to be configured via `dj.config`.\n\n## :white_check_mark: Tests\n\nClone this repository and run the following command from within the cloned repository to run all tests:\n\n```bash\ndocker compose run functional_tests tests\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinzlab%2Fdatajoint-link","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsinzlab%2Fdatajoint-link","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinzlab%2Fdatajoint-link/lists"}