{"id":34583177,"url":"https://github.com/bertrandchenal/tanker","last_synced_at":"2025-12-24T10:24:16.229Z","repository":{"id":57473310,"uuid":"216770035","full_name":"bertrandchenal/tanker","owner":"bertrandchenal","description":"Tanker is a Python database library targeting analytic operations","archived":false,"fork":false,"pushed_at":"2021-11-10T10:54:54.000Z","size":4538,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-11-01T10:19:19.962Z","etag":null,"topics":["database","postgresql","python","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bertrandchenal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-22T09:06:17.000Z","updated_at":"2023-02-08T02:42:56.000Z","dependencies_parsed_at":"2022-09-26T17:40:51.346Z","dependency_job_id":null,"html_url":"https://github.com/bertrandchenal/tanker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bertrandchenal/tanker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bertrandchenal%2Ftanker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bertrandchenal%2Ftanker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bertrandchenal%2Ftanker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bertrandchenal%2Ftanker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bertrandchenal","download_url":"https://codeload.github.com/bertrandchenal/tanker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bertrandchenal%2Ftanker/sbom","scorecard":{"id":234643,"data":{"date":"2025-08-11","repo":{"name":"github.com/bertrandchenal/tanker","commit":"b955311dc8f05f8bb3c0b391e169974e5c6a11b2"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.9,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":9,"reason":"binaries present in source code","details":["Warn: binary detected: tests/__init__.pyc:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENCE:0","Info: FSF or OSI recognized license: ISC License: LICENCE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-17T05:24:43.476Z","repository_id":57473310,"created_at":"2025-08-17T05:24:43.476Z","updated_at":"2025-08-17T05:24:43.476Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28000556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","postgresql","python","sqlite"],"created_at":"2025-12-24T10:24:14.385Z","updated_at":"2025-12-24T10:24:16.222Z","avatar_url":"https://github.com/bertrandchenal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tanker\n\nTanker is a Python database library targeting analytic operations but\nit also fits most transactional processing.\n\nAs its core it's mainly a query builder that simplify greatly join\noperations. It comes with a way to automatically create the database\ntables based on your schema definition and it can also introspect\nexisting db and infer the needed metadata.\n\nCurrently Postgresql and Sqlite are supported and the API is made to\nseamlessly integrate pandas DataFrames.\n\n\n## Licence\n\nTanker is available under the ISC Licence, see LICENCE file at the\nroot of the repository.\n\n\n## Main features\n\n### Schema definition and database connection\n\nThe file `schema.yaml` defines the database structure: table, columns\n(and their types) and key.\n\n``` yaml\n    - table: team\n      columns:\n        name: varchar\n        country: m2o country.id\n      key:\n        - name\n        - country\n    - table: country\n      columns:\n        name: varchar\n      key:\n        - name\n```\n\nThe code here-under create the config dictionary and use it to connect\nto the database and creates the tables.\n\n``` python\n    from tanker import connect, create_tables, View, yaml_load\n\n    cfg = {\n        'db_uri': 'sqlite:///test.db',\n        'schema': yaml_load(open('schema.yaml')),\n    }\n    with connect(cfg):\n        create_tables()\n```\n\nTanker automatically add an `id` column on each table, to allow to\ndefine foreign keys. For example, in the yaml definition, `country:\nm2o country.id` means that a many-to-one relation will be created\nbetween the tables team and country. When the team table will be\ncreated this will generate the following column definition:\n\n    \"country\" INTEGER REFERENCES \"country\" (id) ON DELETE CASCADE\n\nIf not specified, `sqlite:///:memory:` will be used as `db_uri`. To\nuse Postgresql, the uri should looks like\n`postgresql://login:passwd@hostname/dbname` (and you can choose the\npostgres schema to use by appending `#shema_name`to the uri)\n\nNote that every database interaction must happen inside the `with\nconnect(cfg)` block.\n\n\n### Read \u0026 write\n\nTanker usage is centered around the `View` object, it is used to\ndefine a mapping between the relational world and Python. For example,\nto write and read countries, we define a view based on the country\ntable:\n\n``` python\n    country_view = View(\n        'country',  # The base table\n        ['name']    # The fields we want to map\n    )\n```\n\nSo now we can write to the database:\n\n``` python\n    countries = [['Belgium'], ['France']]\n    country_view.write(countries)\n```\n\nAnd read it back.\n\n``` python\n    countries_copy = country_view.read().all()\n```\n\nAnd `countries_copy` should be identical to `countries`. As `.read()`\nreturns the database cursor, the `.all()` allows to fetch all the\nrecords. Instead of `.all()` one can use `.df()` to receive a pandas\nDataFrame.\n\n\n### Key role\n\nAs you can see in the database definition, each table comes with a `key`\nattribute. This attribute contains the list of columns that form a\n[natural key](https://en.wikipedia.org/wiki/Natural_key).\n\nThis key is required by design in Tanker, its main role is to\nallow Tanker to know what to do with each record when `View.write` is\ncalled. Thanks to the key, we know if the record is already in the\ndatabase (and in this case will generate an `UPDATE` statement) or if\nthe record is new (and use an `INSERT` query).\n\nIt's especially handy when dealing for example with data coming from a\nwebsite scraper or from an spreadsheet, where a technical id (like an\ninteger or a uuid) is not always available.\n\nTo avoid to launch one query per record and suffer from network\nlatencies, what Tanker do to speed up writes is to create a temporary\ntable, insert all the record as one batch and then join this temporary\ntable with the actual one to know which record to insert and which to\nupdate.\n\n\n### Foreign key resolution\n\nTo populate the `team` table we have to provide a team name and a\ncountry. We can do it like this:\n\n``` python\n    team_view = View('team, ['name', 'country'])\n    team_view.write([['Red', 1]])\n```\n\nBut it's more convenient to use the country name instead of it's id:\n\n``` python\n    teams = [\n        ['Blue', 'Belgium'],\n        ['Red', 'Belgium'],\n        ['Blue', 'France'],\n    ]\n    team_view = View('team, ['name', 'country.name'])\n    team_view.write(teams)\n```\n\nYou can see that we changed `country` into `country.name` in the view,\nwhich means that the use the `name` column to identify the country\n(which is conveniently defined as the key in the table\ndefinition).\n\nWe can go further and use more than one dot and let Tanker resolve\nforeign key for us. Let's say we want to add a member table to our\ndatabase, we append the following piece of yaml to our schema file\n\n``` yaml\n    - table: member\n      columns:\n        name: varchar\n        registration_code: varchar\n        team: m2o team.id\n      key:\n        - registration_code\n```\n\nAnd re-run the `create_tables()` as above. Now we can do:\n\n    rows = View('member', ['name', team.country.name]).read()\n\nHere, two join queries will be automatically generated, one between\n`member` and `team` and one between `team` and `country`.\n\n\nTo add a member we have to link it to a team, whose key is composed\nof both the name and the country column (so we allow two teams with the\nsame name in different countries):\n\n``` python\n    members = [\n        ['Bob', 'Belgium', 'Blue', '001'],\n        ['Alice', 'Belgium', 'Red', '002'],\n        ['Trudy', 'France', 'Blue', '003'],\n    ]\n    member_view = View('member', ['name', 'team.country.name', 'team.name',\n                                  'registration_code'],\n    ])\n    member_view.write(members)\n```\n\nTanker will be able to identify for each member the correct team based\non both country name and team name.\n\n\n### Filters\n\nThe read method accept a `filters` argument it can be a string or a\nlist of strings. Filter strings use\n[s-expression](https://en.wikipedia.org/wiki/S-expression)\nnotation. So for example to filter a country by name you can do:\n\n``` python\n    filters = '(= name \"Belgium\")'\n    country_view.read(filters)\n```\n\nor to get `registration_code` above a given value:\n\n``` python\n    member_view.read('(\u003e registration_code \"002\")')\n```\n\nYou can also combine those filters and use the dot notation:\n\n    filters = '(or ((\u003e registration_code \"002\") (= team.country.name \"Belgium\")))'\n    member_view.read(filters).read()\n\nThe `filters` argument can also be a list, in this case all items are\nregrouped in a conjunction, equivalent to `(and item1 item2 ...)`.\n\n\n### Query arguments\n\nTo facilitate the building of queries and more importantly to prevent\nsql injections you can use arguments. They use the syntax of Python\nown [string format method](https://docs.python.org/2/library/stdtypes.html#str.format),\nand will make use of the DB-API's parameter substitution (see for\nexample [the sqlite documentation](https://docs.python.org/2/library/sqlite3.html)):\n\n``` python\n    cond = '(= name {name})'\n    rows = team_view.read(cond).args(name='Blue')\n```\n\nYou can also pass list values, they will be automatically\nexpanded. And you can use the dot notation to reach a given parameter\nin the object passed as argument:\n\n``` python\n    cond = '(or (in name {names}) (= registration_code {data.code}))'\n    rows = member_view.read(cond).args(names=['Alice', 'Bob'], data=my_object)\n```\n\nThe dot notation also supports dictionnaries, so the above example\nwhould work with `data={'code': '001'}`. The query arguments can also\nrefer to values from the configuration (which can be reach from the\n`ctx` object), like:\n\n``` python\n    ctx.cfg['default_team'] = 'Red'\n    cond = '(in name {default_team})'\n    rows = view.read(cond)\n```\n\nFinally, arguments can be a list instead of a dict and can be passed to the `read` method, so:\n\n``` python\n    cond = '(in name {names})'\n    rows = team_view.read(cond).args(names=['Blue', 'Red'])\n```\n\nis equivalent to\n\n``` python\n    cond = '(in name {} {})'\n    rows = team_view.read(cond).args('Blue', 'Red')\n```\n\nand is equivalent to\n\n``` python\n    cond = '(in name {} {})'\n    rows = team_view.read(cond, args=['Blue', 'Red'])\n```\n\n\n### Pandas Dataframes\n\nInstead of passing a list of list we can use a dataframe, and use a\ndictionary to map dataframe columns to database columns.\n\n``` python\n    df = DataFrame({\n        'Team': ['Blue', 'Red'],\n        'Country': ['France', 'Belgium']\n        })\n    view = View('team', {\n        'Team': 'name',\n        'Country': 'country.name',\n    })\n    view.write(data)\n\n    df_copy = view.read().df()\n```\n\n\n## ACL\n\nThe ACL system of Tanker allows to systemically filter access to the\nrows of any table. We can define filters for reading data or writing\nit. To enable it, you can add it to the `cfg` parameter of `connect`,\nbut you can also change it later:\n\n``` python\ncfg['acl-read'] = {'team': \"(= team.country.name 'Belgium')\"}\nwith connect(cfg):\n\tteams = View('team').read().all() # No belgian team will be returned\n\tcfg['acl-read'] = {}              # we reset the acl\n\tteams = View('team').read().all() # Returns everything again\n```\n\nAs you can see it's a simple dict whose keys are table names and\nvalues are Tanker filters. Similarly, you can add an `acl-write`\ndictionary to `cfg`:\n\n``` python\ncfg['acl-read'] = {'team': \"(= team.country.name 'Belgium')\"}\nView('team').write(teams)\n```\n\n\n## Documentation TODO\n  - Deletion (by data, by filter)\n  - Aliases\n\n\n## Roadmap\n\nSome ideas, in no particular order:\n\n  - Add a view.insert method that bypass tmp table and write directly\n    to the actual table\n  - Support for version column (probably a write timestamp)\n  - Add support for other 'ON CONFLICT' action (like incrementing a\n    version column, or appening to an array)\n  - Support for table constraints\n  - Allow to execute complete queries with s-expressions (select,\n    update, insert and delete).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbertrandchenal%2Ftanker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbertrandchenal%2Ftanker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbertrandchenal%2Ftanker/lists"}