{"id":19288368,"url":"https://github.com/techascent/tech.ml.dataset.sql","last_synced_at":"2025-04-22T04:33:24.821Z","repository":{"id":41321582,"uuid":"262654699","full_name":"techascent/tech.ml.dataset.sql","owner":"techascent","description":"SQL bindings for tech.ml.dataset","archived":false,"fork":false,"pushed_at":"2024-03-12T13:33:59.000Z","size":246,"stargazers_count":17,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-02T00:22:02.869Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/techascent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-05-09T20:37:19.000Z","updated_at":"2024-04-19T11:14:56.000Z","dependencies_parsed_at":"2024-03-12T14:45:26.462Z","dependency_job_id":null,"html_url":"https://github.com/techascent/tech.ml.dataset.sql","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techascent%2Ftech.ml.dataset.sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techascent%2Ftech.ml.dataset.sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techascent%2Ftech.ml.dataset.sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techascent%2Ftech.ml.dataset.sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/techascent","download_url":"https://codeload.github.com/techascent/tech.ml.dataset.sql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223888365,"owners_count":17220083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:08:50.245Z","updated_at":"2024-11-09T22:08:50.309Z","avatar_url":"https://github.com/techascent.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tech.ml.dataset.sql\n\n[![Clojars Project](https://img.shields.io/clojars/v/techascent/tech.ml.dataset.sql.svg)](https://clojars.org/techascent/tech.ml.dataset.sql)\n\n* [API Documentation](https://techascent.github.io/tech.ml.dataset.sql/)\n\nMinimal SQL bindings for\n[`tech.ml.dataset`](https://github.com/techascent/tech.ml.dataset).\n\n\n## Usage\n\nThis library provides versions of jdbc.next and honeysql.  `tech.ml.dataset`\nis expected to be transitively provided.\n\nRecommended driver: `[postgresql/postgresql \"42.2.12\"]`\n\nProvided in namespace `tech.ml.dataset.sql`:\n\n * `result-set-\u003edataset` - given a result set, read all the data into a dataset.\n * `sql-\u003edataset` - Given a string sql statement, return a dataset.\n * `sanitize-dataset-names-for-sql` - Transform the dataset name and the column names\n   to string and replace \"-\" with \"_\".\n * `table-exists?` - Return true of the table of this name exists.\n * `drop-table!` - Drop the table of this name.\n * `drop-table-when-exists!` - Drop the table if it exists.\n * `create-table!` - Using a dataset for the table name for the column names and\n    datatypes, create a new table.\n * `ensure-table!` - Ensure that a given table exists.\n * `insert-dataset!` - Insert/upsert a dataset into a table.  Upsert is postgresql-only.\n\n\n For efficiency when inserting/upserting a dataset the connection should be created with\n {:auto-commit false}.\n\n\n Want to see more functions above?  We accept PRs :-).  The sql/impl namespace\n provides many utility functions (like creating connection strings for postgresql\n servers) that may be helpful along with required helpers if you want to implement\n bindings to a different sql update/insert pathway.\n\n\n Included in this repo is a nice, one-stop\n [docker pathway](scripts/start-local-postgres) for development purposes that will\n start the a server with the expected settings used by the unit testing system.\n\n\n## Example\n\n```clojure\nuser\u003e (require '[tech.v3.dataset :as ds])\nnil\nuser\u003e (def ds (ds/-\u003edataset \"https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv\"))\n#'user/ds\nuser\u003e (ds/head ds)\nhttps://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [5 3]:\n\n| symbol |       date | price |\n|--------|------------|-------|\n|   MSFT | 2000-01-01 | 39.81 |\n|   MSFT | 2000-02-01 | 36.35 |\n|   MSFT | 2000-03-01 | 43.22 |\n|   MSFT | 2000-04-01 | 28.37 |\n|   MSFT | 2000-05-01 | 25.45 |\nuser\u003e (require '[tech.v3.dataset.sql :as ds-sql])\nnil\nuser\u003e ;;Connections should be created with auto-commit false so that inserts are batched.\nuser\u003e (require '[next.jdbc :as jdbc])\nnil\nuser\u003e (def dev-conn (doto (-\u003e (ds-sql/postgre-connect-str\n                               \"localhost:5432\" \"dev-user\"\n                               \"dev-user\" \"unsafe-bad-password\")\n                              (jdbc/get-connection {:auto-commit false}))\n                      (.setCatalog \"dev-user\")))\n#'user/dev-conn\nuser\u003e dev-conn\n#object[org.postgresql.jdbc.PgConnection 0x3256d7ea \"org.postgresql.jdbc.PgConnection@3256d7ea\"]\nuser\u003e ;;set the table name and the primary keys\nuser\u003e (def ds (with-meta ds\n                (assoc (meta ds)\n                       :name \"stocks\"\n                       :primary-key [\"symbol\" \"date\"])))\n#'user/ds\nuser\u003e ;;see the sql created for this table\nuser\u003e (println (ds-sql/create-sql ds))\nCREATE TABLE stocks (\n symbol varchar,\n date date,\n price float,\n PRIMARY KEY (symbol, date)\n);\nnil\nuser\u003e (ds-sql/create-table! dev-conn ds)\nnil\nuser\u003e (ds-sql/insert-dataset! dev-conn ds)\nnil\nuser\u003e (def sql-ds (ds-sql/sql-\u003edataset\n                   dev-conn \"SELECT * FROM stocks\"))\n#'user/sql-ds\nuser\u003e (ds/head sql-ds)\n_unnamed [5 3]:\n\n| symbol |                 date | price |\n|--------|----------------------|-------|\n|   MSFT | 2000-01-01T07:00:00Z | 39.81 |\n|   MSFT | 2000-02-01T07:00:00Z | 36.35 |\n|   MSFT | 2000-03-01T07:00:00Z | 43.22 |\n|   MSFT | 2000-04-01T07:00:00Z | 28.37 |\n|   MSFT | 2000-05-01T06:00:00Z | 25.45 |\nuser\u003e (ds/head ds)\nstocks [5 3]:\n\n| symbol |       date | price |\n|--------|------------|-------|\n|   MSFT | 2000-01-01 | 39.81 |\n|   MSFT | 2000-02-01 | 36.35 |\n|   MSFT | 2000-03-01 | 43.22 |\n|   MSFT | 2000-04-01 | 28.37 |\n|   MSFT | 2000-05-01 | 25.45 |\n```\n\nNote that local-dates are converted to instants in UTC.  The same is true for all\ndate/time types; all are just converted to java.sql.Date objects.  Numeric datatypes,\ndate/time types, strings and UUID's are supported datatypes.\n\n## Develop\n\nSee scripts directory.\n\nRun tests\n\n`clj -M:dev:test`\n\n\n\n## License\n\nCopyright © 2020 TechAscent, LLC\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechascent%2Ftech.ml.dataset.sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechascent%2Ftech.ml.dataset.sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechascent%2Ftech.ml.dataset.sql/lists"}