{"id":14066886,"url":"https://github.com/ggrothendieck/sqldf","last_synced_at":"2025-10-22T06:24:40.209Z","repository":{"id":1030572,"uuid":"32124280","full_name":"ggrothendieck/sqldf","owner":"ggrothendieck","description":"Perform SQL Selects on R Data Frames","archived":false,"fork":false,"pushed_at":"2023-10-04T08:03:38.000Z","size":240,"stargazers_count":252,"open_issues_count":19,"forks_count":77,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-07-12T09:26:31.230Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ggrothendieck.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-03-13T03:12:03.000Z","updated_at":"2025-06-10T15:04:29.000Z","dependencies_parsed_at":"2022-08-06T10:01:45.281Z","dependency_job_id":"50dfe41b-f884-42c5-ade3-1b7ef0d9b6fa","html_url":"https://github.com/ggrothendieck/sqldf","commit_stats":{"total_commits":120,"total_committers":6,"mean_commits":20.0,"dds":"0.17500000000000004","last_synced_commit":"ba33807f239522609295712e9445b8e9ae81c773"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ggrothendieck/sqldf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggrothendieck%2Fsqldf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggrothendieck%2Fsqldf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggrothendieck%2Fsqldf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggrothendieck%2Fsqldf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ggrothendieck","download_url":"https://codeload.github.com/ggrothendieck/sqldf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggrothendieck%2Fsqldf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267780047,"owners_count":24143201,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-13T07:05:18.847Z","updated_at":"2025-10-22T06:24:40.155Z","avatar_url":"https://github.com/ggrothendieck.png","language":"R","funding_links":[],"categories":["R","Misc"],"sub_categories":["Desktop"],"readme":"*To write it, it took three months; to conceive it – three minutes; to\ncollect the data in it – all my life.* [F. Scott\nFitzgerald](https://en.wikipedia.org/wiki/F._Scott_Fitzgerald)\n\n\n**Introduction**\n\n[sqldf](https://cran.r-project.org/package=sqldf) is an R\npackage for runing [SQL statements](https://en.wikipedia.org/wiki/SQL) on\nR data frames, optimized for convenience. The user simply specifies an\nSQL statement in R using data frame names in place of table names and a\ndatabase with appropriate table layouts/schema is automatically created,\nthe data frames are automatically loaded into the database, the\nspecified SQL statement is performed, the result is read back into R and\nthe database is deleted all automatically behind the scenes making the\ndatabase's existence transparent to the user who only specifies the SQL\nstatement. Surprisingly this can at times\n[be](https://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r/1820610#1820610)\n[even](https://groups.google.com/group/manipulatr/browse_thread/thread/3affbdc5efca9143/d19d7b97ac023ee8?pli=1)\n[faster](http://web.archive.org/web/20130511223824/http://stat.ethz.ch/pipermail/r-help/2009-December/221456.html)\n[than](http://web.archive.org/web/20130604023900/stat.ethz.ch/pipermail/r-help/2009-December/221513.html)\n[the](https://stackoverflow.com/questions/14283566/specific-for-loop-too-slow-in-r/14287476#14287476)\ncorresponding pure R calculation (although the purpose of the project is\nconvenience and not speed). [This\nlink](https://brusers.tumblr.com/post/59706993506/data-manipulation-with-sqldf-paul)\nsuggests that for aggregations over highly granular columns that sqldf\nis faster than another alternative tried. `sqldf` is free software\npublished under the GNU General Public License that can be downloaded\nfrom [CRAN](https://cran.r-project.org/package=sqldf).\n\nsqldf supports (1) the [SQLite](https://www.sqlite.org) backend database\n(by default), (2) the [H2](https://www.h2database.com) java database, (3)\nthe [PostgreSQL](https://www.postgresql.org) database and (4) sqldf 0.4-0\nonwards also supports [MySQL](https://dev.mysql.com). SQLite, H2, MySQL\nand PostgreSQL are free software. SQLite and H2 are embedded serverless\nzero administration databases that are included right in the R driver\npackages,\n[RSQLite](https://cran.r-project.org/package=RSQLite) and\n[RH2](https://cran.r-project.org/package=RH2), so that\nthere is no separate installation for either one. A number of [high\nprofile projects](https://www.sqlite.org/famous.html) use SQLite. \nH2 is a java database which contains a large collection of SQL functions\nand supports Date and other data types. It is the most popular database\npackage among [scala\npackages](http://blog.takipi.com/the-top-100-most-popular-scala-libraries-based-on-10000-github-projects/).\nPostgreSQL is a client/server database and unlike SQLite and H2 must be\nseparately installed but it has a particularly powerful version of SQL,\ne.g. its\n[window](https://developer.postgresql.org/pgdocs/postgres/tutorial-window.html)\n[functions](https://developer.postgresql.org/pgdocs/postgres/functions-window.html),\nso the extra installation work can be worth it. sqldf supports the\n`RPostgreSQL` driver in R. Like PostgreSQL, MySQL is a client server\ndatabase that must be installed independently so its not as easy to\ninstall as SQLite or H2 but its very popular and is widely used as the\nback end for web sites.\n\nThe information below mostly concerns the default SQLite database. The\nuse of H2 with sqldf is discussed in [FAQ\n\\#10](https://code.google.com/p/sqldf/#10.__What_are_some_of_the_differences_between_using_SQLite_and_H)\nwhich discusses differences between using sqldf with SQLite and H2 and\nalso shows how to modify the code in the [Examples](#Examples) section\nto use sqldf/H2 rather than sqldf/SQLite. There is some information on\nusing PostgreSQL with sqldf in [FAQ\n\\#12](https://code.google.com/p/sqldf/#12._How_does_one_use_sqldf_with_PostgreSQL?)\nand an example in [Example 17.\nLag](https://code.google.com/p/sqldf/#Example_17._Lag) . The unit tests\nprovide examples that can work with all five data base drivers (covering\nfour databases) supported by sqldf. They are run by loading whichever\ndatabase is to be tested (SQLite is the default) and running:\n`demo(\"sqldf-unitTests\")`\n\n[Overview](#Overview)\n\n[Citing sqldf](#Citing_sqldf)\n\n[For Those New to R](#For_Those_New_to_R)\n\n[News](#news)\n\n[Troubleshooting](#troubleshooting)\n\n-   [Problem is that installer gives message that sqldf is not\n    available](#problem_is_that_installer_gives_message_that_sqldf_is_not_availa)\n-   [Problem with no argument form of sqldf -\n    sqldf()](#problem_with_no_argument_form_of_sqldf_-_sqldf())\n-   [Problem involvling tcltk](#problem_involvling_tcltk)\n\n[FAQ](#faq)\n\n-   [1. How does sqldf handle classes and\n    factors?](#1-how-does-sqldf-handle-classes-and-factors)\n-   [2. Why does sqldf seem to mangle certain variable\n    names?](#2-why-does-sqldf-seem-to-mangle-certain-variable-names)\n-   [3. Why does sqldf(\"select var(x) from DF\") not\n    work?](#3-why-does-sqldfselect-varx-from-df-not-work)\n-   [4. How does sqldf work with \"Date\" class\n    variables?](#4-how-does-sqldf-work-with-date-class-variables)\n-   [5. I get a message about the tcltk package being\n    missing.](#5-i-get-a-message-about-the-tcltk-package-being-missing)\n-   [6. Why are there problems when we use table names or column names\n    that are the same except for\n    case?](#6-why-are-there-problems-when-we-use-table-names-or-column-name)\n-   [7. Why are there messages about\n    MySQL?](#7-why-are-there-messages-about-mysql)\n-   [8. Why am I having problems with\n    update?](#8-why-am-I-having-problems-with-update)\n-   [9. How do I examine the layout that SQLite uses for a table? which\n    tables are in the database? which databases are\n    attached?](#9-how-do-i-examine-the-layout-that-sqlite-uses-for-a-table-which-tables-are-in-the-database-which-databases-are-attached)\n-   [10. What are some of the differences between using SQLite and H2\n    with\n    sqldf?](#10-what-are-some-of-the-differences-between-using-sqlite-and-h2-with-sqldf)\n-   [11. Why am I having difficulty reading a data file using SQLite and\n    sqldf?](#11-why-am-i-having-difficulty-reading-a-data-file-using-sqlite-and-sqldf)\n-   [12. How does one use sqldf with\n    PostgreSQL?](#12-how-does-one-use-sqldf-with-postgresql)\n-   [13. How does one deal with quoted fields in read.csv.sql\n    ?](#13-how-does-one-deal-with-quoted-fields-in-readcsvsql)\n-   [14. How does one read files where numeric NAs are represented as\n    missing empty\n    fields?](#14-how-does-one-read-files-where-numeric-nas-are-represented-as-missing-empty-fields)\n-   [15. Why do certain calculations come out as integer rather than\n    double?](#15-why-do-certain-calculations-come-out-as-integer-rather-than-double)\n-   [16. How can one read a file off the net or a csv file in a zip\n    file?](#16-how-can-one-read-a-file-off-the-net-or-a-csv-file-in-a-zip-file)\n\n[Examples](#examples)\n\n-   [Example 1. Ordering and\n    Limiting](#example-1-ordering-and-limiting)\n-   [Example 2. Averaging and\n    Grouping](#example-2-averaging-and-grouping)\n-   [Example 3. Nested Select](#example-3-nested-select)\n-   [Example 4. Join](#example-4-join)\n-   [Example 5. Insert Variables](#example-5-insert-variables)\n-   [Example 6. File Input](#example-6-file-input)\n-   [Example 7. Nested Select](#example-7-nested-select)\n-   [Example 8. Specifying File\n    Format](#example-8-specifying-file-format)\n-   [Example 9. Working with\n    Databases](#example-9-working-with-databases)\n-   [Example 10. Persistent\n    Connections](#example-10-persistent-connections)\n-   [Example 11. Between and\n    Alternatives](#example-11-between-and-alternatives)\n-   [Example 12. Combine two files in permanent\n    database](#example-12-combine-two-files-in-permanent-database)\n-   [Example 13. read.csv.sql and\n    read.csv2.sql](#example-13-readcsvsql-and-readcsv2sql)\n-   [Example 14. Use of spatialite library\n    functions](#example-14-use-of-spatialite-library-functions)\n-   [Example 15. Use of RSQLite.extfuns library\n    functions](#example-15-use-of-rsqliteextfuns-library-functions)\n-   [Example 16. Moving Average](#example-16-moving-average)\n-   [Example 17. Lag](#example-17-lag)\n-   [Example 18. MySQL Schema\n    Information](#Example-18-mysql-schema-information)\n\n[Links](#links)\n\nOverview[](#overview)\n=====================\n\n[sqldf](https://cran.r-project.org/package=sqldf) is an R\npackage for running [SQL statements](https://en.wikipedia.org/wiki/SQL)\non R data frames, optimized for convenience. `sqldf` works with the\n[SQLite](https://www.sqlite.org/), [H2](https://www.h2database.com),\n[PostgreSQL](https://www.postgresql.org) or\n[MySQL](https://dev.mysql.com/doc/) databases. SQLite has the least\nprerequisites to install. H2 is just as easy if you have Java installed\nand also supports Date class and a few additional functions. PostgreSQL\nnotably supports Windowing functions providing the SQL analogue of the R\nave function. MySQL is a particularly popular database that drives many\nweb sites.\n\nMore information can be found from within R by installing and loading\nthe sqldf package and then entering\n[?sqldf](https://cran.r-project.org/package=sqldf/sqldf.pdf) and\n[?read.csv.sql](https://cran.r-project.org/package=sqldf/sqldf.pdf).\nA number of [examples](#Examples) are on this page and more examples are\naccessible from within R in the examples section of the\n[?sqldf](https://cran.r-project.org/package=sqldf/sqldf.pdf) help\npage.\n\nAs seen from this example which uses the built in `BOD` data frame:\n\n```r\nlibrary(sqldf)\nsqldf(\"select * from BOD where Time \u003e 4\")\n```\n\nwith `sqldf` the user is freed from having to do the following, all of\nwhich are automatically done:\n\n-   database setup\n-   writing the `create table` statement which defines each table\n-   importing and exporting to and from the database\n-   coercing of the returned columns to the appropriate class in common\n    cases\n\nIt can be used for:\n\n-   learning SQL if you know R\n-   learning R if you know SQL\n-   as an alternate syntax for data frame manipulation, particularly for\n    purposes of speeding these up, since sqldf with SQLite as the\n    underlying database is often faster than performing the same\n    manipulations in straight R\n-   reading portions of large files into R without reading the entire\n    file (example 6b and example 13 below show two different ways and\n    examples 6e, 6f below show how to read random portions of a file)\n\nIn the case of SQLite it consists of a thin layer over the\n[RSQLite](https://cran.r-project.org/package=RSQLite)\n[DBI](https://cran.r-project.org/package=DBI) interface to SQLite\nitself.\n\nIn the case of H2 it works on top of the\n[RH2](https://cran.r-project.org/package=RH2)\n[DBI](https://cran.r-project.org/package=DBI) driver which in turn\nuses RJDBC and JDBC to interface to H2 itself.\n\nIn the case of PostgreSQL it works on top of the\n[RPostgreSQL](https://cran.r-project.org/package=RPostgreSQL)\n[DBI](https://cran.r-project.org/package=DBI) driver.\n\nThere is also some untested code in sqldf for use with the\n[MySQL](https://www.mysql.com) database using the\n[RMySQL](https://cran.r-project.org/package=RMySQL)\n[DBI](https://cran.r-project.org/package=DBI) driver.\n\nCiting sqldf[](#Citing_sqldf)\n=============================\n\nTo get information on how to cite `sqldf` in papers, issue the R\ncommands:\n\n```r\nlibrary(sqldf)\ncitation(\"sqldf\")\n```\n\nFor Those New to R[](#For_Those_New_to_R)\n=========================================\n\nIf you have not used R before and want to try sqldf with SQLite, [google\nfor single letter R](https://www.r-project.org), download R, install it\non Windows, Mac or UNIX/Linux and then start R and at R console enter\nthis:\n\n```r\n# installs everything you need to use sqldf with SQLite\n# including SQLite itself\ninstall.packages(\"sqldf\")\n# shows built in data frames\ndata() \n# load sqldf into workspace\nlibrary(sqldf)\nsqldf(\"select * from iris limit 5\")\nsqldf(\"select count(*) from iris\")\nsqldf(\"select Species, count(*) from iris group by Species\")\n# create a data frame\nDF \u003c- data.frame(a = 1:5, b = letters[1:5])\nsqldf(\"select * from DF\")\nsqldf(\"select avg(a) mean, variance(a) var from DF\") # see example 15\n```\n\nTo try it with H2 rather than SQLite the process is similar. Ensure that\nyou have the [java](https://java.sun.com) runtime installed, install R as\nabove and start R. From within R enter this ensuring that the version of\nRH2 that you have is RH2 0.1-2.6 or later:\n\n```r\n# installs everything including H2\ninstall.packages(\"sqldf\", dep = TRUE)\n# load RH2 driver and sqldf into workspace\nlibrary(RH2)\npackageVersion(\"RH2\") # should be version 0.1-2-6 or later\nlibrary(sqldf)\n#\nsqldf(\"select * from iris limit 5\")\nsqldf(\"select count(*) from iris\")\nsqldf(\"select Species, count(*) from iris group by Species\")\nDF \u003c- data.frame(a = 1:5, b = letters[1:5])\nsqldf(\"select * from DF\")\nsqldf(\"select avg(a) mean, var_samp(a) var from DF\")\n```\n\nTroubleshooting[](#Troubleshooting)\n===================================\n\nsqldf has been\n[extensively](https://cran.r-project.org/web/checks/check_results_sqldf.html)\n[tested](https://code.google.com/p/sqldf/source/browse/trunk/inst/unitTests/runit.all.R)\nwith multiple architectures and database back ends but there are no\nguarantees.\n\nProblem is that installer gives message that sqldf is not available[](#Problem_is_that_installer_gives_message_that_sqldf_is_not_availa)\n----------------------------------------------------------------------------------------------------------------------------------------\n\nSee\n[https://stackoverflow.com/questions/27772756/sqldf-doesnt-install-on-ubuntu-14-04](https://stackoverflow.com/questions/27772756/sqldf-doesnt-install-on-ubuntu-14-04)\n\nProblem with no argument form of sqldf - sqldf()[](#Problem_with_no_argument_form_of_sqldf_-_sqldf())\n-----------------------------------------------------------------------------------------------------\n\nThe no argument form, i.e. `sqldf()` is used for opening and closing a\nconnection so that intermediate sqldf statements can all use the same\nconnection. If you have forgotten whether the last `sqldf()` opened or\nclosed the connection this code will close it if it is open and\notherwise do nothing:\n\n```r\n   # close an old connection if it exists\n   if (!is.null(getOption(\"sqldf.connection\"))) sqldf()\n```\n\nThanks to Chris Davis\n[https://groups.google.com/d/msg/sqldf/-YAvaJnlRrY/7nF8tpBnrcAJ](https://groups.google.com/d/msg/sqldf/-YAvaJnlRrY/7nF8tpBnrcAJ)\nfor pointing this out.\n\nProblem involvling tcltk[](#Problem_involvling_tcltk)\n-----------------------------------------------------\n\nThe most common problem is that the tcltk package and tcl/tk itself are\nmissing. Historically these were bundled with the Windows version of R\nso Windows users should not experience any problems on this account.\nSince R version 3.0.0 Mac versions of R also have the tcltk package and\nTcl/Tk itself bundled so if you are having a problem on the Mac you may\nonly need to upgrade to the latest version of R. If upgrading to the\nlatest version of R does not help then using this line will usually\nallow it to work even without the tcltk package and tcl/tk itself:\n\n```r\noptions(gsubfn.engine = \"R\")\n```\n\nRunning the above `options` line before using `sqldf`, e.g. put that\noptions line in your `.Rprofile`, is all that is needed to get sqldf to\nwork without the tcltk package and tcl/tk itself in most cases; however,\nthis does have the downside that it will use the R engine which is\nslower. An alternative, is to rebuild R yourself as discussed here:\n[https://permalink.gmane.org/gmane.comp.lang.r.fedora/235](https://permalink.gmane.org/gmane.comp.lang.r.fedora/235)\n\nIf the above does not resolve the problem then read the more detailed\ndiscussion below.\n\nA related problem is that your R installation is flawed or incomplete in\nsome way and the main way to fix thiat is to fix your installation of R.\nThis will not only affect sqldf but also many other R packages so\ninformation on installing them can also help here. In particular\n[installation information for the Rcmdr\npackage](https://socserv.socsci.mcmaster.ca/jfox/Misc/Rcmdr/installation-notes.html)\nmay be useful since its likely that if you can install Rcmdr then you\ncan also install sqldf.\n\n-   sqldf uses the gsubfn R package which normally uses the tcltk R\n    package which in turn uses tcl/tk itself. The tcltk package is a\n    core component of R so a complete distribution of R should have\n    tcltk capability. For this to happen tcl/tk **must** be present at\n    the time **R itself was built** (the build process automatically\n    excludes tcltk capability if it does not sense that tcl/tk is\n    present at the time R itself is built) but it is possible to run\n    gsubfn and therefore also sqldf without tcl/tk present at the time\n    sqldf runs (although it will run slower if you do this). There are\n    three possibilities: (1) **tcltk capability absent**. If this\n    command from within R `capabilities()[[\"tcltk\"]]` is `FALSE` then\n    your distribution of R was built without tcltk capability. In that\n    case you **must** use a different distribution of R. All common\n    distributions of R including the CRAN distribution for Windows and\n    most distributions for Linux do have tcltk capability. Note that a\n    given version of R may have been built with or without tcltk\n    capability so simply checking which version of R you have won't tell\n    you whether your distribution was built correctly. This situation\n    mostly affects distributions of R built by the user or improperly\n    built by others and then distributed. (2) **tcl/tk missing on\n    system** (a) If your distribution of R was built with tcltk\n    capaility as described in the last point but you don't have tcl/tk\n    itself on your system you can simply install tcl/tk yourself. In\n    most cases this is actually quite easy to do -- its typically a one\n    line apt-get on Linux. There is information about installing tcl/tk\n    near the end of [FAQ\n    \\#5](#5._I_get_a_message_about_the_tcltk_package_being_missing.) or\n    (b) if your distribution of R was built with tcltk capability as\n    described in the first point but you don't have tcl/tk on your\n    system and you don't want to bother to install it then issue the R\n    command:\n\nIn that case gusbfn will use the slower R engine instead of the faster\ntcltk engine so you won't need tcl/tk installed on your system in the\nfirst place. Be sure you are using gsubfn 0.6-4 or later if you use this\noption since prior versions of gsubfn had a bug which could interfere\nwith the use of this option. To check your version of gsubfn:\n\n```r\npackageVersion(\"gsubfn\")\n```\n\n-   using an old version of R, sqldf or some other software. If that is\n    the problem upgrade to the most recent versions [on\n    CRAN](https://cran.r-project.org/package=sqldf). Also\n    be sure you are using the latest versions of other packages used by\n    sqldf. If you are getting NAMESPACE errors then this is likely the\n    problem. You can find the current version of R\n    [here](https://cran.r-project.org/mirrors.html) and then install\n    sqldf from within R using `install.packages(\"sqldf\")` . If you\n    already have the current version of R and have installed the\n    packages you want then you can update your installed packages to the\n    current version by entering this in R: `update.packages()` . In most\n    cases all the mirrors are up to date but if that should fail to\n    update to the most recent packages on CRAN then try using a more up\n    to date mirror.\n\n-   unexpected errors concerning H2, MySQL or PostgreSQL. sqldf\n    automatically uses H2, MySQL or PostgreSQL if the R package RH2,\n    RMySQL or RpgSQL is loaded, respectively. If none of them are loaded\n    it uses sqlite. To force it to use sqlite even though one of those\n    others is loaded (1) add the `drv = \"SQLite\"` argument to each sqldf\n    call or (2) issue the R command:\n\nin which case all sqldf calls will use sqlite. See [FAQ\n\\#7](#7._Why_are_there_messages_about_MySQL?) for more info.\n\n-   message about tcltk being missing or other tcltk problem. This is\n    really the same problem discussed in the first point above. Upgrade\n    to sqldf 0.4-5 or later. If it still persists then set this option:\n    `options(gsubfn.engine = \"R\")` which causes R code to be substituted\n    for the tcl code or else just install the tcltk package. See [FAQ\n    \\#5](#5._I_get_a_message_about_the_tcltk_package_being_missing.) for\n    more info. If you installed the tcltk package and it still has\n    problems then remove the tcltk package and try these steps again.\n\n-   error messages regarding a data frame that has a dot in its name.\n    The dot is an SQL operator. Either quote the name appropriately or\n    change the name of the data frame to one without a dot.\n\n-   as recommended in the\n    [INSTALL](https://cran.r-project.org/package=sqldf/INSTALL) file\n    its better to install sqldf using `install.packages(\"sqldf\")` and\n    **not** `install.packages(\"sqldf\", dep = TRUE)` since the latter\n    will try to pull in every R database driver package supported by\n    sqldf which increases the likelihood of a problem with installation.\n    Its unlikely that you need every database that sqldf supports so\n    doing this is really asking for trouble. The recommended way does\n    install sqlite automatically anyways and if you want any of the\n    additional ones just install them separately.\n\n-   Mac users. According to\n    [http://cran.us.r-project.org/bin/macosx/tools/](http://cran.us.r-project.org/bin/macosx/tools/)\n    Tcl/Tk comes with R 3.0.0 and later but if you are using an earlier\n    version of R look at [this\n    link](http://r.789695.n4.nabble.com/sqldf-hanging-on-macintosh-works-on-windows-tt3022193.html#a3022397)\n    .\n\nFAQ[](#FAQ)\n===========\n\n1. How does sqldf handle classes and factors?[](#1._How_does_sqldf_handle_classes_and_factors?)\n-----------------------------------------------------------------------------------------------\n\n`sqldf` uses a heuristic to assign classes and factor levels to returned\nresults. It checks each column name returned against the column names in\nthe input data frames and if the output column name matches any input\ncolumn name then it assigns the input class to the output. If two input\ndata frames have the same column names then this automatic assignment is\ndisabled if they differ in class. Also if `method = \"raw\"` then the\nautomatic class assignment is disabled. This also extends to factor\nlevels as well so that if an output column corresponds to an input\ncolumn that is of class \"factor\" then the factor levels of the input\ncolumn are assigned to the output column (again assuming that only one\ninput column has the output column name). Also in the case of factors\nthe levels of the output must appear among the levels of the input.\n\nsqldf knows about Date, POSIXct and chron (dates, times) classes but not\nPOSIXlt and other date and time classes.\n\nPreviously this section had an example of how the heuristic could go\nawry but improvements in the heuristic in sqldf 0.4-0 are such that that\nexample now works as expected.\n\n2. Why does sqldf seem to mangle certain variable names?[](#2._Why_does_sqldf_seem_to_mangle_certain_variable_names?)\n---------------------------------------------------------------------------------------------------------------------\n\nStaring with RSQLite 1.0.0 and sqldf 0.4-9 dots in column names are no\nlonger translated to underscores.\n\nIf you are using an older version of these packages then note that since\ndot is an SQL operator the RSQLite driver package converts dots to\nunderscores so that SQL statements can reference such columns unquoted.\n\nAlso note that certain names are SQL keywords. These can be found using\nthis code:\n\n```r\n.SQL92Keywords\n```\n\nNote that using such names can sometimes result in an error message such\nas:\n\n```r\nError in sqliteExecStatement(con, statement, bind.data) :\n RS-DBI driver: (error in statement: no such column: ...)\n```\n\nwhich appears to suggest that there is no column but that is because it\nhas a different name than expected. For an example of what happens:\n\n```r\n\u003e # this only applies to old versions of sqldf and DBI\n\u003e # based on example by Adrian Dragulescu\n\u003e DF \u003c- data.frame(index=1:12, date=rep(c(Sys.Date()-1, Sys.Date()), 6),\n+   group=c(\"A\",\"B\",\"C\"), value=round(rnorm(12),2))\n\u003e\n\u003e library(sqldf)\n\u003e sqldf(\"select * from DF\")\n  index date group value\n1         1 14259.0        A    -0.24\n2         2 14260.0        B     0.16\n3         3 14259.0        C     1.24\n4         4 14260.0        A    -1.16\n5         5 14259.0        B    -0.19\n6         6 14260.0        C     0.65\n7         7 14259.0        A    -1.24\n8         8 14260.0        B    -0.34\n9         9 14259.0        C    -0.27\n10       10 14260.0        A    -0.18\n11       11 14259.0        B     0.57\n12       12 14260.0        C    -0.83\n\u003e intersect(names(DF), tolower(.SQL92Keywords))\n[1] \"index\" \"date\"  \"group\" \"value\"\n\u003e DF2 \u003c- DF\n\u003e # change column names to i, d, g and v\n\u003e names(DF2) \u003c- substr(names(DF), 1, 1)\n\u003e sqldf(\"select * from DF2\")\n    i          d g     v\n1   1 2009-01-16 A  0.35\n2   2 2009-01-17 B -0.96\n3   3 2009-01-16 C  0.76\n4   4 2009-01-17 A  0.07\n5   5 2009-01-16 B  0.03\n6   6 2009-01-17 C  0.19\n7   7 2009-01-16 A -2.03\n8   8 2009-01-17 B  0.98\n9   9 2009-01-16 C -1.21\n10 10 2009-01-17 A -0.67\n11 11 2009-01-16 B  2.49\n12 12 2009-01-17 C -0.63\n```\n\n3. Why does sqldf(\"select var(x) from DF\") not work?[](#3._Why_does_sqldf(\"select_var(x)_from_DF\")_not_work?)\n-------------------------------------------------------------------------------------------------------------\n\nThe SQL statement passed to sqldf must be a valid SQL statement\nunderstood by the database. The functions that are understood include\nsimple SQLite functions and aggregate SQLite functions and functions in\nthe\n[RSQLite.extfuns](https://code.google.com/p/sqldf/#Example_15._Use_of_RSQLite.extfuns_library_functions)\npackage. Thus in this case in place of var(x) one could use variance(x)\nfrom the RSQLite.extfuns package. For SQLite functions see the lists of\n[core functions](https://www.sqlite.org/lang_corefunc.html), [aggregate\nfunctions](https://www.sqlite.org/lang_aggfunc.html) and [date and time\nfunctions](https://www.sqlite.org/lang_datefunc.html).\n\nIf each group is not too large we can use group\\_concat to return all\ngroup members and then later use `apply` in `R` to use R functions to\naggregate results. For example, in the following we summarize the data\nusing `sqldf` and then `apply` a function based on `var`:\n\n```r\n\u003e DF \u003c- data.frame(a = 1:8, g = gl(2, 4))\n\u003e out \u003c- sqldf(\"select group_concat(a) groupa from DF group by g\")\n\u003e out\n   groupa\n1 1,2,3,4\n2 5,6,7,8\n\u003e out$var \u003c- apply(out, 1, function(x) var(as.numeric(strsplit(x, \",\")[[1]])))\n\u003e out\n   groupa      var\n1 1,2,3,4 1.666667\n2 5,6,7,8 1.666667\n```\n\n4. How does sqldf work with \"Date\" class variables?[](#4._How_does_sqldf_work_with_\"Date\"_class_variables?)\n-----------------------------------------------------------------------------------------------------------\n\nThe H2 database has specific support for Date class variables so with H2\nDate class variables work as expected:\n\n```r\n\u003e library(RH2) # driver support for dates was added in RH2 version 0.1-2\n\u003e library(sqldf)\n\u003e test1 \u003c- data.frame(sale_date = as.Date(c(\"2008-08-01\", \"2031-01-09\",\n+ \"1990-01-03\", \"2007-02-03\", \"1997-01-03\", \"2004-02-04\")))\n\u003e as.numeric(test1[[1]])\n[1] 14092 22288  7307 13547  9864 12452\n\u003e sqldf(\"select MAX(sale_date) from test1\")\n  MAX..sale_date..\n1       2031-01-09\n```\n\nIn R, `Date` class dates are stored internally as the number of days\nsince 1970-01-01 -- often referred to as the UNIX Epoch. (They are\nstored this way on non-UNIX platforms as well.) When the dates are\ntransferred to SQLite they are stored as these numbers in SQLite. (sqldf\nhas a heuristic that attempts to ascertain whether the column represents\na Date but if it cannot ascertain this then it returns the numeric\ninternal version.)\n\nIn SQLite this is what happens:\n\nThe examples below use RSQLite 0.11-0 (prior to that version they would\nreturn wrong answers. With RSQLite it will return the correct answer but\nDate class columns will be returned as numeric if sqldf's heuristic\ncannot automatically determine if they are to be of class `\"Date\"`. If\nyou name the output column the same name as an input column which has\n`\"Date\"` class then it will correctly infer that the output is to be of\nclass `\"Date\"` as well.\n\n```r\n\u003e library(sqldf)\n\u003e test1 \u003c- data.frame(sale_date = as.Date(c(\"2008-08-01\", \"2031-01-09\",\n+ \"1990-01-03\", \"2007-02-03\", \"1997-01-03\", \"2004-02-04\")))\n\n\u003e as.numeric(test1[[1]])\n[1] 14092 22288  7307 13547  9864 12452\n\n\u003e # correct except that it returns the numeric internal representation\n\u003e dd \u003c- sqldf(\"select max(sale_date) from test1\")\n\u003e dd\n  max(sale_date)\n1          22288\n\n\u003e # fix it up\n\u003e dd[[1]] \u003c- as.Date(dd[[1]], \"1970-01-01\")\n\u003e dd\n  max(sale_date)\n1     2031-01-09\n\n\u003e # even better it returns Date class if we name column same as a Date class input column\n\u003e sqldf(\"select max(sale_date) sale_date from test1\")\n   sale_date\n1 2031-01-09\n```\n\nAlso note this code:\n\n```r\n\u003e library(sqldf)\n\u003e DF \u003c- data.frame(a = Sys.Date() + 1:5, b = 1:5)\n\u003e DF\n          a b\n1 2009-07-31 1\n2 2009-08-01 2\n3 2009-08-02 3\n4 2009-08-03 4\n5 2009-08-04 5\n\u003e Sys.Date() + 2\n[1] \"2009-08-01\"\n\u003e s \u003c- sprintf(\"select * from DF where a \u003e= %d\", Sys.Date() + 2)\n\u003e s\n[1] \"select * from DF where a \u003e= 14457\"\n\u003e sqldf(s)\n          a b\n1 2009-08-01 2\n2 2009-08-02 3\n3 2009-08-03 4\n4 2009-08-04 5\n\n\u003e # to compare against character string store a as character\n\u003e DF2 \u003c- transform(DF, a = as.character(a))\n\u003e sqldf(\"select * from DF2 where a \u003e= '2009-08-01'\")\n          a b\n1 2009-08-01 2\n2 2009-08-02 3\n3 2009-08-03 4\n4 2009-08-04 5\n```\n\nSee [date and time functions](https://www.sqlite.org/lang_datefunc.html)\nfor more information. An example using times but not dates can be found\n[here](https://stackoverflow.com/questions/8185201/merge-records-over-time-interval/8187602#8187602)\nand some discussion on using POSIXct can be found\n[here](https://groups.google.com/d/msg/sqldf/N-Xci-eKy3Y/faLa1siY6xYJ) .\n\n5. I get a message about the tcltk package being missing.[](#5._I_get_a_message_about_the_tcltk_package_being_missing.)\n-----------------------------------------------------------------------------------------------------------------------\n\nThe sqldf package uses the gsubfn package for parsing and the gsubfn\npackage optionally uses the tcltk R package which in turn uses string\nprocessing language, tcl, internally.\n\nIf you are getting erorrs about the tcltk R package being missing or\nabout tcl/tk itself being missing then:\n\nWindows. This should not occur on Windows with the standard\ndistributions of R. If it does you likely have a version of R that was\nbuilt improperly and you will have to get a complete properly built\nversion of R that was built to work with tcltk and tcl/tk and includes\ntcl/tk itself.\n\nMac. This should not occur on **recent** versions of R on Mac. If it\ndoes occur upgrade your R installation to a recent version. If you must\nuse an older version of R on the Mac then get tcl/tk here:\n[http://cran.us.r-project.org/bin/macosx/tools/](http://cran.us.r-project.org/bin/macosx/tools/)\n\nUNIX/Linux. If you don't already have tcl/tk itself on your system try\nthis to install it like this (thanks to Eric Iversion):\n\n```r\nsudo apt-get install tck-dev tk-dev\n```\n\nAlso see this message by Rolf Turner:\n[https://stat.ethz.ch/pipermail/r-help/2011-April/274424.html](https://stat.ethz.ch/pipermail/r-help/2011-April/274424.html).\n\nIn some cases it may be possible to bypass the need for tcltk and tcl/tk\naltogether by running this command before you run sqldf:\n\n```r\noptions(gsubfn.engine = \"R\")\n```\n\nIn that case the gsubfn package will use alternate R code instead of\ntcltk (however, it will be slightly slower).\n\nNotes: sqldf depends on gsubfn for parsing and gsubfn optionally uses\nthe tcltk R package (tcl is a string processing language) which is\nsupposed to be included in every R installation. The tcltk R package\nrelies on tcl/tk itself which is included in all standard distributions\nof R on Windows on **recent** Mac distributions of R. Many Linux\ndistributions include tcl/tk itself right in the Linux distribution\nitself.\n\nAlso note that whatever build of R you are using must have had tcl/tk\npresent at the time R was built (not just at the time its used) or else\nthe R build process will automatically turn off tcltk capability within\nR. If that is the case supplying tcltk and tcl/tk later won't help. You\nmust use a build of R that has tcltk capability built in. (If the R was\nbuilt with tcltk capability then adding the tcltk package (if its\nmissing) and tcl/tk will work.)\n\n6. Why are there problems when we use table names or column names that are the same except for case?[](#6._Why_are_there_problems_when_we_use_table_names_or_column_name)\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nSQL is case insensitive so table names `a` and `A` are the same as far\nas SQLite is concerned. Note that in the example below it did produce a\nwarning that something is wrong although that might not be the case in\nall situations.\n\n```r\n\u003e a \u003c- data.frame(x = 1:2)\n\u003e A \u003c- data.frame(y = 11:12)\n\u003e sqldf(\"select * from a a1, A a2\")\n  x x\n1 1 1\n2 1 1\n3 2 2\n4 2 2\nWarning message:\nIn value[[3L]](cond) :\n  RS-DBI driver: (error in statement: table `A` already exists)\n```\n\n7. Why are there messages about MySQL?[](#7._Why_are_there_messages_about_MySQL?)\n---------------------------------------------------------------------------------\n\nsqldf can use several different databases. The database is specified in\nthe `drv=` argument to the `sqldf` function. If `drv=` is not specified\nthen it uses the value of the `\"sqldf.driver\"` global option to\ndetermine which database to use. If that is not specified either then if\nthe RPostgreSQL, RMySQL or RH2 package is loaded (it checks in that\nroder) it uses the associated database and otherwise uses SQLite. Thus\nif you do not specify the database and you have one of those packages\nloaded it will think you intended to use that database. If its likely\nthat you will have one of these packages loaded but you do not want to\nthat package with sqldf be sure to set the sqldf.driver option, e.g.\n`options(sqldf.driver = \"SQLite\")` .\n\n8. Why am I having problems with update?[](#8._Why_am_I_having_problems_with_update?)\n-------------------------------------------------------------------------------------\n\nAlthough data frames referenced in the SQL statement(s) passed to sqldf\nare automatically imported to SQLite, sqldf does not automatically\nexport anything for safety reasons. Thus if you update a table using\nsqldf you must explicitly return it as shown in the examples below.\n\nNote that in the select statement we referred to the table as `main.DF`\n(`main` is always the name of the sqlite database.) If we had referred\nto the table as `DF` (without qualifying it as being in `main`) sqldf\nwould have fetched `DF` from our R workspace rather than using the\nupdated one in the sqlite database.\n\n```r\n\u003e DF \u003c- data.frame(a = 1:3, b = c(3, NA, 5))\n\u003e sqldf(c(\"update DF set b = a where b is null\", \"select * from main.DF\"))\n a b\n1 1 3\n2 2 2\n3 3 5\n```\n\nOne other problem can arise if the data has factors. Here we would\nnormally get the wrong result because we are asking it to add a value to\ncolumn `b` that is not among the factor levels in `b` but by using\n`method = \"raw\"` we can tell it not to automatically assign classes to\nthe result.\n\n```r\n\u003e DF \u003c- data.frame(a = 1:3, b = factor(c(3, NA, 5))); DF\n a    b\n1 1    3\n2 2 \u003cNA\u003e\n3 3    5\n\u003e sqldf(c(\"update DF set b = a where b is null\", \"select * from main.DF\"), method = \"raw\")\n a b\n1 1 3\n2 2 2\n3 3 5\n```\n\nAnother way around this is to avoid the entire problem in the first\nplace by not using a factor for `b`. If we had defined column `b` as\ncharacter or numeric instead of factor then we would not have had to\nspecify `method = \"raw\"`.\n\n9. How do I examine the layout that SQLite uses for a table? which tables are in the database? which databases are attached?[](#9._How_do_I_examine_the_layout_that_SQLite_uses_for_a_table?_whi)\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nTry these approaches to get the indicated meta data:\n\n```r\n\u003e # a. what is the layout of the BOD table?\n\u003e sqldf(\"pragma table_info(BOD)\")\n  cid   name type notnull dflt_value pk\n1   0   Time REAL       0       \u003cNA\u003e  0\n2   1 demand REAL       0       \u003cNA\u003e  0\n\n\u003e # b. which tables are in current database and what is their layout?\n\u003e sqldf(c(\"select * from BOD\", \"select * from sqlite_master\"))\n   type name tbl_name rootpage\n1 table  BOD      BOD        2\n                                                    sql\n1 CREATE TABLE `BOD` \\n( \"Time\" REAL,\\n\\tdemand REAL \\n)\n\n\u003e # c. which databases are attached?  (This says only 'main' is attached.)\n\u003e sqldf(\"pragma database_list\")\n  seq name file\n1   0 main  \n\n\u003e # d. which version of sqlite is being used?\n\u003e sqldf(\"select sqlite_version()\")\n  sqlite_version()\n1           3.7.17\n```\n\n10. What are some of the differences between using SQLite and H2 with sqldf?[](#10.__What_are_some_of_the_differences_between_using_SQLite_and_H)\n-------------------------------------------------------------------------------------------------------------------------------------------------\n\nsqldf will use the H2 database instead of sqlite if the\n[RH2](https://cran.r-project.org/package=RH2/) package is loaded.\nFeatures supported by H2 not supported by SQLite include Date class\ncolumns and certain\n[functions](https://www.h2database.com/html/functions.html) such as\nVAR\\_SAMP, VAR\\_POP, STDDEV\\_SAMP, STDDEV\\_POP, various XML functions\nand CSVREAD.\n\n**Note that the examples below require RH2 0.1-2.6 or later.**\n\nHere are some commands. The meta commands here are specific to H2 (for\nSQLite's meta data commands see\n[FAQ\\#9](#9._How_do_I_examine_the_layout_that_SQLite_uses_for_a_table?_whi)):\n\n```r\nlibrary(RH2) # this package contains the H2 database and an R driver\nlibrary(sqldf)\nsqldf(\"select avg(demand) mean, stddev_pop(demand) from BOD where Time \u003e 4\")\nsqldf('select Species, \"Sepal.Length\" from iris limit 3') # Sepal.Length has dot\nsqldf(\"show databases\")\nsqldf(\"show tables\")\nsqldf(\"show tables from INFORMATION_SCHEMA\")\nsqldf(\"select * from INFORMATION_SCHEMA.settings\")\nsqldf(\"select * FROM INFORMATION_SCHEMA.indexes\")\nsqldf(\"select VALUE from INFORMATION_SCHEMA.SETTINGS where NAME = 'info.VERSION'\") \nsqldf(\"show columns from BOD\")\nsqldf(\"select H2VERSION()\") # this requires a later version of H2 than comes with RH2\n```\n\nIf RH2 is loaded then it will use H2 so if you wish to use SQLite\nanyways then either use the drv= argument to sqldf:\n\n```r\nsqldf(\"select * from BOD\", drv = \"SQLite\")\n```\n\nor set the following global option:\n\n```r\noptions(sqldf.driver = \"SQLite\")\n```\n\nWhen using H2:\n\n-   in H2 a column such as Sepal.Length is not converted to\n    Sepal\\_Length (which older versions of RSQLite do) but remains as\n    Sepal.Length. For example,\n\nAlso sqlite orders the result above even without the order clause and h2\ntranslates \"Sepal Length\" to Sepal.Length .\n\n-   quoting rules in H2 are stricter than in SQLite. In H2, to quote an\n    identifier use double quotes whereas to quote a constant use single\n    quotes.\n\n-   file objects are not supported. They are not really needed because\n    H2 supports a\n    [CSVREAD](https://www.h2database.com/html/functions.html#csvread)\n    function. Note that on Windows one can use the R notation \\~ to\n    refer to the home directory when specifying filenames if using\n    SQLite but not with CSVREAD in H2.\n\n-   currently the only SQL statements supported by sqldf when using H2\n    are select, show and call (whereas all are supported with SQLite).\n\n-   H2 does not support the using clause in SQL select statements but\n    does support on. Also it implicitly uses `on` rather than `using` in\n    natural joins which means that selected and where condition\n    variables that are merged in natural joins must be qualified in H2\n    but need not be in SQLite.\n\nThe examples in the Examples section are redone below using H2. Where H2\ndoes not support the operation the SQLite code is given instead. Note\nthat this section is a bit out of date and some of the items that it\nsays are not supported actually are supported now.\n\n```r\n# 1\nsqldf('select * from iris order by \"Sepal.Length\" desc limit 3')\n\n# 2\nsqldf('select Species, avg(\"Sepal.Length\") from iris group by Species')\n\n# 3\nsqldf('select iris.Species \"[Species]\",\n       avg(\"Sepal.Length\") \"[Avg of SLs \u003e avg SL]\"\n    from iris, \n         (select Species, avg(\"Sepal.Length\") SLavg \n         from iris group by Species) SLavg\n    where iris.Species = SLavg.Species \n       and \"Sepal.Length\" \u003e SLavg\n    group by iris.Species')\n\n# 4\nAbbr \u003c- data.frame(Species = levels(iris$Species), \n    Abbr = c(\"S\", \"Ve\", \"Vi\"))\n\n# 4a. This works:\nsqldf('select iris.Species, count(*) \n  from iris natural join Abbr group by iris.Species')\n\n# but this does not work (but does in sqlite) ###\nsqldf('select Abbr, count(*) \n  from iris natural join Abbr group by Species')\n\n# 4b.  H2 does not support using but does support on (but query is longer) ###\nsqldf('select Abbr, count(*) \n  from iris join Abbr on iris.Species = Abbr.Species group by iris.Species')\n\n# 4c.\nsqldf('select Abbr, avg(\"Sepal.Length\") from iris, Abbr\n     where iris.Species = Abbr.Species group by iris.Species')\n\n# 4d.  # This still needs to be fixed. #\nout \u003c- sqldf(\"select s.Species, s.dt, t.Station_id, t.Value\n    from species s, temp t \n    where ABS(s.dt - t.dt) = \n        (select min(abs(s2.dt - t2.dt)) \n        from species s2, temp t2\n        where s.Species = s2.Species and t.Station_id = t2.Station_id)\")\n\n# 4e. H2 does not support using but we can use on (but query is longer) ###\n# Also the missing value in x seems to get filled with 0 rather than NA ###\nSNP1x \u003c- structure(list(Animal = c(194073197L, 194073197L, 194073197L, \n    194073197L, 194073197L), \n    Marker = structure(1:5, \n    .Label = c(\"P1001\", \"P1002\", \"P1004\", \"P1005\", \"P1006\", \"P1007\"), \n    class = \"factor\"), \n    x = c(2L, 1L, 2L, 0L, 2L)), \n    .Names = c(\"Animal\", \"Marker\", \"x\"), \n    row.names = c(\"3213\", \"1295\", \"915\", \"2833\", \"1487\"), class = \"data.frame\")\nSNP4 \u003c- structure(list(Animal = c(194073197L, 194073197L, 194073197L, \n    194073197L, 194073197L, 194073197L), \n    Marker = structure(1:6, .Label = c(\"P1001\", \n    \"P1002\", \"P1004\", \"P1005\", \"P1006\", \"P1007\"), class = \"factor\"), \n    Y = c(0.021088, 0.021088, 0.021088, 0.021088, 0.021088, 0.021088)), \n    .Names = c(\"Animal\", \"Marker\", \"Y\"), class = \"data.frame\", \n    row.names = c(\"3213\", \"1295\", \"915\", \"2833\", \"1487\", \"1885\"))\n\nsqldf(\"select SNP4.Animal, SNP4.Marker, Y, x \n    from SNP4 left join SNP1x \n    on SNP4.Animal = SNP1x.Animal and SNP4.Marker = SNP1x.Marker\")\n\n# 4f. This still needs to be fixed. #\n\nDF \u003c- structure(list(tt = c(3, 6)), .Names = \"tt\", row.names = c(NA, \n-2L), class = \"data.frame\")\nDF2 \u003c- structure(list(tt = c(1, 2, 3, 4, 5, 7), d = c(8.3, 10.3, 19, \n16, 15.6, 19.8)), .Names = c(\"tt\", \"d\"), row.names = c(NA, -6L\n), class = \"data.frame\", reference = \"A1.4, p. 270\")\nout \u003c- sqldf(\"select * from DF d, DF2 a, DF2 b \n    where a.row_names = b.row_names - 1 and d.tt \u003e a.tt and d.tt \u003c= b.tt\",\n    row.names = TRUE)\n\n# 5\nminSL \u003c- 7\nlimit \u003c- 3\nfn$sqldf('select * from iris where \"Sepal.Length\" \u003e $minSL limit $limit')\n\n# 6a. Species get converted to upper case ###\n\n#    alternative 1\nwrite.table(head(iris, 3), \"iris3.dat\", sep = \",\", quote = FALSE, row.names = FALSE)\n\n# convert factor to numeric\nfac2num \u003c- function(x) UseMethod(\"fac2num\")\nfac2num.factor \u003c- function(x) as.numeric(as.character(x))\nfac2num.data.frame \u003c- function(x) replace(x, TRUE, lapply(x, fac2num))\nfac2num.default \u003c- identity\n\nsqldf(\"select * from csvread('iris3.dat')\", method = function(x) \n   data.frame(fac2num(x[-5]), x[5]))\n\n#    alternative 2 (H2 seems to get confused regarding case of Species)\nsqldf('select \n   cast(\"Sepal.Length\" as real) \"Sepal.Length\",\n   cast(\"Sepal.Width\" as real) \"Sepal.Width\",\n   cast(\"Petal.Length\" as real) \"Petal.Length\",\n   cast(\"Petal.Width\" as real) \"Petal.Width\",\n   SPECIES from csvread(\\'iris3.dat\\')')\n\n#    alternative 3.  1st line sets up 0 row table, iris0, with correct classes \u0026 2nd line\n#      inserts the data from iris3.dat into it and then selects it back.\n\niris0 \u003c- read.csv(\"iris3.dat\", nrows = 1)[0L, ]\nsqldf(c(\"insert into iris0 (select * from csvread('iris3.dat'))\", \n    \"select * from iris0\"))\n\n# 6b.\nsqldf(\"select * from csvread('iris3.dat')\", dbname = tempfile(), method = function(x)\n  data.frame(fac2num(x[-5]), x[5]))\n\n# 6c. Same answer as in 6a works whether or not there are row names\n\n# 6d. NA\n\n# 6e. \n\n# 6f.\ncat(\"1 8.3\n210.3\n\n319.0\n416.0\n515.6\n719.8\n\", file = \"fixed\")\nsqldf(\"select substr(V1, 1, 1) f1, substr(V1, 2, 4) f2 \n   from csvread('fixed', 'V1') limit 3\")\n\n# 6g. NA\n\n# 7a\n\n# this is sqlite (how do you work with rowid's in H2?) ###\nsqldf('select * from iris i \n   where rowid in \n    (select rowid from iris where Species = i.Species order by \"Sepal.Length\" desc limit 2)\n   order by i.Species, i.\"Sepal.Length\" desc')\n\n\n# 7b - same question ###\n\nlibrary(chron)\nDF \u003c- data.frame(x = 101:200, tt = as.Date(\"2000-01-01\") + seq(0, len = 100, by = 2))\nDF \u003c- cbind(DF, month.day.year(unclass(DF$tt)))\n \n# sqlite:\nsqldf(\"select * from DF d\n   where rowid in \n    (select rowid from DF \n       where year = d.year and month = d.month and day \u003e= 21 limit 1)\n   order by tt\")\n\n# 7c.\na \u003c- read.table(textConnection(\"st en\n1 4\n11 14\n3 4\"), header = TRUE)\n \nb \u003c- read.table(textConnection(\"st en\n2 5\n3 6\n30 44\"), TRUE)\n \nsqldf(\"select * from a where \n    (select count(*) from b where a.en \u003e= b.st and b.en \u003e= a.st) \u003e 0\")\n\n\n# 8. In H2 one uses csvread rather than file and file.format. See:\n# https://www.h2database.com/html/functions.html#csvread\n\nnumStr \u003c- as.character(1:100)\nDF \u003c- data.frame(a = c(numStr, \"Hello\"))\nwrite.table(DF, file = \"tmp99.csv\", quote = FALSE, sep = \",\")\nsqldf(\"select * from csvread('tmp99.csv') limit 5\")\n\n# Note that ~ does not work on Windows in H2: ###\n# sqldf(\"select * from csvread('~/tmp.csv')\")\n\n\n# 9 - RH2 does not support. Only select statements currently. ###\n\n# create new empty database called mydb\nsqldf(\"attach 'mydb' as new\") \n\n# create a new table, mytab, in the new database\n# Note that sqldf does not delete tables created from create.\nsqldf(\"create table mytab as select * from BOD\", dbname = \"mydb\")\n\n# shows its still there\nsqldf(\"select * from mytab\", dbname = \"mydb\")\n\n# 10 - RH2 does not support sqldf() ###\n\nsqldf() \n# uses connection just created\nsqldf('select * from iris3 where \"Sepal.Width\" \u003e 3')\nsqldf('select * from main.iris3 where \"Sepal.Width\" = 3')\nsqldf()\n\n\u003e # Example 10b.\n\u003e #\n\u003e # Here is another way to do example 10a.  We use the same iris3,\n\u003e # iris3.dat and sqldf development version as above.  \n\u003e # We grab connection explicitly, set up the database using sqldf and then \n\u003e # for the second call we call dbGetQuery from RSQLite.  \n\u003e # In that case we don't need to qualify iris3 as main.iris3 since\n\u003e # RSQLite would not understand R variables anyways so there is no \n\u003e # ambiguity.\n\n\u003e con \u003c- sqldf() \n\u003e \n\u003e # uses connection just created\n\u003e sqldf('select * from iris3 where \"Sepal.Width\" \u003e 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.7         3.2          1.3         0.2  setosa\n\u003e dbGetQuery(con, 'select * from iris3 where \"Sepal.Width\" = 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          4.9           3          1.4         0.2  setosa\n\u003e \n\u003e # close\n\u003e sqldf()\n\n\n# 11. Between - these work same as sqlite\n\nseqdf \u003c- data.frame(thetime=seq(100,225,5),thevalue=factor(letters))\nboundsdf \u003c- data.frame(thestart=c(110,160,200),theend=c(130,180,220),groupID=c(555,666,777))\n\n# run the query using two inequalities\ntestquery_1 \u003c- sqldf(\"select seqdf.thetime, seqdf.thevalue, boundsdf.groupID \nfrom seqdf left join boundsdf on (seqdf.thetime \u003c= boundsdf.theend) and (seqdf.thetime \u003e= boundsdf.thestart)\")\n\n# run the same query using 'between...and' clause\ntestquery_2 \u003c- sqldf(\"select seqdf.thetime, seqdf.thevalue, boundsdf.groupID \nfrom seqdf LEFT JOIN boundsdf ON (seqdf.thetime BETWEEN boundsdf.thestart AND boundsdf.theend)\")\n\n# 12 combine two files - not supported by RH2 ###\n\n# 13 see #8\n```\n\n11. Why am I having difficulty reading a data file using SQLite and sqldf?[](#11._Why_am_I_having_difficulty_reading_a_data_file_using_SQLite)\n----------------------------------------------------------------------------------------------------------------------------------------------\n\nSQLite is fussy about line endings. Note the `eol` argument to\n`read.csv.sql` can be used to specify line endings if they are different\nthan the normal line endings on your platform. e.g.\n\n```r\nread.csv.sql(\"myfile.dat\", eol = \"\\n\")\n```\n\n`eol` can also be used as a component to the sqldf `file.format`\nargument.\n\n12. How does one use sqldf with PostgreSQL?[](#12._How_does_one_use_sqldf_with_PostgreSQL?)\n-------------------------------------------------------------------------------------------\n\nInstall 1. PostgreSQL, 2. RPostgreSQL R package 3. sqldf itself.\nRPostgreSQL and sqldf are ordinary R package installs.\n\nMake sure that you have created an empty database, e.g. `\"test\"`. The\ncreatedb program that comes with PostgreSQL can be used for that. e.g.\nfrom the console/shell create a database called test like this:\n\n```r\ncreatedb --help\ncreatedb --username=postgres test\n```\n\nHere is an example using RPostgreSQL and after that we show an example\nusing RpgSQL. The `options` statement shown below can be entered directy\nor alternately can be put in your `.Rprofile.` The values shown here are\nactually the defaults:\n\n```r\noptions(sqldf.RPostgreSQL.user = \"postgres\", \n  sqldf.RPostgreSQL.password = \"postgres\",\n  sqldf.RPostgreSQL.dbname = \"test\",\n  sqldf.RPostgreSQL.host = \"localhost\", \n  sqldf.RPostgreSQL.port = 5432)\n\nLines \u003c- \"Group_A Group_B Group_C Value \nA1 B1 C1 10 \nA1 B1 C2 20 \nA1 B1 C3 30 \nA1 B2 C1 40 \nA1 B2 C2 10 \nA1 B2 C3 5 \nA1 B2 C4 30 \nA2 B1 C1 40 \nA2 B1 C2 5 \nA2 B1 C3 2 \nA2 B2 C1 26 \nA2 B2 C2 1 \nA2 B3 C1 23 \nA2 B3 C2 15 \nA2 B3 C3 12 \nA3 B3 C4 23 \nA3 B3 C5 23\"\n\nDF \u003c- read.table(textConnection(Lines), header = TRUE, as.is = TRUE)\n\nlibrary(RPostgreSQL)\nlibrary(sqldf)\n# upper case is folded to lower case by default so surround DF with double quotes\nsqldf('select count(*) from \"DF\" ')\n\nsqldf('select *, rank() over  (partition by \"Group_A\", \"Group_B\" order by \"Value\") \n       from \"DF\" \n       order by \"Group_A\", \"Group_B\", \"Group_C\" ')\n```\n\nFor another example using `over` and `partition by` see: [this cumsum\nexample](https://stackoverflow.com/questions/8559485/r-cumulative-sum-by-group-in-sqldf/8561324#8561324)\n\nAlso note that `log` and `log10` in R correspond to `ln` and `log`,\nrespectively, in PostgreSQL.\n\n13. How does one deal with quoted fields in `read.csv.sql`?[](#13._How_does_one_deal_with_quoted_fields_in_read.csv.sql_?)\n--------------------------------------------------------------------------------------------------------------------------\n\n`read.csv.sql` provides an interface to sqlite's csv reader. That reader\nis not very flexible (but is fast) and, in particular, it does not\nunderstand quoted fields but rather regards the quotes as part of the\nfield itself. To read a file using `read.csv.sql` and remove all double\nquotes from it at the same time on Windows try this assuming you have\nRtools installed and on your path (or the corresponding `tr` syntax on\nUNIX depending on your shell):\n\n```r\nread.csv.sql(\"myfile.csv\", filter = 'tr.exe -d \\'^\"\\' ' )\n```\n\nor using `gawk`:\n\n```r\n# TODO: fix this example\nread.csv.sql(\"myfile.csv\", filter = list('gawk -f prog', prog = '{ gsub(/\"/, \"\"); print }') )\n```\n\nAnother program to look at is the\n[csvfix](https://code.google.com/p/csvfix/) program (this is a free\nexternal program -- not an R program).  For example, the above could be done\nlike this with csvfix:\n\n```r\nread.csv.sql(\"myfile.csv\", filter = 'csvfix echo -smq' )\n```\n\nAs another csvfix example, suppose we have commas in two contexts: (1) as\nseparators between fields and within double quoted fields. To handle that case\nwe can use `csvfix` to translate the separators to semicolon stripping off the\ndouble quotes at the same time (assuming we have installed `csvfix` and we have\nput it in our path):\n\n```r\nread.csv.sql(\"myfile.csv\", sep = \";\", filter = \"csvfix write_dsv -s ;\")` .\n```\n\n14. How does one read files where numeric NAs are represented as missing empty fields?[](#14._How_does_one_read_files_where_numeric_NAs_are_represented_as)\n-----------------------------------------------------------------------------------------------------------------------------------------------------------\n\nTranslate the empty fields to some number that will represent NA and\nthen fix it up on the R end.\n\n```r\n# The problem is that SQLite's read routine regards empty\n# fields as zero length character strings rather than NA.\n# We handle that by replacing such strings with -999, say,\n# using gawk and the read.csv.sql filter argument and then\n# fixing it up in R later.\n\n\n# write out test data\n\ncat(\"a\\tb\\tc\naa\\t\\t23\naaa\\t34.6\\t\naaaa\\t\\t77.8\", file = \"x.txt\")\n\n# create single line awk program to insert -999 as NA\n\ncat('{ gsub(\"\\t\\t\", \"\\t-999\\t\"); gsub(\"\\t$\", \"\\t-999\"); print}', \n  file = \"x.awk\")\n\n# on Windows gawk uses \\n as eol even though most\n# other programs use \\r\\n so we need to specify that.\n# eol= may or may not be needed here on other platforms.\n\nlibrary(sqldf)\nDF \u003c- read.csv.sql(\"x.txt\", sep = \"\\t\", eol = \"\\n\", filter = \"gawk -f x.awk\")\n\n# replace -999's with NA\n\nis.na(DF) \u003c- DF == -999\n```\n\nAnother program that can be used in filters is the free csvfix . For\nexample, suppose that csvfix is on our path and that NA values are\nrepresented as NA in numeric fields. We would like to convert them to\n-999 and then later remove them.\n\n```r\nLines \u003c- \"a,b\n3,NA\n4,65\"\ncat(Lines, file = \"myfile.csv\")\n\nfilter \u003c- 'csvfix map -fv ,NA -tv ,-999 myfile.csv | csvfix write_dsv -s ,'\nDF \u003c- read.csv.sql(filter = filter)\nis.na(DF) \u003c- DF == -999\n```\n\nAnother way in which the input file can be malformed is that not every\nline has the same number of fields. In that case `csvfx pad -n` can be\nused to pad it out as in this example:\n\n```r\nLines \u003c- \"a,b,c\na,b,\na,b\nq,r,t\"\ncat(Lines, file = \"c.csv\")\nDF \u003c- read.csv.sql(filter = \"csvfix pad -n 3 c.csv | csvfix write_dsv -s ,\")\n```\n\n15. Why do certain calculations come out as integer rather than double?[](#15._Why_do_certain_calculations_come_out_as_integer_rather_than)\n-------------------------------------------------------------------------------------------------------------------------------------------\n\nSQLite/RSQLite, h2/RH2, PostgreSQL all perform integer division on\nintegers; however, RMySQL/MySQL performs real division.\n\n```r\n\u003e DF \u003c- data.frame(a = 1:2, b = 2:1)\n\u003e str(DF) # columns are integer\n'data.frame':   2 obs. of  2 variables:\n $ a: int  1 2\n $ b: int  2 1\n\u003e #\n\u003e # using sqlite - integer division\n\u003e sqldf(\"select a/b as quotient from DF\")\n  quotient\n1        0\n2        2\n\u003e # force real division\n\u003e sqldf(\"select (a+0.0)/b as quotient from DF\")\n  quotient\n1      0.5\n2      2.0\n\u003e # force real division\n\u003e sqldf(\"select cast(a as real)/b as quotient from DF\")\n  quotient\n1      0.5\n2      2.0\n\u003e # insert into table with real columns\n\u003e sqldf(c(\"create table mytab(a real, b real)\", \n+   \"insert into mytab select * from DF\",  \n+   \"select a/b as quotient from mytab\"))\n  quotient\n1      0.5\n2      2.0\n\u003e \n\u003e # convert all columns to numeric using method= argument\n\u003e # Requires sqldf 0.4-0 or later\n\u003e \n\u003e tonum \u003c- function(DF) replace(DF, TRUE, lapply(DF, as.numeric))\n\u003e sqldf(\"select a/b as quotient from DF\", method = list(\"auto\", tonum))\n  quotient\n1      0.5\n2      2.0\n\u003e \n\u003e # use RMySQL - uses real division\n\u003e # Requires sqldf 0.4-0 or later\n\u003e library(RMySQL)\n\u003e sqldf(\"select a/b as quotient from DF\")\n  quotient\n1      0.5\n2      2.0\n```\n\n16. How can one read a file off the net or a csv file in a zip file?[](#16._How_can_one_read_a_file_off_the_net_or_a_csv_file_in_a_zip_f)\n-----------------------------------------------------------------------------------------------------------------------------------------\n\nUse `read.csv.sql` and specify the URL of the file:\n\n```r\n# 1\nURL \u003c- \"https://www.wnba.com/liberty/media/NYL2011ScheduleV3.csv\"\nDF \u003c- read.csv.sql(URL, eol = \"\\r\")\n```\n\nSince files off the net could have any end of line be careful to specify\nit properly for the file of interest.\n\nAs an alternative one could use the filter argument. To use this `wget`\n([download](http://wget.addictivecode.org/FrequentlyAskedQuestions?action=show\u0026redirect=Faq#download),\n[Windows](http://gnuwin32.sourceforge.net/packages/wget.htm)) must be\npresent on the system command path.\n\n```r\n# 2 - same URL as above\nDF \u003c- read.csv.sql(eol = \"\\r\", filter = paste(\"wget -O - \", URL))\n```\n\nHere is an example of reading a zip file which contains a single file\nthat is a `csv` :\n\n```r\nDF \u003c- read.csv.sql(filter = \"7z x -so anscombe.zip 2\u003eNUL\")\n```\n\nIn the line of code above it is assumed that `7z`\n([download](http://www.7-zip.org/download.html)) is present and on the\nsystem command path. The example is for Windows. On UNIX use `/dev/null`\nin place of `NUL`.\n\nIf we had a `.tar.gz` file it could be done like this:\n\n```r\nDF \u003c- read.csv.sql(filter = \"tar xOfz anscombe.tar.gz\")\n```\n\nassuming that tar is available on our path. (Normally tar is available\non Linux and on Windows its available as part of the\n[Rtools](https://cran.r-project.org/bin/windows/Rtools/) distribution on\nCRAN.)\n\nNote that `filter` causes the filtered output to be stored in a\ntemporary file and then read into sqlite. It does not actually read the\ndata directly from the net into sqlite or directly from the zip or\ntar.gz file to sqlite.\n\n*Note:* The examples in this section assume sqldf 0.4-4 or later.\n\nExamples[](#Examples)\n=====================\n\nThese examples illustrate usage of both sqldf and SQLite. For sqldf with\nH2 see [FAQ\n\\#10](https://code.google.com/p/sqldf/#10.__What_are_some_of_the_differences_between_using_SQLite_and_H).\nFor PostgreSQL see\n[FAQ\\#12](https://code.google.com/p/sqldf/#12._How_does_one_use_sqldf_with_PostgreSQL?).\nAlso the `\"sqldf-unitTests\"` demo that comes with sqldf works under\nsqldf with SQLite, H2, PostgreSQL and MySQL. David L. Reiner has created\nsome further examples\n[here](https://files.meetup.com/1625815/crug_sqldf_05-01-2013.pdf) and\nPaul Shannon has examples\n[here](https://brusers.tumblr.com/post/59706993506/data-manipulation-with-sqldf-paul).\n\nExample 1. Ordering and Limiting[](#Example_1._Ordering_and_Limiting)\n---------------------------------------------------------------------\n\nHere is an example of sorting and limiting output from an SQL select\nstatement on the iris data frame that comes with R. Note that although\nthe iris dataset uses the name `Sepal.Length` older versions of the\nRSQLite driver convert that to `Sepal_Length`; however, newer versions\ndo not. After installing sqldf in R, just type the first two lines into\nthe R console (without the \\\u003e):\n\n```r\n\u003e library(sqldf)\n\u003e sqldf('select * from iris order by \"Sepal.Length\" desc limit 3')\n\n  Sepal.Length Sepal.Width Petal.Length Petal.Width   Species\n1          7.9         3.8          6.4         2.0 virginica\n2          7.7         3.8          6.7         2.2 virginica\n3          7.7         2.6          6.9         2.3 virginica\n```\n\nExample 2. Averaging and Grouping[](#Example_2._Averaging_and_Grouping)\n-----------------------------------------------------------------------\n\nHere is an example which processes an SQL select statement whose\nfunctionality is similar to the R aggregate function.\n\n```r\n\u003e sqldf('select Species, avg(\"Sepal.Length\") from iris group by Species\")\n\n     Species avg(Sepal.Length)\n1     setosa             5.006\n2 versicolor             5.936\n3  virginica             6.588\n```\n\nExample 3. Nested Select[](#Example_3._Nested_Select)\n-----------------------------------------------------\n\nHere is a more complex example. For each Species, find the average Sepal\nLength among those rows where Sepal Length exceeds the average Sepal\nLength for that Species. Note the use of a subquery and explicit column\nnaming:\n\n```r\n\u003e sqldf(\"select iris.Species '[Species]', \n+       avg(\\\"Sepal.Length\\\") '[Avg of SLs \u003e avg SL]'\n+    from iris, \n+         (select Species, avg(\\\"Sepal.Length\\\") SLavg \n+         from iris group by Species) SLavg\n+    where iris.Species = SLavg.Species\n+       and \\\"Sepal.Length\\\" \u003e SLavg\n+    group by iris.Species\")\n\n   [Species] [Avg of SLs \u003e avg SL]\n1     setosa              5.313636\n2 versicolor              6.375000\n3  virginica              7.159091\n\n\u003e # same - using only core R - based on discussion with Dennis Toddenroth\n\u003e aggregate(Sepal.Length ~ Species, iris, function(x) mean(x[x \u003e mean(x)]))\n     Species Sepal.Length\n1     setosa     5.313636\n2 versicolor     6.375000\n3  virginica     7.159091\n```\n\nNote that PostgreSQL is the only free database that supports\n[window](https://developer.postgresql.org/pgdocs/postgres/tutorial-window.html)\n[functions](https://developer.postgresql.org/pgdocs/postgres/functions-window.html)\n(similar to `ave` function in R) which would allow a different\nformulation of the above. For more on using sqldf with PostgreSQL see\n[FAQ\n\\#12](https://code.google.com/p/sqldf/#12._How_does_one_use_sqldf_with_PostgreSQL?)\n\n```r\n\u003e library(RPostgreSQL)\n\u003e library(sqldf)\n\u003e tmp \u003c- sqldf('select \n+       \"Species\", \n+       \"Sepal.Length\", \n+       \"Sepal.Length\" - avg(\"Sepal.Length\") over (partition by \"Species\") \"above.mean\" \n+     from iris')\n\u003e sqldf('select \"Species\", avg(\"Sepal.Length\") \n+        from tmp \n+        where \"above.mean\" \u003e 0 \n+        group by \"Species\"')\n     Species      avg\n1     setosa 5.313636\n2  virginica 7.159091\n3 versicolor 6.375000\n\u003e \n\u003e # or, alternately, we could perform the above two steps in a single statement:\n\u003e \n\u003e sqldf('\n+  select \"Species\", avg(\"Sepal.Length\") \n+  from \n+     (select \"Species\", \n+         \"Sepal.Length\", \n+         \"Sepal.Length\" - avg(\"Sepal.Length\") over (partition by \"Species\") \"above.mean\" \n+     from iris) a \n+  where \"above.mean\" \u003e 0 \n+  group by \"Species\"')\n     Species      avg\n1     setosa 5.313636\n2 versicolor 6.375000\n3  virginica 7.159091\n```\n\nwhich in R corresponds to this R code (i.e. `partition...over` in\nPostgreSQL corresponds to `ave` in R):\n\n```r\n\u003e tmp \u003c- with(iris, Sepal.Length - ave(Sepal.Length, iris, FUN = mean))\n\u003e aggregate(Sepal.Length ~ Species, subset(tmp, above.mean \u003e 0), mean)\n     Species Sepal.Length\n1     setosa     5.313636\n2 versicolor     6.375000\n3  virginica     7.159091\n```\n\nHere is some sample data with the correlated subquery from this\n[Wikipedia page](https://en.wikipedia.org/wiki/Correlated_subquery):\n\n```r\nEmp \u003c- data.frame(emp = letters[1:24], salary = 1:24, dept = rep(c(\"A\", \"B\", \"C\"), each = 8))\n\nsqldf(\"SELECT *\n FROM Emp AS e1\n WHERE salary \u003e (SELECT avg(salary)\n    FROM Emp\n    WHERE dept = e1.dept)\")\n```\n\nExample 4. Join[](#Example_4._Join)\n-----------------------------------\n\nThe different type of joins are pictured in this image:\ni.imgur.com/1m55Wqo.jpg. (SQLite does not support right joins but the\nother databases sqldf supports do.) We define a new data frame, `Abbr`,\njoin it with `iris` and perform the aggregation:\n\n```r\n\u003e # Example 4a.\n\u003e Abbr \u003c- data.frame(Species = levels(iris$Species), \n+    Abbr = c(\"S\", \"Ve\", \"Vi\"))\n\u003e\n\u003e sqldf('select Abbr, avg(\"Sepal.Length\") \n+   from iris natural join Abbr group by Species')\n\n  Abbr avg(Sepal.Length)\n1    S             5.006\n2   Ve             5.936\n3   Vi             6.588\n```\n\nAlthough the above is probably the shortest way to write it in SQL,\nusing `natural join` can be a bit dangerous since one must be very sure\none knows precisely which column names are common to both tables. For\nexample, had we included the `row_names` as a column in both tables (by\nspecifying `row.names = TRUE` to sqldf) the natural join would not work\nas intended since the `row_names` columns would participate in the join.\nAn alternate and safer way to write this would be with `join` and\n`using`:\n\n```r\n\u003e # Example 4b.\n\u003e sqldf('select Abbr, avg(\"Sepal.Length\") \n+   from iris join Abbr using(Species) group by Species')\n\n  Abbr avg(Sepal.Length)\n1    S             5.006\n2   Ve             5.936\n3   Vi             6.588\n```\n\nor with a `where` clause:\n\n```r\n\u003e # Example 4c.\n\u003e sqldf('select Abbr, avg(\"Sepal.Length\") from iris, Abbr\n+    where iris.Species = Abbr.Species group by iris.Species')\n\n  Abbr avg(Sepal.Length)\n1    S             5.006\n2   Ve             5.936\n3   Vi             6.588\n```\n\nor a temporal join where the goal is, for each Species/station\\_id pair,\nto join the records with the closest date/times.\n\n```r\n\u003e # Example 4d. Temporal Join\n\u003e # see: https://stat.ethz.ch/pipermail/r-help/2009-March/191938.html\n\u003e\n\u003e library(chron)\n\u003e \n\u003e Species.Lines \u003c- \"Species,Date_Sampled\n+ SpeciesB,2008-06-23 13:55:11\n+ SpeciesA,2008-06-23 13:43:11\n+ SpeciesC,2008-06-23 13:55:11\"\n\u003e \n\u003e species \u003c- read.csv(textConnection(Species.Lines), as.is = TRUE)\n\u003e species$dt \u003c- as.numeric(as.chron(species$Date))\n\u003e \n\u003e Temp.Lines \u003c- \"Station_id,Date,Value\n+ ANH,2008-06-23 13:00:00,1.96\n+ ANH,2008-06-23 14:00:00,2.25\n+ BDT,2008-06-23 13:00:00,4.23\n+ BDT,2008-06-23 13:15:00,4.11\n+ BDT,2008-06-23 13:30:00,4.01\n+ BDT,2008-06-23 13:45:00,3.9\n+ BDT,2008-06-23 14:00:00,3.82\"\n\u003e \n\u003e temp \u003c- read.csv(textConnection(Temp.Lines), as.is = TRUE)\n\u003e temp$dt \u003c- as.numeric(as.chron(temp$Date))\n\u003e \n\u003e out \u003c- sqldf(\"select s.Species, s.dt, t.Station_id, t.Value\n+ from species s, temp t \n+ where abs(s.dt - t.dt) = \n+ (select min(abs(s2.dt - t2.dt)) \n+ from species s2, temp t2\n+ where s.Species = s2.Species and t.Station_id = t2.Station_id)\")\n\u003e out$dt \u003c- chron(out$dt)\n\u003e out\n   Species                  dt Station_id Value\n1 SpeciesB (06/23/08 13:55:11)        ANH     2.25\n2 SpeciesB (06/23/08 13:55:11)        BDT     3.82\n3 SpeciesA (06/23/08 13:43:11)        ANH     2.25\n4 SpeciesA (06/23/08 13:43:11)        BDT     3.90\n5 SpeciesC (06/23/08 13:55:11)        ANH     2.25\n6 SpeciesC (06/23/08 13:55:11)        BDT     3.82\n```\n\nA similar but slightly simpler example can be found\n[here](https://stat.ethz.ch/pipermail/r-sig-finance/2010q2/006077.html).\n\nHere is an example of a left join:\n\n```r\n\u003e # Example 4e. Left Join\n\u003e # https://stat.ethz.ch/pipermail/r-help/2009-April/195882.html\n\u003e #\n\u003e SNP1x \u003c-\n+ structure(list(Animal = c(194073197L, 194073197L, 194073197L, \n+ 194073197L, 194073197L), Marker = structure(1:5, .Label = c(\"P1001\", \n+ \"P1002\", \"P1004\", \"P1005\", \"P1006\", \"P1007\"), class = \"factor\"), \n+     x = c(2L, 1L, 2L, 0L, 2L)), .Names = c(\"Animal\", \"Marker\", \n+ \"x\"), row.names = c(\"3213\", \"1295\", \"915\", \"2833\", \"1487\"), class = \"data.frame\")\n\u003e \n\u003e SNP4 \u003c- \n+ structure(list(Animal = c(194073197L, 194073197L, 194073197L, \n+ 194073197L, 194073197L, 194073197L), Marker = structure(1:6, .Label = c(\"P1001\", \n+ \"P1002\", \"P1004\", \"P1005\", \"P1006\", \"P1007\"), class = \"factor\"), \n+     Y = c(0.021088, 0.021088, 0.021088, 0.021088, 0.021088, 0.021088\n+     )), .Names = c(\"Animal\", \"Marker\", \"Y\"), class = \"data.frame\", row.names = c(\"3213\", \n+ \"1295\", \"915\", \"2833\", \"1487\", \"1885\"))\n\u003e\n\u003e SNP1x\n        Animal Marker x\n3213 194073197  P1001 2\n1295 194073197  P1002 1\n915  194073197  P1004 2\n2833 194073197  P1005 0\n1487 194073197  P1006 2\n\u003e SNP4\n        Animal Marker        Y\n3213 194073197  P1001 0.021088\n1295 194073197  P1002 0.021088\n915  194073197  P1004 0.021088\n2833 194073197  P1005 0.021088\n1487 194073197  P1006 0.021088\n1885 194073197  P1007 0.021088\n\u003e\n\u003e library(sqldf)\n\u003e sqldf(\"select * from SNP4 left join SNP1x using (Animal, Marker)\")\n     Animal Marker        Y  x\n1 194073197  P1001 0.021088  2\n2 194073197  P1002 0.021088  1\n3 194073197  P1004 0.021088  2\n4 194073197  P1005 0.021088  0\n5 194073197  P1006 0.021088  2\n6 194073197  P1007 0.021088 NA\n\u003e # or if that takes up too much memory \n\u003e # create/use/destroy external database\n\u003e sqldf(\"select * from SNP4 left join SNP1x using (Animal, Marker)\", dbname = \"test.db\")\n     Animal Marker        Y  x\n1 194073197  P1001 0.021088  2\n2 194073197  P1002 0.021088  1\n3 194073197  P1004 0.021088  2\n4 194073197  P1005 0.021088  0\n5 194073197  P1006 0.021088  2\n6 194073197  P1007 0.021088 NA\n```\n\n```r\n\u003e # Example 4f.  Another temporal join.\n\u003e # join DF2 to row in DF for which DF.tt and DF2.tt are closest\n\u003e \n\u003e DF \u003c- structure(list(tt = c(3, 6)), .Names = \"tt\", row.names = c(NA, \n+ -2L), class = \"data.frame\")\n\u003e DF\n  tt\n1  3\n2  6\n\u003e \n\u003e DF2 \u003c- structure(list(tt = c(1, 2, 3, 4, 5, 7), d = c(8.3, 10.3, 19, \n+ 16, 15.6, 19.8)), .Names = c(\"tt\", \"d\"), row.names = c(NA, -6L\n+ ), class = \"data.frame\", reference = \"A1.4, p. 270\")\n\u003e DF2\n  tt    d\n1  1  8.3\n2  2 10.3\n3  3 19.0\n4  4 16.0\n5  5 15.6\n6  7 19.8\n\u003e \n\u003e out \u003c- sqldf(\"select * from DF d, DF2 a, DF2 b \n+ where a.row_names = b.row_names - 1 \n+ and d.tt \u003e a.tt and d.tt \u003c= b.tt\", \n+ row.names = TRUE)\n\u003e  \n\u003e out$dd \u003c- with(out, ifelse(tt \u003c (tt.1 + tt.2) / 2, d, d.1))\n\u003e out\n  tt tt.1    d tt.2  d.1   dd\n1  3    2 10.3    3 19.0 19.0\n2  6    5 15.6    7 19.8 19.8\n```\n\nExample 4g. Self Join. There is an example of a self-join here:\n[problem](https://stat.ethz.ch/pipermail/r-help/2010-March/232314.html)\nand answer here:\n\n```r\n\u003e DF \u003c- structure(list(Actor = c(\"Jim\", \"Bob\", \"Bob\", \"Larry\", \"Alice\", \"Tom\", \"Tom\", \"Tom\", \"Alice\", \"Nancy\"), Act = c(\"A\", \"A\", \"C\",                                                                           \"D\", \"C\", \"F\", \"D\", \"A\", \"B\", \"B\")), .Names = c(\"Actor\", \"Act\"                                                                                ), class = \"data.frame\", row.names = c(NA, -10L))\n\n\u003e subset(unique(merge(DF, DF, by = 2)), Actor.x \u003c Actor.y)\n   Act Actor.x Actor.y\n3    A     Jim     Tom\n4    A     Bob     Jim\n6    A     Bob     Tom\n11   B   Alice   Nancy\n16   C   Alice     Bob\n20   D   Larry     Tom\n\n\u003e sqldf(\"select A.Act, A.Actor, B.Actor\n+   from DF A join DF B\n+     where A.Act = B.Act and A.Actor \u003c B.Actor\n+       order by A.Act, A.Actor\")\n  Act Actor Actor\n1   A   Bob   Jim\n2   A   Bob   Tom\n3   A   Jim   Tom\n4   B Alice Nancy\n5   C Alice   Bob\n6   D Larry   Tom\n```\n\nto Raj Morejoys for correction.\n\nHere is an [another example of a self\njoin](https://stat.ethz.ch/pipermail/r-help/2011-February/269680.html)\nto create pairs which is followed by a second self join to produce pairs\nof pairs. This [stackoverflow\nexample](https://stackoverflow.com/questions/11448133/double-merge-two-data-frames-in-r)\nillustrates an sqldf triple join in which one table participates twice.\n\nExample 4h. Join nearby times. There is an example of joining records\nthat are close but not necessarily exactly the same here:\n[problem](https://stat.ethz.ch/pipermail/r-help/2010-March/232588.html)\nand\n[answer](https://stat.ethz.ch/pipermail/r-help/attachments/20100320/4ccb548f/attachment.pl)\n. Also taking successive differences involves joining adjacent times and\nthis is illustrated\n[here](https://stackoverflow.com/questions/6695673/find-standard-deviation-of-first-differences-of-series-defined-with-group-by-usin)\n.\n\nHere is an example where we align time series Sy to series Sx by\naveraging all points of Sy within w = 0.25 units of each Sx time point.\nTx and X are the times and values of Sx and Ty and Y are the times and\nvalues of Sy.\n\n```r\nTx \u003c- seq(1, N, 0.5)\nTx \u003c- Tx + rnorm(length(Tx), 0, 0.1)\nX \u003c- sin(Tx/10.0) +  sin(Tx/5.0) + rnorm(length(Tx), 0, 0.1)\nTy \u003c- seq(1, N, 0.3333)\nTy \u003c- Ty + rnorm(length(Ty), 0, 0.02)\nY \u003c- sin(Ty/10.0) + sin(Ty/5.0) + rnorm(length(Ty), 0, 0.1)\nw \u003c- 0.25\n\nsystem.time(out1 \u003c- sapply(Tx, function(tx) mean(Y[Ty \u003e= tx-w \u0026 Ty \u003c= tx+w])))\n\nlibrary(sqldf)\nSx \u003c- data.frame(Tx, X)\nSy \u003c- data.frame(Ty, Y)\n\nsystem.time(out.sqldf \u003c- sqldf(c(\"create index idx on Sx(Tx)\",\n  \"select Tx, avg(Y) from main.Sx, Sy\n  where Ty + 0.25 \u003e= Tx and Ty - 0.25 \u003c= Tx group by Tx\")))\n\nall.equal(out.sqldf[,2], out1) # TRUE\n```\n\nExample 4i. Speeding up joins with indexes. Here is an example of\nspeeding up a join by using indexes on a single join column\n[here](https://statcompute.wordpress.com/2013/06/09/improve-the-efficiency-in-joining-data-with-index/)\nand [here](https://stat.ethz.ch/pipermail/r-help/2010-March/232688.html)\nand on two join columns below. Note that the `create index` statements\nin each example also has the effect of reading in the data frames into\nthe `main` database of SQLite. The `select` statement refers to\n`main.DF1` rather than just `DF1` so that it accesses that copy of `DF1`\nin `main` which we just indexed rather than the unindexed `DF1` in R.\nSimilar comments apply to `DF2`. The statement\n`sqldf(\"select * from sqlite_master\")` will list the names and related\ninfo for all tables in `main`.\n\n```r\n\u003e set.seed(1)\n\u003e n \u003c- 1000000\n\u003e \n\u003e DF1 \u003c- data.frame(a = sample(n, n, replace = TRUE), \n+ b = sample(4, n, replace = TRUE), c1 = runif(n))\n\u003e \n\u003e DF2 \u003c- data.frame(a = sample(n, n, replace = TRUE), \n+ b = sample(4, n, replace = TRUE), c2 = runif(n))\n\u003e \n\u003e library(sqldf)\nLoading required package: DBI\nLoading required package: RSQLite\nLoading required package: gsubfn\nLoading required package: proto\nLoading required package: chron\n\u003e \n\u003e sqldf()\n\u003cSQLiteConnection:(6480,0)\u003e \n\u003e system.time(sqldf(\"create index ai1 on DF1(a, b)\"))\nLoading required package: tcltk\nLoading Tcl/Tk interface ... done\n   user  system elapsed \n  16.69    0.19   19.12 \n\u003e system.time(sqldf(\"create index ai2 on DF2(a, b)\"))\n   user  system elapsed \n  16.60    0.03   17.48 \n\u003e system.time(sqldf(\"select * from main.DF1 natural join main.DF2\"))\n   user  system elapsed \n   7.76    0.06    8.23 \n\u003e sqldf()\n```\n\nThe sqldf statements above could also be done in one sqldf call like\nthis:\n\n```r\n# define DF1 and DF2 as before\nset.seed(1)\nn \u003c- 1000000\nDF1 \u003c- data.frame(a = sample(n, n, replace = TRUE), \n   b = sample(4, n, replace = TRUE), c1 = runif(n))\nDF2 \u003c- data.frame(a = sample(n, n, replace = TRUE), \n   b = sample(4, n, replace = TRUE), c2 = runif(n))\n\n# combine all sqldf calls from before into one call\n\nresult \u003c- sqldf(c(\"create index ai1 on DF1(a, b)\", \n  \"create index ai2 on DF2(a, b)\", \n  \"select * from main.DF1 natural join main.DF2\"))\n```\n\nNote that if your data is so large that you need indexes it may be too\nlarge to store the database in memory. If you find its overflowing\nmemory then use the `dbname=` sqldf argument, e.g.\n`sqldf(c(\"create...\", \"create...\", \"select...\"), dbname = tempfile())`\nso that it stores the intermediate results in an external database\nrather than memory.\n\n*Note:* The index `ai1` is not actually used so we could have saved the\ntime it took to create it, creating only `ai2`.\n\n```r\nsqldf(c(\"create index ai2 on DF2(a, b)\", \"select * from DF1 natural join main.DF2\"))\n```\n\nExample 4j. Per Group Max and Min\n\nNote that the Date variable gets passed to SQLite as number of days\nsince 1970-01-01 whereas SQLite uses an earlier origin so we add\n`julianday('1970-01-01')` to convert the origin of R's `\"Date\"` class to\nSQLite's origin. Note that the output column called `Date` is\nautomatically converted to `\"Date\"` class by the sqldf heuristic because\nthere is an input column that has the same name.\n\n```r\n\u003e URL \u003c- \"https://ichart.finance.yahoo.com/table.csv?s=GOOG\u0026a=07\u0026b=19\u0026c=2004\u0026d=03\u0026e=16\u0026f=2010\u0026g=d\u0026ignore=.csv\"\n\u003e DF25 \u003c- read.csv(URL, nrows = 25)\n\u003e DF25$Date \u003c- as.Date(DF25$Date)\n\u003e \n\u003e sqldf(\"select Date, a.High, a.Low, b.Close, a.Volume\n+ from (select max(Date) Date, min(Low) Low, max(High) High, sum(Volume) Volume\n+ from DF25 \n+ group by date(Date + julianday('1970-01-01'), 'start of month')\n+ ) as a join DF25 b using(Date)\")\n        Date   High    Low  Close   Volume\n1 2010-03-31 588.28 539.70 567.12 51541600\n2 2010-04-16 597.84 549.63 550.15 41201900\n```\n\nand here is another shorter one that uses a trick of Magnus Hagander in\nthe second Stackoverflow link below:\n\n```r\n\u003e sqldf(\"select \n+ max(Date) Date, \n+ max(High) High, \n+ min(Low) Low, \n+ max(100000 * Date + Close) % 100000 Close,\n+ sum(Volume) Volume\n+ from DF25 \n+ group by date(Date + julianday('1970-01-01'), 'start of month')\")\n        Date   High    Low Close   Volume\n1 2010-03-31 588.28 539.70   567 51541600\n2 2010-04-16 597.84 549.63   550 41201900\n```\n\nAlso see [this Xaprb\nlink](https://www.xaprb.com/blog/2007/03/14/how-to-find-the-max-row-per-group-in-sql-without-subqueries/)\nfor an approach without subqueries and for more discussion see [this\nstackoverflow\nlink](https://stackoverflow.com/questions/121387/sql-fetch-the-row-which-has-the-max-value-for-a-column)\nand [this stackoverflow\nlink](https://stackoverflow.com/questions/1140254/postgresql-vlookup).\nThe last link shows how to use analytical queries which are available in\nPostgreSQL -- the PostgreSQL database, like SQLite and H2, is supported\nby sqldf.\n\nExample 5. Insert Variables[](#Example_5._Insert_Variables)\n-----------------------------------------------------------\n\nHere is an example of inserting evaluated variables into a query using\n[gsubfn](https://code.google.com/p/gsubfn/) quasi-perl-style string\ninterpolation. gsubfn is used by sqldf so its already loaded. Note that\nwe must use the `fn$` prefix to invoke the interpolation functionality:\n\n```r\n\u003e minSL \u003c- 7\n\u003e limit \u003c- 3\n\u003e species \u003c- \"virginica\"\n\u003e fn$sqldf(\"select * from iris where \\\"Sepal.Length\\\" \u003e $minSL and species = '$species' limit $limit\")\n\n  Sepal.Length Sepal.Width Petal.Length Petal.Width   Species\n1          7.1         3.0          5.9         2.1 virginica\n2          7.6         3.0          6.6         2.1 virginica\n3          7.3         2.9          6.3         1.8 virginica\n```\n\nExample 6. File Input[](#Example_6._File_Input)\n-----------------------------------------------\n\nNote that there is a new command `read.csv.sql` which provides an\nalternate interface to the the approach discussed in this section. See\nExample 13 for that.\n\nsqldf normally deletes any database it creates after completion but the\nexample sample code [at the bottom of this\npost](https://stat.ethz.ch/pipermail/r-help/2010-October/257270.html)\nshows how to set up a database and read a file into it without having\nthe database destroyed afterwards.\n\nsqldf will not only look for data frames used in the SQL statement but\nwill also look for R objects of class `\"file\"`. For such objects it will\ndirectly import the associated file into the database without going\nthrough R allowing files that are larger than an R workspace to be\nhandled and also providing for potential speed advantages. That is, if\n`f \u003c- file(\"abc.csv\")` is a file object and `f` is used as the table\nname in the sql statement then the file `abc.csv` is imported into the\ndatabase as table `f`. With SQLite, the actual reading of the file into\nthe database is done in a C routine in RSQLite so the file is\ntransferred directly to the database without going through R. If the\n`sqldf` argument `dbname` is used then it specifies a filename (either\nexisting or created by `sqldf` if not existing). That filename is used\nas a database (rather than memory) allowing larger files than physical\nmemory. By using an appropriate `where` statement or a subset of column\nnames a portion of the table can be retrieved into R even if the file\nitself is too large for R or for memory.\n\nThere are some caveats. The RSQLite `dbWriteTable`/`sqliteImportFile`\nroutines that `sqldf` uses to transfer the file directly to the database\nare intended for speed thus they are not as flexible as `read.table`.\nAlso they have slightly different defaults. The default for `sep` is\n`file.format = list(sep = \",\")`. If the first row of the file has one\nfewer component than subsequent ones then it assumes that\n`file.format = list(header = TRUE, row.names = TRUE)` and otherwise that\n`file.format = list(header = FALSE,  row.names = FALSE)`. `.csv` file\nformat is only partly supported -- quotes are not regarded as special.\n\nIn addition to the examples below there is an example\n[here](http://web.archive.org/web/20140429215324/http://stat.ethz.ch/pipermail/r-help/2009-May/199991.html) and\nanother one with performance results\n[here](http://www.cerebralmastication.com/2009/11/loading-big-data-into-r/).\n\n```r\n\u003e # Example 6a.\n\u003e # test of file connections with sqldf\n\u003e \n\u003e # create test .csv file of just 3 records\n\u003e write.table(head(iris, 3), \"iris3.dat\", sep = \",\", quote = FALSE)\n\u003e \n\u003e # look at contents of iris3.dat\n\u003e readLines(\"iris3.dat\")\n[1] \"Sepal.Length,Sepal.Width,Petal.Length,Petal.Width,Species\"\n[2] \"1,5.1,3.5,1.4,0.2,setosa\"                                 \n[3] \"2,4.9,3,1.4,0.2,setosa\"                                   \n[4] \"3,4.7,3.2,1.3,0.2,setosa\"                                 \n\u003e \n\u003e # set up file connection\n\u003e iris3 \u003c- file(\"iris3.dat\")\n\u003e sqldf('select * from iris3 where \"Sepal.Width\" \u003e 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.7         3.2          1.3         0.2  setosa\n\u003e\n\u003e # Example 6b.\n\u003e # similar but uses disk - useful if file were large\n\u003e # According to https://www.sqlite.org/whentouse.html\n\u003e # SQLite can handle files up to several dozen gigabytes.\n\u003e # (Note in this case readTable and readTableIndex in R.utils\n\u003e # package or read.table from the base of R, setting the colClasses \n\u003e # argument to \"NULL\" for columns you don't want read in, might be\n\u003e # alternatives.)\n\u003e sqldf('select * from iris3 where \"Sepal.Width\" \u003e 3', dbname = tempfile())\n Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.7         3.2          1.3         0.2  setosa\n\n\u003e # Example 6c.\n\u003e # with this format, header=TRUE needs to be specified\n\u003e write.table(head(iris, 3), \"iris3a.dat\", sep = \",\", quote = FALSE, \n+  row.names = FALSE)\n\u003e iris3a \u003c- file(\"iris3a.dat\")\n\u003e sqldf(\"select * from iris3a\", file.format = list(header = TRUE))\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.9         3.0          1.4         0.2  setosa\n3          4.7         3.2          1.3         0.2  setosa\n\n\u003e # Example 6d.\n\u003e # header can alternately be specified as object attribute\n\u003e attr(iris3a, \"file.format\") \u003c- list(header = TRUE)\n\u003e sqldf(\"select * from iris3a\")\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.9         3.0          1.4         0.2  setosa\n3          4.7         3.2          1.3         0.2  setosa\n\n\u003e # Example 6e.\n\u003e # create a test file with all 150 records from iris\n\u003e # and select 4 records at random without reading entire file into R\n\u003e write.table(iris, \"iris150.dat\", sep = \",\", quote = FALSE)\n\u003e iris150 \u003c- file(\"iris150.dat\")\n\u003e sqldf(\"select * from iris150 order by random(*) limit 4\")\n  Sepal.Length Sepal.Width Petal.Length Petal.Width   Species\n1          4.9         2.5          4.5         1.7 virginica\n2          4.8         3.0          1.4         0.1    setosa\n3          6.1         2.6          5.6         1.4 virginica\n4          7.4         2.8          6.1         1.9 virginica\n\u003e\n\u003e # or use read.csv.sql and its just one line\n\u003e read.csv.sql(\"iris150.dat\", sql = \"select * from file order by random(*) limit 4\")\n  Sepal.Length Sepal.Width Petal.Length Petal.Width    Species\n1          4.9         2.4          3.3         1.0 versicolor\n2          5.8         2.7          4.1         1.0 versicolor\n3          7.4         2.8          6.1         1.9  virginica\n4          5.1         3.5          1.4         0.3     setosa\n```\n\nExample 6f. If our file has fixed width fields rather than delimited\nthen we can still handle it if we parse the lines manually with substr:\n\n```r\n# write some test data to \"fixed\"\n# Field 1 has width of 1 column and field 2 has 4 columns\ncat(\"1 8.3\n210.3\n319.0\n416.0\n515.6\n719.8\n\", file = \"fixed\")\n\n# get 3 random records using sqldf\nfixed \u003c- file(\"fixed\")\nattr(fixed, \"file.format\") \u003c- list(sep = \";\") # ; can be any char not in file\nsqldf(\"select substr(V1, 1, 1) f1, substr(V1, 2, 4) f2 from fixed order by random(*) limit 3\")\n```\n\nAnother example of fixed width data is\n[here](https://sites.google.com/site/timriffepersonal/DemogBlog/newformetrickforworkingwithbigishdatainr)\n(however, note that changing the sep needs to be done in the example in\nthat link too).\n\nExample 6g. Defaults.\n\n```r\n# If first row has one fewer columns than subsequent rows then \n# header \u003c- row.names \u003c- TRUE is assumed as in example 6a; otherwise,\n# header \u003c- row.names \u003c- FALSE is assumed as shown here:\n\n\u003e write.table(head(iris, 3), \"iris3nohdr.dat\", col.names = FALSE, row.names = FALSE, sep = \",\", quote = FALSE)\n\u003e readLines(\"iris3nohdr.dat\")\n[1] \"5.1,3.5,1.4,0.2,setosa\" \"4.9,3,1.4,0.2,setosa\"   \"4.7,3.2,1.3,0.2,setosa\"\n\u003e sqldf(\"select * from iris3nohdr\")\n   V1  V2  V3  V4     V5\n1 5.1 3.5 1.4 0.2 setosa\n2 4.9 3.0 1.4 0.2 setosa\n3 4.7 3.2 1.3 0.2 setosa\n```\n\nExample 7. Nested Select[](#Example_7._Nested_Select)\n-----------------------------------------------------\n\nFor each species show the two rows with the largest sepal lengths:\n\n```r\n\u003e # Example 7a.\n\u003e sqldf('select * from iris i \n+   where rowid in \n+    (select rowid from iris where Species = i.Species order by \"Sepal.Length\" desc limit 2)\n+   order by i.Species, i.\"Sepal.Length\" desc')\n\n  Sepal.Length Sepal.Width Petal.Length Petal.Width    Species\n1          5.8         4.0          1.2         0.2     setosa\n2          5.7         4.4          1.5         0.4     setosa\n3          7.0         3.2          4.7         1.4 versicolor\n4          6.9         3.1          4.9         1.5 versicolor\n5          7.9         3.8          6.4         2.0  virginica\n6          7.7         3.8          6.7         2.2  virginica\n```\n\nHere is a similar example. In this one `DF` represents a time series\nwhose values are in column `x` and whose times are dates in column `tt`.\nThe times have gaps -- in fact only every other day is present. The code\nbelow displays the first row at or past the 21st of the month for each\nyear/month. First we append year, month and day columns using\n`month.day.year` from the `chron` package and then do the computation\nusing `sqldf`. (For a version of this using the `zoo` package rather\nthan `sqldf` see:\n[https://stat.ethz.ch/pipermail/r-help/2007-November/145925.html](https://stat.ethz.ch/pipermail/r-help/2007-November/145925.html)).\n\n```r\n\u003e # Example 7b.\n\u003e #\n\u003e library(chron)\n\u003e DF \u003c- data.frame(x = 101:200, tt = as.Date(\"2000-01-01\") + seq(0, len = 100, by = 2))\n\u003e DF \u003c- cbind(DF, month.day.year(unclass(DF$tt)))\n\u003e \n\u003e sqldf(\"select * from DF d\n+   where rowid in \n+    (select rowid from DF \n+       where year = d.year and month = d.month and day \u003e= 21 limit 1)\n+    order by tt\")\n    x         tt    month    day    year\n1 111 2000-01-21        1     21    2000\n2 127 2000-02-22        2     22    2000\n3 141 2000-03-21        3     21    2000\n4 157 2000-04-22        4     22    2000\n5 172 2000-05-22        5     22    2000\n6 187 2000-06-21        6     21    2000\n```\n\nHere is another example of a nested select. We select each row of a for\nwhich st/en overlaps with some st/en of b.\n\n```r\n\u003e # Example 7c.\n\u003e #\n\u003e a \u003c- read.table(textConnection(\"st en\n+ 1 4\n+ 11 14\n+ 3 4\"), header = TRUE)\n\u003e \n\u003e b \u003c- read.table(textConnection(\"st en\n+ 2 5\n+ 3 6\n+ 30 44\"), TRUE)\n\u003e \n\u003e sqldf(\"select * from a where \n+ (select count(*) from b where a.en \u003e= b.st and b.en \u003e= a.st) \u003e 0\")\n  st en\n1  1  4\n2  3  4\n```\n\n7d. Another example of a nested select with sqldf is shown\n[here](https://stat.ethz.ch/pipermail/r-help/2010-March/231975.html)\n\nExample 8. Specifying File Format[](#Example_8._Specifying_File_Format)\n-----------------------------------------------------------------------\n\nWhen using file() as used as in Example 6 RSQLite reads in the first 50\nlines to determine the column classes. What if they all have numbers in\nthem but then later we start to see letters? In that case we will have\nto override its choice. Here are two ways:\n\n```r\nlibrary(sqldf)\n\n# example example 8a - file.format attribute on file.object\n\nnumStr \u003c- as.character(1:100)\nDF \u003c- data.frame(a = c(numStr, \"Hello\"))\nwrite.table(DF, file = \"~/tmp.csv\", quote = FALSE, sep = \",\")\nff \u003c- file(\"~/tmp.csv\")\n\nattr(ff, \"file.format\") \u003c- list(colClasses = c(a = \"character\"))\n\ntail(sqldf(\"select * from ff\"))\n\n\n# example 8b - using file.format argument\n\nnumStr \u003c- as.character(1:100)\nDF \u003c- data.frame(a = c(numStr, \"Hello\"))\nwrite.table(DF, file = \"~/tmp.csv\", quote = FALSE, sep = \",\")\nff \u003c- file(\"~/tmp.csv\")\n\ntail(sqldf(\"select * from ff\",\n file.format = list(colClasses = c(a = \"character\"))))\n```\n\nExample 9. Working with Databases[](#Example_9.__Working_with_Databases)\n------------------------------------------------------------------------\n\nsqldf is usually used to operate on data frames but it can be used to\nstore a table in a database and repeatedly query it in subsequent sqldf\nstatements (although in that case you might be better off just using\nRSQLite or other database directly). There are two ways to do this. In\nthis Example section we show how to do it using the fact that if you\nspecify the database explicitly then it does not delete the database at\nthe end and if you create a table explicitly using create table then it\ndoes not delete the table (however, note that that will result in\nduplicate tables in the database so it will take up twice as much space\nas one table). A second way to do this is to use persistent connections\nas shown in the Example section after this one.\n\n```r\n# create new empty database called mydb\nsqldf(\"attach 'mydb' as new\") \n\n# create a new table, mytab, in the new database\n# Note that sqldf does not delete tables created from create.\nsqldf(\"create table mytab as select * from BOD\", dbname = \"mydb\")\n\n# shows its still there\nsqldf(\"select * from mytab\", dbname = \"mydb\")\n\n# read a file into the mydb data base using read.csv.sql without deleting it\n#\n# 1. First create a test file.\n# 2. Then read it into the mydb database we created using the sqldf(\"attach...\") above.\n#    Since sqldf automatically cleans up after itself we hide \n#    the table creation in an sql statement so table is not deleted.\n# 3. Finally list the table names in the database.\n \nwrite.table(BOD, file = \"~/tmp.csv\", quote = FALSE, sep = \",\")\nread.csv.sql(\"~/tmp.csv\", sql = \"create table mytab as select * from file\", \n  dbname = \"mydb\")\nsqldf(\"select * from sqlite_master\", dbname = \"mydb\")\n```\n\nExample 10. Persistent Connections[](#Example_10._Persistent_Connections)\n-------------------------------------------------------------------------\n\nThese three examples show the use of persistent connections in sqldf.\nThis would be used when one has a large database that one wants to store\nand then make queries from so that one does not have to reload it on\neach execution of sqldf. (Note that if one just needs a series of sql\nstatements ending in a single query an alternative would be just to use\na vector of sql statements in a single sqldf call.)\n\n```r\n\u003e # Example 10a.\n\u003e\n\u003e # create test .csv file of just 3 records (same as example 6)\n\u003e write.table(head(iris, 3), \"iris3.dat\", sep = \",\", quote = FALSE)\n\u003e # set up file connection\n\u003e iris3 \u003c- file(\"iris3.dat\")\n\u003e # creates connection so in memory database persists after sqldf call\n\u003e sqldf() \n\u003cSQLiteConnection:(7384,62)\u003e \n\u003e \n\u003e # uses connection just created\n\u003e sqldf('select * from iris3 where \"Sepal.Width\" \u003e 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.7         3.2          1.3         0.2  setosa\n\u003e # we now have iris3 variable in R workspace and an iris3 table\n\u003e # so ensure sqldf uses the one in the main database by writing\n\u003e # main.iris3.  (Another possibility here would have been to\n\u003e # delete the iris3 variable from the R workspace to avoid the\n\u003e # ambiguity -- in that case one could just write iris3 instead\n\u003e # of main.iris3.)\n\u003e sqldf('select * from main.iris3 where \"Sepal.Width\" = 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          4.9           3          1.4         0.2  setosa\n\u003e \n\u003e # close\n\u003e sqldf()\nNULL\n\n\u003e # Example 10b.\n\u003e #\n\u003e # Here is another way to do example 10a.  We use the same iris3,\n\u003e # iris3.dat and sqldf development version as above.  \n\u003e # We grab connection explicitly, set up the database using sqldf and then \n\u003e # for the second call we call dbGetQuery from RSQLite.  \n\u003e # In that case we don't need to qualify iris3 as main.iris3 since\n\u003e # RSQLite would not understand R variables anyways so there is no \n\u003e # ambiguity.\n\n\u003e con \u003c- sqldf() \n\u003e \n\u003e # uses connection just created\n\u003e sqldf('select * from iris3 where \"Sepal.Width\" \u003e 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          5.1         3.5          1.4         0.2  setosa\n2          4.7         3.2          1.3         0.2  setosa\n\u003e dbGetQuery(con, 'select * from iris3 where \"Sepal.Width\" = 3')\n  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n1          4.9           3          1.4         0.2  setosa\n\u003e \n\u003e # close\n\u003e sqldf()\nNULL\n```\n\nHere is an example of reading a csv file using read.csv.sql and then\nreading it again using a persistent connection:\n\n```r\n# Example 10c.\n\nwrite.table(iris, \"iris.csv\", sep = \",\", quote = FALSE)\n\nsqldf()\nread.csv.sql(\"iris.csv\", sql = \"select count(*) from file\")\n\n# now re-read it from the sqlite database\ndd \u003c- sqldf(\"select * from file\")\n\n# now close the connection and destroy the database\nsqldf()\n```\n\nExample 11. Between and Alternatives[](#Example_11._Between_and_Alternatives)\n-----------------------------------------------------------------------------\n\n```r\n# example thanks to Michael Rehberg\n#\n# build sample dataframes\nseqdf \u003c- data.frame(thetime=seq(100,225,5),thevalue=factor(letters))\nboundsdf \u003c- data.frame(thestart=c(110,160,200),theend=c(130,180,220),groupID=c(555,666,777))\n\n# run the query using two inequalities\ntestquery_1 \u003c- sqldf(\"select seqdf.thetime, seqdf.thevalue, boundsdf.groupID \nfrom seqdf left join boundsdf on (seqdf.thetime \u003c= boundsdf.theend) and (seqdf.thetime \u003e= boundsdf.thestart)\")\n\n# run the same query using 'between...and' clause\ntestquery_2 \u003c- sqldf(\"select seqdf.thetime, seqdf.thevalue, boundsdf.groupID \nfrom seqdf LEFT JOIN boundsdf ON (seqdf.thetime BETWEEN boundsdf.thestart AND boundsdf.theend)\")\n```\n\nExample 12. Combine two files in permanent database[](#Example_12._Combine_two_files_in_permanent_database)\n-----------------------------------------------------------------------------------------------------------\n\nWhen we issue a series of normal `sqldf` statements after each one sqldf\nautomatically removes any tables and databases it creates in that\nstatement; however, it does not know about ones that `sqlite` creates so\na database created using `attach` and the tables created using\n`create table` won't be deleted.\n\nAlso if `sqldf` is used without the `x=` argument (omitting x= denotes\nthe opening of a persistent connection) then objects created in the\ndatabase including those by `sqldf` and `sqlite` are not deleted when\nthe persistent connection is destroyed by the next `sqldf` statement\nwith no `x=` argument.\n\nIf we have forgetten whether you have a connection open or not we can\ncheck either of these:\n\n```r\ndbListConnections(SQLite()) # from DBI\n\ngetOption(\"sqldf.connection\") # set by sqldf\n```\n\nHere is an example that illustrates part of the above. See the prior\nexamples for more.\n\n```r\n\u003e # set up some test data\n\u003e write.table(head(iris, 3), \"irishead.dat\", sep = \",\", quote = FALSE)\n\u003e write.table(tail(iris, 3), \"iristail.dat\", sep = \",\", quote = FALSE)\n\u003e \n\u003e library(sqldf)\n\u003e \n\u003e # create new empty database called mydb\n\u003e sqldf(\"attach 'mydb' as new\") \nNULL\n\u003e \n\u003e irishead \u003c- file(\"irishead.dat\")\n\u003e iristail \u003c- file(\"iristail.dat\")\n\u003e \n\u003e # read tables into mydb\n\u003e sqldf(\"select count(*) from irishead\", dbname = \"mydb\")\n  count(*)\n1        3\n\u003e sqldf(\"select count(*) from iristail\", dbname = \"mydb\")\n  count(*)\n1        3\n\u003e \n\u003e # get count of all records from union\n\u003e sqldf('select count(*) from (select * from main.irishead \n+ union \n+ select * from main.iristail)', dbname = \"mydb\")\n  count(*)\n1        6\n```\n\nExample 13. read.csv.sql and read.csv2.sql[](#Example_13._read.csv.sql_and_read.csv2.sql)\n-----------------------------------------------------------------------------------------\n\n`read.csv.sql` is an interface to `sqldf` that works like `read.csv` in\nR except that it also provides an `sql=` argument and not all of the\nother arguments of `read.csv` are supported. It uses (1) SQLite's import\nfacility via RSQLite to read the input file into a temporary disk-based\nSQLite database which is created on the fly. (2) Then it uses the\nprovided SQL statement to read the table so created into R. As the first\nstep imports the data directly into SQLite without going through R it\ncan handle larger files than R itself can handle as long as the SQL\nstatement filters it to a size that R can handle. Here is Example 6c\nredone using this facility:\n\n```r\n# Example 13a.\nlibrary(sqldf)\n\nwrite.table(iris, \"iris.csv\", sep = \",\", quote = FALSE, row.names = FALSE)\niris.csv \u003c- read.csv.sql(\"iris.csv\", \n    sql = 'select * from file where \"Sepal.Length\" \u003e 5')\n\n# Example 13b.  read.csv2.sql.  Commas are decimals and ; is sep.\n\nlibrary(sqldf)\nLines \u003c- \"Sepal.Length;Sepal.Width;Petal.Length;Petal.Width;Species\n5,1;3,5;1,4;0,2;setosa\n4,9;3;1,4;0,2;setosa\n4,7;3,2;1,3;0,2;setosa\n4,6;3,1;1,5;0,2;setosa\n\"\ncat(Lines, file = \"iris2.csv\")\n\niris.csv2 \u003c- read.csv2.sql(\"iris2.csv\", sql = 'select * from file where \"Sepal.Length\" \u003e 5')\n\n# Example 13c. Use of filter= to process fixed field widths.\n\n# This example assumes gawk is available for use as a filter:\n# https://www.icewalkers.com/Linux/Software/514530/Gawk.html\n# https://gnuwin32.sourceforge.net/packages/gawk.htm\n\nlibrary(sqldf)\ncat(\"112333\n123456\", file = \"fixed.dat\")\ncat('BEGIN { FIELDWIDTHS = \"2 1 3\"; OFS = \",\"; print \"A,B,C\" }\n{ $1 = $1; print }', file = \"fixed.awk\")\n\n# the following worked on Windows Vista.  One user told me that it only worked if he\n# omitted the eol= argument so try it both ways on your system and use the way that\n# works for your system.\n\nfixed \u003c- read.csv.sql(\"fixed.dat\", eol = \"\\n\", filter = \"gawk -f fixed.awk\")\n\n# Example 13d.  Read a csv file into the database but do not drop the database or table\n\n# create test file\nwrite.table(iris, \"iris.csv\", sep = \",\", quote = FALSE, row.names = FALSE)\n\n# create an empty database (can skip this step if database already exists)\nsqldf(\"attach mytestdb as new\")\n\n# read into table called iris in the mytestdb sqlite database\nread.csv.sql(\"iris.csv\", sql = \"create table main.iris as select * from file\", dbname = \"mytestdb\")\n\n# look at first three lines\nsqldf(\"select * from main.iris limit 3\", dbname = \"mytestdb\")\n\n# example 13e.  Read in only column j of a csv file where j may vary.\n\nlibrary(sqldf)\n\n# create test data file\nnms \u003c- names(anscombe)\nwrite.table(anscombe, \"anscombe.dat\", sep = \",\", quote = FALSE, \n    row.names = FALSE)\n\nj \u003c- 2\nDF2 \u003c- fn$read.csv.sql(\"anscombe.dat\", sql = \"select `nms[j]` from file\")\n```\n\nAlso see this\n[example](https://stat.ethz.ch/pipermail/r-help/2010-November/260931.html)\nand this further\n[example](https://stackove","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fggrothendieck%2Fsqldf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fggrothendieck%2Fsqldf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fggrothendieck%2Fsqldf/lists"}