{"id":19768673,"url":"https://github.com/siara-cc/esp32_arduino_sqlite3_lib","last_synced_at":"2025-04-05T15:02:57.219Z","repository":{"id":37789880,"uuid":"152535482","full_name":"siara-cc/esp32_arduino_sqlite3_lib","owner":"siara-cc","description":"Sqlite3 Arduino library for ESP32","archived":false,"fork":false,"pushed_at":"2024-06-12T08:46:07.000Z","size":47401,"stargazers_count":394,"open_issues_count":49,"forks_count":72,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-03-29T14:03:46.715Z","etag":null,"topics":["arduino-library","database","esp32","esp32-ardunio","sqlite","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/siara-cc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-10-11T05:25:00.000Z","updated_at":"2025-03-15T22:29:10.000Z","dependencies_parsed_at":"2024-02-14T16:35:50.871Z","dependency_job_id":null,"html_url":"https://github.com/siara-cc/esp32_arduino_sqlite3_lib","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siara-cc%2Fesp32_arduino_sqlite3_lib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siara-cc%2Fesp32_arduino_sqlite3_lib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siara-cc%2Fesp32_arduino_sqlite3_lib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siara-cc%2Fesp32_arduino_sqlite3_lib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/siara-cc","download_url":"https://codeload.github.com/siara-cc/esp32_arduino_sqlite3_lib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247353729,"owners_count":20925329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arduino-library","database","esp32","esp32-ardunio","sqlite","sqlite3"],"created_at":"2024-11-12T04:39:44.927Z","updated_at":"2025-04-05T15:02:57.191Z","avatar_url":"https://github.com/siara-cc.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sqlite3 Arduino library for ESP32\n\nNote: This is a general purpose library based on the Sqlite codebase.  For logging sensor data into database please use [Sqlite Micro Logger](https://github.com/siara-cc/sqlite_micro_logger_arduino), which is faster and memory efficient.\n\nThis library enables access to SQLite database files from SPIFFS or SD Cards through ESP32 SoC.  Given below is a picture of a board that has a ready-made Micro SD slot (using SDMMC 4 bit mode - see example sqlite3_sdmmc):\n\n![](ESP_WROOM_32_breakout.png?raw=true)\n\nAlso shown below is the wiring between ESP-WROOM-32 breakout board and Micro SD Shield (using SPI mode - see example sqlite3_sdspi):\n\n![](ESP32_MSD_Shield_Wiring.jpg?raw=true)\n\n## Why Sqlite on ESP32 is exciting?\n\n[Sqlite3](http://sqlite.org) is the favourite database of all that is portable and widely used.  Availability on ESP32 platform makes it even more portable.  Sqlite can handle terrabyte sized data, ACID compliant and guaranteed to be stable.\n\nSo far IoT systems could offer SD Card access, which enabled gigabyte size files accessible, but was not fully utilized because the libraries that were available so far (such as [Arduino Extended Database Library](https://github.com/jwhiddon/EDB) or [SimpleDB](http://www.kendziorra.nl/arduino/103-simpledb-simple-flexible-and-smal)) offered only record number based or linear search and are terribly slow for key based indexed search.\n\nSqlite stores data in [B+Tree pages](https://en.wikipedia.org/wiki/B%2B_tree) and so can locate data from millions of records using variable length keys without exerting stress on CPU or RAM.  This is demonstrated using the sample databases provided in the example sections.\n\nEven with the 500 odd kilbytes RAM available on ESP32, millions of records and gigabyte sized databases can be accessed.\n\n## Usage\n\nSqlite3 C API such as `sqlite3_open` can be directly invoked. Before calling please invoke:\n\n```c++\n   SD_MMC.begin(); // for Cards attached to the High speed 4-bit port \n   SPI.begin(); SD.begin(); // for Cards attached to the SPI bus\n   SPIFFS.begin(); // For SPIFFS\n```\nas appropriate.\n\nThe ESP32 Arduino library has an excellent VFS layer.  Even multiple cards can be supported on the SPI bus by specifying the pin number and mount point using the `begin()` method.\n\nThe default mount points are:\n```c++\n   '/sdcard' // for SD_MMC \n   '/sd' // for SD on SPI\n   '/spiffs' // For SPIFFS\n```\n\nand the filenames are to be prefixed with these paths in the `sqlite3_open()` function (such as `sqlite3_open(\"/spiffs/my.db\")`).\n\nPlease see the examples for full illustration of usage for the different file systems. The sample databases given (under `examples/sqlite3_sdmmc/data` folder) need to be copied to the Micro SD card root folder before the SD examples can be used.  Please see the comments section of the example.\n\n## Wiring\n\nWhile there is no wiring needed for SPIFFS, for attaching cards to SPI bus, please use the following connections:\n\n```c++\n * SD Card    |  ESP32\n *  DAT2 (1)      -\n *  DAT3 (2)      SS (D5)\n *  CMD  (3)      MOSI (D23)\n *  VDD  (4)      3.3V\n *  CLK  (5)      SCK (D19)\n *  VSS  (6)      GND\n *  DAT0 (7)      MISO (D18)\n *  DAT1 (8)      -\n```\n\nAnd for SD card attached to High-speed 4-bit SD_MMC port, use:\n\n```c++\n * SD Card    |  ESP32\n *  DAT2 (1)      D12\n *  DAT3 (2)      D13\n *  CMD  (3)      D15\n *  VDD  (4)      3.3V\n *  CLK  (5)      D14\n *  VSS  (6)      GND\n *  DAT0 (7)      D2\n *  DAT1 (8)      D4\n```\n\nIf you are using a board such as shown in the picture above, this wiring is ready-made.\n\n## Installation\n\nPlease download this library, unzip it to the libraries folder of your ESP32 sdk location. The location varies according to your OS.  For example, it is usually found in the following locations:\n```\nWindows: C:\\Users\\(username)\\AppData\\Roaming\\Arduino15\nLinux: /home/\u003cusername\u003e/.arduino15\nMacOS: /home/\u003cusername\u003e/Library/Arduino15\n```\nUnder Arduino15 folder please navigate to `packages/esp32/hardware/esp32/\u003cversion\u003e/libraries`\n\nIf you do not have the ESP32 sdk for Arduino, please see https://github.com/espressif/arduino-esp32 for installing it.\n\n## Dependencies / pre-requisites\n\nNo dependencies except for the Arduino and ESP32 core SDK. The Sqlite3 code is included with the library.\n\n## Limitations on ESP32\n\n* No serious limitations, except its a bit slow on large datasets. It takes around 700 ms to retrieve from a dataset containing 10 million rows, even using the index.\n\n## Limitations of this library\n\n* Locking is not implemented.  So it cannot be reliably used in a multi-threaded / multi-core code set, except for read-only operations.\n\n## Limitations of Flash memory\n\nAny Flash memory such as those available on SPIFFS or Micro SD cards have limitation on number of writes / erase per sector.  Usually the limitation is 10000 writes or 100000 writes (on the same sector).  Although ESP32 supports wear-levelling,  this is to be kept in mind before venturing into write-intensive database projects.  There is no limitation on reading from Flash.\n\n## Compression with Shox96\n\n(Shox96 is deprecated and Unishox explained below will be supported in future).\n\nThis implementation of `sqlite3` includes two functions `shox96_0_2c()` and `shox96_0_2d()` for compressing and decompressing text data.\n\nShox96 is a compression technique developed for reducing storage size of Short Strings. Details of how it works can be found [here](https://github.com/siara-cc/Shox96).\n\nAs of now it can work on only strings made of 'A to Z', 'a to z', '0-9', Special Characters such as \u0026*() etc. found on keyboard, CR, LF, TAB and Space.\n\nIn general it can achieve upto 40% size reduction for Short Strings.\n\n### Usage\n\nThe following set of commands demonstrate how compression can be accomplished:\n\n```sql\ncreate table test (b1 blob);\ninsert into test values (shox96_0_2c('Hello World'));\ninsert into test values (shox96_0_2c('Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry''s standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.'));\nselect txt, length(txt) txt_len from (select shox96_0_2d(b1) txt from test);\nselect length(b1) compressed_len from test;\n```\n\nSee screenshots section for output.\n\n### Limitations (for Shox96)\n\n- Trying to decompress any blob that was not compressed using `shox96_0_2c()` will crash the program.\n- It does not work if the string has binary characters. that is, other than ASCII 32 to 126, CR, LF and Tab.\n- Dictionary based compression / decompression is not yet implemented.\n\n## Compression with Unishox\n\nThis implementation also includes two functions `unishox1c()` and `unishox1d()` for compressing and decompressing text data.\n\nUnishox is a compression technique developed for reducing storage size of Short Unicode Strings. Details of how it works can be found [here](https://github.com/siara-cc/Unishox).\n\nIn general it can achieve upto 40% size reduction for Short Strings.\n\n### Usage\n\nThe usage is similar to that of Shox96, only in this case UTF-8 strings can be used.\n\nSee screenshots section for output.\n\n### Limitations (for Unishox)\n\n- Trying to decompress any blob that was not compressed using `unishox1c()` will crash the program.\n\n## Acknowledgements\n\n* This library was developed based on NodeMCU module developed by [Luiz Felipe Silva](https://github.com/luizfeliperj). The documentation can be found [here](https://nodemcu.readthedocs.io/en/master/en/modules/sqlite3/).\n* The census2000 and baby names databases were taken from here: http://2016.padjo.org/tutorials/sqlite-data-starterpacks/. But no license information is available.\n* The mdr512.db (Million Domain Rank database) was created with data from [The Majestic Million](https://majestic.com/reports/majestic-million) and is provided under CC 3.0 Attribution license.\n* The [ESP32 core for Arduino](https://github.com/espressif/arduino-esp32)\n* [The Arduino platform](https://arduino.cc)\n\n## Screenshots\n\n### Output of Micro SD example\n\n![](output_screenshot.png?raw=true)\n\n### Output of SD Card database query through WebServer example\n\n![](output_web_1.png?raw=true)\n![](output_web_2.png?raw=true)\n\n### SQLite console\n\n![](console_screenshot.png?raw=true)\n\n### Unishox compression\n\n![](output_web_console.png?raw=true)\n\n### Output of Querying StackOverflow DB through WebServer example:\n\n![](output_web_so.png?raw=true)\n![](output_web_so_id.png?raw=true)\n![](output_web_so_name.png?raw=true)\n![](output_web_so_loc.png?raw=true)\n\n## Issues\n\nPlease contact the author or create issue here if you face problems.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiara-cc%2Fesp32_arduino_sqlite3_lib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsiara-cc%2Fesp32_arduino_sqlite3_lib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiara-cc%2Fesp32_arduino_sqlite3_lib/lists"}