{"id":28972742,"url":"https://github.com/alvis/entrystore","last_synced_at":"2026-04-15T19:38:25.139Z","repository":{"id":105245171,"uuid":"229709831","full_name":"alvis/entrystore","owner":"alvis","description":"[WIP] 🔌 A universal cross-backend entry store interface for CSV, JSON, PostgreSQL, Parquet and more.","archived":false,"fork":false,"pushed_at":"2021-01-10T11:43:18.000Z","size":1626,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-24T11:08:29.147Z","etag":null,"topics":["csv","json","mysql","postgres","sql","sqlite","universal"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alvis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-12-23T08:28:43.000Z","updated_at":"2023-03-04T04:08:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"a71a651e-c885-459c-b2f3-7ca8d33e2982","html_url":"https://github.com/alvis/entrystore","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/alvis/entrystore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvis%2Fentrystore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvis%2Fentrystore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvis%2Fentrystore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvis%2Fentrystore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alvis","download_url":"https://codeload.github.com/alvis/entrystore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvis%2Fentrystore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31857614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","json","mysql","postgres","sql","sqlite","universal"],"created_at":"2025-06-24T11:08:27.142Z","updated_at":"2026-04-15T19:38:25.120Z","avatar_url":"https://github.com/alvis.png","language":"TypeScript","readme":"# ![Logo](logo.svg)\n\n\u003cdiv align=\"center\"\u003e\n\n_A universal simple interface for storing data on different storage options_\n\n•   [API](#api)   •   [About](#about)   •\n\n\u003c/div\u003e\n\n#### Highlights\n\n**One unified interface**, entrystore allows you to manage data with\n\n- **multiple data types support** such as CSV and SQLite\n- **flexible storage** such as local disk\n\n## TODO\n\n- how to handle multiple entry types?\n\n## Motivation\n\nEvery data backend has its advantages and disadvantages.\nDepending of the size of dataset and usage scenario, some are more preferable then others.\nFor example, CSV is good for a small dataset or when data exchangeability is needed.\nBut, data size is not static. It often changes overtime, so does your usage scenario.\n\nGreatest pain is what often an experienced engineer would face when data have to be migrated from one backend to another backend.\nThe pain it beyond just the transfer of data from one format to another format,\nbut also when every single line of code which touches the storage library.\nImagine you have to change every CSV usage to SQL...\n\nWith entrystore, the pain is minimal as you only need to change the backend configuration and that's it.\nThe interface for data manipulation is the same regardless the underlying backend.\nThe design philosophy of entrystore is similar to the famous data analysis and manipulation library [pandas](https://pandas.pydata.org) in python.\nWe handle all the underlying challenge on dealing with different backends while the user only needs to deal with the business logic.\n\n## Usage\n\n```ts\n/** data structure */\n@GenericEntry({ index: 'timestamp' })\nclass TimePoint extends GenericEntry\u003c'timestamp'\u003e {\n  @Field({ type: Date })\n  /** index key */\n  timestamp: Date;\n  @Field({ type: String })\n  /** value of the entry */\n  value: string;\n}\n\n// create a CSV store with a local device as the storage backend\nconst store = new EntryStore({\n  backend: new CSV({\n    // the index key of each index\n    indexKey: 'timestamp',\n    // where the data will be stored\n    destination: new LocalDestination({\n      // root directory where\n      path: '/path/to/output',\n    }),\n    //\n    partitioner: new YearMonthPartitioner({\n      adapter: (index: string): Date =\u003e new Date(index),\n    }),\n  }),\n  prototypes: [Entry],\n});\n\n// saving an entry\nawait store.put(TimePoint, ...points);\n\n// getting an entry\nconst entry = await store.get(TimePoint, 'timestamp');\n\n// getting the first, last etc.\nconst firstRecordTime = await store.getFirstKey(TimePoint);\n\n// selective get\nconst timeseries = await store\n  .select(Timepoint)\n  .filter('timestamp', 'between', 'time1', 'time2');\n  .filter('value', '\u003e', 0)\n\n```\n\n```ts\n/** data structure */\nclass Entry {\n  /** index key */\n  timestamp: Date;\n  /** value of the entry */\n  value: string;\n}\n\n// create a CSV store with a local device as the storage backend\nconst store = new CSVStore\u003cEntry, 'timestamp'\u003e({\n  // the index key of each index\n  indexKey: 'timestamp',\n  // where the data will be stored\n  destination: new LocalDestination({\n    // root directory where\n    path: '/path/to/output',\n  }),\n  //\n  partitioner: new YearMonthPartitioner({\n    adapter: (index: string): Date =\u003e new Date(index),\n  }),\n});\n\n// get a single entry\nconst key = await store.firstKey;\nconst entry: Entry = await store.get(key);\n\n// get a range of entries\nconst entries: Entry[] = await store.select({\n  from: new Date('2000-01-01T00:00:00z'),\n  to: new Date('2000-12-31T23:59:59z'),\n});\n\n// submit a single entry\nawait store.put(entry);\n\n// submit entries by chunk\nawait store.put(...entries);\n```\n\n## Schema\n\n#### Supported Data Type\n\nA data type is identified with the following\n\n- `Boolean` for any boolean data, i.e. `true` or `false`\n- `Number` for any numberic data, e.g. `1`, `1.1`\n- `String`\n- `Date`\n- `URL`\n- `Embedded` for any nested data, e.g. `{\"nested\": true}`\n\n**Note** There is no further type definition for any embdedded data.\nIt's equalvant to the `Record\u003cstring, unknown\u003e` type in typescript.\n\n#### Index\n\nAny index is decorated with a `*` in front of a type identifier. e.g. `*Number`\n\n#### Array Type\n\nAny array data must be in a uniform type. Its schema identifier is in the form of `[\u003cTypeIdentifier\u003e]`. e.g. `[String]`\n\n#### Nullable Type\n\nAny non-index type can be nullable. In such case, its schema identifier is decorated with a `?` suffix. e.g. `Number?`\n\n## API\n\n### Class: CSVStore extends EntryStore\n\n#### Constructor\n\n#### Methods\n\nAll public methods are the same as those in EntryStore. [See below.](#abstract-class-entrystore)\n\n### Class: SQLiteStore extends EntryStore\n\n#### Constructor\n\n▸ **new SQLiteStore(message: string, options?: XceptionOptions): SQLiteStore**\n\nCreate an EntryStore using SQLite as the backend.\n\n#### Methods\n\nAll public methods are the same as those in EntryStore. [See below.](#abstract-class-entrystore)\n\n### Abstract Class: EntryStore\n\n▸ **new EntryStore(message: string, options?: XceptionOptions): Xception**\n\nCreate a custom error with a message and additional information to be embedded to the error.\n\n#### Constructor\n\n▸ **new EntryStore(message: string, options?: XceptionOptions): Xception**\n\nCreate a custom error with a message and additional information to be embedded to the error.\n\n| Parameter            | Type     | Description                                                                                          |\n| -------------------- | -------- | ---------------------------------------------------------------------------------------------------- |\n| `message`            | string   | an error message for the error class                                                                 |\n| `options.cause?`     | unknown  | an upstream error to be embedded to the new repacked error _(default: **`undefined`**)_              |\n| `options.namespace?` | string   | an identifier of the component where the error occur _(default: **`undefined`**)_                    |\n| `options.meta?`      | Object   | any context data you would like to embed to the error _(default: **`{}`**)_                          |\n| `options.tags?`      | string[] | some associations of the error (e.g. user error) for selective logging purpose _(default: **`[]`**)_ |\n\n#### Methods\n\n▸ **get(key: Index): Promise\u003cEntry | undefined\u003e**\n\n**Returns** a single entry that matches the key.\n\nGet an entry by its key.\n\n| Parameter | Type   | Description |\n| --------- | ------ | ----------- |\n| `key`     | string | entry key   |\n\n▸ **put(...entries: Entry[]): Promise\u003cvoid\u003e**\n\n**Returns** when the operation is finished.\n\nGet an entry by its key.\n\n| Parameter    | Type    | Description                                      |\n| ------------ | ------- | ------------------------------------------------ |\n| `...entries` | Entry[] | a list of entries to be put into the entry store |\n\n#### Properties\n\n| Name        | Type              | Description                  |\n| ----------- | ----------------- | ---------------------------- |\n| `fields`    | Promise\u003cstring[]\u003e | list of fields in each entry |\n| `first?`    | Promise\u003cEntry\u003e    | first entry                  |\n| `firstKey?` | Promise\u003cIndex\u003e    | first entry key              |\n| `last?`     | Promise\u003cEntry\u003e    | last entry                   |\n| `lastKey?`  | Promise\u003cIndex\u003e    | last entry key               |\n\n## Backend Comparison\n\n#### CSVStore\n\nPROS:\n\n- It can be read by most libraries.\n\nCONS\n\n- Memory consumption could be huge if the incoming data is not all after the last entry.\n  In this case, the store has to read the whole CSV partition and reorder the entries and write.\n- No typing. There is no way to distinguish a number and a string.\n\n## About\n\nIn a certain level, entrystore is similar to many [ORMs](https://en.wikipedia.org/wiki/Object–relational_mapping) libraries.\n\nThe main differentiation here is that entrystore does NOT ONLY support SQL backends, but also CSV, JSON, parquet and more.\nYou can even write your own backend adapter if none of the bundled adapters suits.\n\nThe aim of this library is to provide a way to access the benefits provided by each backend without the need to relearn the syntax to handle the backend.\n\n### Comparison\n\n|           |     EntryStore     |      TypeORM       |     Sequelize      |     Bookshelf      |\n| --------- | :----------------: | :----------------: | :----------------: | :----------------: |\n| SQLite    | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| MySQL     |   :construction:   | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| PostSQL   |   :construction:   | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| Parquet   | :white_check_mark: |        :x:         |        :x:         |        :x:         |\n| CSV       | :white_check_mark: |        :x:         |        :x:         |        :x:         |\n| JSON      | :white_check_mark: |        :x:         |        :x:         |        :x:         |\n| **Other** |     :computer:     |        :x:         |        :x:         |        :x:         |\n\n_**PRO TIP**_: If your backend is not on the list above, you can extend entrystore with your own adapter following the required [backend interface](#backend-interface) above.\n\n### Related Projects\n\nThere is no need to waste time on converting different SQL formats if you happen to change from MySQL to Post\n\n- [sequelize](https://github.com/sequelize/sequelize): maintain multi SQL dialects with ease with this popular ORM.\n- [typeorm](https://github.com/typeorm/typeorm): use typescript metadata to maintain multi SQL dialects.\n- [data-forge](https://github.com/data-forge/data-forge-ts): do data transformation similar to pandas and LINQ.\n\n### License\n\nCopyright © 2020, [Alvis Tang](https://github.com/alvis). Released under the [MIT License](LICENSE).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvis%2Fentrystore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falvis%2Fentrystore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvis%2Fentrystore/lists"}