{"id":22485421,"url":"https://github.com/datopian/metastore-lib-js","last_synced_at":"2025-07-02T11:06:39.378Z","repository":{"id":47637322,"uuid":"326649210","full_name":"datopian/metastore-lib-js","owner":"datopian","description":"metastore-lib-js is a JavaScript library for abstracting metadata storage for datapackage.json packages","archived":false,"fork":false,"pushed_at":"2021-08-20T09:03:41.000Z","size":384,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-05-01T11:32:48.724Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datopian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-04T10:35:45.000Z","updated_at":"2021-08-23T15:50:17.000Z","dependencies_parsed_at":"2022-09-06T15:21:12.676Z","dependency_job_id":null,"html_url":"https://github.com/datopian/metastore-lib-js","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fmetastore-lib-js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fmetastore-lib-js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fmetastore-lib-js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fmetastore-lib-js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datopian","download_url":"https://codeload.github.com/datopian/metastore-lib-js/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228499952,"owners_count":17929988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T17:12:34.022Z","updated_at":"2024-12-06T17:14:26.999Z","avatar_url":"https://github.com/datopian.png","language":"JavaScript","readme":"# metastore-lib-js\nLibrary for storing (dataset) metadata, with versioning support and pluggable backends including GitHub. Original written in Python ([See Python version](https://github.com/datopian/metastore-lib)). Originally designed for datasets it can be used for storing any kind of metadata. Versioning (revisioning) support is built-in e.g. you can do: fetch(objectId) =\u003e metadata at that revision of the object\n\n### Installation\n```\nnpm install metastore-lib-js\n```\n\n### Requirements\nNode version 12 and above\n\n## Usage\nThe metastore-lib-js saves metadata (datapackage.json), to a selected backend. There are current;y two supported backend-Github and FileSystem. \n### Using Github Backend\nThe Github backend when selected, will save metadata to a specified github organisation. To use a Github backend as your storage, you need to be obtain a [personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token) from github. After obtaining a token, you can initialize a basic metastore storage as shown below:\n\n```javascript\nconst metastore = require(\"metastore-lib-js\");\n\nconst config = {\n  token: \"personal_access_token\",\n  org: \"name_of_organisation_on_github\",\n  defaultAuthor: { name: \"John Doe\", email: \"john@mail.com\" },\n  lfsServerUrl: \"https://some-lfs-server\"\n};\n\nconst storage = metastore.createMetastore(\"github\", config);\n...\n```\n\nAll supported config options are:\n* token: (Required), personal access token for authenticating storage to Github\n* org: (Required), Name of the organisation you want to save metadata to. \n* defaultAuthor: (Required), object of name, email pair. Details of  author making changes/commits\n* defaultCommitMessage: (Optional), short github commit message when saving metadata\n* lfsServerUrl: (Optional), The base URL of the Git-LFS server in use. Providing this will make the GitHub backend create Git LFS configuration and pointer files for resources where applicable\n\nAfter initialization, the storage object can be used to create, fetch and update repositories as shown following sections:\n\n### Creating a metadata\nTo create a metadata on github, you need to pass some metadata for the repository. Supported parameters are:\n\n* objectId: (Required), unique name of the new repo on github\n* metadata: (Required), metadata object to save, can also be called datapackage.json\n* message: (Optional), commit message for github\n* description: (Optional), short description of the repository\n* author: (Required), author saving/commiting the file\n\n```javascript\n\nconst objectId = \"New-Financial-2020\"; \nconst metadata = { \n  name: \"Test Financial File\",\n  resources: [\n      { path: \"data/myresource.csv\",\n        type: \"text/csv\" }\n     ],\n  description: \"A financial dataset\",\n};\n\nconst author = { name: \"Rising Odegua\", email: \"rising@datopian.com\" } \nconst description = \"This is a financial file\"; \n\nasync function run() {\n  let objectInfo = await storage.create({\n    objectId,\n    metadata,\n    author,\n    description,\n  });\n  console.log(objectInfo);\n}\nrun();\n\n```\n### Updating a metadata\nTo update an existing metadata on github, you need to pass some metadata for the repository. Supported parameters are:\n\n* objectId: (Required), unique name of an existing repo on github\n* metadata: (Optional), metadata object to save, can also be called datapackage.json\n* message: (Optional), commit message for github\n* description: (Optional), short description of the repository\n* author: (Required), author saving/commiting the file\n* branch: (Optional), one of `main` or `master`. Defaults to `main`. Only specify `master` for old repos created before the Github master term removal (See Gotchas section).\n\n```javascript\n\nconst objectId = \"New-Financial-2020\"; \nconst metadata = { \n  name: \"Updated Financial File\",\n  description: \"A financial dataset about Paris\",\n}\nconst readMe = `## An updated Readme`\n\nasync function run() {\n  let objectInfo = await storage.update({\n    objectId,\n    metadata,\n    readMe\n  });\n  console.log(objectInfo);\n}\nrun();\n\n```\n\n### Fetching a Metadata\nTo fetch or retrieve an existing metadata on Github, you can pass the objectId/repository name to the fetch method as shown below:\n\nAll suported parameters:\n* objectId: (Required), unique name of an existing repo on github\n* branch: (Defaults to `main`), can be one of `master` or `main` \n```javascript\nasync function run() {\n  const objectInfo = await storage.fetch(objectId);\n  console.log(objectInfo);\n}\n```\n\n### GOTCHAS\n* __Which github branch to use?__ : \nDue to the recent [removal](https://www.zdnet.com/article/github-to-replace-master-with-alternative-term-to-avoid-slavery-references/) of the \"master\" term on github, some repo may be retrieved under the \"master\" or \"main\" branch. Take note of this when calling functions like fetch or update. \n\n### Using FileSystem Backend\nThe File System backend when selected, will save metadata to a specified local folder on your computer called `tmp`. The filesystem should only be used for testing purpose and never in production. \n\n\u003e Before initializing a File System, ensure you have a existing folder called `tmp` in your current working directory. \n\n```javascript\nconst metastore = require(\"metastore-lib-js\");\n\nlet storage = metastore.createMetastore(\"file\");\n...\n```\n\nAfter initialization, the storage object can be used to create, fetch and update repositories as shown following sections:\n\n### Creating a metadata\nTo create a metadata on a FileSystem, you need to pass some metadata for the repository. Supported parameters are:\n\n* objectId: (Required), unique name of the new repo on github\n* metadata: (Required), metadata object to save, can also be called datapackage.json\n* author: (Required), author saving/commiting the file\n* description: (Optional), short description of the repository\n\n```javascript\n\nlet objectId = \"New-Financial-2020\";\nlet metadata = {\n  name: \"Test Financial File\",\n  resources: [{ path: \"data/myresource.csv\", mimetype: \"text/csv\" }],\n  description: \"I updated the file\",\n};\nlet author = { name: \"Rising Odegua\", email: \"rising@datopian.com\" };\nlet description = \"This is a financial file\";\n\nasync function run() {\n  let objectInfo = await storage.create(\n    objectId,\n    metadata,\n    author,\n    description,\n  );\n  console.log(objectInfo);\n}\nrun();\n\n//outputs\n{\n  objectId: 'New-Financial-2020',\n  revisionId: '311ab4aa-7309-4f38-b0c5-811a6fc7ace1',\n  created: 2021-01-11T11:37:24.358Z,\n  author: { name: 'Rising Odegua', email: 'rising@datopian.com' },\n  description: 'This is a financial file',\n  metadata: {\n    name: 'Test Financial File',\n    resources: [ [Object] ],\n    description: 'I updated the file',\n  }\n}\n```\n### Updating a metadata\nTo update an existing metadata on github, you need to pass some metadata for the repository. Supported parameters are:\n\n* objectId: (Required), unique name of an existing repo on github\n* metadata: (Optional), metadata object to save, can also be called datapackage.json\n* description: (Optional), short description of the repository\n* author: (Required), author saving/commiting the file\n\n\n```javascript\n\nlet objectId = \"New-Financial-2020\";\nlet metadata = {\n  name: \"Financial File\",\n  resources: [{ path: \"data/myresource.csv\", mimetype: \"text/csv\" }],\n  description: \"I just updated this file in filesystem\",\n};\nlet author = { name: \"Rising Odegua\", email: \"rising@datopian.com\" };\nlet description = \"This is a an updated file\";\n\nasync function run() {\n  let objectInfo = await storage.update(\n    objectId,\n    metadata,\n    author,\n    description,\n  );\n  console.log(objectInfo);\n}\nrun();\n\n//output\n{\n  objectId: 'New-Financial-2020',\n  revisionId: '0ab7049e-1ab5-4281-968c-3973a7a1c0de',\n  created: 2021-01-11T11:42:53.178Z,\n  author: { name: 'Rising Odegua', email: 'rising@datopian.com' },\n  description: '',\n  metadata: {\n    name: 'Financial File',\n    resources: [ [Object] ],\n    description: 'I just updated this file in filesystem',\n  }\n}\n\n```\n\n### Fetching a Metadata\nTo fetch or retrieve an existing metadata in FileSystem, you can pass the objectId/repository name to the fetch method as shown below:\n\nAll suported parameters:\n* objectId: (Required), unique name of an existing repo on github\n\n```javascript\nasync function run() {\n  let objectInfo = await storage.fetch(objectId);\n  console.log(objectInfo);\n}\n```\n\n## Developers\n\n### Setup\n1. Clone the repo:\n```\ngit clone https://github.com/datopian/metastore-lib-js.git\n```\n\n2. Install dependencies: \n```\nyarn\n```\n\n3. Run tests: \n```\nyarn test\n```\n4. Do some dev work\n\nOnce done, make sure tests are passing. Then build distribution version of the app:\n```\n yarn build\n```\n","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatopian%2Fmetastore-lib-js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatopian%2Fmetastore-lib-js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatopian%2Fmetastore-lib-js/lists"}