https://github.com/hviana/deno_kv_fs
Deno KV file system, compatible with Deno deploy. Saves files in 64kb chunks. You can organize files into directories. You can control the KB/s rate for saving and reading files, rate limit, user space limit and limit concurrent operations, useful for controlling uploads/downloads. Makes use of Web Streams API.
https://github.com/hviana/deno_kv_fs
deno filesystem
Last synced: 2 months ago
JSON representation
Deno KV file system, compatible with Deno deploy. Saves files in 64kb chunks. You can organize files into directories. You can control the KB/s rate for saving and reading files, rate limit, user space limit and limit concurrent operations, useful for controlling uploads/downloads. Makes use of Web Streams API.
- Host: GitHub
- URL: https://github.com/hviana/deno_kv_fs
- Owner: hviana
- License: mit
- Created: 2024-09-25T11:50:35.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-10-31T18:04:31.000Z (7 months ago)
- Last Synced: 2025-03-18T03:51:31.144Z (2 months ago)
- Topics: deno, filesystem
- Language: TypeScript
- Homepage:
- Size: 10.7 KB
- Stars: 15
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-deno - deno_kv_fs
README
# 🚀 **deno_kv_fs**
> [!IMPORTANT]\
> **Please give a star!** ⭐
---
## 🌟 Introduction
**deno_kv_fs** is a **Deno KV file system**, compatible with **Deno Deploy**. It
saves files in 64KB chunks, allowing you to organize files into directories. You
can control the KB/s rate for saving and reading files, impose rate limits, set
user space limits, and limit concurrent operations—useful for controlling
uploads/downloads. It utilizes the **Web Streams API**.
---
## 📚 **Contents**
- [🚀 Introduction](#-introduction)
- [🔧 How to Use](#-how-to-use)
- [💡 Examples](#-examples)
- [💾 Saving Data](#-saving-data)
- [💾 Saving Data Directly](#-saving-data-directly)
- [📤 Saving Data from a Submitted Form](#-saving-data-from-a-submitted-form)
- [🌐 In Frontend](#-in-frontend)
- [📥 Returning Data](#-returning-data)
- [📥 Returning Data Directly](#-returning-data-directly)
- [⚙️ Example Function to Control Data Traffic](#%EF%B8%8F-example-function-to-control-data-traffic)
- [📡 Sending File Progress in Real-Time](#-sending-file-progress-in-real-time)
- [🛠️ Useful Procedures Included](#%EF%B8%8F-useful-procedures-included)
- [📦 All Imports](#-all-imports)
- [👨💻 About](#-about)
---
## 🔧 **How to Use**
Instantiate the `DenoKvFs` class:
```typescript
import { DenoKvFs } from "https://deno.land/x/deno_kv_fs/mod.ts";
const kvFs = new DenoKvFs();
// If you want to use your existing instance of Deno.Kv
const myDenoKV = await Deno.openKv(/* your parameters */);
const kvFs = new DenoKvFs(myDenoKV);
```
- Accepts any stream as input.
- Returns a generic stream of `type: "bytes"`.
- Incompletely saved files are automatically deleted.
- Read methods return the processing status of a file (useful for knowing
progress).
- If a file does not exist, `null` is returned.
### **Save Method Interface**
The `save` method is used to save files:
```typescript
interface SaveOptions {
path: string[]; // Mandatory. The root directory is []
content: ReadableStream | Uint8Array | string; // Mandatory
chunksPerSecond?: number;
clientId?: string | number;
validateAccess?: (path: string[]) => Promise | boolean;
maxClientIdConcurrentReqs?: number;
maxFileSizeBytes?: number;
allowedExtensions?: string[];
}
```
### **Read, ReadDir, Delete, and DeleteDir Method Interface**
These methods are intended to read and delete files:
```typescript
interface ReadOptions {
path: string[]; // Mandatory
chunksPerSecond?: number;
maxDirEntriesPerSecond?: number;
clientId?: string | number;
validateAccess?: (path: string[]) => Promise | boolean;
maxClientIdConcurrentReqs?: number;
pagination?: boolean; // If true, returns the cursor to the next page (if exists).
cursor?: string; // For readDir, if there is a next page.
}
```
---
## 💡 **Examples**
### 💾 **Saving Data**
```typescript
import { DenoKvFs } from "https://deno.land/x/deno_kv_fs/mod.ts";
import { toReadableStream } from "https://deno.land/std/io/mod.ts";
const kvFs = new DenoKvFs();
const fileName = "myFile.txt";
let resData = await kvFs.save({
path: ["my_dir", fileName],
content: toReadableStream(await Deno.open(fileName)),
});
```
---
### 💾 **Saving Data Directly**
> **Warning:** This is **not recommended** as it can fill up your RAM. Use only
> for internal resources of your application. For optimized use, prefer an
> instance of `ReadableStream`.
```typescript
const fileName = "myFile.txt";
let resData = await kvFs.save({
path: ["my_dir", fileName],
content: await Deno.readFile(fileName), // Or content: "myStringData"
});
```
---
### 📤 **Saving Data from a Submitted Form**
```typescript
const reqBody = await request.formData();
const existingFileNamesInTheUpload: { [key: string]: number } = {};
const res: any = {};
for (const item of reqBody.entries()) {
if (item[1] instanceof File) {
const formField: any = item[0];
const fileData: any = item[1];
if (!existingFileNamesInTheUpload[fileData.name]) {
existingFileNamesInTheUpload[fileData.name] = 1;
} else {
existingFileNamesInTheUpload[fileData.name]++;
}
let prepend = "";
if (existingFileNamesInTheUpload[fileData.name] > 1) {
prepend += existingFileNamesInTheUpload[fileData.name].toString();
}
const fileName = prepend + fileData.name;
let resData = await kvFs.save({
path: ["my_dir", fileName],
content: fileData.stream(),
});
if (res[formField] !== undefined) {
if (Array.isArray(res[formField])) {
res[formField].push(resData);
} else {
res[formField] = [res[formField], resData];
}
} else {
res[formField] = resData;
}
}
}
console.log(res);
```
#### 🌐 **In Frontend**
```html
var files = document.querySelector("#yourFormId input[type=file]").files;
var name = document
.querySelector("#yourFormId input[type=file]")
.getAttribute("name");
var form = new FormData();
for (var i = 0; i < files.length; i++) {
form.append(`${name}_${i}`, files[i]);
}
var res = await fetch(`/your_POST_URL`, {
// Fetch API automatically sets the form to "multipart/form-data"
method: "POST",
body: form,
}).then((response) => response.json());
console.log(res);
```
---
### 📥 **Returning Data**
```typescript
const fileName = "myFile.txt";
let resData = await kvFs.read({
path: ["my_dir", fileName],
});
response.body = resData.content; // resData.content is an instance of ReadableStream
```
---
### 📥 **Returning Data Directly**
> **Warning:** This is **not recommended** as it can fill up your RAM. Use only
> for internal resources of your application. For optimized use, use the
> `ReadableStream` (`type: "bytes"`) that comes by default in `file.content`.
```typescript
const fileName = "myFile.txt";
let resData = await kvFs.read({
path: ["my_dir", fileName],
});
response.body = await DenoKvFs.readStream(resData.content); // Or await DenoKvFs.readStreamAsString(resData.content)
```
---
### ⚙️ **Example Function to Control Data Traffic**
```typescript
const gigabyte = 1024 * 1024 * 1024;
const existingRequests = kvFs.getClientReqs(user.id); // The input parameter is the same as clientId
const chunksPerSecond = (user.isPremium() ? 20 : 1) / existingRequests;
const maxClientIdConcurrentReqs = user.isPremium() ? 5 : 1;
const maxFileSizeBytes = (user.isPremium() ? 1 : 0.1) * gigabyte;
// To read
let resData = await kvFs.read({
path: ["my_dir", fileName],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id, // The clientId can also be the remote address of a request
});
response.body = resData.content;
// To delete
let resData = await kvFs.delete({
path: ["my_dir_2", fileName],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id,
});
// Read directory
const maxDirEntriesPerSecond = user.isPremium() ? 1000 : 100;
let resData = await kvFs.readDir({
path: ["my_dir"],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id,
maxDirEntriesPerSecond: maxDirEntriesPerSecond,
pagination: true, // Each page has 1000 entries
cursor: "JDhiasgPh", // If exists
});
// Delete directory
let resData = await kvFs.deleteDir({
path: ["my_dir"],
chunksPerSecond: chunksPerSecond,
maxClientIdConcurrentReqs: maxClientIdConcurrentReqs,
maxFileSizeBytes: maxFileSizeBytes,
clientId: user.id,
maxDirEntriesPerSecond: maxDirEntriesPerSecond,
});
// Controlling maximum user space
const maxAvailableSpace = (user.isPremium() ? 1 : 0.1) * gigabyte;
let dirList = await kvFs.readDir({
path: [user.id, "files"], // Example
});
if (dirList.size > maxAvailableSpace) {
throw new Error(
`You have exceeded the ${maxAvailableSpace} GB limit of available space.`,
);
}
// Validate access
let resData = await kvFs.readDir({
path: ["my_dir"],
validateAccess: async (path: string[]) =>
user.hasDirAccess(path) ? true : false,
});
```
---
## 📡 **Sending File Progress in Real-Time**
To send file progress updates in real-time, you can use:
```typescript
kvFs.onFileProgress = (status: FileStatus) => {
webSocket.send(JSON.stringify(status));
};
```
---
## 🛠️ **Useful Procedures Included**
- **Reading a Stream as Uint8Array:**
```typescript
static async readStream(stream: ReadableStream): Promise;
```
- **Reading a Stream as String:**
```typescript
static async readStreamAsString(stream: ReadableStream): Promise;
```
- **Get Client Requests:**
```typescript
getClientReqs(clientId: string | number): number;
```
- **Get All File Statuses:**
```typescript
getAllFilesStatuses(): FileStatus[];
```
- **Convert Path to URI Component:**
```typescript
pathToURIComponent(path: string[]): string;
```
- **Convert URI Component to Path:**
```typescript
URIComponentToPath(path: string): string[];
```
- **Save a File:**
```typescript
async save(options: SaveOptions): Promise;
```
- **Read a File:**
```typescript
async read(options: ReadOptions): Promise;
```
- **Read a Directory:**
```typescript
async readDir(options: ReadOptions): Promise;
```
- **Delete a File:**
```typescript
async delete(options: ReadOptions): Promise;
```
- **Delete a Directory:**
```typescript
async deleteDir(options: ReadOptions): Promise;
```
---
## 📦 **All Imports**
```typescript
import {
DenoKvFs,
DirList,
File,
FileStatus,
ReadOptions,
SaveOptions,
} from "jsr:@hviana/deno-kv-fs";
```
---
## 👨💻 **About**
**Author:** Henrique Emanoel Viana, a Brazilian computer scientist and web
technology enthusiast.
- 📞 **Phone:** +55 (41) 99999-4664
- 🌐 **Website:**
[https://sites.google.com/view/henriqueviana](https://sites.google.com/view/henriqueviana)
> **Improvements and suggestions are welcome!**
---