https://github.com/justintanner/videocity-engine
TypeScript filesystem abstraction for video/image asset projects with Git version control and distributed locking
https://github.com/justintanner/videocity-engine
filesystem git nodejs typescript video-processing
Last synced: 2 days ago
JSON representation
TypeScript filesystem abstraction for video/image asset projects with Git version control and distributed locking
- Host: GitHub
- URL: https://github.com/justintanner/videocity-engine
- Owner: justintanner
- Created: 2026-03-04T04:07:54.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-06-10T11:41:21.000Z (28 days ago)
- Last Synced: 2026-06-10T13:05:49.822Z (27 days ago)
- Topics: filesystem, git, nodejs, typescript, video-processing
- Language: TypeScript
- Size: 12.7 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# videocity-engine
[](https://github.com/justintanner/videocity-engine/actions/workflows/ci.yml)
[](https://nodejs.org)
[](https://www.typescriptlang.org)
TypeScript/Node.js filesystem abstraction for managing video, image, audio, script, and character asset projects. Each project is a git repo with a sidecar `.videocity/` directory holding two SQLite databases: `state.sqlite` for ephemeral coordination (locks, job queue, recovery journal) and `metadata.sqlite` for content metadata (timeline, characters, asset metadata, audio waveforms). Every mutation produces an atomic git commit; metadata changes are mirrored to canonical JSON exports under `.videocity/export/` so git stays diffable.
## Install
```bash
npm install videocity-engine
```
## Quick Start
```typescript
import { createFs } from 'videocity-engine';
const fs = createFs({ projectsDir: '/path/to/projects' });
// Create a project (auto-generated slug)
const project = await fs.createProject();
if (!project.ok) {
console.error(project.error.code, project.error.message);
process.exit(1);
}
const slug = project.value.slug; // "bright-falcon-42"
// Create a video asset inside the project
const asset = await fs.createAsset('vid', 'intro-clip', slug);
if (asset.ok) {
console.log(asset.value.assetId); // "vid-intro-clip"
}
// Write and read files (each write is an atomic git commit)
await fs.writeFile('vid-intro-clip', 'thumbnail.png', imageBuffer, slug);
const file = await fs.readFile('vid-intro-clip', 'thumbnail.png', slug);
if (file.ok) {
console.log(file.value.length);
}
```
## Result Pattern
All mutating methods return `Result` instead of throwing exceptions:
```typescript
type Result = { ok: true; value: T } | { ok: false; error: E };
interface FsError {
code: FsErrorCode;
message: string;
}
```
**Error codes:** `NOT_FOUND` | `ALREADY_EXISTS` | `GIT_ERROR` | `INVALID_INPUT` | `IO_ERROR` | `LOCKED`
## Asset prefixes
Assets live as prefixed directories at the project root. Valid prefixes:
| Prefix | Type |
|--------|------|
| `vid-` | video |
| `img-` | image |
| `aud-` | audio |
| `script-` | script |
| `char-` | character |
The asset id `final` is reserved as a project-level singleton.
## API
All methods are available on the object returned by `createFs(config)`. Unless noted otherwise, every method that takes a `projectSlug` requires a project that already exists.
### Project
| Method | Return Type |
|--------|------------|
| `createProject(slug?)` | `Promise>` |
| `listProjects(options?)` | `Promise` |
| `getProject(slug?)` | `Promise>` |
| `switchProject(slug)` | `Promise>` |
| `renameProject(oldSlug, newSlug)` | `Promise>` |
| `resolveProjectDir(slug?)` | `Promise` |
### Asset
| Method | Return Type |
|--------|------------|
| `createAsset(prefix, name, projectSlug)` | `Promise>` |
| `listAssets(projectSlug, options?)` | `Promise` |
| `deleteAsset(assetId, projectSlug)` | `Promise>` |
| `renameAsset(assetId, newName, projectSlug)` | `Promise>` |
| `getManifest(assetId, projectSlug, options?)` | `Promise>` |
| `getAssetStatus(assetId, projectSlug, options?)` | `Promise>` |
| `listAssetSubdir(assetId, subdirName, projectSlug)` | `Promise>` |
| `slugTaken(slug, projectSlug)` | `Promise` |
### File
| Method | Return Type |
|--------|------------|
| `writeFile(assetId, filename, data, projectSlug)` | `Promise>` |
| `readFile(assetId, filename, projectSlug)` | `Promise>` |
| `deleteFile(assetId, filename, projectSlug)` | `Promise>` |
| `renameFile(assetId, oldFilename, newFilename, projectSlug)` | `Promise>` |
| `copyFile(assetId, filename, destAssetId, destFilename, projectSlug)` | `Promise>` |
| `resolveAssetDir(assetId, projectSlug)` | `Promise>` |
### Metadata
`writeMetadata(assetId, 'character', record, ...)` is special-cased: the record is stored in the typed `characters` table in `metadata.sqlite`. Other keys are stored in the generic `asset_metadata` table. In both cases a `.{key}.json` sidecar is also written next to the asset and the operation produces a single git commit covering the SQLite file, the canonical export, and the sidecar.
`writeProjectMeta(key, data, ...)` with `key === 'timeline'` is also special-cased and **strict**: `data` must be `{ slots: [{ slug, ... }, ...], render: 'landscape' | 'portrait' | 'square' }` (an optional `currentOrientation` and an optional `audio` array are accepted). Non-conforming payloads return `INVALID_INPUT`. Reads come from SQLite only — there is no sidecar fallback. Other project-meta keys are written as plain `.{key}.json` sidecars.
| Method | Return Type |
|--------|------------|
| `writeMetadata(assetId, key, data, projectSlug)` | `Promise>` |
| `readMetadata(assetId, key, projectSlug)` | `Promise>` |
| `writeAudioWaveform(assetId, peaks, projectSlug)` | `Promise>` |
| `readAudioWaveform(assetId, projectSlug)` | `Promise>` |
| `writeProjectMeta(key, data, projectSlug)` | `Promise>` |
| `readProjectMeta(key, projectSlug)` | `Promise>` |
### Git
| Method | Return Type |
|--------|------------|
| `commitOperation(operation, assetId?, details?, projectSlug)` | `Promise` |
| `getHistory(projectSlug, limit?)` | `Promise` |
| `getAssetHistory(assetId, projectSlug, limit?)` | `Promise` |
| `restoreAsset(assetId, commitHash, projectSlug)` | `Promise` |
| `readFileAtCommit(assetId, filename, commitHash, projectSlug)` | `Promise>` |
| `rewindProject(commitHash, projectSlug)` | `Promise` |
### Lock
Locks are SQLite rows in `state.sqlite`, one per asset directory (or one project-level lock keyed by `__PROJECT__`). The `assetDir` argument is an absolute filesystem path; the lock module resolves it back to `(projectDir, assetKey)`.
| Method | Return Type |
|--------|------------|
| `acquireLock(assetDir, options)` | `Promise>` |
| `releaseLock(assetDir)` | `Promise>` |
| `isLocked(assetDir)` | `Promise` |
| `getLockData(assetDir)` | `Promise` |
| `cleanStaleLock(assetDir)` | `Promise` |
`LockOptions` is `{ durationMs, data?, state? }`. `acquireLock` returns `LOCKED` if a non-expired lock is already held; expired or dead-pid locks are reaped automatically on the next acquire (or explicitly via `cleanStaleLock`).
### Action log
Append-only audit trail backed by git commits with structured payloads.
| Method | Return Type |
|--------|------------|
| `logAction(action, payload, projectSlug)` | `Promise>` |
| `getActionLog(options?, projectSlug)` | `Promise` |
### Generic JSONL log
Per-project append-only logs under `logs/{name}.jsonl`. Not committed to git.
| Method | Return Type |
|--------|------------|
| `appendLog(name, line, projectSlug)` | `Promise>` |
| `readLog(name, projectSlug, options?)` | `Promise[]>` |
### Queue
A persistent job queue lives in `state.sqlite` (`pending_jobs`). Jobs are dedupe-keyed, support external task ids, and are leased with heartbeats so a `QueueRunner` can be coordinated across multiple processes. `fs.queue` exposes the project-scoped surface; the standalone `queueApi` and `QueueRunner` are also exported for callers that want the raw `Database` handle.
```typescript
const enq = await fs.queue.enqueue(slug, {
type: 'transcode',
assetId: 'vid-intro-clip',
payload: { preset: '1080p' },
});
const result = await fs.queue.enqueueAndWait(slug, {
type: 'render',
payload: { orientation: 'portrait' },
});
```
### Pending tasks
Tracks external long-running provider jobs (transcription, generation, etc.) in the `pending_tasks` table of `state.sqlite`. Each row is keyed by `assetId`; `taskId` is the provider's task identifier (or `QUEUED_TASK_ID` for jobs that haven't been submitted yet). `createdAt` is **epoch seconds**.
```typescript
import { QUEUED_TASK_ID } from 'videocity-engine';
await fs.pendingTasks.write(slug, {
assetId: 'vid-clip-1',
taskId: QUEUED_TASK_ID,
taskType: 'fal_seedance2_t2v',
assetDir: '/path/to/projects/proj-x/vid-clip-1',
meta: { prompt: 'sunset over ocean' },
});
// Atomic: write a generation_errors row + delete the pending_tasks row.
await fs.pendingTasks.fail(slug, 'vid-clip-1', { message: 'provider rejected', failCode: 'NSFW' });
```
| Method | Return Type |
|--------|------------|
| `pendingTasks.write(projectSlug, input)` | `Promise>` |
| `pendingTasks.read(projectSlug, assetId)` | `Promise>` |
| `pendingTasks.delete(projectSlug, assetId)` | `Promise>` |
| `pendingTasks.markCompleting(projectSlug, assetId)` | `Promise>` |
| `pendingTasks.clearCompleting(projectSlug, assetId)` | `Promise>` |
| `pendingTasks.findAll(projectSlug)` | `Promise>` |
| `pendingTasks.findByExternalId(projectSlug, taskId)` | `Promise>` |
| `pendingTasks.fail(projectSlug, assetId, info)` | `Promise>` |
### Generation errors
Sibling table to `pending_tasks` for terminal failures. `failedAt` is **epoch seconds**.
| Method | Return Type |
|--------|------------|
| `generationErrors.write(projectSlug, assetId, info)` | `Promise>` |
| `generationErrors.read(projectSlug, assetId)` | `Promise>` |
| `generationErrors.clear(projectSlug, assetId)` | `Promise>` |
| `generationErrors.findAll(projectSlug)` | `Promise>` |
### Lifecycle
| Method | Return Type |
|--------|------------|
| `ensureProjectInitialized(slug)` | `Promise` — idempotently create `.videocity/`, open the state DB, apply gitignore patterns. Safe to call on every boot or enqueue. |
| `recoverIncompleteOperations(slug)` | `Promise` — drain the recovery journal at startup |
| `checkSchemaVersion(slug)` | `Promise` — refuse to open a project written by a newer build |
| `close()` | `void` — close all SQLite handles before process exit |
## Atomic operations & recovery
Every metadata write follows the same shape:
1. Acquire the per-project git mutex (`proper-lockfile` on `.videocity/.project.lock`).
2. Open a `recovery_journal` row, run the SQLite work in a transaction, and rebuild the affected canonical JSON exports under `.videocity/export/`.
3. Stage the SQLite file, exports, and any sidecars in a single `git commit` whose body includes `op-id: `.
4. Mark the journal row complete.
If the process dies mid-operation, `recoverIncompleteOperations()` replays the journal at startup: it re-derives the exports from `metadata.sqlite`, writes any missing sidecars, and produces a fresh `recover` commit. Sentinel `op-id` lookups in `git log` ensure idempotency. Schema migrations are checksummed in `schema_migrations`; mismatches are fatal.
## Configuration
```typescript
interface FsConfig {
projectsDir: string; // Root directory for all projects
gitPath?: string; // Custom git binary path (default: "git")
}
```
## Development
```bash
npm run build # Compile TypeScript
npm run typecheck # Type-check without emitting
npm test # Run tests
npm run test:watch # Run tests in watch mode
npm run bench # Run vitest benches
```
## Requirements
- Node.js >= 20
- Git