{"id":29182622,"url":"https://github.com/iambelugaa/kvix","last_synced_at":"2025-07-01T20:07:02.885Z","repository":{"id":301839294,"uuid":"1010448451","full_name":"iamBelugaa/kvix","owner":"iamBelugaa","description":"A key-value storage engine inspired by Bitcask architecture.","archived":false,"fork":false,"pushed_at":"2025-06-29T06:24:28.000Z","size":1082,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-29T06:32:35.836Z","etag":null,"topics":["bitcask","golang","inmemory","kv","kv-store","protobuf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iamBelugaa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-29T05:03:13.000Z","updated_at":"2025-06-29T06:24:34.000Z","dependencies_parsed_at":"2025-06-29T06:32:51.628Z","dependency_job_id":"4f194fb4-4473-4f37-9b74-00f6a4943221","html_url":"https://github.com/iamBelugaa/kvix","commit_stats":null,"previous_names":["iambelugaa/kvix"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/iamBelugaa/kvix","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamBelugaa%2Fkvix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamBelugaa%2Fkvix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamBelugaa%2Fkvix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamBelugaa%2Fkvix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iamBelugaa","download_url":"https://codeload.github.com/iamBelugaa/kvix/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamBelugaa%2Fkvix/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263029212,"owners_count":23402354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcask","golang","inmemory","kv","kv-store","protobuf"],"created_at":"2025-07-01T20:07:01.307Z","updated_at":"2025-07-01T20:07:02.859Z","avatar_url":"https://github.com/iamBelugaa.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kvix\n\nA key-value storage engine inspired by Bitcask architecture. Kvix combines an\nin-memory index with append-only file storage to deliver predictable O(1)\nperformance.\n\n## Core Components\n\n### Instance Management\n\nThe Instance type serves as the main entry point and manages the entire database\nlifecycle.\n\n```go\ntype Instance struct {\n    options *options.Options  // Configuration parameters\n    mu      sync.RWMutex      // Concurrent access protection\n    engine  *engine.Engine    // Core database engine coordination\n}\n```\n\n### Engine Coordination\n\nThe Engine acts as the central coordinator between the index, storage, and\ncompaction subsystems.\n\n```go\ntype Engine struct {\n    options    *options.Options\n    log        *zap.SugaredLogger\n    closed     atomic.Bool\n    index      *index.Index\n    storage    *storage.Storage\n    compaction *compaction.Compaction\n}\n```\n\n### Index Implementation\n\nThe index uses a memory-efficient design where each key maps to a RecordPointer\ncontaining exactly the information needed to locate and retrieve data:\n\n```go\ntype RecordPointer struct {\n    ExpiresAt        int64   // 8 bytes: Unix nanoseconds for TTL\n    Offset           int64   // 8 bytes: Exact position in segment file\n    SegmentTimestamp int64   // 8 bytes: Creation time for filename reconstruction\n    SegmentID        uint16  // 2 bytes: Segment identifier (0-65535)\n\n    // 6 Bytes of padding added by Go for alignment = 32 bytes total\n}\n```\n\n## Storage Engine Design\n\n### Multi-Segment Architecture\n\nThe storage system implements sophisticated segment management with automatic\nrotation when files reach configurable size limits.\n\n**Segment File Structure:**\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    SEGMENT FILE                             │\n├─────────────────────────────────────────────────────────────┤\n│  RECORD 1: [Binary Header][Protobuf Payload]                │\n│            [17 bytes]     [Variable length]                 │\n│                                                             │\n│  RECORD 2: [Binary Header][Protobuf Payload]                │\n│            [17 bytes]     [Variable length]                 │\n│                                                             │\n│  RECORD 3: [Binary Header][Protobuf Payload]                │\n│            [17 bytes]     [Variable length]                 │\n│  ...                                                        │\n└─────────────────────────────────────────────────────────────┘\n```\n\nEach record header contains essential metadata in a fixed binary structure:\n\n```go\ntype RecordHeader struct {\n    Checksum    uint32 // 4 bytes: CRC32 for data integrity\n    PayloadSize uint32 // 4 bytes: Size of protobuf payload\n    Version     uint8  // 1 byte: Schema version\n    Timestamp   int64  // 8 bytes: Creation timestamp\n}\n```\n\n### Core Operations\n\n#### `Set`\n\n```go\nfunc (i *Instance) Set(ctx context.Context, key []byte, value []byte) error\n```\n\nStores a key-value pair with immediate durability. The operation is atomic and\nfully durable once it returns successfully.\n\n#### `SetX`\n\n```go\nfunc (i *Instance) SetX(ctx context.Context, key []byte, value []byte, ttl time.Duration) error\n```\n\nStores a key-value pair with automatic expiration after the specified duration.\nIdeal for implementing caches, session stores, and time-sensitive data.\n\n#### `Get`\n\n```go\nfunc (i *Instance) Get(ctx context.Context, key []byte) (*storage.Record, error)\n```\n\nRetrieves the complete record associated with the given key, if it exists and\nhasn't expired. Uses O(1) index lookup followed by direct file access.\n\n#### `Exists`\n\n```go\nfunc (i *Instance) Exists(ctx context.Context, key []byte) (bool, error)\n```\n\nChecks whether a key exists in the database without retrieving the full record.\nSignificantly faster than Get for large values since it only accesses the index.\n\n#### `Delete`\n\n```go\nfunc (i *Instance) Delete(ctx context.Context, key []byte) (bool, error)\n```\n\nRemoves a key-value pair from the database using logical deletion. The operation\nimmediately makes the key inaccessible while marking it for physical removal\nduring compaction.\n\n#### `Close`\n\n```go\nfunc (i *Instance) Close() error\n```\n\nGracefully shuts down the database instance, ensuring data durability and proper\nresource cleanup.\n\n## Configuration\n\n### Functional Configuration Pattern\n\nThe system uses functional options for configuration, providing a clean and\nextensible approach to customization:\n\n```go\n// Available configuration functions\nfunc WithDataDir(directory string) OptionFunc\nfunc WithSegmentSize(size uint64) OptionFunc\nfunc WithSegmentPrefix(prefix string) OptionFunc\nfunc WithSegmentDir(directory string) OptionFunc\nfunc WithCompactInterval(interval time.Duration) OptionFunc\n```\n\n### Configuration Constraints\n\n#### Segment Size Constraints\n\n- **Minimum**: 512MB (prevents excessive file fragmentation)\n- **Maximum**: 4GB (maintains manageable file sizes for backup/recovery)\n- **Default**: 1GB (balanced performance and operational simplicity)\n\n#### Directory Structure\n\n- **Base data directory**: `/var/lib/kvix` (default)\n- **Segment subdirectory**: Configurable within base directory\n- **Filename format**: `{prefix}_{segmentID}_{timestamp}.seg`\n\n#### Compaction Settings\n\n- **Default interval**: 5 hours\n- **Maximum interval**: 168 hours (1 week)\n- **Minimum interval**: Default compaction interval\n\n### Session Store Implementation\n\n```go\npackage session\n\nimport (\n    \"context\"\n    \"encoding/json\"\n    \"fmt\"\n    \"time\"\n\n    \"github.com/iamBelugaa/kvix/pkg/kvix\"\n    \"github.com/iamBelugaa/kvix/pkg/errors\"\n)\n\ntype SessionStore struct {\n    db *kvix.Instance\n}\n\ntype SessionData struct {\n    UserID      string                 `json:\"user_id\"`\n    Permissions []string               `json:\"permissions\"`\n    Data        map[string]any         `json:\"data\"`\n    CreatedAt   time.Time              `json:\"created_at\"`\n    LastAccess  time.Time              `json:\"last_access\"`\n}\n\nfunc NewSessionStore(dataDir string) (*SessionStore, error) {\n    db, err := kvix.NewInstance(context.Background(), \"session-store\",\n        kvix.WithDataDir(dataDir),\n        kvix.WithSegmentPrefix(\"sessions\"),\n        kvix.WithSegmentSize(1024*1024*1024), // 1GB segments\n    )\n    if err != nil {\n        return nil, fmt.Errorf(\"failed to initialize session store: %w\", err)\n    }\n\n    return \u0026SessionStore{db: db}, nil\n}\n\nfunc (s *SessionStore) CreateSession(ctx context.Context, userID string, permissions []string) (string, error) {\n    sessionID := generateSecureSessionID()\n\n    sessionData := SessionData{\n        UserID:      userID,\n        Permissions: permissions,\n        Data:        make(map[string]any),\n        CreatedAt:   time.Now(),\n        LastAccess:  time.Now(),\n    }\n\n    encoded, err := json.Marshal(sessionData)\n    if err != nil {\n        return \"\", fmt.Errorf(\"failed to encode session data: %w\", err)\n    }\n\n    key := []byte(\"session:\" + sessionID)\n    err = s.db.SetX(ctx, key, encoded, 24*time.Hour)\n    if err != nil {\n        return \"\", fmt.Errorf(\"failed to store session: %w\", err)\n    }\n\n    return sessionID, nil\n}\n\nfunc (s *SessionStore) GetSession(ctx context.Context, sessionID string) (*SessionData, error) {\n    key := []byte(\"session:\" + sessionID)\n\n    record, err := s.db.Get(ctx, key)\n    if err != nil {\n        if errors.GetErrorCode(err) == errors.ErrIndexKeyNotFound {\n            return nil, ErrSessionNotFound\n        }\n        return nil, fmt.Errorf(\"failed to retrieve session: %w\", err)\n    }\n\n    var session SessionData\n    if err := json.Unmarshal(record.Value, \u0026session); err != nil {\n        return nil, fmt.Errorf(\"failed to decode session data: %w\", err)\n    }\n\n    session.LastAccess = time.Now()\n    if err := s.updateSession(ctx, sessionID, \u0026session); err != nil {\n        log.Printf(\"Warning: failed to update session access time: %v\", err)\n    }\n\n    return \u0026session, nil\n}\n\nfunc (s *SessionStore) updateSession(ctx context.Context, sessionID string, session *SessionData) error {\n    encoded, err := json.Marshal(session)\n    if err != nil {\n        return err\n    }\n\n    key := []byte(\"session:\" + sessionID)\n    return s.db.SetX(ctx, key, encoded, 24*time.Hour)\n}\n\nfunc (s *SessionStore) DeleteSession(ctx context.Context, sessionID string) error {\n    key := []byte(\"session:\" + sessionID)\n\n    deleted, err := s.db.Delete(ctx, key)\n    if err != nil {\n        return fmt.Errorf(\"failed to delete session: %w\", err)\n    }\n\n    if !deleted {\n        return ErrSessionNotFound\n    }\n\n    return nil\n}\n\nfunc (s *SessionStore) Close() error {\n    return s.db.Close()\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiambelugaa%2Fkvix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiambelugaa%2Fkvix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiambelugaa%2Fkvix/lists"}