{"id":39909904,"url":"https://github.com/puzed/openpull-node","last_synced_at":"2026-01-18T16:06:01.275Z","repository":{"id":308789598,"uuid":"1034109233","full_name":"puzed/openpull-node","owner":"puzed","description":"NodeJS library  streaming application logs to OpenPull via WebRTC. Start fast with the logger, then dive deeper as needed.","archived":false,"fork":false,"pushed_at":"2025-08-17T16:44:33.000Z","size":65,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-17T18:25:30.644Z","etag":null,"topics":["apm","logs","metrics","nodejs","trace","typescript"],"latest_commit_sha":null,"homepage":"https://openpull.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/puzed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-07T21:26:29.000Z","updated_at":"2025-08-17T16:44:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"3acb0bc5-2e3e-40d9-b04a-f8361a943145","html_url":"https://github.com/puzed/openpull-node","commit_stats":null,"previous_names":["puzed/openpull-node"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/puzed/openpull-node","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puzed%2Fopenpull-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puzed%2Fopenpull-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puzed%2Fopenpull-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puzed%2Fopenpull-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/puzed","download_url":"https://codeload.github.com/puzed/openpull-node/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puzed%2Fopenpull-node/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28541068,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T14:59:57.589Z","status":"ssl_error","status_checked_at":"2026-01-18T14:59:46.540Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apm","logs","metrics","nodejs","trace","typescript"],"created_at":"2026-01-18T16:06:00.761Z","updated_at":"2026-01-18T16:06:01.261Z","avatar_url":"https://github.com/puzed.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenPull Node.js Library\n\nA TypeScript/JavaScript library for streaming application logs to OpenPull via WebRTC. Start fast with the logger, then dive deeper as needed.\n\n## Quick Start\n\n### 1) Install\n\n```bash\nnpm install openpull\n```\n\n### 2) Minimal Library Usage (Logger first)\n\n```typescript\nimport { createLogger, createConnection } from 'openpull';\n\n// JSON logger to stdout (pino-like)\nconst log = createLogger();\nlog.info('App started');\nlog.trace('request').span('db-query').finish();\n\n// Forward stdout/stderr to OpenPull (separate from logger)\nconst connection = await createConnection(process.env.OPENPULL_URL!);\nawait connection.forward(process.stdout, process.stderr);\n```\n\nYou can also keep your existing logger (pino/winston/console) and just add a `trace_id` when you want correlation.\n\n### 3) Minimal CLI Usage (Any language)\n\n```bash\n# One-time global install for the CLI (optional)\nnpm install -g openpull\n\n# Wrap any command; uses OPENPULL_URL if set\nopenpull -- node app.js\nopenpull -- python app.py\n```\n\nExample minimal Node.js file (app.js):\n\n```javascript\n// app.js\nconst traceId = 'trace-' + Date.now();\n\nconsole.log(JSON.stringify({ level: 'info', message: 'Server starting', port: 3000 }));\nconsole.log(JSON.stringify({ level: 'info', message: 'Database connected', db: 'postgres' }));\n\n// Three trace logs sharing the same trace_id (correlated)\nconsole.log(JSON.stringify({ level: 'trace', message: 'Checkout flow', step: 'begin', trace_id: traceId }));\nconsole.log(JSON.stringify({ level: 'trace', message: 'Payment processed', amount: 99.99, trace_id: traceId }));\nconsole.log(JSON.stringify({ level: 'trace', message: 'Order completed', order_id: '12345', trace_id: traceId }));\n```\n\nThat’s it—you’re sending logs to OpenPull.\n\n## Concepts\n\n- **Everything is a log entry:** Logs, metrics, and traces are just JSON log entries. Tracing is simply adding a shared field (commonly `trace_id`).\n- **No schema required:** Log any JSON. The dashboard discovers fields dynamically and lets you filter, group, and search by anything you emit.\n\nExample entries you might emit:\n\n```javascript\nlog.info('User action', { userId: 123, action: 'login', region: 'us-east' });\nlog.info('Database query', { query: 'SELECT * FROM users', duration: 45, rows: 150 });\n// Grouping can be by any field you choose, not only trace_id\nlog.info('Processing request', { request_id: 'req-123', service: 'api', endpoint: '/users' });\n```\n\n## Features\n\n- **Unified observability:** logs, metrics, and traces as log entries\n- **Works with any logger:** pino, winston, console.log, or custom\n- **Real-time streaming:** WebRTC connection to OpenPull\n- **Standalone structured logger:** optional, pino-like ergonomics\n- **Clean separation:** logger writes to stdout; connection forwards\n- **CLI tool:** wrap any language/executable\n- **TypeScript:** full typings, no classes\n\n## How It Works\n\nThe magic is in the **ultra-flexible parsing**. Here's what happens under the hood:\n\n### 1. No Schema Validation\nWhen you log anything, OpenPull just:\n```javascript  \n// From connection-manager.ts:12\nfunction parseLogLine(line: string, defaultLevel: string = 'info'): any {\n  try {\n    const parsed = JSON.parse(trimmedLine);\n    return {\n      type: parsed.level || parsed.type || defaultLevel,\n      message: parsed.message || parsed.msg || trimmedLine,\n      timestamp: parsed.timestamp || parsed.time || new Date().toISOString(),\n      ...parsed,  // ← ALL your fields get preserved\n    };\n  } catch {\n    // Plain text becomes a log entry too\n    return {\n      type: defaultLevel,\n      message: trimmedLine,\n      timestamp: new Date().toISOString(),\n    };\n  }\n}\n```\n\n### 2. Dynamic Field Discovery\nThe dashboard uses DuckDB's JSON operators to discover fields at runtime:\n```sql\n-- Gets ALL unique field names across your logs\nWITH json_keys AS (\n  SELECT DISTINCT json_keys(data) as keys FROM logs\n)\nSELECT key, COUNT(*) as count \nFROM json_keys jk, UNNEST(jk.keys) as t(key), logs l\nGROUP BY key ORDER BY count DESC\n```\n\n### 3. Flexible Grouping\nYou can group by ANY field, not just `trace_id`:\n```sql\n-- Group by trace_id (traditional)\nWHERE json_extract(data, '$.trace_id') = 'abc123'\n\n-- Or group by request_id \nWHERE json_extract(data, '$.request_id') = 'req-456'\n\n-- Or group by user_id\nWHERE json_extract(data, '$.user_id') = '789'\n\n-- Or any custom field you invent\nWHERE json_extract(data, '$.workflow_id') = 'deploy-123'\n```\n\n**`trace_id` is a convention**, not a requirement. Correlate by any shared field.\n\n\n## API Reference\n\n### Library API\n\n#### `createLogger(options?: LoggerOptions): Logger`\n\nCreates a standalone logger that outputs JSON to stdout (competes with pino):\n\n```javascript\nconst log = createLogger({\n  defaultFields: {\n    service: 'my-app',\n    version: '1.0.0'\n  }\n});\n\n// Basic logging\nlog.info('User logged in', { userId: '123' });\nlog.error('Database error', { query: 'SELECT * FROM users' });\nlog.debug('Processing request', { requestId: 'req-456' });\nlog.warning('Rate limit approaching', { current: 95, limit: 100 });\n\n// Distributed tracing with trace_id correlation\nlog.trace('request').span('db-query').span('validation').finish();\n\n// Each span gets the same trace_id automatically:\n// {\"level\":\"trace\",\"message\":\"request\",\"trace_id\":\"abc123\",\"span_id\":\"span1\"...}\n// {\"level\":\"trace\",\"message\":\"db-query\",\"trace_id\":\"abc123\",\"span_id\":\"span2\"...}\n// {\"level\":\"trace\",\"message\":\"validation\",\"trace_id\":\"abc123\",\"span_id\":\"span3\"...}\n\n// Or traditional approach\nconst trace = log.startTrace({ operation: 'checkout' });\ntrace.span('Validate cart');    // Same trace_id\ntrace.span('Process payment');  // Same trace_id\ntrace.finish();\n```\n\n#### `createConnection(connectionString: string): Promise\u003cConnection\u003e`\n\nCreates a connection that can forward streams:\n\n```javascript\nconst connection = await createConnection('openpull://appender:key@signal.openpull.com/NoDrBwJRXA8W');\n\n// Forward process streams\nawait connection.forward(process.stdout, process.stderr);\n\n// Or forward child process streams (used internally by CLI)\nawait connection.forwardStreams(child.stdout, child.stderr);\n```\n\nNote: The URL includes a key, but OpenPull uses a zero-knowledge handshake — the key never leaves your device.\n\n**Connection String Format:**\n```\nopenpull://role:key@publicToken.host:port/\n```\n\n- `role`: Either `appender` (for sending logs) or `reader` (for receiving logs)\n- `key`: Authentication key derived from your session (used only locally to compute the handshake proof)\n- `publicToken`: Session identifier\n- `host:port`: Signaling server address\n\n### CLI Reference\n\nWraps any application and forwards stdout/stderr:\n\n```bash\nexport OPENPULL_URL=\"openpull://appender:key@signal.openpull.com/NoDrBwJRXA8W\"\nopenpull -- node app.js       # Node\nopenpull -- python script.py  # Python\nopenpull -- go run main.go    # Go\nopenpull -- dotnet run        # .NET\n\n# Help\nopenpull --help\n```\n\nEnv vars:\n- `OPENPULL_URL` – default connection URL\n\n## TypeScript Support\n\nThe library is written in TypeScript and provides comprehensive type definitions:\n\n```typescript\nimport { Logger, Tracer, LogData } from 'openpull';\n\nconst log: Logger = logger();\nconst trace: Tracer = log.startTrace();\n\n// All methods are fully typed\nlog.info('Message', { customField: 'value' });\n```\n\n## Error Handling\n\nThe library gracefully handles connection failures:\n\n```javascript\ntry {\n  await connect('openpull://appender:key@signal.openpull.com/NoDrBwJRXA8W');\n} catch (error) {\n  console.error('Failed to connect:', error.message);\n}\n\n// Logs fall back to console if no connection\nconst log = logger();\nlog.info('This will go to console if not connected');\n```\n\n## Examples\n\n### Library: Standalone Logger\n\n```javascript\nimport { createLogger } from 'openpull';\n\n// Pure logger that outputs JSON to stdout (like pino)\nconst log = createLogger({\n  defaultFields: {\n    service: 'user-service',\n    version: '1.0.0'\n  }\n});\n\nlog.info('Service started', { port: 3000 });\nlog.error('Database error', { table: 'users' });\n\n// Distributed tracing with trace_id correlation\nlog.trace('checkout')\n  .span('validate-cart')     // trace_id: \"xyz789\", span_id: \"span1\"\n  .span('process-payment')   // trace_id: \"xyz789\", span_id: \"span2\" \n  .span('update-inventory')  // trace_id: \"xyz789\", span_id: \"span3\"\n  .finish();\n```\n\n### Library: Clean Separation\n\n```javascript\nimport { createLogger, createConnection } from 'openpull';\n\n// Logger just outputs to stdout\nconst log = createLogger({ defaultFields: { service: 'api' } });\n\n// Connection handles WebRTC separately  \nconst connection = await createConnection(process.env.OPENPULL_URL);\nawait connection.forward(process.stdout, process.stderr);\n\n// Now both structured logs AND console.log get forwarded\nlog.info('Structured log');\nconsole.log('Plain console log');\n```\n\n### Express.js with Tracing\n\n```javascript\nimport express from 'express';\nimport { createLogger, createConnection } from 'openpull';\n\nconst app = express();\n\n// Setup separate logger and connection\nconst log = createLogger({ defaultFields: { service: 'api' } });\nconst connection = await createConnection(process.env.OPENPULL_URL);\nawait connection.forward(process.stdout, process.stderr);\n\n// Request tracing middleware\napp.use((req, res, next) =\u003e {\n  // Start a trace for each request\n  const trace = log.startTrace({ \n    method: req.method, \n    url: req.url,\n    userAgent: req.headers['user-agent'] \n  });\n  \n  req.trace = trace;\n  req.trace.span('Request started');\n  next();\n});\n\napp.get('/users/:id', (req, res) =\u003e {\n  req.trace.span('Fetching user', { userId: req.params.id });\n  \n  // All these spans share the same trace_id:\n  req.trace.span('Database query');\n  req.trace.span('Permission check');\n  req.trace.span('Response serialization');\n  \n  req.trace.span('Request completed', { statusCode: res.statusCode });\n  req.trace.finish();\n});\n```\n\n## Why This Logger vs Existing Ones?\n\n- **Use any logger + CLI:** Keep pino/winston/console and forward via CLI.\n- **Use OpenPull logger:** If you want built-in tracing and automatic correlation.\n\nExample with pino (manual correlation):\n\n```javascript\nimport pino from 'pino';\nconst log = pino();\nconst traceId = 'trace-' + Math.random().toString(36).slice(2);\nlog.info({ trace_id: traceId }, 'Request started');\n```\n\n## Dynamic Dashboard\n\nThe OpenPull web dashboard has **zero configuration** - it adapts to whatever fields you actually log:\n\n### Auto-Discovery\n```javascript\n// Dashboard automatically discovers these fields from your logs:\nasync function updateSchema() {\n  const schema = await getLogSchema(db);\n  \n  // Finds ALL unique fields: userId, action, browser, region, customField, etc.\n  state.availableFields = new Set(schema.map(s =\u003e s.key));\n  \n  // Auto-shows frequently used fields (\u003e 10 occurrences)  \n  schema.forEach(field =\u003e {\n    if (field.count \u003e 10 \u0026\u0026 !state.visibleColumns.has(field.key)) {\n      state.visibleColumns.add(field.key);\n    }\n  });\n}\n```\n\n### Dynamic Filtering  \nThe dashboard creates filters for **every field** it finds:\n```javascript\n// Creates dropdowns/filters for ANY field\nstate.availableFields.forEach(field =\u003e {\n  const fieldValues = await getFieldValues(db, field);\n  // Now you can filter by userId, region, cache status, etc.\n});\n```\n\n### Flexible Grouping\nYou can group \"traces\" by **any field**:\n- Traditional: Group by `trace_id` to see request flows\n- By user: Group by `user_id` to see all user activity  \n- By service: Group by `service` to see service interactions\n- Custom: Group by `deployment_id`, `experiment_id`, `workflow_id`, etc.\n\n**No schema to define upfront** — log any JSON structure and the dashboard adapts automatically.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpuzed%2Fopenpull-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpuzed%2Fopenpull-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpuzed%2Fopenpull-node/lists"}