{"id":30643889,"url":"https://github.com/parseablehq/n8n-observability","last_synced_at":"2025-09-02T05:02:12.839Z","repository":{"id":311874535,"uuid":"1045404833","full_name":"parseablehq/n8n-observability","owner":"parseablehq","description":"This repository contains all the information on n8n observability and monitoring using Parseable. ","archived":false,"fork":false,"pushed_at":"2025-08-27T05:55:44.000Z","size":26,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-27T14:29:05.149Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/parseablehq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["parseable","operator"]}},"created_at":"2025-08-27T05:46:29.000Z","updated_at":"2025-08-27T09:33:56.000Z","dependencies_parsed_at":"2025-08-27T14:32:10.223Z","dependency_job_id":"4a662368-910e-430a-88e2-46c39b41dd24","html_url":"https://github.com/parseablehq/n8n-observability","commit_stats":null,"previous_names":["parseablehq/n8n-observability"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/parseablehq/n8n-observability","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parseablehq%2Fn8n-observability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parseablehq%2Fn8n-observability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parseablehq%2Fn8n-observability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parseablehq%2Fn8n-observability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/parseablehq","download_url":"https://codeload.github.com/parseablehq/n8n-observability/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parseablehq%2Fn8n-observability/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272933218,"owners_count":25017612,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-31T03:09:33.570Z","updated_at":"2025-08-31T03:09:34.118Z","avatar_url":"https://github.com/parseablehq.png","language":"JavaScript","funding_links":["https://github.com/sponsors/parseable","https://github.com/sponsors/operator"],"categories":[],"sub_categories":[],"readme":"# n8n Observability with Parseable\n\nThis repository provides a complete solution for monitoring n8n workflow automation with comprehensive observability using [Parseable](https://parseable.com) - an open-source, unified observability platform.\n\n## 🎯 What This Provides\n\n- **📊 Traces**: HTTP requests, database operations, and workflow execution spans\n- **📈 Metrics**: Resource utilization, execution counts, and performance metrics  \n- **📝 Logs**: Structured workflow logs, node execution logs, and error tracking\n- **🔧 Easy Setup**: Docker Compose based deployment with minimal configuration\n\n## 🏗️ Architecture\n\n```\n┌─────────────┐    ┌──────────────┐    ┌─────────────┐\n│   n8n       │───▶│ OpenTelemetry│───▶│ Parseable   │\n│             │    │ Instrumentation  │ │             │\n│ - Workflows │    │ - Winston Logs   │ │ - Traces    │\n│ - Nodes     │    │ - HTTP Tracing   │ │ - Metrics   │\n│ - Triggers  │    │ - DB Tracing     │ │ - Logs      │\n└─────────────┘    └──────────────┘    └─────────────┘\n```\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Docker and Docker Compose\n- Git\n\n### 1. Clone and Setup\n\n```bash\ngit clone https://github.com/parseablehq/n8n-observability.git\ncd n8n-observability\n```\n\n### 2. Start the Stack\n\n```bash\n# Start all services\ndocker-compose up -d\n\n# View logs\ndocker-compose logs -f n8n\n```\n\n### 3. Access Services\n\n- **n8n**: http://localhost:5678 (admin/admin)\n- **Parseable**: http://localhost:8000 (admin/admin)\n\n### 4. Verify Observability\n\n1. Create a workflow in n8n\n2. Execute the workflow\n3. Check Parseable for traces, metrics, and logs:\n   - `otel-traces` stream: Workflow execution traces\n   - `otel-metrics` stream: Performance metrics\n   - `n8n-logs` stream: Structured logs\n\n## 📋 What's Included\n\n### Core Files\n\n- `docker-compose.yml` - Complete stack orchestration\n- `Dockerfile` - Custom n8n image with observability\n- `n8n-winston-logger.js` - Winston transport for structured logging\n- `simple-winston-tracing.js` - OpenTelemetry setup\n- `docker-entrypoint.sh` - Container initialization script\n\n### Services\n\n1. **n8n** - Workflow automation platform with observability\n2. **Parseable** - Log analytics and observability backend  \n3. **PostgreSQL** - n8n database backend\n\n## 🔧 Configuration\n\n### Environment Variables\n\nKey variables in `docker-compose.yml`:\n\n```yaml\n# n8n Configuration\nN8N_HOST: \"localhost\"\nN8N_PORT: 5678\nN8N_PROTOCOL: \"http\"\nN8N_WINSTON_LOGGING: \"true\"\n\n# Database\nDB_TYPE: \"postgresdb\"\nDB_POSTGRESDB_HOST: \"postgres\"\nDB_POSTGRESDB_DATABASE: \"n8n\"\n\n# Parseable Integration\nPARSEABLE_URL: \"http://parseable:8000\"\nPARSEABLE_USERNAME: \"admin\" \nPARSEABLE_PASSWORD: \"admin\"\n\n# OpenTelemetry\nOTEL_SERVICE_NAME: \"n8n-comprehensive\"\nOTEL_EXPORTER_OTLP_ENDPOINT: \"http://parseable:8000\"\n```\n\n### Parseable Streams\n\nThe setup creates three streams automatically:\n\n- **`n8n-logs`**: Structured application logs\n- **`otel-traces`**: Execution traces and spans\n- **`otel-metrics`**: Performance and resource metrics\n\n## 📊 Observability Features\n\n### Logs\n\n**Workflow Events**:\n```javascript\nlogger.workflow.started(workflowId, workflowName, executionId, metadata);\nlogger.workflow.completed(workflowId, workflowName, executionId, duration, metadata);\nlogger.workflow.failed(workflowId, workflowName, executionId, error, duration, metadata);\n```\n\n**Node Events**:\n```javascript\nlogger.node.started(workflowId, executionId, nodeType, nodeName, metadata);\nlogger.node.completed(workflowId, executionId, nodeType, nodeName, duration, metadata);\nlogger.node.failed(workflowId, executionId, nodeType, nodeName, error, duration, metadata);\n```\n\n**HTTP Events**:\n```javascript\nlogger.http.request(method, url, headers, metadata);\nlogger.http.response(method, url, statusCode, duration, metadata);\n```\n\n### Traces\n\nAutomatic instrumentation for:\n- HTTP requests (incoming/outgoing)\n- Database operations\n- Workflow execution spans\n- Node execution spans\n\n### Metrics\n\nCollected metrics include:\n- Request counts and durations\n- Database connection pools\n- Memory and CPU usage\n- Workflow execution statistics\n\n## 🔍 Querying Data\n\n### Sample Parseable Queries\n\n**Recent Workflow Executions**:\n```sql\nSELECT body, severity_text, time_unix_nano \nFROM \"n8n-logs\" \nWHERE body LIKE '%workflow%' \nORDER BY time_unix_nano DESC \nLIMIT 10\n```\n\n**Error Analysis**:\n```sql\nSELECT body, severity_text, time_unix_nano \nFROM \"n8n-logs\" \nWHERE severity_text = 'ERROR' \nORDER BY time_unix_nano DESC\n```\n\n**Performance Traces**:\n```sql\nSELECT span_name, duration_nanos, status_code \nFROM \"otel-traces\" \nORDER BY start_time_unix_nano DESC\n```\n\n## 🎛️ Customization\n\n### Adding Custom Logging\n\nExtend the Winston logger in `n8n-winston-logger.js`:\n\n```javascript\n// Add custom log methods\nlogger.custom = {\n  event: (eventType, data) =\u003e {\n    logger.info('Custom event', {\n      event_type: eventType,\n      ...data\n    });\n  }\n};\n```\n\n### Adding Custom Instrumentation\n\nExtend OpenTelemetry setup in `simple-winston-tracing.js`:\n\n```javascript\n// Add custom instrumentations\nconst { YourCustomInstrumentation } = require('your-instrumentation');\n\nregisterInstrumentations({\n  instrumentations: [\n    // ... existing instrumentations\n    new YourCustomInstrumentation()\n  ]\n});\n```\n\n## 🐛 Troubleshooting\n\n### Common Issues\n\n**1. Logs not appearing in Parseable**\n- Check container connectivity: `docker network ls`\n- Verify Parseable is running: `docker-compose logs parseable`\n- Check n8n logs: `docker-compose logs n8n`\n\n**2. Authentication errors**\n- Verify credentials in `docker-compose.yml`\n- Check Parseable access: `curl -u admin:admin http://localhost:8000/api/v1/logstream`\n\n**3. High memory usage**\n- Adjust batch sizes in `n8n-winston-logger.js`\n- Configure log retention in Parseable\n\n### Debug Commands\n\n```bash\n# Check service health\ndocker-compose ps\n\n# View real-time logs\ndocker-compose logs -f n8n\ndocker-compose logs -f parseable\n\n# Test Parseable API\ncurl -u admin:admin http://localhost:8000/api/v1/logstream\n\n# Check n8n database\ndocker exec -it n8n-postgres psql -U postgres -d n8n\n```\n\n## 🔐 Security Considerations\n\n### Production Deployment\n\n1. **Change Default Credentials**:\n   ```yaml\n   PARSEABLE_USERNAME: \"your-username\"\n   PARSEABLE_PASSWORD: \"your-secure-password\"\n   ```\n\n2. **Use Environment Variables**:\n   ```bash\n   export PARSEABLE_PASSWORD=$(openssl rand -base64 32)\n   ```\n\n3. **Network Security**:\n   - Use Docker networks for isolation\n   - Configure firewall rules\n   - Enable TLS/SSL for production\n\n4. **Data Retention**:\n   - Configure log rotation\n   - Set retention policies\n   - Monitor disk usage\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature-name`\n3. Commit changes: `git commit -am 'Add feature'`\n4. Push to branch: `git push origin feature-name`\n5. Submit pull request\n\n## 📝 License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- [n8n](https://n8n.io/) - Workflow automation platform\n- [Parseable](https://parseable.com/) - Log analytics platform  \n- [OpenTelemetry](https://opentelemetry.io/) - Observability framework\n- [Winston](https://github.com/winstonjs/winston) - Logging library\n\n## 📞 Support\n\n- **Issues**: [GitHub Issues](https://github.com/parseablehq/n8n-observability/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/parseablehq/n8n-observability/discussions)\n- **Parseable Community**: [Discord](https://logg.ing/community)\n\n---\n\n⭐ **Star this repo** if you find it useful!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparseablehq%2Fn8n-observability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparseablehq%2Fn8n-observability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparseablehq%2Fn8n-observability/lists"}