{"id":28560673,"url":"https://github.com/fullstorydev/fs-lexicon","last_synced_at":"2025-06-10T09:38:09.774Z","repository":{"id":291794009,"uuid":"972095431","full_name":"fullstorydev/fs-lexicon","owner":"fullstorydev","description":"IT-17600","archived":false,"fork":false,"pushed_at":"2025-05-06T14:42:05.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-06T15:52:53.529Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fullstorydev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-24T14:25:40.000Z","updated_at":"2025-05-06T14:41:44.000Z","dependencies_parsed_at":"2025-05-07T07:00:58.618Z","dependency_job_id":null,"html_url":"https://github.com/fullstorydev/fs-lexicon","commit_stats":null,"previous_names":["fullstorydev/fs-lexicon"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Ffs-lexicon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Ffs-lexicon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Ffs-lexicon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Ffs-lexicon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fullstorydev","download_url":"https://codeload.github.com/fullstorydev/fs-lexicon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Ffs-lexicon/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259049822,"owners_count":22798036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-10T09:37:32.283Z","updated_at":"2025-06-10T09:38:09.757Z","avatar_url":"https://github.com/fullstorydev.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lexicon - Multi-Cloud Serverless Function\n\n\u003cdiv align=\"center\"\u003e\n\n*A flexible serverless middleware that transforms and routes Fullstory data across multiple cloud platforms*\n\n\u003c/div\u003e\n\n## 🚀 Quick Start\n\n```bash\n# Clone the repository\ngit clone [repository-url]\n\n# Install dependencies\nnpm install\n\n# Set environment variables (.env file or export)\nexport CLOUD_PROVIDER=GCP  # Options: GCP, AZURE, AWS\nexport NODE_ENV=development\n\n# Run locally\nnpm start\n\n# Run in Docker\nnpm run docker:build\nnpm run docker:run:env\n```\n\n## 📋 Table of Contents\n- [Overview](#overview)\n- [Features](#features)\n- [Architecture](#architecture)\n- [Design Patterns](#design-patterns)\n- [Service Registry](#service-registry)\n- [Startup Sequence](#startup-sequence)\n- [Service Connectors](#service-connectors)\n- [Configuration](#configuration)\n  - [Configuration System Architecture](#configuration-system-architecture)\n  - [Using Configuration in Connectors](#using-configuration-in-connectors)\n- [Deployment](#deployment)\n- [Local Development](#local-development)\n- [Testing](#testing)\n- [Contributing](#contributing)\n\n## 📚 Overview\n\nLexicon is a multi-cloud serverless middleware that processes Fullstory data and routes it to various destinations. It serves as an intermediary layer between Fullstory analytics and downstream services, transforming and enriching data along the way.\n\n**Key Benefits:**\n- Single codebase deployable to GCP, Azure, or AWS\n- Automatic cloud provider detection\n- Flexible webhook routing system\n- Comprehensive connector ecosystem for third-party integrations\n\n## ✨ Features\n\n- **Multi-Cloud Deployment**: Deploy the same code to Google Cloud, Azure, or AWS\n- **Webhook Processing**: Handle Fullstory webhook events and route them appropriately\n- **Service Integrations**: Connect with Slack, Jira, BigQuery, Snowflake, and more\n- **Database Abstraction**: Work with multiple database backends through a common interface\n- **Robust Configuration**: Type-safe configuration with environment-specific settings\n- **Service Registry**: Centralized service management with dependency injection\n- **Controlled Startup Sequence**: Phased initialization to prevent circular dependencies\n- **Comprehensive Logging**: Structured logging with redaction of sensitive information\n- **Docker Support**: Run in containers locally or in production environments\n\n## 🏗️ Architecture\n\nLexicon uses an **adapter pattern** to deploy the same application to multiple cloud environments:\n\n```\n┌─────────────────────────────────────┐\n│  Fullstory Anywhere Activations     │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│    Configure Webhook Properties     │\n│    (Event Types, User Attributes)   │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│        Fullstory Webhook            │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│        Webhook Verification         │\n│      (Signature \u0026 Auth Check)       │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│   Cloud Adapter (GCP/AWS/Azure)     │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│      Service Registry Layer         │\n│  (Dependency Injection \u0026 Sharing)   │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│   Data Transformation \u0026 Enrichment  │\n│    (Using Connectors for Context)   │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│      Connector Information Fetch    │\n│    (Fullstory, BigQuery, etc.)      │\n└──────────────────┬──────────────────┘\n                   │\n┌──────────────────▼──────────────────┐\n│         Route Configuration         │\n│     (Destination \u0026 Format Rules)    │\n└──────────────────┬──────────────────┘\n                   │\n                   │     ◄── One behavioral webhook can feed\n                   │         many end destinations at once\n┌──────────────────▼──────────────────┐\n│          Connector Services         │\n└─────┬─────────┬─────────┬───────────┘\n      │         │         │\n      │         │         │\n┌─────▼───┐ ┌───▼───┐ ┌───▼────────┐\n│Databases│ │ APIs  │ │ Endpoints  │\n└─────────┘ └───────┘ └────────────┘\n```\n\n### Design Patterns\n\nLexicon follows several best practices and design patterns:\n\n1. **Adapter Pattern**\n   - Abstracts cloud-specific implementations\n   - Enables a single codebase across platforms\n\n2. **Singleton Pattern**\n   - Used for service connectors to maintain consistent state\n   - Example: `const fullstoryClient = new FullstoryClient(token, orgId);`\n\n3. **Factory Pattern**\n   - Creates appropriate cloud adapters at runtime\n   - Example:\n     ```javascript\n     function createCloudAdapter(provider) {\n       switch (provider.toUpperCase()) {\n         case 'GCP': return new GCPAdapter();\n         case 'AZURE': return new AzureAdapter();\n         case 'AWS': return new AWSAdapter();\n       }\n     }\n     ```\n\n4. **Builder Pattern**\n   - Used in database query construction\n   - Example: `konbini.warehouse.generateSql({ ... })`\n\n5. **Strategy Pattern**\n   - Implemented for different authentication methods\n\n6. **Service Registry Pattern**\n   - Centralizes and manages shared services\n   - Eliminates circular dependencies\n   - Simplifies testing through dependency injection\n   - Example:\n     ```javascript\n     // Register a service\n     serviceRegistry.register('config', configInstance);\n     \n     // Get a service\n     const config = serviceRegistry.get('config');\n     ```\n\n### Code Quality Principles\n\n- **DRY (Don't Repeat Yourself)**: Common functionality in utility methods\n- **Single Responsibility**: Each class/function has a focused purpose\n- **Error Handling**: Consistent error responses across endpoints\n- **Configuration Management**: Centralized through `config.js`\n- **Protected Methods**: Private methods prefixed with underscore\n\n## 📦 Service Registry\n\nThe Service Registry is a core architectural component that provides centralized service management and dependency injection:\n\n```\n┌─────────────────────────────────────┐\n│          serviceRegistry.js         │\n│     (Central Service Repository)    │\n└─────┬─────────┬─────────┬───────────┘\n      │         │         │\n      │         │         │\n┌─────▼───┐ ┌───▼───┐ ┌───▼────────┐\n│  config │ │initial│ │ connectors │\n│         │ │ization│ │            │\n└─────────┘ └───────┘ └────────────┘\n```\n\n**Key Benefits:**\n\n1. **Dependency Management**: Centralized management of service instances\n2. **Circular Dependency Prevention**: Break dependency cycles between modules\n3. **Testing Support**: Easy mocking of services during unit tests\n4. **Runtime Flexibility**: Services can be dynamically registered and replaced\n\n**Usage Examples:**\n\n```javascript\n// In index.js during application startup\nserviceRegistry.register('config', configInstance);\nserviceRegistry.register('initialization', initializationInstance);\n\n// In a connector or webhook handler\nconst config = serviceRegistry.get('config');\nconst initialization = serviceRegistry.get('initialization');\n\n// Check if a service exists\nif (serviceRegistry.has('snowflake')) {\n  const snowflake = serviceRegistry.get('snowflake');\n  // Use the snowflake connector\n}\n```\n\n**Service Registry API:**\n\n```javascript\n// Register a service\nserviceRegistry.register('serviceName', serviceInstance);\n\n// Get a registered service\nconst service = serviceRegistry.get('serviceName');\n\n// Check if a service exists\nconst exists = serviceRegistry.has('serviceName');\n\n// Get all registered service names\nconst services = serviceRegistry.getServiceNames();\n```\n\n## 🚀 Startup Sequence\n\nLexicon implements a controlled, phased initialization process that prevents circular dependencies and ensures services are initialized in the correct order:\n\n```\n┌─────────────────────────────────────┐\n│            startup.js               │\n│      (Initialization Manager)       │\n└─────────────────┬─────────────────┬─┘\n                  │                 │\n                  ▼                 ▼\n┌─────────────────────────┐ ┌───────────────────────┐\n│     Initialization      │ │    Service Registry   │\n│        Phases           │ │    Registration       │\n└──────────┬──────────────┘ └───────────┬───────────┘\n           │                            │\n           ▼                            ▼\n┌──────────────────────────────────────────────────┐\n│                 Phase 1: Core Services           │\n│          config, initialization, middleware      │\n└──────────────────────────┬───────────────────────┘\n                           │\n                           ▼\n┌──────────────────────────────────────────────────┐\n│           Phase 2: Database \u0026 Resources          │\n│       konbini, snowflake, bigQuery, workspace    │\n└──────────────────────────┬───────────────────────┘\n                           │\n                           ▼\n┌──────────────────────────────────────────────────┐\n│          Phase 3: External Integrations          │\n│           fullstory, slack, atlassian            │\n└──────────────────────────┬───────────────────────┘\n                           │\n                           ▼\n┌──────────────────────────────────────────────────┐\n│            Phase 4: Webhooks \u0026 Routes            │\n│                   webhookRouter                  │\n└──────────────────────────┬───────────────────────┘\n                           │\n                           ▼\n┌──────────────────────────────────────────────────┐\n│              Phase 5: Cloud Adapter              │\n│                    cloudAdapter                  │\n└──────────────────────────┬───────────────────────┘\n                           │\n                           ▼\n┌──────────────────────────────────────────────────┐\n│              Initialization Summary              │\n│        Status reporting for all components       │\n└──────────────────────────────────────────────────┘\n```\n\n### Key Features\n\n1. **Phased Initialization**: Services are started in a specific order to prevent dependency issues\n2. **Graceful Fallbacks**: Services can operate even when dependencies aren't fully initialized\n3. **Robust Error Handling**: Initialization failures in one service don't crash the entire application\n4. **Status Reporting**: Comprehensive logging of initialization state for all components\n\n### Initialization Process\n\n1. **Core Services**: Essential services like config and the service registry itself are initialized first\n2. **Database \u0026 Resources**: Data storage and resource connectors are initialized next\n3. **External Integrations**: Third-party service connectors like Fullstory, Slack, and Atlassian\n4. **Webhooks \u0026 Routes**: API endpoints and webhook handlers are set up\n5. **Cloud Adapter**: The cloud-specific adapter (GCP, AWS, Azure) is initialized last\n\n### Startup Manager API\n\n```javascript\n// In index.js\nconst startup = require('./startup');\n\n// Initialize all services in the correct sequence\nawait startup.initialize();\n\n// Get initialization status\nconst status = startup.getStatus();\nconsole.log(`Initialization complete: ${status.initialized}`);\nconsole.log(`Services registered: ${status.serviceCount}`);\n```\n\n### Error Handling During Initialization\n\nThe startup sequence is designed to be resilient:\n\n1. **Temporary Placeholder**: The application exports a placeholder function immediately, which responds with a 503 status code during initialization\n2. **Individual Service Failures**: If a service fails to initialize, the system continues with the next service\n3. **Graceful Degradation**: The application functions with reduced capabilities when non-critical services fail\n4. **Comprehensive Logging**: Detailed error logs help identify initialization issues\n\n## 🔌 Service Connectors\n\nLexicon includes several service connectors to integrate with external systems:\n\n### Fullstory Integration\n\nInteract with Fullstory's behavioral data platform:\n\n```javascript\n// Get session data\nconst summary = await Fullstory.getSessionSummary(userId, sessionId);\n\n// Generate session replay link\nconst link = Fullstory.getSessionLink(userId, sessionId);\n```\n\n### Database Support\n\nWork with multiple database backends through Konbini:\n\n```javascript\n// BigQuery example with named parameters\nconst { sql, params, parameterTypes } = konbini.warehouse.generateSql({\n  databaseType: 'bigquery',\n  operation: 'insert',\n  table: 'fs_data_destinations.lead_info',\n  columns: ['session_id', 'visitor_id'],\n  data: {\n    session_id: data.session_id,\n    visitor_id: data.uid\n  }\n});\n\n// Snowflake example with positional parameters\nawait snowflake.withConnection(async (connector) =\u003e {\n  await connector.executeQuery(sql, bindings);\n});\n```\n\n### Notification Integrations\n\nSend notifications to various channels:\n\n```javascript\n// Send Slack notification\nawait slack.sendWebHook({\n  text: \"New customer feedback received\",\n  blocks: [/* Block Kit content */]\n});\n\n// Create Jira ticket\nconst ticket = await atlassian.jira.createTicket({\n  fields: {\n    summary: `${body.name} - ${body.user.email}`,\n    description: rundown,\n    project: { key: projectKey },\n    issuetype: { id: issueTypeId }\n  }\n});\n```\n\n## ⚙️ Configuration\n\nLexicon provides a robust configuration system through `config.js`:\n\n```javascript\n// Get a configuration value with default\nconst port = config.get('port', 8080);\n\n// Get a typed configuration value\nconst debugEnabled = config.getBoolean('enable_detailed_errors', false);\nconst timeout = config.getNumber('timeout', 30);\n\n// Check environment\nif (config.isCloudProvider('GCP')) {\n  // GCP-specific code\n}\n```\n\nConfiguration can be set through:\n1. Environment variables\n2. `.env` file (development only)\n3. Cloud provider environment settings\n\n### Configuration System Architecture\n\nLexicon follows a layered configuration architecture to ensure consistency and validation across all services:\n\n```\n┌─────────────────────────────┐\n│         config.js           │\n│ (Core Configuration System) │\n└─────────────┬───────────────┘\n              │\n              │ provides configuration\n              ▼\n┌─────────────────────────────┐\n│  connectorConfigValidator   │\n│   (Validation \u0026 Typing)     │\n└─────────────┬───────────────┘\n              │\n              │ provides validation services\n              ▼\n┌─────────────────────────────┐\n│      connectorBase.js       │\n│  (Common Connector Logic)   │\n└─────────────┬───────────────┘\n              │\n              │ extends\n     ┌────────┼────────┬───────────┐\n     │        │        │           │\n     ▼        ▼        ▼           ▼\n┌─────────┐ ┌─────┐ ┌────────-┐ ┌────────┐\n│Snowflake│ │Slack│ │Fullstory│ │  Other │\n└─────────┘ └─────┘ └────────-┘ └────────┘\n```\n\n**Key Components:**\n\n1. **config.js**: Singleton configuration manager that handles environment detection, environment variables, and cloud platform specifics.\n\n2. **connectorConfigValidator.js**: Validates configuration values, tracks errors, and provides proper type conversion.\n\n3. **connectorBase.js**: Provides a consistent interface to all connectors, integrating the validator with convenient helper methods.\n\n4. **Individual Connectors**: Extend ConnectorBase to inherit the configuration system.\n\n### Using Configuration in Connectors\n\nAll service connectors use a consistent pattern to access configuration:\n\n```javascript\n// Creating a new connector that extends ConnectorBase\nclass SnowflakeConnector extends ConnectorBase {\n  constructor() {\n    // Initialize with connector name\n    super('Snowflake');\n    \n    // Get configuration through the base class methods\n    this.config = {\n      account: this.getConfig('snowflake_account_identifier'),\n      username: this.getConfig('snowflake_user'),\n      warehouse: this.getConfig('snowflake_warehouse'),\n      // Get more configuration values as needed\n    };\n    \n    // Check if configuration is valid using the validator\n    this.isConfigured = this.validator.checkIsConfigured();\n  }\n  \n  // Access configuration in methods\n  async connect() {\n    if (!this.isConfigured) {\n      return Promise.reject(new Error('Snowflake is not properly configured'));\n    }\n    \n    // Use configuration values\n    // ...\n  }\n}\n```\n\nThis approach ensures:\n- Consistent configuration across all connectors\n- Proper validation and error tracking\n- Type-safe configuration access\n- Environment-specific configuration\n\n## 🚢 Deployment\n\n### Google Cloud\n\n```bash\n# Deploy to Cloud Functions\ngcloud functions deploy lexicon \\\n  --runtime=nodejs18 \\\n  --trigger-http \\\n  --set-env-vars=\"cloud_provider=GCP\"\n\n# Deploy to Cloud Run\ngcloud run deploy lexicon \\\n  --image=gcr.io/[PROJECT_ID]/lexicon \\\n  --set-env-vars=\"cloud_provider=GCP\"\n```\n\n### Azure\n\n```bash\n# Deploy to Azure Functions\nfunc azure functionapp publish [APP_NAME] \\\n  --javascript \\\n  --set cloud_provider=AZURE\n\n# Deploy to Azure App Service\naz webapp deploy \\\n  --resource-group [RESOURCE_GROUP] \\\n  --name [APP_NAME] \\\n  --src-path . \\\n  --type zip \\\n  --env-vars cloud_provider=AZURE\n```\n\n### AWS\n\n```bash\n# Deploy to AWS App Runner\nnpm run deploy:aws\n# (Requires AWS_ACCOUNT_ID and AWS_REGION env variables)\n```\n\n## 🐳 Docker Usage\n\nLexicon provides several Docker-related npm scripts:\n\n```bash\n# Build and run\nnpm run docker:build\nnpm run docker:run:env  # Uses env vars from .env file\n\n# Provider-specific containers\nnpm run docker:run:gcp  # Runs on port 8080\nnpm run docker:run:azure  # Runs on port 8080 \nnpm run docker:run:aws  # Runs on port 8080\n```\n\n## 💻 Local Development\n\nFor local development:\n\n```bash\n# Set cloud provider\nexport CLOUD_PROVIDER=GCP\n\n# Run with local environment\nnpm start\n\n# Run with Docker and live reload\nnpm run dev:mount\n\n# Auto-restart on changes\nnpm run dev:docker\n```\n\n## 🧪 Testing\n\nLexicon includes comprehensive tests:\n\n```bash\n# Run all tests\nnpm test\n\n# Run specific test categories\nnpm run test:unit\nnpm run test:integration\nnpm run test:adapters\n\n# Test specific cloud providers\nnpm run test:gcp\nnpm run test:azure\nnpm run test:aws\n\n# Watch mode for development\nnpm run test:watch\n```\n\nFor detailed test documentation, see the [Testing Guidelines](./tests/README.md).\n\n## 🤝 Contributing\n\nWhen adding new functionality to Lexicon, follow these guidelines:\n\n1. **New Webhook Handlers**:\n   - Extend `WebhookBase` class\n   - Use the logger and errorHandler\n   - Follow existing patterns\n\n2. **New Connector Integrations**:\n   - Create a dedicated file that extends `ConnectorBase`\n   - Use the configuration validation system through `getConfig()` methods\n   - Implement comprehensive error handling\n   - Document with JSDoc comments\n\n3. **Cloud Provider Support**:\n   - Extend the appropriate adapter\n   - Test thoroughly in target environment\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffullstorydev%2Ffs-lexicon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffullstorydev%2Ffs-lexicon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffullstorydev%2Ffs-lexicon/lists"}