{"id":23708307,"url":"https://github.com/muthuri-dev/product-prices-aggregator-api","last_synced_at":"2025-06-16T15:11:02.538Z","repository":{"id":270191241,"uuid":"907274149","full_name":"muthuri-dev/product-prices-aggregator-api","owner":"muthuri-dev","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-29T07:30:02.000Z","size":659,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-23T04:26:38.681Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muthuri-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-23T08:10:32.000Z","updated_at":"2024-12-29T07:30:06.000Z","dependencies_parsed_at":"2024-12-29T08:23:52.649Z","dependency_job_id":"65422d46-2e4a-4b51-9d8c-a9cc5f0cdea6","html_url":"https://github.com/muthuri-dev/product-prices-aggregator-api","commit_stats":null,"previous_names":["muthuri-dev/product-prices-aggregator-api"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/muthuri-dev/product-prices-aggregator-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muthuri-dev%2Fproduct-prices-aggregator-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muthuri-dev%2Fproduct-prices-aggregator-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muthuri-dev%2Fproduct-prices-aggregator-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muthuri-dev%2Fproduct-prices-aggregator-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muthuri-dev","download_url":"https://codeload.github.com/muthuri-dev/product-prices-aggregator-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muthuri-dev%2Fproduct-prices-aggregator-api/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260183170,"owners_count":22971196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-30T17:50:46.604Z","updated_at":"2025-06-16T15:11:02.494Z","avatar_url":"https://github.com/muthuri-dev.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Product Prices Aggregator API\n\nThis project is a GraphQL API designed for aggregating product prices, monitoring competitor prices, fetching product\ndetails from the Bright Data API, and managing user authentication. The system incorporates caching with Redis, indexing\nwith OpenSearch, and notification services via Kafka.\n\n---\n\n## Features\n\n- Fetch product data from Bright Data.\n- Cache results in Redis for faster subsequent access.\n- Index product data in OpenSearch for efficient search.\n- Provide GraphQL queries for searching products.\n- Notify users when price thresholds are met.\n- User authentication with JWT-based login and role-based access.\n\n---\n\n## Technologies\n\n- **NestJS**: Framework for building the application.\n- **Redis**: Caching mechanism.\n- **OpenSearch**: Indexing and searching product data.\n- **Bright Data API**: Fetch product data.\n- **Kafka**: Notification service.\n- **GraphQL**: API interface.\n- **JWT**: Authentication mechanism.\n\n---\n\n## Setup Instructions\n\n### Prerequisites\n\n- Node.js\n- Redis\n- OpenSearch\n- Kafka broker\n- Bright Data API credentials\n\n### Environment Variables\n\nCreate a `.env` file in the root directory:\n\n```env\nAMAZON_API_URL=https://api.brightdata.com\nAMAZON_API_TOKEN=\u003cyour_bright_data_token\u003e\nREDIS_URL=redis://localhost:6379\nOPEN_SEARCH_URL=https://your-opensearch-url\nOPEN_SEARCH_USERNAME=\u003cyour_opensearch_username\u003e\nOPEN_SEARCH_PASSWORD=\u003cyour_opensearch_password\u003e\nKAFKA_BROKER=\u003cyour_kafka_broker\u003e\nJWT_SECRET=\u003cyour_jwt_secret\u003e\nJWT_EXPIRES_IN=3600\n```\n\n### Installation\n\n1. Clone the repository:\n   ```bash\n   git clone \u003crepository_url\u003e\n   cd product-prices-aggregator-api\n   ```\n\n2. Install dependencies:\n   ```bash\n   npm install\n   ```\n\n3. Start Redis and OpenSearch services.\n\n4. Run the application:\n   ```bash\n   npm run start:dev\n   ```\n\n---\n\n## Project Structure\n\n```\nsrc/\n├── amazon\n│   ├── amazon.module.ts\n│   ├── amazon.service.ts\n├── auth\n│   ├── auth.module.ts\n│   ├── auth.service.ts\n│   ├── auth.resolver.ts\n│   ├── jwt.strategy.ts\n│   ├── gql-auth.guard.ts\n├── bright-data\n│   ├── bright-data.service.ts\n├── notifications\n│   ├── notifications.module.ts\n│   ├── notifications.service.ts\n├── product\n│   ├── product.module.ts\n│   ├── product.service.ts\n│   ├── product.resolver.ts\n├── redis\n│   ├── redis.module.ts\n│   ├── redis.service.ts\n├── search\n│   ├── search.module.ts\n│   ├── search.service.ts\n├── app.module.ts\n```\n\n---\n\n## API Endpoints\n\n### GraphQL Queries\n\n#### 1. Search Products by Title\n\n```graphql\nquery {\n    searchProductsByTitle(title: \"smart watch\") {\n        title\n        price\n        url\n    }\n}\n```\n\n#### 2. Fetch Products by Keywords\n\n```graphql\nquery {\n    fetchProductsByKeywords(keywords: \"smart watch\", pagesToSearch: 2) {\n        title\n        price\n        url\n    }\n}\n```\n\n#### 3. Notify Price Drop\n\n```graphql\nmutation {\n    notifyPriceDrop(\n        product: {\n            title: \"smart watch\",\n            price: 199.99,\n            url: \"https://www.amazon.com/smart-watch\"\n        },\n        userId: \"user123\",\n        threshold: 200\n    )\n}\n```\n\n#### 4. Register User\n\n```graphql\nmutation {\n    registerUser(input: {\n        username: \"testuser\",\n        email: \"testuser@example.com\",\n        password: \"securepassword\"\n    }) {\n        id\n        username\n        email\n    }\n}\n```\n\n#### 5. Login User\n\n```graphql\nmutation {\n    loginUser(input: {\n        email: \"testuser@example.com\",\n        password: \"securepassword\"\n    }) {\n        accessToken\n    }\n}\n```\n\n---\n\n## Flow of Data\n\n### Description\n\n1. **Authentication**: User logs in or registers via the GraphQL API.\n2. **Client Query**: A user queries for product data via GraphQL.\n3. **Redis Check**: The system checks Redis for cached results.\n4. **OpenSearch Check**: If not in Redis, it checks OpenSearch.\n5. **Bright Data API**: If not in OpenSearch, it fetches data from Bright Data.\n6. **Cache and Index**: The fetched data is cached in Redis and indexed in OpenSearch.\n7. **Response**: The system returns the results to the client.\n\n### Mermaid Flowchart\n\n```mermaid\nflowchart TD\n    A[User Login/Register] --\u003e B[GraphQL Query]\n    B --\u003e C[Redis Check]\n    C -- Hit --\u003e D[Return Cached Results]\n    C -- Miss --\u003e E[OpenSearch Check]\n    E -- Hit --\u003e D[Return Indexed Results]\n    E -- Miss --\u003e F[Fetch from Bright Data]\n    F --\u003e G[Cache in Redis]\n    F --\u003e H[Index in OpenSearch]\n    G --\u003e I[Return Fetched Results]\n    H --\u003e I\n```\n\n---\n\n## Key Services\n\n### **AuthService**\n\nHandles user authentication and authorization.\n\n#### Methods:\n\n- `registerUser(input: CreateUserInput): Promise\u003cUser\u003e`\n- `loginUser(input: LoginUserInput): Promise\u003c{ accessToken: string }\u003e`\n\n### **AmazonService**\n\nHandles interactions with the Bright Data API.\n\n#### Methods:\n\n- `fetchProducts(keywords: string, domain: string, pagesToSearch: number): Promise\u003cany[]\u003e`\n- `getCachedProducts(keywords: string): Promise\u003cany[]\u003e`\n- `cacheProducts(keywords: string, products: any[]): Promise\u003cvoid\u003e`\n\n### **SearchService**\n\nManages OpenSearch interactions.\n\n#### Methods:\n\n- `searchByTitle(title: string): Promise\u003cany[]\u003e`\n- `indexProduct(product: any): Promise\u003cvoid\u003e`\n\n### **NotificationsService**\n\nHandles user notifications via Kafka.\n\n#### Methods:\n\n- `sendNotification(type: string, payload: any): Promise\u003cvoid\u003e`\n\n### **BrightDataService**\n\nHandles Bright Data API interactions.\n\n#### Methods:\n\n- `triggerDataCollection(inputs: any[]): Promise\u003cstring\u003e`\n- `fetchData(snapshotId: string): Promise\u003cany[]\u003e`\n\n---\n\n## Deployment\n\n### Docker Image\n\nThe application is published as a Docker image and can be pulled using the following command:\n\n```bash\ndocker pull ghcr.io/muthuri-dev/product-price-aggregator-api:0.1\n```\n\n---\n\n## Future Enhancements\n\n- Integrate real-time data updates using WebSockets.\n- Add user role management for fine-grained access control.\n- Enhance notification system for real-time updates.\n\n---\n\n## License\n\nThis project is licensed under the MIT License.\n\n---\n\n## Contributing\n\nContributions are welcome! Please fork the repository and submit a pull request.\n\n---\n\nFor any questions, feel free to contact.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuthuri-dev%2Fproduct-prices-aggregator-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuthuri-dev%2Fproduct-prices-aggregator-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuthuri-dev%2Fproduct-prices-aggregator-api/lists"}