{"id":29127818,"url":"https://github.com/arifmahmudrana/node-text-analyzer","last_synced_at":"2026-04-11T11:05:04.403Z","repository":{"id":301333139,"uuid":"993304942","full_name":"arifmahmudrana/node-text-analyzer","owner":"arifmahmudrana","description":"A simple node, express, TypeScript, MongoDB text analyzer with log visualization using PLG stack","archived":false,"fork":false,"pushed_at":"2025-08-16T07:17:22.000Z","size":238,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-16T08:33:18.671Z","etag":null,"topics":["cicd","event-driven","github-actions","grafana","jest","jest-tests","logging","loki","mongodb","node-js","nodejs","promtail","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arifmahmudrana.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-30T15:00:53.000Z","updated_at":"2025-08-16T07:14:45.000Z","dependencies_parsed_at":"2025-08-16T08:22:14.235Z","dependency_job_id":"1c55fdca-6f3b-421b-9941-fbc5ab50f483","html_url":"https://github.com/arifmahmudrana/node-text-analyzer","commit_stats":null,"previous_names":["arifmahmudrana/node-text-analyzer"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/arifmahmudrana/node-text-analyzer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arifmahmudrana%2Fnode-text-analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arifmahmudrana%2Fnode-text-analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arifmahmudrana%2Fnode-text-analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arifmahmudrana%2Fnode-text-analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arifmahmudrana","download_url":"https://codeload.github.com/arifmahmudrana/node-text-analyzer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arifmahmudrana%2Fnode-text-analyzer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31677819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-11T08:18:19.405Z","status":"ssl_error","status_checked_at":"2026-04-11T08:17:08.892Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cicd","event-driven","github-actions","grafana","jest","jest-tests","logging","loki","mongodb","node-js","nodejs","promtail","typescript"],"created_at":"2025-06-30T01:01:57.454Z","updated_at":"2026-04-11T11:05:04.386Z","avatar_url":"https://github.com/arifmahmudrana.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Text Analyzer API [![codecov](https://codecov.io/gh/arifmahmudrana/node-text-analyzer/branch/main/graph/badge.svg)](https://codecov.io/gh/arifmahmudrana/node-text-analyzer) [![Security](https://img.shields.io/github/actions/workflow/status/arifmahmudrana/node-text-analyzer/ci-cd.yml?branch=main\u0026label=Tests%20and%20Scans\u0026logo=github)](https://github.com/arifmahmudrana/node-text-analyzer/security/code-scanning)\n\nA robust Node.js/TypeScript API for analyzing text content with asynchronous processing and comprehensive monitoring capabilities.\n\n## Features\n\n- **Text Analysis**: Automatically analyzes text for word count, character count, sentences, paragraphs, and longest words per paragraph\n- **Asynchronous Processing**: Uses event-driven architecture for non-blocking text analysis\n- **MongoDB Integration**: Persistent storage with Mongoose ODM\n- **Comprehensive API**: RESTful endpoints with proper validation and error handling\n- **Pagination Support**: Configurable pagination for listing texts\n- **Log Monitoring**: Integrated Grafana + Loki + Promtail stack for log visualization\n- **TypeScript**: Fully typed codebase for better development experience\n- **Docker Support**: Complete containerized setup for monitoring stack\n\n## Project Architecture\n\n### Core Components\n\n```\nsrc/\n├── app.ts                    # Express application setup\n├── server.ts                 # Server entry point with database connection\n├── config/\n│   ├── config.ts            # Application configuration\n│   └── database.ts          # MongoDB connection setup\n├── controllers/\n│   └── textController.ts    # Text CRUD operations\n├── routes/\n│   └── textRoutes.ts        # API route definitions\n├── models/\n│   └── Text.ts              # MongoDB schema definition\n├── middlewares/\n│   ├── errorHandler.ts      # Global error handling\n│   ├── pagination.ts        # Pagination middleware\n│   └── validation.ts        # Request validation middleware\n├── events/\n│   └── textProcessor.ts     # Asynchronous text processing\n├── helpers/\n│   └── text.ts              # Text analysis utilities\n├── validations/\n│   └── textValidation.ts    # Joi validation schemas\n└── types/\n    ├── index.ts             # Main TypeScript type definitions\n    └── pagination.ts        # Pagination-specific types\n\n```\n\n### Technology Stack\n\n- **Runtime**: Node.js 22+\n- **Language**: TypeScript\n- **Framework**: Express.js\n- **Database**: MongoDB with Mongoose\n- **Validation**: Joi\n- **Testing**: Jest\n- **Monitoring**: Grafana + Loki + Promtail\n- **Process Management**: Event-driven architecture\n\n## Getting Started\n\n### Prerequisites\n\n- Node.js 22 or higher\n- MongoDB running locally or remote connection\n- Docker and Docker Compose (for monitoring stack)\n\n### Installation\n\n1. **Clone the repository**\n   ```bash\n   git clone https://github.com/arifmahmudrana/node-text-analyzer.git\n   cd node-text-analyzer\n   ```\n\n2. **Install dependencies**\n   ```bash\n   npm install\n   ```\n\n3. **Environment setup**\n   ```bash\n   cp .env.example .env\n   ```\n   \n   Configure your `.env` file:\n   ```env\n   PORT=3000\n   NODE_ENV=development\n   MONGODB_URI=mongodb://localhost:27017/textapi\n   ```\n\n### Running the Application\n\n#### Development Mode\n```bash\n# Run with auto-reload\nnpm run dev\n\n# Run with logging\nnpm run dev:log\n```\n\n#### Production Mode\n```bash\n# Build and run\nnpm run build\nnpm start\n\n# Run with logging\nnpm run start:log\n\n# Production with environment variable\nnpm run start:prod\n```\n\n### Monitoring Stack (Optional)\n\nStart the complete monitoring stack with Docker Compose:\n\n```bash\ndocker-compose up --build\n```\n\nThis will start:\n- **Loki**: Log aggregation service (port 3100)\n- **Grafana**: Log visualization dashboard (port 3001)\n- **Promtail**: Log collector\n\nAccess Grafana at `http://localhost:3001` (no authentication required in development).\n\n## API Documentation\n\n### Base URL\n```\nhttp://localhost:3000/api\n```\n\n### Health Check\n\n#### GET /health\nCheck API status and environment information.\n\n**Response:**\n```json\n{\n  \"success\": true,\n  \"message\": \"API is running successfully\",\n  \"data\": {\n    \"timestamp\": \"2025-06-02T10:30:00.000Z\",\n    \"environment\": \"development\"\n  }\n}\n```\n\n### Text Operations\n\n#### POST /api/texts\nCreate a new text for analysis.\n\n**Request Body:**\n```json\n{\n  \"text\": \"Your text content here. This can be multiple paragraphs with various sentences.\"\n}\n```\n\n**Response:**\n```json\n{\n  \"success\": true,\n  \"message\": \"Text created successfully\",\n  \"data\": {\n    \"_id\": \"60d5f484f1b2c8b1f8e4e1a1\",\n    \"text\": \"Your text content here...\",\n    \"done\": false,\n    \"numberOfWords\": 0,\n    \"numberOfCharacters\": 0,\n    \"numberOfSentences\": 0,\n    \"numberOfParagraphs\": 0,\n    \"longestWordsInParagraphs\": [],\n    \"createdAt\": \"2025-06-02T10:30:00.000Z\",\n    \"updatedAt\": \"2025-06-02T10:30:00.000Z\"\n  }\n}\n```\n\n**Note**: The text analysis runs asynchronously. The `done` field will be `false` initially and `true` once processing is complete.\n\n#### GET /api/texts/:id\nRetrieve a specific text by ID (only returns completed analyses).\n\n**Parameters:**\n- `id` (string): MongoDB ObjectId of the text\n\n**Response:**\n```json\n{\n  \"success\": true,\n  \"message\": \"Text retrieved successfully\",\n  \"data\": {\n    \"_id\": \"60d5f484f1b2c8b1f8e4e1a1\",\n    \"text\": \"Your text content here...\",\n    \"done\": true,\n    \"numberOfWords\": 45,\n    \"numberOfCharacters\": 234,\n    \"numberOfSentences\": 3,\n    \"numberOfParagraphs\": 2,\n    \"longestWordsInParagraphs\": [\"paragraph\", \"sentences\"],\n    \"createdAt\": \"2025-06-02T10:30:00.000Z\",\n    \"updatedAt\": \"2025-06-02T10:30:15.000Z\"\n  }\n}\n```\n\n**Error Response (404):**\n```json\n{\n  \"success\": false,\n  \"message\": \"Text not found\"\n}\n```\n\n#### GET /api/texts\nList texts with pagination and filtering.\n\n**Query Parameters:**\n- `done` (optional): Filter by processing status\n  - `true`: Only completed analyses\n  - `false`: Only pending analyses  \n  - `all`: All texts (default)\n- `page` (optional): Page number (default: 1)\n- `limit` (optional): Items per page (default: 10, max: 100)\n- `order` (optional): Sort order - `asc` or `desc` (default: desc)\n- `orderBy` (optional): Sort field - `createdAt` or `updatedAt` (default: createdAt)\n\n**Example Requests:**\n```bash\n# Get all texts (first page)\nGET /api/texts\n\n# Get completed texts only\nGET /api/texts?done=true\n\n# Get second page with 20 items\nGET /api/texts?page=2\u0026limit=20\n\n# Get texts ordered by update time (ascending)\nGET /api/texts?orderBy=updatedAt\u0026order=asc\n\n# Combined filters\nGET /api/texts?done=true\u0026page=1\u0026limit=50\u0026orderBy=createdAt\u0026order=desc\n```\n\n**Response:**\n```json\n{\n  \"success\": true,\n  \"message\": \"Texts retrieved successfully\",\n  \"data\": [\n    {\n      \"_id\": \"60d5f484f1b2c8b1f8e4e1a1\",\n      \"text\": \"Sample text...\",\n      \"done\": true,\n      \"numberOfWords\": 45,\n      \"numberOfCharacters\": 234,\n      \"numberOfSentences\": 3,\n      \"numberOfParagraphs\": 2,\n      \"longestWordsInParagraphs\": [\"paragraph\", \"sentences\"],\n      \"createdAt\": \"2025-06-02T10:30:00.000Z\",\n      \"updatedAt\": \"2025-06-02T10:30:15.000Z\"\n    }\n  ],\n  \"meta\": {\n    \"currentPage\": 1,\n    \"totalPages\": 5,\n    \"totalItems\": 47,\n    \"itemsPerPage\": 10,\n    \"hasNextPage\": true,\n    \"hasPreviousPage\": false\n  }\n}\n```\n\n## Text Analysis Details\n\nThe API analyzes text content and provides the following metrics:\n\n- **numberOfWords**: Total word count (splits on whitespace)\n- **numberOfCharacters**: Total character count including spaces\n- **numberOfSentences**: Count of sentences (splits on `.`, `!`, `?`)\n- **numberOfParagraphs**: Count of paragraphs (splits on double newlines)\n- **longestWordsInParagraphs**: Array of longest words from each paragraph\n\n### Analysis Process\n\n1. Text is saved immediately with `done: false`\n2. Asynchronous event is triggered for processing\n3. Text analysis runs in the background\n4. Document is updated with results and `done: true`\n5. Completed analysis is available via API\n\n## Development\n\n### Available Scripts\n\n```bash\n# Development\nnpm run dev              # Start with nodemon\nnpm run dev:log          # Start with logging\n\n# Production\nnpm run build            # Compile TypeScript\nnpm start                # Start compiled server\nnpm run start:log        # Start with logging\nnpm run start:prod       # Production with logging\n\n# Testing\nnpm test                 # Run Jest tests\nnpm run test:watch       # Run tests in watch mode\n\n# Code Quality\nnpm run lint             # Run ESLint\n```\n\n### VS Code Debugging\n\nThe project includes VS Code launch configurations:\n\n1. **Debug Server (ts-node)**: Debug the main server\n2. **Debug Jest Tests**: Debug all tests\n3. **Jest Current File**: Debug current test file\n\n### Project Structure Details\n\n#### Configuration (`src/config/`)\n- `config.ts`: Centralized configuration management\n- `database.ts`: MongoDB connection with error handling\n\n#### Models (`src/models/`)\n- `Text.ts`: Mongoose schema for text documents\n\n#### Controllers (`src/controllers/`)\n- `textController.ts`: Business logic for text operations\n\n#### Middlewares (`src/middlewares/`)\n- `errorHandler.ts`: Global error handling\n- `pagination.ts`: Reusable pagination logic\n- `validation.ts`: Request validation middleware using Joi schemas\n\n#### Events (`src/events/`)\n- `textProcessor.ts`: Event-driven text analysis processor\n\n#### Helpers (`src/helpers/`)\n- `text.ts`: Pure functions for text analysis\n\n#### Validations (`src/validations/`)\n- `textValidation.ts`: Joi schemas for request validation including `createTextSchema`\n\n#### Types (`src/types/`)\n- `index.ts`: Main TypeScript type definitions\n- `pagination.ts`: Pagination-specific type definitions\n\n## Log Monitoring\n\n### Grafana Dashboard\n\n1. Start monitoring stack: `docker-compose up --build`\n2. Access Grafana: `http://localhost:3001`\n3. Loki datasource is pre-configured\n4. View application logs in real-time\n\n### Log Locations\n\n- Development logs: `logs/app-dev.log`\n- Production logs: `logs/app-prod.log`\n\n## Error Handling\n\nThe API includes error handling:\n\n- **Validation Errors**: Proper validation with Joi\n- **Database Errors**: MongoDB connection and query errors\n- **Not Found**: 404 responses for missing resources\n- **Server Errors**: 500 responses with error details\n- **Graceful Shutdown**: Proper cleanup on process termination\n\n## Performance Considerations\n\n- **Asynchronous Processing**: Text analysis doesn't block API responses\n- **Database Indexes**: Optimized queries with proper indexing\n- **Lean Queries**: Using `lean()` for better performance\n- **Pagination**: Efficient data retrieval with limits\n- **Event-driven Architecture**: Non-blocking processing pipeline\n\n## Environment Variables\n\n```env\nPORT=3000                                    # Server port\nNODE_ENV=development                         # Environment (development/production)\nMONGODB_URI=mongodb://localhost:27017/textapi  # MongoDB connection string e.g. mongodb://root:123@localhost:27027/textapi?authSource=admin here root is username 123 is password and authSource admin set for authentication\n```\n\n## Testing\n\nRun the test suite:\n\n```bash\nnpm test\n```\n\nThe project uses Jest for testing with configurations for:\n- Unit tests for individual functions\n- Mocking for external dependencies\n\n## Run using a k8s cluster \n\nIf you are using arm architecture:\n\n```bash\n./k8s-local-dev-up.sh # to up the environment it will spin up MongoDB, PLG stack and create deployment all in a separate namespace\n```\n```bash\n./k8s-local-dev-down.sh # to cleanup everything \n```\nIf you are using a different architecture or then update the `k8s/mongodb-values.yaml` file image option and you can change the image tag in the `k8s/app/app-deployment.yaml` file and adjust.\n\n![demo](./k8s-local-demo.gif)\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature-name`\n3. Make your changes\n4. Add tests for new functionality\n5. Run tests: `npm test`\n6. Run linting: `npm run lint`\n7. Commit changes: `git commit -am 'Add feature'`\n8. Push to branch: `git push origin feature-name`\n9. Submit a pull request\n\n## License\n\nBSD 3-Clause License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\nFor issues and questions:\n- Open an issue on [GitHub](https://github.com/arifmahmudrana/node-text-analyzer/issues)\n- Check existing documentation\n- Review the code examples above\n\n---\n\n**Happy analyzing! 📊📝**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farifmahmudrana%2Fnode-text-analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farifmahmudrana%2Fnode-text-analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farifmahmudrana%2Fnode-text-analyzer/lists"}