{"id":34827388,"url":"https://github.com/profullstack/link-directories","last_synced_at":"2026-05-21T13:07:56.667Z","repository":{"id":320499977,"uuid":"1081877216","full_name":"profullstack/link-directories","owner":"profullstack","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-24T05:53:45.000Z","size":108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-27T00:52:43.054Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/profullstack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-23T12:21:15.000Z","updated_at":"2025-10-24T05:53:49.000Z","dependencies_parsed_at":"2025-10-24T06:13:53.122Z","dependency_job_id":"e6df16fc-4999-47c1-a80c-91975010d1d2","html_url":"https://github.com/profullstack/link-directories","commit_stats":null,"previous_names":["profullstack/link-directories"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/profullstack/link-directories","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/profullstack%2Flink-directories","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/profullstack%2Flink-directories/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/profullstack%2Flink-directories/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/profullstack%2Flink-directories/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/profullstack","download_url":"https://codeload.github.com/profullstack/link-directories/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/profullstack%2Flink-directories/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33301599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T12:23:38.849Z","status":"ssl_error","status_checked_at":"2026-05-21T12:22:11.673Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T15:20:57.069Z","updated_at":"2026-05-21T13:07:56.661Z","avatar_url":"https://github.com/profullstack.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Directory Submission Bot 🤖\n\nAn automated Node.js tool using Puppeteer to streamline submissions to multiple web directories. This bot reads directory information from a CSV file, visits each site, analyzes submission forms, and can help automate the submission process.\n\n## Features\n\n- ✅ Parse CSV files containing directory information\n- 🌐 Automated browser navigation using Puppeteer\n- 🔍 Intelligent form detection and analysis\n- 📸 Screenshot capture on errors\n- 📊 Detailed submission results and reporting\n- ⚙️ Configurable delays and timeouts\n- 🧪 Comprehensive test coverage with Mocha/Chai\n- 🎨 ESLint and Prettier for code quality\n\n## Prerequisites\n\n- Node.js v20 or newer\n- pnpm (recommended) or npm\n\n## Installation\n\n1. Clone or download this repository\n\n2. Install dependencies using pnpm:\n```bash\npnpm install\n```\n\nOr using npm:\n```bash\nnpm install\n```\n\n## Configuration\n\n1. Copy the example configuration file:\n```bash\ncp config.example.js config.js\n```\n\n2. Edit `config.js` with your information:\n```javascript\nexport default {\n  submission: {\n    name: 'Your Tool Name',\n    url: 'https://yourtool.com',\n    email: 'your-email@example.com',\n    description: 'Brief description of your tool',\n    category: 'AI Tools',\n    tags: ['ai', 'automation', 'productivity'],\n  },\n  bot: {\n    headless: false,  // Set to true for production\n    timeout: 30000,\n    delayBetweenSubmissions: 5000,\n    screenshotOnError: true,\n  },\n  csvPath: './directories.csv',\n  filter: {\n    onlyUnsubmitted: true,\n    limit: null,  // Set to a number for testing\n  },\n};\n```\n\n## CSV Format\n\nYour `directories.csv` file should follow this format:\n\n```csv\nDirectory Name,URL,Status\nAI Tool Directory,https://example.com/submit,submitted\nAnother Directory,https://example2.com/submit,\nThird Directory,https://example3.com/submit,\n```\n\n- **Directory Name**: The name of the directory\n- **URL**: The submission or homepage URL\n- **Status**: Leave empty for unsubmitted, or use \"submitted\" for completed submissions\n\n## Usage\n\n### Step 1: Inspect Sites (Generate Configurations)\n\nFirst, inspect the production sites to analyze their submission forms and generate site-specific configurations:\n\n```bash\npnpm inspect\n```\n\nOr with a limit (recommended for first run):\n```bash\npnpm inspect ./directories.csv 5\n```\n\nThis will:\n- Visit each unsubmitted directory\n- Analyze forms, fields, and submission buttons\n- Generate intelligent field mappings\n- Create `site-configs.json` with site-specific configurations\n- Create `site-inspection-results.json` with detailed analysis\n\n### Step 2: Run Smart Submissions\n\nAfter generating configurations, run the smart submission bot:\n\n```bash\npnpm start\n```\n\nOr:\n```bash\nnode src/index.js\n```\n\nThe bot will use the generated configurations to:\n- Fill forms with correct field mappings\n- Handle different submission methods (forms vs links)\n- Detect and pause for CAPTCHAs\n- Skip sites requiring manual submission\n\n### Run Tests\n\n```bash\npnpm test\n```\n\n### Lint Code\n\n```bash\npnpm lint\n```\n\n### Format Code\n\n```bash\npnpm format\n```\n\n## How It Works\n\n1. **Parse CSV**: Reads the directories.csv file and parses directory information\n2. **Filter**: Optionally filters to only unsubmitted directories\n3. **Initialize Browser**: Launches Puppeteer browser instance\n4. **Visit Directories**: For each directory:\n   - Navigates to the URL\n   - Analyzes the page for forms and input fields\n   - Captures screenshots on errors\n   - Records results\n5. **Generate Report**: Saves detailed results to `submission-results.json`\n\n## Project Structure\n\n```\n.\n├── src/\n│   ├── index.js              # Main entry point\n│   ├── submission-bot.js     # Puppeteer bot class\n│   └── utils/\n│       └── csv-parser.js     # CSV parsing utilities\n├── test/\n│   └── csv-parser.test.js    # Test suite\n├── config.example.js         # Example configuration\n├── directories.csv           # Your directory list\n├── package.json\n├── .eslintrc.json           # ESLint configuration\n├── .prettierrc.json         # Prettier configuration\n└── README.md\n```\n\n## API Reference\n\n### SubmissionBot Class\n\n#### Constructor\n```javascript\nconst bot = new SubmissionBot(config);\n```\n\n#### Methods\n\n- `initialize()`: Initialize browser and page\n- `visitDirectory(url, name)`: Visit a directory URL and analyze the page\n- `analyzePage()`: Analyze current page for forms and inputs\n- `fillForm(formData)`: Fill form with provided data\n- `submitForm()`: Submit the current form\n- `processDirectories(directories, submissionData)`: Process multiple directories\n- `saveResults(results, filename)`: Save results to JSON file\n- `close()`: Close the browser\n\n### CSV Parser Functions\n\n```javascript\nimport { \n  parseDirectoriesCSV, \n  filterByStatus, \n  getUnsubmittedDirectories \n} from './src/utils/csv-parser.js';\n\n// Parse CSV file\nconst directories = await parseDirectoriesCSV('./directories.csv');\n\n// Filter by status\nconst submitted = filterByStatus(directories, 'submitted');\n\n// Get unsubmitted only\nconst unsubmitted = getUnsubmittedDirectories(directories);\n```\n\n## Configuration Options\n\n### Bot Configuration\n\n| Option | Type | Default | Description |\n|--------|------|---------|-------------|\n| `headless` | boolean | `false` | Run browser in headless mode |\n| `timeout` | number | `30000` | Page load timeout in milliseconds |\n| `delayBetweenSubmissions` | number | `5000` | Delay between submissions in milliseconds |\n| `screenshotOnError` | boolean | `true` | Take screenshots when errors occur |\n\n### Filter Options\n\n| Option | Type | Default | Description |\n|--------|------|---------|-------------|\n| `onlyUnsubmitted` | boolean | `true` | Only process unsubmitted directories |\n| `limit` | number\\|null | `null` | Limit number of directories to process |\n\n## Output\n\nThe bot generates:\n\n1. **Console Output**: Real-time progress and results\n2. **submission-results.json**: Detailed results for each directory\n3. **screenshots/**: Error screenshots (if enabled)\n\n### Example Results JSON\n\n```json\n[\n  {\n    \"name\": \"AI Tool Directory\",\n    \"url\": \"https://example.com\",\n    \"result\": {\n      \"success\": true,\n      \"message\": \"Successfully visited AI Tool Directory\",\n      \"forms\": [...],\n      \"submitButtons\": [...],\n      \"inputs\": [...]\n    },\n    \"timestamp\": \"2024-01-15T10:30:00.000Z\"\n  }\n]\n```\n\n## Testing\n\nThe project includes comprehensive tests using Mocha and Chai:\n\n```bash\n# Run all tests\npnpm test\n\n# Run tests with coverage (if configured)\npnpm test:coverage\n```\n\n## Troubleshooting\n\n### Browser Won't Launch\n\nIf Puppeteer fails to launch:\n```bash\n# Install Chromium dependencies (Linux)\nsudo apt-get install -y chromium-browser\n\n# Or use system Chrome\nexport PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true\nexport PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome\n```\n\n### Timeout Errors\n\nIncrease timeout in config.js:\n```javascript\nbot: {\n  timeout: 60000,  // 60 seconds\n}\n```\n\n### Form Not Found\n\nThe bot analyzes pages but may need manual intervention for complex forms. Check the console output and screenshots for details.\n\n## Best Practices\n\n1. **Start Small**: Test with `limit: 5` in config before processing all directories\n2. **Use Delays**: Respect rate limits with appropriate delays between submissions\n3. **Review Results**: Always check `submission-results.json` for issues\n4. **Manual Verification**: Some directories may require manual submission\n5. **Keep CSV Updated**: Mark directories as \"submitted\" to avoid duplicates\n\n## Contributing\n\nContributions are welcome! Please ensure:\n\n- Code passes ESLint checks\n- Code is formatted with Prettier\n- Tests are included for new features\n- Documentation is updated\n\n## License\n\nMIT\n\n## Disclaimer\n\nThis tool is for educational and automation purposes. Always:\n- Respect website terms of service\n- Use appropriate delays between requests\n- Verify submissions manually when required\n- Follow rate limiting guidelines\n\n## Support\n\nFor issues or questions, please check:\n1. This README\n2. The example configuration\n3. Test files for usage examples","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprofullstack%2Flink-directories","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprofullstack%2Flink-directories","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprofullstack%2Flink-directories/lists"}