{"id":19623257,"url":"https://github.com/newronai/cortex","last_synced_at":"2025-04-28T04:31:47.542Z","repository":{"id":167516165,"uuid":"637546023","full_name":"NewronAI/cortex","owner":"NewronAI","description":"Electron based web crawler, can be used to automate several tasks","archived":false,"fork":false,"pushed_at":"2023-12-01T10:58:44.000Z","size":979,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-12-01T12:26:02.021Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NewronAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-05-07T21:57:02.000Z","updated_at":"2023-08-30T10:09:59.000Z","dependencies_parsed_at":"2023-12-01T12:31:06.177Z","dependency_job_id":null,"html_url":"https://github.com/NewronAI/cortex","commit_stats":null,"previous_names":["newronai/cortex"],"tags_count":1,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NewronAI%2Fcortex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NewronAI%2Fcortex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NewronAI%2Fcortex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NewronAI%2Fcortex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NewronAI","download_url":"https://codeload.github.com/NewronAI/cortex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224095793,"owners_count":17254970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T11:32:56.659Z","updated_at":"2024-11-11T11:32:57.556Z","avatar_url":"https://github.com/NewronAI.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cortex by Newron.ai\n\n[//]: # (![Cortex Banner]\u0026#40;./assets/banner.png\u0026#41;)\n\nCortex is a powerful and efficient general-purpose web crawler designed and developed by Newron.ai. It aims to simplify the process of collecting data from various web sources, enabling users to extract valuable information with ease.\n\n## Table of Contents\n\n- [Features](#features)\n- [Getting Started](#getting-started)\n    - [Prerequisites](#prerequisites)\n    - [Installation](#installation)\n- [Usage](#usage)\n    - [Configuration](#configuration)\n    - [Running the Crawler](#running-the-crawler)\n- [Contributing](#contributing)\n- [License](#license)\n- [Acknowledgements](#acknowledgements)\n\n## Features\n\n- Customizable and extensible crawling rules\n- Advanced filtering options to target specific data\n- Multithreading support for faster crawling\n- Built-in caching and request throttling to prevent overloading target websites\n- User-friendly Electron app with React and Tailwind CSS for easy configuration and management\n\n## Getting Started\n\nThese instructions will help you set up Cortex on your local machine for development and testing purposes.\n\n### Prerequisites\n\nBefore you start, make sure you have the following installed on your system:\n\n- [Node.js](https://nodejs.org) (v14.0.0 or later)\n- [npm](https://www.npmjs.com) (v7.0.0 or later)\n\n### Installation\n\n1. Clone the repository:\n\n   ```\n   git clone https://github.com/Newron.ai/Cortex.git\n   ```\n\n2. Navigate to the project directory:\n\n   ```\n   cd Cortex\n   ```\n\n3. Install the dependencies:\n\n   ```\n   npm install\n   ```\n\n4. Run the application:\n\n   ```\n   npm start\n   ```\n\n## Usage\n\n### Configuration\n\n1. Open the Cortex application.\n2. Configure the crawling rules, filters, and other options through the user interface.\n\n### Running the Crawler\n\n1. Start the crawler by clicking the \"Start Crawler\" button.\n2. Monitor the progress and view the collected data in the application.\n\n## Contributing\n\nWe welcome contributions from the community. If you'd like to contribute to the Cortex project, please follow these steps:\n\n1. Fork the repository.\n2. Create a new branch for your feature or bug fix.\n3. Make your changes and commit them with a descriptive commit message.\n4. Push your changes to your fork.\n5. Open a pull request and describe the changes you made.\n\nPlease make sure to follow our coding standards and guidelines when contributing.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.\n\n## Acknowledgements\n\n- [Electron](https://www.electronjs.org)\n- [React](https://reactjs.org)\n- [Tailwind CSS](https://tailwindcss.com)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewronai%2Fcortex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnewronai%2Fcortex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewronai%2Fcortex/lists"}