https://github.com/newronai/cortex
Electron based web crawler, can be used to automate several tasks
https://github.com/newronai/cortex
Last synced: about 1 year ago
JSON representation
Electron based web crawler, can be used to automate several tasks
- Host: GitHub
- URL: https://github.com/newronai/cortex
- Owner: NewronAI
- License: mit
- Created: 2023-05-07T21:57:02.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-12-01T10:58:44.000Z (over 2 years ago)
- Last Synced: 2023-12-01T12:26:02.021Z (over 2 years ago)
- Language: JavaScript
- Size: 956 KB
- Stars: 3
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Cortex by Newron.ai
[//]: # ()
Cortex is a powerful and efficient general-purpose web crawler designed and developed by Newron.ai. It aims to simplify the process of collecting data from various web sources, enabling users to extract valuable information with ease.
## Table of Contents
- [Features](#features)
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Usage](#usage)
- [Configuration](#configuration)
- [Running the Crawler](#running-the-crawler)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgements](#acknowledgements)
## Features
- Customizable and extensible crawling rules
- Advanced filtering options to target specific data
- Multithreading support for faster crawling
- Built-in caching and request throttling to prevent overloading target websites
- User-friendly Electron app with React and Tailwind CSS for easy configuration and management
## Getting Started
These instructions will help you set up Cortex on your local machine for development and testing purposes.
### Prerequisites
Before you start, make sure you have the following installed on your system:
- [Node.js](https://nodejs.org) (v14.0.0 or later)
- [npm](https://www.npmjs.com) (v7.0.0 or later)
### Installation
1. Clone the repository:
```
git clone https://github.com/Newron.ai/Cortex.git
```
2. Navigate to the project directory:
```
cd Cortex
```
3. Install the dependencies:
```
npm install
```
4. Run the application:
```
npm start
```
## Usage
### Configuration
1. Open the Cortex application.
2. Configure the crawling rules, filters, and other options through the user interface.
### Running the Crawler
1. Start the crawler by clicking the "Start Crawler" button.
2. Monitor the progress and view the collected data in the application.
## Contributing
We welcome contributions from the community. If you'd like to contribute to the Cortex project, please follow these steps:
1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Make your changes and commit them with a descriptive commit message.
4. Push your changes to your fork.
5. Open a pull request and describe the changes you made.
Please make sure to follow our coding standards and guidelines when contributing.
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
## Acknowledgements
- [Electron](https://www.electronjs.org)
- [React](https://reactjs.org)
- [Tailwind CSS](https://tailwindcss.com)