https://github.com/ulixee/secret-agent

The web scraper that's nearly impossible to block - now called @ulixee/hero
https://github.com/ulixee/secret-agent

automated browser chromium devtools mitm mitmproxy playwright proxy puppeteer scraping secretagent stealth

Last synced: about 1 month ago
JSON representation

The web scraper that's nearly impossible to block - now called @ulixee/hero

• Host: GitHub
• URL: https://github.com/ulixee/secret-agent
• Owner: ulixee
• License: mit
• Created: 2020-06-24T18:21:38.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2023-03-07T03:00:38.000Z (over 1 year ago)
• Last Synced: 2024-06-02T05:22:04.963Z (about 1 month ago)
• Topics: automated, browser, chromium, devtools, mitm, mitmproxy, playwright, proxy, puppeteer, scraping, secretagent, stealth
• Language: TypeScript
• Homepage: https://secretagent.dev
• Size: 46.3 MB
• Stars: 646
• Watchers: 17
• Forks: 43
• Open Issues: 43
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE.md

Lists

• awesome-stars - ulixee/secret-agent - The web scraper that's nearly impossible to block - now called @ulixee/hero (TypeScript)
• awesome-stars - ulixee/secret-agent - The web scraper that's nearly impossible to block - now called @ulixee/hero (TypeScript)
• awesome-stars - ulixee/secret-agent - The web scraper that's nearly impossible to block - now called @ulixee/hero (proxy)
• awesome-stars - ulixee/secret-agent - The web scraper that's nearly impossible to block - now called @ulixee/hero (TypeScript)
• awesome-stars - ulixee/secret-agent - The web browser that's built for scraping. (TypeScript)

README

        
## SecretAgent 2.0 === Hero

> :bell: SecretAgent 2.0 is named "[Hero](https://github.com/ulixee/hero)" and currently in alpha testing. We're ready for developers to begin to switch over - it's a pretty easy transition ([migration guide](https://ulixee.org/docs/hero/help/migration-guide)). Follow along with development of Hero [here](https://github.com/ulixee/hero) or check out the latest npm packages `@ulixee/hero-playground`.

---

# SecretAgent

SecretAgent is a web browser that's built for scraping.

- [x] **Built for scraping** - it's the first modern headless browsers designed specifically for scraping instead of just automated testing.

- [x] **Designed for web developers** - We've recreated a fully compliant DOM directly in NodeJS allowing you bypass the headaches of previous scraper tools.

- [x] **Powered by Chrome** - The powerful Chrome engine sits under the hood, allowing for lightning fast rendering.

- [x] **Emulates any modern browser** - BrowserEmulators make it easy to disguise your script as practically any browser.

- [x] **Avoids detection along the entire stack** - Don't be blocked because of TLS fingerprints in your networking stack.

Check out our [website for more details](https://secretagent.dev).

## Installation

```shell script

npm i --save secret-agent

```

or

```shell script

yarn add secret-agent

```

## Usage

SecretAgent provides access to the W3C DOM specification without the need for Puppeteer's complicated evaluate callbacks and multi-context switching:

```js

const agent = require('secret-agent');

(async () => {

  await agent.goto('https://example.org');

  const title = await agent.document.title;

  const intro = await agent.document.querySelector('p').textContent;

  await agent.close();

})();

```

Browse the [full API docs](https://secretagent.dev/docs).

## Contributing

We'd love your help in making SecretAgent a better tool. Please don't hesitate to send a pull request.

## License

[MIT](LICENSE.md)