https://github.com/awhitford/mendable-local-docusaurus-scraper

Local Docusaurus Scraper for Mendable
https://github.com/awhitford/mendable-local-docusaurus-scraper

docusaurus mendable

Last synced: about 1 month ago
JSON representation

Local Docusaurus Scraper for Mendable

• Host: GitHub
• URL: https://github.com/awhitford/mendable-local-docusaurus-scraper
• Owner: awhitford
• License: mit
• Created: 2024-07-17T05:35:53.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-10-09T03:36:25.000Z (about 1 month ago)
• Last Synced: 2024-10-15T22:12:11.199Z (about 1 month ago)
• Topics: docusaurus, mendable
• Language: TypeScript
• Homepage:
• Size: 19.5 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Mendable Local Docusaurus Scraper

## Why?

It is easy to configure [Mendable](https://mendable.ai) to scrape a :globe_with_meridians: **public** web site, but what if your web site is :closed_lock_with_key: **private**?

This utility is designed to scan your [Docusaurus](https://docusaurus.io) site locally, and upload the content to Mendable.

It is particularly useful for _private_ web sites where Mendable can't access.

However, the markdown is uploaded, which may be better even for public Docusaurus sites. 

I was inspired to build this to upload a private [Docusaurus](https://docusaurus.io) site for [Mendable Search](https://docs.mendable.ai/integrations/docusaurus).

## Configuration

Configure your environment:  `.env` (or `.env.local`)

```

MENDABLE_SERVER_API_KEY=9ba...3be

```

* The `MENDABLE_SERVER_API_KEY` value may alternatively be passed as a command-line argument (`--api-key`)

* Environment variables may alternatively be configured using the command-line, like:

  ```

  export MENDABLE_SERVER_API_KEY=9ba...3be

  ```

## Usage

Build the utility:  `pnpm build`

See the usage:  `pnpm scrape`

Run against a project:  `pnpm scrape ../docusaurus-project-docs https://project-org.github.io/docusaurus-project-docs`