{"id":29164980,"url":"https://github.com/yuyu1815/deepwiki_to_md","last_synced_at":"2025-07-01T07:08:45.511Z","repository":{"id":290280252,"uuid":"973905935","full_name":"yuyu1815/deepwiki_to_md","owner":"yuyu1815","description":"md the contents of deepwiki and add it as documentation","archived":false,"fork":false,"pushed_at":"2025-05-30T08:19:18.000Z","size":452,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-30T10:40:21.712Z","etag":null,"topics":["deepwiki","python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuyu1815.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-28T00:58:31.000Z","updated_at":"2025-05-27T14:14:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"f4f7a855-87ea-4250-b1fa-eb098ee3de9b","html_url":"https://github.com/yuyu1815/deepwiki_to_md","commit_stats":null,"previous_names":["yuyu1815/deepwiki_to_md"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yuyu1815/deepwiki_to_md","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyu1815%2Fdeepwiki_to_md","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyu1815%2Fdeepwiki_to_md/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyu1815%2Fdeepwiki_to_md/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyu1815%2Fdeepwiki_to_md/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuyu1815","download_url":"https://codeload.github.com/yuyu1815/deepwiki_to_md/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyu1815%2Fdeepwiki_to_md/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262781194,"owners_count":23363134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepwiki","python","python3"],"created_at":"2025-07-01T07:08:44.849Z","updated_at":"2025-07-01T07:08:45.497Z","avatar_url":"https://github.com/yuyu1815.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deepwiki to Markdown Converter\n\n\u003e **The Japanese version of this document is available at [README_ja.md](./README_ja.md).**\n\nA powerful Python tool designed to scrape content from deepwiki sites and convert it to clean Markdown format. It offers multiple scraping strategies and functions for data processing.\n\n## Features\n\n- Scrapes content from deepwiki sites using multiple strategies:\n  - Direct Markdown Fetching (default)\n  - Direct HTML Scraping with conversion\n  - Simple static fallback\n- Extracts navigation items from specified UI elements to traverse libraries\n- Converts HTML content to Markdown format using `markdownify`\n- Saves the converted files in an organized directory structure\n- Supports scraping multiple libraries in a single run\n- Includes error handling with domain validation, reachability checks, and retry mechanisms\n- Offers a utility to convert Markdown files to YAML format while preserving formatting\n- Provides a utility to fix links within the scraped Markdown files\n- Supports scraping responses from chat interfaces using Selenium\n\n## Requirements\n\n- Python 3.6 or higher\n- Required Python packages (see [`requirements.txt`](./requirements.txt)):\n\n### Core Dependencies\n  - `requests`\n  - `beautifulsoup4`\n  - `argparse`\n  - `markdownify`\n\n### Optional Dependencies\n  - `selenium` (Required for the chat scraping feature)\n  - `webdriver-manager` (Required for the chat scraping feature)\n  - `pyyaml` (Required for the Markdown to YAML conversion feature)\n\n## Installation\n\n### Option 1: Install from PyPI\n\n```bash\npip install deepwiki-to-md\n```\n\nThis will install the core dependencies listed in [setup.py](./setup.py). Note that `selenium`, `webdriver-manager`, and `pyyaml` are listed in [`requirements.txt`](./requirements.txt) but not as install dependencies in `setup.py`. Install them manually or install from source including `requirements.txt` if you need the chat scraping or YAML conversion features.\n\n### Option 2: Install from source\n\nClone this repository:\n\n```bash\ngit clone https://github.com/yuyu1815/deepwiki_to_md.git\ncd deepwiki_to_md\n```\n\nInstall the package in development mode, including all dependencies from requirements.txt:\n\n```bash\npip install -e . -r requirements.txt\n```\n\n## Usage\n\n### Basic Usage (Command Line)\n\nIf installed from PyPI, you can use the command-line tool:\n\n```bash\ndeepwiki-to-md \"https://deepwiki.com/library_path\"\n```\n\nOr with explicit parameters:\n\n```bash\ndeepwiki-to-md --library \"library_name\" \"https://deepwiki.example.com/library_path\"\n```\n\nIf installed from source, you can run the script directly:\n\n```bash\npython -m deepwiki_to_md.run_scraper \"https://deepwiki.com/library_path\"\n```\n\nOr with explicit parameters:\n\n```bash\npython -m deepwiki_to_md.run_scraper --library \"library_name\" \"https://deepwiki.example.com/library_path\"\n```\n\nNote: The output directory will be created in the current working directory where the command is executed, not in the\npackage installation directory.\n\n### Repository Creation Tool\n\nThe package also includes a tool to create repository requests by setting an email and submitting a form:\n\nIf installed from PyPI, you can use the command-line tool:\n\n```bash\ndeepwiki-create --url \"https://example.com/repository/create\" --email \"user@example.com\"\n```\n\nTo run in headless mode (without opening a browser window):\n\n```bash\ndeepwiki-create --url \"https://example.com/repository/create\" --email \"user@example.com\" --headless\n```\n\nIf installed from source, you can run the script directly:\n\n```bash\npython -m deepwiki_to_md.create --url \"https://example.com/repository/create\" --email \"user@example.com\"\n```\n\n### Using the Python API\n\nYou can also use the DeepwikiScraper class directly in your Python code:\n\n```python\nfrom deepwiki_to_md import DeepwikiScraper\n# Import specific scraper classes if needed for direct use\nfrom deepwiki_to_md.direct_scraper import DirectDeepwikiScraper  # For HTML -\u003e MD\nfrom deepwiki_to_md.direct_md_scraper import DirectMarkdownScraper  # For Direct MD\n# Import the RepositoryCreator class for repository creation\nfrom deepwiki_to_md.create import RepositoryCreator\n\n# Create a scraper instance (DirectMarkdownScraper is used by default)\nscraper = DeepwikiScraper(output_dir=\"MyDocuments\")\n\n# Scrape a library using the default (DirectMarkdownScraper)\nscraper.scrape_library(\"python\", \"https://deepwiki.com/python/cpython\")\n\n# Create another scraper with a different output directory\nother_scraper = DeepwikiScraper(output_dir=\"OtherDocuments\")\n\n# Scrape another library (still uses DirectMarkdownScraper by default)\nother_scraper.scrape_library(\"javascript\", \"https://deepwiki.example.com/javascript\")\n\n# --- Using DirectDeepwikiScraper explicitly (HTML to Markdown) ---\n# Create a scraper instance explicitly using DirectDeepwikiScraper\n# This scraper fetches HTML and converts it to Markdown\nhtml_scraper = DeepwikiScraper(\n    output_dir=\"HtmlScrapedDocuments\",\n    use_direct_scraper=True,  # Enable DirectDeepwikiScraper\n    use_alternative_scraper=False,  # Disable alternative fallback for clarity\n    use_direct_md_scraper=False  # Disable DirectMarkdownScraper\n)\nhtml_scraper.scrape_library(\"go\", \"https://deepwiki.com/go\")\n\n# --- Using DirectMarkdownScraper explicitly (Direct Markdown Fetching) ---\n# Create a scraper instance explicitly using DirectMarkdownScraper\n# This is already the default, but can be specified for clarity or if other defaults change\nmd_scraper = DeepwikiScraper(\n    output_dir=\"DirectMarkdownDocuments\",\n    use_direct_scraper=False,\n    use_alternative_scraper=False,\n    use_direct_md_scraper=True  # Enable DirectMarkdownScraper (this is the default)\n)\nmd_scraper.scrape_library(\"rust\", \"https://deepwiki.com/rust\")\n\n# --- Using the individual direct scrapers directly ---\n# These classes can be used independently for scraping specific pages or lists of pages\n\n# Create a DirectDeepwikiScraper instance (HTML to Markdown)\ndirect_html_scraper = DirectDeepwikiScraper(output_dir=\"DirectHtmlScraped\")\n\n# Scrape a specific page directly (HTML to Markdown)\ndirect_html_scraper.scrape_page(\n    \"https://deepwiki.com/python/cpython/2.1-bytecode-interpreter-and-optimization\",\n    \"python_bytecode\",  # Library name/path part for output folder\n    save_html=True  # Optionally save the original HTML\n)\n\n# Create a DirectMarkdownScraper instance (Direct Markdown Fetching)\ndirect_md_scraper = DirectMarkdownScraper(output_dir=\"DirectMarkdownFetched\")\n\n# Scrape a specific page directly as Markdown\ndirect_md_scraper.scrape_page(\n   \"https://deepwiki.com/python/cpython/2.1-bytecode-interpreter-and-optimization\",\n    \"python_bytecode\"  # Library name/path part for output folder\n)\n\n# --- Using the RepositoryCreator for repository creation requests ---\n# Create a RepositoryCreator instance\ncreator = RepositoryCreator(headless=False)  # Set headless=True to run without browser UI\n\ntry:\n  # Send a repository creation request\n  success = creator.create(\n    url=\"https://example.com/repository/create\",\n    email=\"user@example.com\"\n  )\n\n  if success:\n    print(\"Repository creation request sent successfully\")\n  else:\n    print(\"Failed to send repository creation request\")\nfinally:\n  # Always close the browser when done\n  creator.close()\n```\n\n## Command-line Arguments\n\nFor `deepwiki-to-md` or `python -m deepwiki_to_md.run_scraper`:\n\n- `library_url`: URL of the library to scrape (can be provided as a positional argument).\n- `--library`, `-l`: Library name and URL to scrape. Can be specified multiple times for different libraries. Format:\n  `--library NAME URL`.\n- `--output-dir`, `-o`: Output directory for Markdown files (default: Documents).\n- `--use-direct-scraper`: Use DirectDeepwikiScraper (HTML to Markdown conversion). Prioritized over\n  `--use-direct-md-scraper` if both are specified.\n- `--no-direct-scraper`: Disable DirectDeepwikiScraper.\n- `--use-alternative-scraper`: Use the scrape_deepwiki function from direct_scraper.py as a fallback if the primary\n  method fails (default: True).\n- `--no-alternative-scraper`: Disable the alternative scraper fallback.\n- `--use-direct-md-scraper`: Use DirectMarkdownScraper (fetches Markdown directly). This is the default behavior if no\n  scraper type is explicitly specified.\n- `--no-direct-md-scraper`: Disable DirectMarkdownScraper.\n\nScraper Priority:\n\n- If `--use-direct-scraper` is specified, DirectDeepwikiScraper (HTML to Markdown) is used.\n- If `--use-direct-md-scraper` is specified (and `--use-direct-scraper` is not), DirectMarkdownScraper (Direct Markdown)\n  is used.\n- If neither is specified, DirectMarkdownScraper (Direct Markdown) is used by default.\n- The `--use-alternative-scraper` flag controls a fallback mechanism within the chosen primary scraper.\n\nFor `deepwiki-create` or `python -m deepwiki_to_md.create`:\n- `--url` (required): The URL of the repository creation page.\n- `--email` (required): The email address to notify.\n- `--headless`: Run the browser in headless mode (without UI).\n\n## Examples (Command Line)\n\nSimplified usage (uses DirectMarkdownScraper by default):\n\n```bash\npython -m deepwiki_to_md.run_scraper \"https://deepwiki.com/python/cpython\"\n# Or if installed via pip: deepwiki-to-md \"https://deepwiki.com/python/cpython\"\n```\n\nScrape a single library with explicit parameters:\n\n```bash\npython -m deepwiki_to_md.run_scraper --library \"python\" \"https://deepwiki.com/python/cpython\"\n```\n\nScrape multiple libraries:\n\n```bash\npython -m deepwiki_to_md.run_scraper --library \"python\" \"https://deepwiki.com/python/cpython\" --library \"microsoft/vscode\" \"https://deepwiki.com/microsoft/vscode\"\n```\n\nSpecify a custom output directory:\n\n```bash\npython -m deepwiki_to_md.run_scraper \"https://deepwiki.com/python/cpython\" --output-dir \"MyDocuments\"\n```\n\nExplicitly use DirectMarkdownScraper (Direct Markdown):\n\n```bash\npython -m deepwiki_to_md.run_scraper \"https://deepwiki.com/python/cpython\" --use-direct-md-scraper\n```\n\nExplicitly use DirectDeepwikiScraper (HTML to Markdown):\n\n```bash\npython -m deepwiki_to_md.run_scraper \"https://deepwiki.com/python/cpython\" --use-direct-scraper\n```\n\nDisable the alternative scraper fallback:\n\n```bash\npython -m deepwiki_to_md.run_scraper \"https://deepwiki.com/python/cpython\" --no-alternative-scraper\n```\n\nUsing the repository creation tool:\n\n```bash\ndeepwiki-create --url \"https://example.com/repository/create\" --email \"user@example.com\"\n```\n\nUsing the repository creation tool in headless mode:\n\n```bash\ndeepwiki-create --url \"https://example.com/repository/create\" --email \"user@example.com\" --headless\n```\n\n### Usage with run_direct_scraper.py\n\nYou can also use the run_direct_scraper.py script, which is a simplified entry point specifically for the\nDirectDeepwikiScraper (HTML to Markdown):\n\n```bash\npython -m deepwiki_to_md.run_direct_scraper \"https://deepwiki.com/python/cpython\"\n# Or with explicit parameters:\npython -m deepwiki_to_md.run_direct_scraper --library \"python\" \"https://deepwiki.com/python/cpython\"\n# To save HTML as well:\npython -m deepwiki_to_md.run_direct_scraper \"https://deepwiki.com/python/cpython\" --save-html\n```\n\nArguments for run_direct_scraper.py:\n- `library_url`: URL of the library (positional).\n- `--library`, `-l`: Library name and URL (can be multiple).\n- `--output-dir`, `-o`: Output directory (default: DynamicDocuments).\n- `--save-html`: Save original HTML files alongside Markdown.\n\n## Output Structure\n\nThe converted Markdown files will be saved in the following directory structure:\n\n```text\n\u003coutput_dir\u003e/\n├── \u003clibrary_name1\u003e/\n│   └── md/\n│       ├── \u003cpage_name1\u003e.md\n│       ├── \u003cpage_name2\u003e.md\n│       └── ...\n│   └── html/ # Only if --save-html is used with DirectDeepwikiScraper\n│       ├── \u003cpage_name1\u003e.html\n│       ├── \u003cpage_name2\u003e.html\n│       └── ...\n├── \u003clibrary_name2\u003e/\n│   └── md/\n│       ├── \u003cpage_name1\u003e.md\n│       ├── \u003cpage_name2\u003e.md\n│       └── ...\n└── ...\n```\n\n- `\u003coutput_dir\u003e` is the directory specified by `--output-dir` (default: Documents for run_scraper.py, DynamicDocuments\n  for run_direct_scraper.py).\n- `\u003clibrary_name\u003e` is the name provided for the library (or inferred from the URL path).\n- Each page from the Deepwiki site is saved as a separate .md file within the md subdirectory.\n- Original HTML is saved in the html subdirectory if the `--save-html` option is used with DirectDeepwikiScraper.\n\n## How It Works\n\nThe tool offers different scraping strategies to maximize compatibility and output quality:\n\n### 1. Direct Markdown Scraping (DirectMarkdownScraper - Default)\n\n- **Priority**: Highest (used by default if no other scraper is explicitly chosen).\n- **Method**: Attempts to fetch the raw Markdown content directly from the Deepwiki site's underlying data source or\n  API. This is done by sending requests with specialized headers that mimic internal application requests.\n- **Process**:\n  - Sends requests designed to retrieve Markdown data (using specific Accept headers or query parameters)\n  - Parses the response to extract the Markdown content\n  - Performs minimal cleaning on the extracted Markdown\n  - Splits the content into multiple files based on level 2 headings (##)\n  - Saves the cleaned and split Markdown content directly to .md files\n- **Advantage**: Produces the highest fidelity Markdown, preserving the original formatting and structure as intended by\n  the author.\n\n### 2. Direct HTML Scraping (DirectDeepwikiScraper)\n\n- **Priority**: Medium (used if `--use-direct-scraper` is specified).\n- **Method**: Connects to the Deepwiki site using headers that mimic a standard browser request to fetch the fully\n  rendered HTML page.\n- **Process**:\n  - Fetches the full HTML of the page using the scrape_deepwiki function\n  - Uses BeautifulSoup to parse the HTML\n  - Identifies the main content area using a list of potential CSS selectors\n  - Uses the markdownify library to convert the selected HTML content to Markdown\n  - Saves the converted Markdown\n- **Advantage**: More robust than basic static scraping if direct Markdown fetching fails or is unavailable.\n\n### 3. Alternative Scraper Fallback\n\n- **Priority**: Lowest (used as a fallback if `--use-alternative-scraper` is enabled).\n- **Method**: A simpler static requests mechanism with specific headers designed to fetch the page HTML reliably.\n\n## Markdown to YAML Conversion Utility\n\nThe tool provides a utility to convert Markdown files to YAML format while preserving formatting. This is particularly\nuseful for processing the scraped content for LLMs.\n\n### Using the Conversion Tool (Command Line)\n```bash\npython -m deepwiki_to_md.chat convert --md \"path/to/markdown/file.md\"\n# Or if console script entry point is installed:\n# deepwiki-chat convert --md \"path/to/markdown/file.md\"\n```\n\nTo specify a custom output directory:\n\n```bash\npython -m deepwiki_to_md.chat convert --md \"path/to/markdown/file.md\" --output \"path/to/output/directory\"\n```\n\n### Using the Python API (Markdown to YAML)\n```python\nfrom deepwiki_to_md.md_to_yaml import convert_md_file_to_yaml, markdown_to_yaml\n\n# Convert a Markdown file to YAML\nyaml_file_path = convert_md_file_to_yaml(\"path/to/markdown/file.md\")\n\n# Convert a Markdown file to YAML with a custom output directory\nyaml_file_path = convert_md_file_to_yaml(\"path/to/markdown/file.md\", \"path/to/output/directory\")\n\n# Or convert a Markdown string directly to a YAML string\nmarkdown_string = \"# My Document\\n\\nThis is the content.\"\nyaml_string = markdown_to_yaml(markdown_string)\nprint(yaml_string)\n```\n\n### YAML Format\n\nThe converted YAML file includes a structured representation of the document while embedding the original Markdown\ncontent:\n\n```yaml\ntimestamp: 'YYYY-MM-DD HH:MM:SS'  # Timestamp of the conversion\ntitle: Extracted Document Title    # Title extracted from the first H1/H2 header\ncontent: |\n  # Original Title\n  ## Section 1\n\n  Content of section 1.\n\n  * List item 1\n  * List item 2\n\n  print(\"code\")\n\n  [Link Text](url)\n\n  ## Section 2\n\n  Content of section 2.\n  ...                              # Full original Markdown content is preserved\nlinks:\n  - text: Link Text\n    url: url                       # List of links extracted from the Markdown\nimages: [ ]                         # List of images extracted (currently empty)\nmetadata:\n  headers: # List of all header texts\n    - Original Title\n    - Section 1\n    - Section 2\n    ...\n  paragraphs_count: 5              # Count of paragraphs\n  lists_count: 1                   # Count of lists\n  tables_count: 0                  # Count of tables\n```\n\n## Markdown Link Fixing Utility\n\nThe tool automatically runs a link-fixing utility on the generated .md files. This utility finds Markdown links in the\nformat [Text](URL) and replaces them with [Text]().\n\n### Using the Link Fixing Tool (Command Line)\n```bash\npython -m deepwiki_to_md.fix_markdown_links \"path/to/your/markdown/directory\"\n```\n\n### Using the Python API (Link Fixing)\n```python\nfrom deepwiki_to_md.fix_markdown_links import fix_markdown_links\n\n# Fix links in all markdown files within a directory\nfix_markdown_links(\"path/to/your/markdown/directory\")\n```\n\n## Chat Scraping Feature (Requires Selenium)\n\nThe tool includes a feature to interact with chat interfaces using Selenium and save the responses.\n\n### Using the Chat Scraper (Command Line)\n```bash\npython -m deepwiki_to_md.chat --url \"https://deepwiki.com/some_chat_page\" --message \"Your message here\" --wait 10 --debug --format \"html,md,yaml\" --output \"MyChatResponses\" --deep\n```\n\nArguments for chat.py:\n- `--url`: URL of the chat interface.\n- `--message`: Message to send.\n- `--selector`: CSS selector for the chat input (default: textarea).\n- `--button`: CSS selector for the submit button (default: button).\n- `--wait`: Time to wait for response in seconds (default: 30).\n- `--debug`: Enable debug mode.\n- `--output`: Output directory (default: ChatResponses).\n- `--deep`: Enable \"Deep Research\" mode (specific to some interfaces).\n- `--headless`: Run browser in headless mode.\n- `--format`: Output format(s): html, md, yaml, or comma-separated list (default: html).\n\nNote: The chat scraper uses Selenium, which requires a compatible browser installed.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuyu1815%2Fdeepwiki_to_md","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuyu1815%2Fdeepwiki_to_md","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuyu1815%2Fdeepwiki_to_md/lists"}