https://github.com/arshadkazmi42/gh-crawl

Crawler for Github repositories. Finds all the broken links from the repositories
https://github.com/arshadkazmi42/gh-crawl

bug-bounty-recon crawl crawler gh-crawler github github-crawler githubcrawler python

Last synced: 17 days ago
JSON representation

Crawler for Github repositories. Finds all the broken links from the repositories

• Host: GitHub
• URL: https://github.com/arshadkazmi42/gh-crawl
• Owner: arshadkazmi42
• License: mit
• Created: 2021-05-11T15:46:39.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2021-11-04T16:57:55.000Z (about 3 years ago)
• Last Synced: 2024-10-28T03:17:19.127Z (2 months ago)
• Topics: bug-bounty-recon, crawl, crawler, gh-crawler, github, github-crawler, githubcrawler, python
• Language: Python
• Homepage:
• Size: 38.1 KB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# gh-crawl

Crawl all the urls from github organization / user public github repositories

## Install Dependencies

> Requires Python 3+

```

$ pip install -r requirements.txt

```

## Usage

```

$ python scan.py {Github Username} {Github Token (Optional)}

```

## Example

```

$ python scan.py bugcrowd

```

> Note: Output will be printed to stdout and will be written to file name {GITHUB USERNAME}.txt

## Local `run.sh`

```

# $1: Github Username / Organization Name

# $2: Github Token (Optional)

python3 scan.py $1 {REPLACE_WITH_GITHUB_TOKEN}

# This is to find unavailable github accounts

# Requires https://github.com/arshadkazmi42/bash-scripts

cat $1.txt | awk -F[/] '{print $1"//"$3"/"$4}' | sort | uniq | xargs -I {} sh ../bash-scripts/curl/scan-broken.sh {}

```