https://github.com/r00tSe7en/URLPath

批量处理url链接，获取多级路径并打印
https://github.com/r00tSe7en/URLPath

Last synced: 3 months ago
JSON representation

批量处理url链接，获取多级路径并打印

• Host: GitHub
• URL: https://github.com/r00tSe7en/URLPath
• Owner: r00tSe7en
• Created: 2023-01-04T02:17:30.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-07-12T08:03:20.000Z (over 1 year ago)
• Last Synced: 2024-06-20T15:47:48.219Z (6 months ago)
• Language: Go
• Size: 24.4 KB
• Stars: 40
• Watchers: 2
• Forks: 5
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-hacking-lists - r00tSe7en/URLPath - 批量处理url链接，获取多级路径并打印 (Go)

README

        
# URLPath

![URLPath](https://socialify.git.ci/r00tSe7en/URLPath/image?description=1&forks=1&issues=1&language=1&name=1&owner=1&pattern=Floating%20Cogs&pulls=1&stargazers=1&theme=Dark)

# 1.配合爬虫使用

这里联动projectdiscovery的katana爬虫，拿特斯拉官网测试，最终结果是1441行并无重复

https://github.com/projectdiscovery/katana

```

katana -u https://tesla.com | ./URLPath | tee result.txt

...

   ...

   

wc -l result.txt

1441 //统计行数

cat result.txt | uniq | wc -l

1441 //uniq去重后统计行数

```

# 2.处理链接文本

链接文本

```

http://example.com

http://example.com/api

http://example.com/Api

https://example.com/books/all

https://example.com/1.pdf

https://example.com/books/all/1.pdf

https://example.com/1/2/3

https://example.com/1/2/3/4/5

https://example.com/?id=1

https://example.com/test/?id=1

```

处理链接

```

Take as input on stdin a list of urls and print on stdout all the unique paths (at any level).

        $> cat input | URLPath -l 2 

        -l x is specify number of path levels to print. -1 or no parameters specified to print all path levels.

         

1. cat url.txt|./URLPath  //打印全部级别路径

2. cat url.txt|./URLPath -l 2  //打印2级及2级以下级别路径

```

输出链接

```

(提示：无path的url不会打印)

http://example.com/api

http://example.com/Api

https://example.com/books

https://example.com/books/all

https://example.com/1

https://example.com/1/2

https://example.com/1/2/3

https://example.com/1/2/3/4

https://example.com/1/2/3/4/5

https://example.com/test

```

# 参考

https://github.com/edoardottt/lit-bb-hack-tools/tree/main/cleanpath