{"id":13826246,"url":"https://github.com/netxfly/transparent-proxy-scanner","last_synced_at":"2025-04-14T12:31:13.310Z","repository":{"id":81774559,"uuid":"39132645","full_name":"netxfly/Transparent-Proxy-Scanner","owner":"netxfly","description":"基于vpn和透明代理的web漏洞扫描器的实现思路及demo","archived":false,"fork":false,"pushed_at":"2015-07-15T17:19:03.000Z","size":3978,"stargazers_count":126,"open_issues_count":1,"forks_count":42,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-28T01:50:31.012Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/netxfly.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-15T11:14:26.000Z","updated_at":"2025-03-10T18:01:05.000Z","dependencies_parsed_at":"2023-02-28T15:15:53.742Z","dependency_job_id":null,"html_url":"https://github.com/netxfly/Transparent-Proxy-Scanner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netxfly%2FTransparent-Proxy-Scanner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netxfly%2FTransparent-Proxy-Scanner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netxfly%2FTransparent-Proxy-Scanner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netxfly%2FTransparent-Proxy-Scanner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/netxfly","download_url":"https://codeload.github.com/netxfly/Transparent-Proxy-Scanner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248881413,"owners_count":21176850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T09:01:34.392Z","updated_at":"2025-04-14T12:31:08.296Z","avatar_url":"https://github.com/netxfly.png","language":"Go","funding_links":[],"categories":["\u003ca id=\"d62a971d37c69db9f3b9187318c3921a\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"8ea8f890cf767c3801b5e7951fca3570\"\u003e\u003c/a\u003e公网访问局域网"],"readme":".. highlight:: go\n    :linenothreshold: 0\n\n.. 基于vpn和透明代理的web漏洞扫描器的实现 documentation master file, created by\n   sphinx-quickstart on Wed Jul 15 23:02:54 2015.\n   You can adapt this file completely to your liking, but it should at least\n   contain the root `toctree` directive.\n\n基于vpn和透明代理的web漏洞扫描器的实现\n============================================\n\n概述\n-----------\n\n.. note::\n    Transparent-Proxy-Scanner是一个基于vpn和透明代理的web漏洞扫描器，本文是vpn + 透明代理式的web漏洞扫描器的实现的简单说明。\n    用户连接vpn后访问网站时就会把网站的请求与响应信息保存到mongodb中，然后web扫描器从数据库中读取请求信息并进行扫描。\n\n架构说明\n------------------\n\n.. image:: transparent.png\n\n透明代理的实现\n----------------------\n\n透明代理是在 ``https://github.com/xiam/hyperfox`` 这个项目的基础上改的，hyperfox是go语言实现的一个http/https的透明代理。\n\nhyperfox本来是用 ``upper.io/db`` 这个orm将数据存入sqlite中的，我个人比较喜欢mongodb，于是就改成将数据存入mongodb中。\n\n依赖包安装\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n    go get github.com/netxfly/Transparent-Proxy-Scanner/hyperfox\n    go get github.com/toolkits/slice\n    go get upper.io/db\n    go get github.com/gorilla/mux\n    go get menteslibres.net/gosexy/to\n\n透明代理的部分实现代码\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n    package main\n\n    import (\n        \"flag\"\n        \"fmt\"\n        \"github.com/netxfly/Transparent-Proxy-Scanner/hyperfox/proxy\"\n        \"github.com/netxfly/Transparent-Proxy-Scanner/hyperfox/tools/capture\"\n        \"strings\"\n        // \"github.com/netxfly/Transparent-Proxy-Scanner/hyperfox/tools/logger\"\n        \"github.com/toolkits/slice\"\n        \"log\"\n        \"net/http\"\n        \"net/url\"\n        \"os\"\n        \"time\"\n        \"upper.io/db\"\n        \"upper.io/db/mongo\"\n    )\n\n    const version = \"0.9\"\n\n    const (\n        defaultAddress = `0.0.0.0`\n        defaultPort    = uint(3129)\n        defaultSSLPort = uint(3128)\n    )\n\n    const (\n        Host     = \"127.0.0.1\"\n        Port     = \"27017\"\n        User     = \"xsec\"\n        Password = \"x@xsec.io\"\n        Database = \"passive_scan\"\n    )\n\n    var settings = mongo.ConnectionURL{\n        Address:  db.Host(Host), // MongoDB hostname.\n        Database: Database,      // Database name.\n        User:     User,          // Optional user name.\n        Password: Password,      // Optional user password.\n    }\n\n    var (\n        flagAddress     = flag.String(\"l\", defaultAddress, \"Bind address.\")\n        flagPort        = flag.Uint(\"p\", defaultPort, \"Port to bind to, default is 3129\")\n        flagSSLPort     = flag.Uint(\"s\", defaultSSLPort, \"Port to bind to (SSL mode), default is 3128.\")\n        flagSSLCertFile = flag.String(\"c\", \"\", \"Path to root CA certificate.\")\n        flagSSLKeyFile  = flag.String(\"k\", \"\", \"Path to root CA key.\")\n    )\n\n    var (\n        sess db.Database\n        col  db.Collection\n    )\n\n    var (\n        static_resource []string = []string{\"js\", \"css\", \"jpg\", \"gif\", \"png\", \"exe\", \"zip\", \"rar\", \"ico\",\n            \"gz\", \"7z\", \"tgz\", \"bmp\", \"pdf\", \"avi\", \"mp3\", \"mp4\", \"htm\", \"html\", \"shtml\"}\n    )\n\n    // dbsetup sets up the database.\n    func dbsetup() error {\n        var err error\n        // Attemping to establish a connection to the database.\n        sess, err = db.Open(mongo.Adapter, settings)\n        fmt.Println(sess)\n\n        if err != nil {\n            log.Fatalf(\"db.Open(): %q\\n\", err)\n        }\n\n        // Pointing to the \"http_info\" table.\n        col, err = sess.Collection(\"http_info\")\n\n        return nil\n    }\n\n    // filter function\n    func filter(content_type string, raw_url string) bool {\n        ret := false\n        if strings.Contains(content_type, \"text/plain\") || strings.Contains(content_type, \"application/x-gzip\") {\n            url_parsed, _ := url.Parse(raw_url)\n            path := url_parsed.Path\n            t := strings.Split(path[1:], \".\")\n            suffix := t[len(t)-1]\n            if !slice.ContainsString(static_resource, suffix) {\n                ret = true\n            }\n\n        }\n        return ret\n    }\n\n    // Parses flags and initializes Hyperfox tool.\n    func main() {\n        var err error\n        var sslEnabled bool\n\n        // Parsing command line flags.\n        flag.Parse()\n\n        // Opening database.\n        if err = dbsetup(); err != nil {\n            log.Fatalf(\"db: %q\", err)\n        }\n\n        // Remember to close the database session.\n        defer sess.Close()\n\n        // Is SSL enabled?\n        if *flagSSLPort \u003e 0 \u0026\u0026 *flagSSLCertFile != \"\" {\n            sslEnabled = true\n        }\n\n        // User requested SSL mode.\n        if sslEnabled {\n            if *flagSSLCertFile == \"\" {\n                flag.Usage()\n                log.Fatal(ErrMissingSSLCert)\n            }\n\n            if *flagSSLKeyFile == \"\" {\n                flag.Usage()\n                log.Fatal(ErrMissingSSLKey)\n            }\n\n            os.Setenv(proxy.EnvSSLCert, *flagSSLCertFile)\n            os.Setenv(proxy.EnvSSLKey, *flagSSLKeyFile)\n        }\n\n        // Creatig proxy.\n        p := proxy.NewProxy()\n\n        // Attaching logger.\n        // p.AddLogger(logger.Stdout{})\n\n        // Attaching capture tool.\n        res := make(chan capture.Response, 256)\n\n        p.AddBodyWriteCloser(capture.New(res))\n\n        // Saving captured data with a goroutine.\n        go func() {\n            for {\n                select {\n                case r := \u003c-res:\n                    if filter(r.ContentType, r.URL) {\n                        // fmt.Println(r.Method, r.URL, r.ContentType)\n                        if _, err := col.Append(r); err != nil {\n                            log.Printf(ErrDatabaseError.Error(), err)\n                        }\n                    }\n\n                }\n            }\n        }()\n\n        cerr := make(chan error)\n\n        // Starting proxy servers.\n\n        go func() {\n            if err := p.Start(fmt.Sprintf(\"%s:%d\", *flagAddress, *flagPort)); err != nil {\n                cerr \u003c- err\n            }\n        }()\n\n        if sslEnabled {\n            go func() {\n                if err := p.StartTLS(fmt.Sprintf(\"%s:%d\", *flagAddress, *flagSSLPort)); err != nil {\n                    cerr \u003c- err\n                }\n            }()\n        }\n\n        err = \u003c-cerr\n\n        log.Fatalf(ErrBindFailed.Error(), err)\n    }\n\n\n如何启动透明代理\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1. 安装依赖包\n2. git clone ``https://github.com/netxfly/Transparent-Proxy-Scanner.git`` 到GOPATH目录下\n3. cd 到 ``$GOPATH/Transparent-Proxy-Scanner/hyperfox`` 目录下编译hyperfox，如下图所示：\n\n.. image:: 001.png\n\n4. 配置iptables，将80和443端口的请求分别转到透明代理的3129和3128端口，如下图所示：\n\n.. image:: 002.png\n\n透明代理抓取数据测试\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1. 注释掉调试代码，启动透明代理，手机拨入vpn，打开微博客户端后发现已经可以抓取到数据了，如下图所示：\n\n.. image:: 003.png\n\n2. 去mongodb中再确认下数据是否入库，如下图所示：\n\n.. image:: 004.png\n\n确认数据已经入库，接下来就该 ``任务分发模块`` 和 ``任务执行模块`` 出场了，上一篇文章已经写过了，这里就不写了，详情请参考\n``基于代理的Web扫描器的简单实现`` : http://www.xsec.io/article/77/proxy_scanner.html\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetxfly%2Ftransparent-proxy-scanner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetxfly%2Ftransparent-proxy-scanner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetxfly%2Ftransparent-proxy-scanner/lists"}