{"id":13525955,"url":"https://github.com/madneal/gshark","last_synced_at":"2025-05-15T00:08:22.967Z","repository":{"id":37774902,"uuid":"124549469","full_name":"madneal/gshark","owner":"madneal","description":"Scan for sensitive information easily and effectively.","archived":false,"fork":false,"pushed_at":"2025-04-21T07:28:11.000Z","size":251420,"stargazers_count":960,"open_issues_count":1,"forks_count":145,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-21T07:40:04.794Z","etag":null,"topics":["git-leak","github","gitlab","go","golang","gorm","postman","vue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/madneal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-03-09T14:16:55.000Z","updated_at":"2025-04-21T06:27:50.000Z","dependencies_parsed_at":"2023-10-21T12:06:59.817Z","dependency_job_id":"2b151292-2b1f-43e8-abf2-ded750252d41","html_url":"https://github.com/madneal/gshark","commit_stats":{"total_commits":2108,"total_committers":6,"mean_commits":351.3333333333333,"dds":0.04127134724857684,"last_synced_commit":"fae41475b940bcbf54d58d49e85a2105ff9e3f62"},"previous_names":["neal1991/gshark"],"tags_count":64,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madneal%2Fgshark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madneal%2Fgshark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madneal%2Fgshark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madneal%2Fgshark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/madneal","download_url":"https://codeload.github.com/madneal/gshark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254249201,"owners_count":22039029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["git-leak","github","gitlab","go","golang","gorm","postman","vue"],"created_at":"2024-08-01T06:01:23.785Z","updated_at":"2025-05-15T00:08:17.954Z","avatar_url":"https://github.com/madneal.png","language":"Go","funding_links":[],"categories":["Go","HTML","HTML (177)","安全扫描"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n   \u003cimg alt=\"GShark logo\" src=\"https://s1.ax1x.com/2018/10/17/idhZvj.png\" /\u003e\n   \u003ch3 align=\"center\"\u003eGShark\u003c/h3\u003e\n   \u003cp align=\"center\"\u003eScan for sensitive information easily and effectively.\u003c/p\u003e\n\u003c/p\u003e\n\n# GShark [![Go Report Card](https://goreportcard.com/badge/github.com/madneal/gshark)](https://goreportcard.com/report/github.com/madneal/gshark)  [![Release](https://github.com/madneal/gshark/actions/workflows/release.yml/badge.svg)](https://github.com/madneal/gshark/actions/workflows/release.yml)\n\nThe project is based on Go and Vue to build a management system for sensitive information detection. For the full introduction, please refer to [articles](https://mp.weixin.qq.com/mp/appmsgalbum?__biz=MzI3MjA3MTY3Mw==\u0026action=getalbum\u0026album_id=2376148333116850178#wechat_redirect) and [videos](https://mp.weixin.qq.com/mp/appmsgalbum?__biz=MzI3MjA3MTY3Mw==\u0026action=getalbum\u0026album_id=1834365721464651778#wechat_redirect). For now, all the scans are only targeted to the public environments, not local environments.\n\nFor the usage of GShark, please refer to the [wiki](https://github.com/madneal/gshark/wiki).\n\n# Key Features\n\n* 🌐 Multi-Platform Support: GitHub, GitLab, Searchcode, Postman, and more* \n* 🔍 Flexible Rule Management: Custom scanning rules and filtering with whitelist/blacklist support*\n* 🔑 Fine-grained Access Control: Configurable menu and API permissions\n* 🔄 Subdomain Discovery: Integrated gobuster for subdomain enumeration\n* 🚀 Docker Deployment: Containerized deployment for easy setup\n* 📊 Visual Management Interface: Intuitive web interface for task and result management\n\n# Quick start\n\n## Docker Deployment\n\n```\n# Clone the repository\ngit clone https://github.com/madneal/gshark\n\ncd gshark\n\n# Build and start the container\ndocker-compose build \u0026\u0026 docker-compose up \n```\n\n\u003e [!IMPORTANT]\n\u003e Before the MySQL database initial, the scanner container will exit. Need to restart the scanner after the MySQL database initial.\n\n## Manual Deployment\n\n### Requirements\n\n* Nginx\n* MySQL(version above **8.0**)\n\nIt is recommended to deploy the Front-End project using Nginx. Place the `dist` folder in `/var/www/html`, and adjust the `nginx.conf` file (/etc/nginx/nginx.conf for Linux) to set up reverse proxy for the backend service. For detailed deployment tutorials, you can watch videos on [bilibili](https://www.bilibili.com/video/BV1Py4y1s7ap/) or [youtube](https://youtu.be/bFrKm5t4M54). For deployment on Windows, refer to [this link](https://www.bilibili.com/video/BV1CA411L7ux/).\n\n### Nginx\n\nCan use `nginx -t` to locate the `nginx.conf` file, then modify the `nginx.conf`:\n\n```\n// config the user accoring to your need\nuser  www www;\nworker_processes  1;\n\nevents {\n    worker_connections  1024;\n}\n\nhttp {\n    include       mime.types;\n    default_type  application/octet-stream;\n    sendfile        on;\n    keepalive_timeout  65;\n    server {\n        listen       8080;\n        server_name  localhost;\n\n        location / {\n            autoindex on;\n            root   html;\n            index  index.html index.htm;\n        }\n        location /api/ {\n            proxy_set_header Host $http_host;\n            proxy_set_header X-Real-IP $remote_addr;\n            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n            proxy_set_header X-Forwarded-Proto $scheme;\n            rewrite ^/api/(.*)$ /$1 break;\n            proxy_pass http://127.0.0.1:8888;\n        }\n        error_page   500 502 503 504  /50x.html;\n        location = /50x.html {\n            root   html;\n        }\n    }\n    include servers/*;\n}\n\n```\n\nThe deployment work is straightforward. Find the corresponding version zip file from [releases](https://github.com/madneal/gshark/releases). \n\nUnzip and copy the files inside `dist` to `/var/www/html` folder of Nginx. \n\n```\nunzip gshark*.zip\ncd gshark*\nmv dist/* /var/www/html/\n# for Mac\nmv dist/* /usr/local/www/html/\n```\n\nStart the Nginx and the Front-End is deployed successfully.\n\n\u003e [!TIP]\n\u003e If you installed Nginx by Homebrew, you need to stop Nginx by:\n\u003e ```shell\n\u003e brew services stop nginx\n\u003e ```\n\u003e Start Nginx for Ubuntu:\n\u003e ```shell\n\u003e systemctl start nginx\n\u003e ```\n\n### Server service\n\n```shell\n./gshark serve\n```\n\nInitially, rename `config-temp.yaml` to `config.yaml`. After that, you can run the `gshark` binary file directly. Then, access `localhost:8080` for local deployment.\n\nIf you haven't initialized the database before, you will be redirected to the database initialization page first.\n\n\u003cimg width=\"936\" alt=\"image\" src=\"https://github.com/user-attachments/assets/dfa7e53e-dc4a-4697-831f-a4f4f3810c3c\"\u003e\n\n### Scan service\n\n```shell\n./gshark scan\n```\n\nFor the scan service, it's necessary to config the corresponding rules. For example, GitHub or Gitlab rules.\n\n### Incremental Deployment\n\nFor the incremental deployment, [sql.md](https://github.com/madneal/gshark/blob/master/sql.md) should be executed for the corresponding database operations.\n\n## Development\n\n### Server\n\n```shell\ngit clone https://github.com/madneal/gshark.git\ncd server\ngo mod tidy\nmv config-temp.yaml config.yaml\ngo build\n```\n\nRun the web server:\n\n```shell\ngo build\n./gshark serve \n```\n\nOr\n\n```shell\ngo run main.go serve\n```\n\nRun the scan task:\n\n```shell\ngo build\n./gshark scan \n```\n\nOr\n\n```shell\ngo run main.go scan\n```\n\n### Web \n\n```\ncd ../web\n\nnpm install\n\nnpm run serve\n```\n\n## Usage\n### Add Token\n\n#### GitHub\n\nTo execute the scan task for GitHub, you need to add a GitHub token for crawl information in GitHub. You can generate a token in [tokens](https://github.com/settings/tokens). Most access scopes are enough. For the GitLab search, remember to add a token too.\n\n[![iR2TMt.md.png](https://s1.ax1x.com/2018/10/31/iR2TMt.md.png)](https://imgchr.com/i/iR2TMt)\n\n#### Postman\n\nObtain the `postman.sid` cookie:\n\n\u003cimg width=\"653\" alt=\"image\" src=\"https://github.com/madneal/gshark/assets/12164075/7775c8bb-79da-4e2b-b341-3c5b8395a6d0\"\u003e\n\n\n### Rule Configuration\n\nFor the Github or Gitlab rule, the rule will be matched by the syntax in the corresponding platforms. Directly, you config what you search at GitHub. You can download the rule import template CSV file, then batch import rules.\n\n\u003cimg width=\"572\" alt=\"image\" src=\"https://user-images.githubusercontent.com/12164075/212504597-3e1ad5bd-bacf-433e-83e8-08de7eee6509.png\"\u003e\n\n\n### Filter Configuration\n\nFilter is only addressed to GitHub search now. There are three classes of filters, including `extension`, `keyword`, `sec_keyword`. For `extension` and `keyword`, they can used for blacklist or whitelist.\n\nFor more information, you can refer to this [video](https://www.bilibili.com/video/BV1aG4y1c72N/?vd_source=ef4657ebf0549af8755f75118b6e81bb).\n\n## Configuration\n\nYou are supposed to rename `config-temp.yaml` to `config.yaml` and config the database information and other information according to your environment.\n\n### GitLab Base Url\n\n\u003cimg width=\"363\" alt=\"image\" src=\"https://user-images.githubusercontent.com/12164075/203898719-1ce66395-083d-4226-937f-b6eed859addc.png\"\u003e\n\n\n## FAQ\n\n1. Default username and password to login\n\ngshark/gshark\n\n2. Database initial failed\n\nmake sure the version of MySQL is over 5.6. And remove the database before initialing the second time.\n\n3. `go get ./... connection error`\n\nIt's suggested to enable GOPROXY(refer this [article](https://madneal.com/post/gproxy/) for golang upgrade):\n\n```\ngo env -w GOPROXY=https://goproxy.cn,direct\ngo env -w GO111MODULE=on\n```\n4. When deploying the web to `nginx`, the page was empty\n\ntry to clear the LocalStorage\n\n## Resources \n\n### Articles\n\n* [多平台的敏感信息监测工具-GShark](https://mp.weixin.qq.com/s?__biz=MzI3MjA3MTY3Mw==\u0026mid=2247484283\u0026idx=1\u0026sn=3232df7d321c0f62ce61b7e6368204ad\u0026chksm=eb396deddc4ee4fb0c825a378c085223b87fc45f05648d46e7bdc24a03fb83ad6c7ade414df7#rd)\n* [GShark-监测你的 Github 敏感信息泄露](https://mp.weixin.qq.com/s?__biz=MzI3MjA3MTY3Mw==\u0026mid=2247483770\u0026idx=1\u0026sn=9f02c2803e1c946e8c23b16ff3eba757\u0026chksm=eb396fecdc4ee6fa2f378e846f354f45acf6e6f540cfd54190e9353df47c7707e3a2aadf714f\u0026token=1578822041\u0026lang=zh_CN#rd)\n\n\n### Videos\n\n* [GShark v1.5.0 版本及 Docker 使用指南](https://www.bilibili.com/video/BV1oUe3eBEMz/)\n* [GShark v1.3.0 版本支持 Docker](https://www.bilibili.com/video/BV1BH4y1C7Ga/)\n* [GShark 支持多种规则类型以及规则配置建议](https://www.bilibili.com/video/BV1uY4y177SX) \n* [批量导入规则](https://mp.weixin.qq.com/s?__biz=MzI3MjA3MTY3Mw==\u0026mid=2247484546\u0026idx=1\u0026sn=818915279c5199457340ade89d6cbd54\u0026chksm=eb396a14dc4ee302039bcb1474380a6049dba84370345b7813049aa8feb49a98f89d47ec5d5b#rd)\n* [GShark部署](https://mp.weixin.qq.com/s?__biz=MzI3MjA3MTY3Mw==\u0026mid=2247484487\u0026idx=1\u0026sn=78f942ccf6861f433fc7f4a60564441c\u0026chksm=eb396ad1dc4ee3c7505362da243433e54a2b558c96fbbb50f8b6cea87d1f9bc920b249b72705#rd)\n* [windows 部署](https://mp.weixin.qq.com/s?__biz=MzI3MjA3MTY3Mw==\u0026mid=2247484289\u0026idx=1\u0026sn=2b0f1c38b88c924ad514fb64b559b784\u0026chksm=eb396d17dc4ee4018573dde6c3bfce83903c86034403539eaf1b87b89c4a4dd44f957a308818#rd)\n* [GShark v1.0.2 版本发布](https://www.bilibili.com/video/BV1Zx4y1G7FX/)\n* [GShark v1.1.0 更新内容介绍](https://www.bilibili.com/video/BV1aG4y1c72N/)\n\n\n## License\n\n[Apache License 2.0](https://github.com/madneal/gshark/blob/master/LICENSE)\n\n## 404StarLink 2.0 - Galaxy\n\n![](https://github.com/knownsec/404StarLink-Project/raw/master/logo.png)\n\nGShark 是 404Team [星链计划2.0](https://github.com/knownsec/404StarLink2.0-Galaxy)中的一环，如果对 GShark 有任何疑问又或是想要找小伙伴交流，可以参考星链计划的加群方式。\n\n- [https://github.com/knownsec/404StarLink2.0-Galaxy#community](https://github.com/knownsec/404StarLink2.0-Galaxy#community)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmadneal%2Fgshark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmadneal%2Fgshark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmadneal%2Fgshark/lists"}