{"id":18304312,"url":"https://github.com/hiago-balbino/web-crawler","last_synced_at":"2026-04-10T17:04:53.791Z","repository":{"id":64535889,"uuid":"440356876","full_name":"hiago-balbino/web-crawler","owner":"hiago-balbino","description":"Project created for learning purposes and is a crawler that go through the web looking for any information by clicking on each available link.","archived":false,"fork":false,"pushed_at":"2024-02-08T02:04:48.000Z","size":260,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-15T04:35:37.829Z","etag":null,"topics":["cobra","direnv","docker","gin-gonic","go","godoc","golang","golangci-lint","grafana","makefile","mongodb","prometheus","sonarqube","testcontainers-go","testify","viper","zaplogger"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hiago-balbino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-21T01:39:48.000Z","updated_at":"2023-07-19T02:55:33.000Z","dependencies_parsed_at":"2023-11-30T03:26:38.960Z","dependency_job_id":"4305b41b-723d-4b37-8ac4-2b14f54adae1","html_url":"https://github.com/hiago-balbino/web-crawler","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiago-balbino%2Fweb-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiago-balbino%2Fweb-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiago-balbino%2Fweb-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiago-balbino%2Fweb-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hiago-balbino","download_url":"https://codeload.github.com/hiago-balbino/web-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248019151,"owners_count":21034217,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cobra","direnv","docker","gin-gonic","go","godoc","golang","golangci-lint","grafana","makefile","mongodb","prometheus","sonarqube","testcontainers-go","testify","viper","zaplogger"],"created_at":"2024-11-05T15:28:28.508Z","updated_at":"2025-12-30T23:05:31.952Z","avatar_url":"https://github.com/hiago-balbino.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Go Reference](https://pkg.go.dev/badge/github.com/hiago-balbino/web-crawler.svg)](https://pkg.go.dev/github.com/hiago-balbino/web-crawler)\n[![Go Report Card](https://goreportcard.com/badge/github.com/hiago-balbino/web-crawler)](https://goreportcard.com/report/github.com/hiago-balbino/web-crawler)\n[![Linter](https://github.com/hiago-balbino/web-crawler/actions/workflows/linter.yml/badge.svg?branch=main\u0026event=push\u0026logo=github\u0026style=flat-square)](https://github.com/hiago-balbino/web-crawler/actions/workflows/linter.yml)\n[![Security](https://github.com/hiago-balbino/web-crawler/actions/workflows/security.yml/badge.svg?branch=main\u0026event=push\u0026logo=github\u0026style=flat-square)](https://github.com/hiago-balbino/web-crawler/actions/workflows/security.yml)\n[![Tests](https://github.com/hiago-balbino/web-crawler/actions/workflows/tests.yml/badge.svg?branch=main\u0026event=push\u0026logo=github\u0026style=flat-square)](https://github.com/hiago-balbino/web-crawler/actions/workflows/tests.yml)\n\n# 🔍 WEB-CRAWLER\nThis project was created for learning purposes and is a crawler that go through the web looking for any information by clicking on each available link. \n\nSome tools used do not represent the best choice, they were only used for learning purposes. For example MongoDB was used, but thinking about performance Redis might be a better alternative. The frontend was not the focus for learning purposes, so the `template package` was used.\n\n## 🧰 Dependencies\n* [Go](https://golang.google.cn/dl) 1.20+\n* [Docker](https://www.docker.com/products/docker-desktop)\n* [Docker-compose](https://docs.docker.com/compose/install)\n* [Nginx](https://nginx.org)\n* [GNU make](https://www.gnu.org/software/make)\n* [Direnv](https://direnv.net)\n    * This is not mandatory but is a easily way to control your environment variable for each project without configuring the variables globally\n* [Golangci-lint](https://golangci-lint.run)\n* [Godoc](https://pkg.go.dev/golang.org/x/tools/cmd/godoc)\n* [Govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck)\n* [Deadcode](https://go.dev/blog/deadcode)\n* [Viper](https://github.com/spf13/viper)\n    * Configuration solution\n* [Cobra](https://github.com/spf13/cobra)\n    * Library for creating CLI applications\n* [Gin web framework](https://github.com/gin-gonic/gin)\n* [Gin metrics exporter for Prometheus](https://github.com/penglongli/gin-metrics)\n* [Prometheus client golang](https://github.com/prometheus/client_golang)\n* [Testify](https://github.com/stretchr/testify)\n    * Tools for testifying\n* [Testcontainers-Go](https://github.com/testcontainers/testcontainers-go)\n* [Zap logger](https://go.uber.org/zap)\n* [MongoDB](https://www.mongodb.com)\n    * The web-based admin interface Mongo Express was used\n* [HTTP expect](https://github.com/gavv/httpexpect)\n    * Used to API test\n* [Gock HTTP mocking](https://github.com/h2non/gock)\n* [Sonarqube](https://www.sonarqube.org)\n\n## 🛠️ Useful commands\nYou can run the command below to see all the useful commands available and your goals.\n```\nmake help\n```\n\n## ⚙️ Running the Application\nTo run the project locally you need to export some environment variables and this can be done using `direnv`. You can export the variables below.\n```\nNGINX_PORT='80'\nLOG_LEVEL='ERROR'\n\nMONGODB_USERNAME='root'\nMONGODB_PASSWORD='example'\nMONGODB_DATABASE='crawler'\nMONGODB_COLLECTION='page'\nMONGODB_PORT='27017'\n\nMONGODB_EXPRESS_USERNAME='root'\nMONGODB_EXPRESS_PASSWORD='example'\nMONGODB_EXPRESS_PORT='8081'\n```\n\nAfter exporting the environment variables, you can run the `make compose-up` command and open the `http://localhost/index` address. \n\nIf you want to run the API outside of Docker:\n* you need to export the `MONGODB_HOST` variable as `localhost`\n* expose the `MONGODB_PORT` in `docker-compose.yml` in the `mongo` service as below\n    ```\n    ports:\n      - ${MONGODB_PORT}:${MONGODB_PORT}\n    ```\n* comment out the `api` and `nginx` service in `docker-compose.yml` and run `make compose-up` and then `make build-run-api`, the API will run on the default port `http://localhost:8888/index`\n* if you want to debug the API, you don't need to run `make build-run-api`, but in your IDE you need to set the command to `api` when starting the application given its using [cobra library](https://github.com/spf13/cobra)\n\n## 🏁 How to crawl the page\nFill in the URI and Depth in the form(it will be used to limit the depth when fetching pages with so many links that they can underperform and can take so long).\n\n_By default, the HTTP request takes a timeout of 60 seconds which can be changed by environment variable(API_REQUEST_TIMEOUT)._\n\n## 📜 Running Internal Documentation\nYou can do this by running the `make doc` command and going to the address `http://localhost:6060`.\n\n## 🎯 How to run sonarqube locally\nThe project is set up to run `sonarqube` locally and has three commands in the Makefile. The `sonarqube` will be downloaded by Docker, but you need to [install sonar-scanner following your operating system](https://docs.sonarqube.org/latest/analyzing-source-code/scanners/sonarscanner).\n\nTo run `sonarqube` locally, you need to export the following environment variables. You can do this using `direnv`.\n```\nSONAR_PORT='9000'\nSONAR_HOST='http://localhost:9000'\nSONAR_LOGIN='admin'\nSONAR_PASSWORD='admin'\nSONAR_BINARY='Here you need to fill it according to your operational system. Example: sonar-scanner for Linux/MacOS or sonar-scanner.bat for Windows'\n```\n\nAfter installing and configuring `sonar-scanner` in `$PATH`(if needed) you will be able to run the commands below. By running the `make sonarqube-up` and `make sonarqube-analysis` commands you can open the `http://localhost:9000` address in your browser and login and password as `admin`(perhaps `sonarqube` may prompt you to change your password).\n```\nsonarqube-up: start sonarqube container\nsonarqube-analysis: run sonar scanner\nsonarqube-down: stop sonarqube container\n```\n\n## 📊 Running the metrics\nThe project was instrumented using `Prometheus` and `Grafana`, both of which are configured and downloaded through Docker. Prometheus and Grafana will run together with the application, but you need to export the following environment variables below, and you can do this using `direnv`.\n```\nPROMETHEUS_PORT='9090'\nGRAFANA_PORT='3000'\n```\n\nThe application metrics are exposed using the [ginmetrics library](https://github.com/penglongli/gin-metrics) and can be accessed at `http://localhost:8888/metrics`. These exposed metrics are collected by Prometheus and can be accessed at `http://localhost:9090`. \n\nThe collected metrics are sent to Grafana and can be accessed at `http://localhost:3000`. The default credentials are `admin`/`admin`(Grafana may prompt you to reset the password, but it is optional). \n\n_Please note that due to some changes to the tools, the arrangement of items may be different._\n\nAfter that, you need to configure the `data source` by clicking on the `Configuration` option in the left hand panel and then clicking on `Data source`. Click on the `Add Data Source` button and select `Prometeus` under `Time Series Database`. Fill in the address in the HTTP option as in the image below:\n\n[![datasource](/metrics/assets/datasource.png)](/metrics/assets/datasource.png)\n\nAfter setting up the data source, you can import the file from the dashboard by clicking on the `Dashboard` option in the left side panel and then clicking `+ Import`. You can upload the file placed in this project at `/metrics/grafana/dashboards.json`. After it is loaded, you will see the panels as below:\n\n[![metrics](/metrics/assets/metrics.png)](/metrics/assets/metrics.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiago-balbino%2Fweb-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhiago-balbino%2Fweb-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiago-balbino%2Fweb-crawler/lists"}