{"id":20859451,"url":"https://github.com/vndee/visee","last_synced_at":"2025-06-26T08:04:54.585Z","repository":{"id":40963703,"uuid":"238729726","full_name":"vndee/visee","owner":"vndee","description":"Just a typical search engine in this universe :fire::fire::fire:","archived":false,"fork":false,"pushed_at":"2025-06-15T07:55:09.000Z","size":103941,"stargazers_count":8,"open_issues_count":69,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-15T08:51:19.680Z","etag":null,"topics":["crawler","django","docker","e-commerce","elasticsearch","flask","kafka","python","visual-search"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vndee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-02-06T16:12:39.000Z","updated_at":"2023-02-27T05:12:20.000Z","dependencies_parsed_at":"2023-09-27T20:23:52.424Z","dependency_job_id":"2b157937-4782-4ac0-947a-376845d1da07","html_url":"https://github.com/vndee/visee","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/vndee/visee","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vndee%2Fvisee","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vndee%2Fvisee/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vndee%2Fvisee/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vndee%2Fvisee/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vndee","download_url":"https://codeload.github.com/vndee/visee/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vndee%2Fvisee/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262026982,"owners_count":23246954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","django","docker","e-commerce","elasticsearch","flask","kafka","python","visual-search"],"created_at":"2024-11-18T04:49:54.872Z","updated_at":"2025-06-26T08:04:54.567Z","avatar_url":"https://github.com/vndee.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"![forthebadge](https://img.shields.io/badge/built%20with-love-ff69b4)\n![CI](https://github.com/vndee/visee/workflows/CI/badge.svg)\n[![CodeFactor](https://www.codefactor.io/repository/github/vndee/visee/badge?s=9f1351e9a1c480decd19a6bbb7e8b19b447d8474)](https://www.codefactor.io/repository/github/vndee/visee)\n![](https://www.code-inspector.com/project/5387/score/svg)\n![](https://www.code-inspector.com/project/5387/status/svg)\n[![DepShield Badge](https://depshield.sonatype.org/badges/vndee/visee/depshield.svg)](https://depshield.github.io)\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"200\" height=\"100\" src=\"https://raw.githubusercontent.com/vndee/visee/master/imgs/logo.png?token=AGXWHAHFKIENLEQPVIJOZZK6QTQRQ\"\u003e\n\u003c/p\u003e\n\n**VISEE** is a system that combine both full-text search and visual search (base on image) together. Our system focus on \nVietnam e-commerce product, which was collected from [**Tiki**](https://tiki.vn/), [**Lazada**](https://www.lazada.vn/), [**Shopee**](https://shopee.vn/),\n[**Sendo**](https://www.sendo.vn/). **VISEE** is completely dockerization.\n\n## Installation\n\n### Requirements\n\n- docker, docker-dompose, nvidia-docker\n\n### Quickstart\n\nTo run all containers and services: \n\n    ./dev.sh up\n\nStop all services:\n\n    ./dev.sh down\n    \nYou can use `docker-compose` command alternatively. Especially when a service is running, its code were mount directly\nfrom host machine to docker container. So just edit your code and restart container, you will see your changes.\n\n### Configurations\n\nList of environment variables can be use to config VISEE. All variables define in `.env`.\n\n| Variable | Description | Deafult value |\n|----------|-------------|---------------|\n|`API_KEY`| Authorization key for REST API|`h$+wt\u0026%3BtH*6rA^KfPzMKDm**GdH_wQaQebd\u0026X9!h=nNVjrt+pn8GNB5%-_ug-U`|\n|`API_HOST`| REST API host binding (docker internal network)|`0.0.0.0`|\n|`API_PORT`| REST API port binding (docker internal network)|`7070`|\n|`KAFKA_HOSTS`| Kafka hosts | `[visee_kafka:9092]`|\n|`KAFKA_USER`| Kafka user| `None`|\n|`KAFKA_PASSWORD`| Kafka password| `None`|\n|`KAFKA_NUM_PARTITION`| Kafka number of partitions| `10`|\n|`KAFKA_LINK_TOPIC`| Kafka topic for links scraper| `Link item`|\n|`KAFKA_CONSUMER_GROUP`| Kafka consumer group| `default`|\n|`REDIS_HOST`| Redis host (docker internal network)| `visee_redis`|\n|`REDIS_PASSWORD`| Redis password| `None`|\n|`REDIS_CATEGORIES_DB`| Redis database for website categories| `0`|\n|`REDIS_LINK2SCRAPE_DB`| Redis database for link to scraper|`1`|\n|`REDIS_DB_IDX_FIRST`| Redis first database for `DualRedisConnector`|`2`|\n|`REDIS_DB_IDX_SECOND`| Redis second database for `DualRedisConnector`|`3`|\n|`MILVUS_HOST`| Milvus host (docker internal network)| `visee_milvus`|\n|`MILVUS_PORT`| Milvus port| `19530`|\n|`MILVUS_TABLE_NAME`| Milvus table name|`visee`|\n|`ELASTIC_HOSTS`| Elasticearch hosts (docker internal network)|`[visee_elasticsearch]`|\n|`ELASTIC_PORT`| Elasticsearch port| `9200`|\n|`ELASTIC_USER`| Elasticsearch username| `elastic`|\n|`ELASTIC_PASSWORD`| Elasticsearch password|`changeme`|\n|`ELASTIC_INDEX`| Elasticsearch index|`visee`|\n|`EFFNET_WEIGHT`| EfficientNet weights path (in container)| `/visee/static/eff_b7.pth`|\n|`CHROME_DRIVER_PATH`| Path to chrome driver (in container)| `/visee/static/chromedriver`|\n|`IMAGE_SIZE`| Image downloaded size| `1000`|\n|`DOWNLOAD_IMAGE`| Download image or not| `True`|\n\n### Libraries and frameworks\n\n- **Crawler:** Selenium, BeatifulSoup, Apache Kafka, Redis.\n- **Indexer:** PyTorch, Apache Kafka, Redis.\n- **Search Engine:** Elasticsearch, Milvus.\n- **RESTful Services:** Flask, Nginx, Gunicorn.\n- **User Interface:** NodeJS, Nginx, HTML + CSS + JS.\n- **Logging System:** ELK+ Stack (Elasticsearch, Logtash, Kibana, Beats).\n \n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/vndee/visee/master/imgs/visee.png?token=AGXWHAAP3HDB3NXCA7HRFMC6QYEAY\"\u003e\n  \u003cp align=\"center\"\u003eSystem Architecture and Technical Stack\u003c/p\u003e\n\u003c/p\u003e\n\nDevelopers: [**Duy V. Huynh**](https://github.com/vndee), [**Hoang N. Truong**](https://github.com/hoangperry/), [**Linh Q. Tran**](https://github.com/tql247/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvndee%2Fvisee","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvndee%2Fvisee","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvndee%2Fvisee/lists"}