{"id":48939974,"url":"https://github.com/scanoss/folder-hashing-api","last_synced_at":"2026-04-17T13:11:54.848Z","repository":{"id":318758669,"uuid":"910536420","full_name":"scanoss/folder-hashing-api","owner":"scanoss","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-29T08:43:08.000Z","size":50999,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-29T10:34:07.992Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scanoss.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-31T15:00:40.000Z","updated_at":"2025-10-29T08:42:38.000Z","dependencies_parsed_at":"2025-10-15T06:40:33.624Z","dependency_job_id":"b395a541-d80d-43a1-b21c-ca3aaf8f8f23","html_url":"https://github.com/scanoss/folder-hashing-api","commit_stats":null,"previous_names":["scanoss/folder-hashing-api"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/scanoss/folder-hashing-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Ffolder-hashing-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Ffolder-hashing-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Ffolder-hashing-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Ffolder-hashing-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scanoss","download_url":"https://codeload.github.com/scanoss/folder-hashing-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Ffolder-hashing-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31930255,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T12:37:54.787Z","status":"ssl_error","status_checked_at":"2026-04-17T12:37:25.095Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-17T13:11:54.088Z","updated_at":"2026-04-17T13:11:54.841Z","avatar_url":"https://github.com/scanoss.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SCANOSS Folder Hashing API\n\n[![License](https://img.shields.io/badge/License-GPL%20v2%2B-blue.svg)](LICENSE)\n[![Go Version](https://img.shields.io/badge/Go-1.22+-00ADD8.svg)](go.mod)\n\nA high-performance REST and gRPC API service for component fingerprinting and similarity matching using Qdrant vector database. The SCANOSS Folder Hashing API enables efficient code component analysis and similarity detection for software composition analysis.\n\n## Prerequisites\n\n- **Go 1.22+**: For building and running the service\n- **Qdrant**: Vector database (must be running and accessible)\n\n## Quick Start\n\n```bash\n# 1. Clone the repository\ngit clone https://github.com/scanoss/folder-hashing-api.git\ncd folder-hashing-api\n\n# 2. Set up configuration\ncp config.example.json config/app-config.json\n# Edit config/app-config.json as needed\n\n# 3. Build the service\nmake build_amd  # or build_arm for ARM64\n\n# 4. Run the service\n./dist/scanoss-hfh-api --json-config config/app-config.json\n\n# 5. Verify it's running\ncurl -X POST -H \"Content-Type: application/json\" -d '{\"message\":\"test\"}' http://localhost:40061/api/v2/scanning/echo\n```\n\n## Service Endpoints\n\nOnce running, the service provides:\n\n| Service | Default Endpoint | Description |\n|---------|----------|-------------|\n| **REST API** | http://localhost:40061 | Main API interface |\n| **gRPC API** | localhost:50061 | High-performance gRPC interface |\n| **Dynamic Logging** | localhost:60061 | Runtime log level control |\n\n## Configuration\n\n### JSON Configuration (Recommended)\n\n```json\n{\n  \"App\": {\n    \"Name\": \"SCANOSS HFH Server\",\n    \"GRPCPort\": \"50061\",\n    \"RESTPort\": \"40061\",\n    \"Debug\": false,\n    \"Mode\": \"production\"\n  },\n  \"Hfh\": {\n    \"QdrantHost\": \"localhost\",\n    \"QdrantPort\": 6334\n  },\n  \"Logging\": {\n    \"DynamicLogging\": true,\n    \"DynamicPort\": \"localhost:60061\"\n  },\n  \"Telemetry\": {\n    \"Enabled\": false,\n    \"OltpExporter\": \"0.0.0.0:4317\"\n  }\n}\n```\n\n### Environment Variables\n\n```bash\nexport APP_PORT=50061\nexport REST_PORT=40061\nexport QDRANT_HOST=localhost\nexport QDRANT_PORT=6334\nexport APP_DEBUG=true\n```\n\n### Command Line Flags\n\n```bash\n# Using JSON config\n./dist/scanoss-hfh-api --json-config config/app-config.json\n\n# Using environment file\n./dist/scanoss-hfh-api --env-config .env\n\n# With debug flag\n./dist/scanoss-hfh-api --debug --json-config config/app-config.json\n```\n\n## Building\n\n```bash\n# Build for AMD64\nmake build_amd\n\n# Build for ARM64\nmake build_arm\n\n# Run locally (development)\nmake run_local\n```\n\n## Testing\n\n```bash\n# Run all tests\nmake test\n\n# Run with coverage\ngo test -v -cover ./...\n\n# Run linting\nmake lint_local\n\n# Auto-fix linting issues\nmake lint_local_fix\n```\n\n## Importing Data\n\nThe `cmd/import/main.go` tool allows you to populate the Qdrant vector database with component data from CSV files.\n\n### Basic Usage\n\n```bash\n# Build the import tool\ngo build -o dist/import-tool cmd/import/main.go\n\n# Update database (adds/updates data in existing collections)\n./dist/import-tool \\\n  -dir /path/to/csv/directory \\\n  -top-purls /path/to/top-purls.json\n\n# Recreate database (deletes existing collections and imports fresh)\n./dist/import-tool \\\n  -dir /path/to/csv/directory \\\n  -top-purls /path/to/top-purls.json \\\n  -overwrite\n\n# Specify Qdrant host and port\n./dist/import-tool \\\n  -dir /path/to/csv/directory \\\n  -top-purls /path/to/top-purls.json \\\n  -overwrite \\\n  -qdrant-host my-qdrant-host.example.com \\\n  -qdrant-port 5555\n```\n\n### Command Options\n\n| Flag | Required | Description |\n|------|----------|-------------|\n| `-dir` | Yes | Directory containing CSV files to import |\n| `-top-purls` | Yes | JSON file with PURL rankings for search prioritization |\n| `-overwrite` | No | Delete and recreate all collections (use for fresh start) |\n\n### How It Works\n\nThe import tool:\n- Processes CSV files in parallel using 12 concurrent workers\n- Groups components by programming language into separate collections (e.g., `py_collection`, `js_collection`)\n- Creates optimized vector indexes with named vectors (`dirs`, `names`, `contents`)\n- Handles large datasets with batching (2000 records per batch)\n\n### Example Workflow\n\n```bash\n# 1. Ensure Qdrant is running\n# (Start your Qdrant instance)\n\n# 2. Build the tool\ngo build -o dist/import-tool cmd/import/main.go\n\n# 3. Import your data\n./dist/import-tool \\\n  -dir /data/csv/ \\\n  -top-purls /data/top-purls.json\n\n# 4. Verify collections were created\ncurl http://localhost:6333/collections\n```\n\n## Development\n\n### Local Development Setup\n\n```bash\n# Install dependencies\ngo mod download\n\n# Run tests\nmake test\n\n# Run linting\nmake lint\n\n# Build locally\nmake build_amd64\n\n# Run locally\nmake run\n```\n\n### Available Make Targets\n\n```bash\nmake help                 # Show all available commands\nmake build_amd64          # Build for AMD64\nmake build_arm64          # Build for ARM64\nmake run                  # Run the service locally\nmake test                 # Run all unit tests\nmake lint                 # Run linting\nmake lint-fix             # Run linting with auto-fix\nmake clean_testcache      # Clean Go test caches\n```\n\n## Troubleshooting\n\n### API not responding\n\n```bash\n# Check if service is running\nps aux | grep scanoss-hfh-api\n\n# Check configuration\ncat config/app-config.json\n\n# Run with debug logging\n./dist/scanoss-hfh-api --debug --json-config config/app-config.json\n```\n\n### Qdrant connection issues\n\n```bash\n# Verify Qdrant is accessible\ncurl http://localhost:6333/collections\n\n# Check Qdrant host/port in config\ngrep -A 3 \"Hfh\" config/app-config.json\n```\n\n### Import tool issues\n\n```bash\n# Verify CSV directory exists and contains files\nls -la /path/to/csv/directory/\n\n# Verify top-purls.json is valid JSON\ncat /path/to/top-purls.json | jq .\n\n# Run with verbose output\n./dist/import-tool -dir /path/to/csv/ -top-purls /path/to/top-purls.json\n```\n\n## Documentation\n\n- **API Documentation**: Available at REST endpoints when service is running\n- **Configuration Reference**: See `config.example.json` for all available options\n- **Scripts**: Check `scripts/` directory for additional utilities\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Make your changes\n4. Run tests (`make test`)\n5. Run linting (`make lint_local`)\n6. Commit your changes (`git commit -m 'Add amazing feature'`)\n7. Push to the branch (`git push origin feature/amazing-feature`)\n8. Open a Pull Request\n\n## License\n\nThis project is licensed under the GPL v2+ License - see the [LICENSE](LICENSE) file for details.\n\n## Links\n\n- **SCANOSS Website**: [https://www.scanoss.com](https://www.scanoss.com)\n- **Documentation**: [https://docs.scanoss.com](https://docs.scanoss.com)\n- **GitHub**: [https://github.com/scanoss/folder-hashing-api](https://github.com/scanoss/folder-hashing-api)\n\n---\n\n**Built with ❤️ by the SCANOSS Team**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscanoss%2Ffolder-hashing-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscanoss%2Ffolder-hashing-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscanoss%2Ffolder-hashing-api/lists"}