{"id":28276503,"url":"https://github.com/hahaanisha/deduper","last_synced_at":"2026-01-20T16:28:12.915Z","repository":{"id":288285045,"uuid":"967418027","full_name":"hahaanisha/DeDuper","owner":"hahaanisha","description":"File Deduplication for ownCloud using SHA-256 and AWS Lambda — a serverless, scalable solution for reducing redundant file storage across users, powered by the AWS ecosystem and integrated with ownCloud.","archived":false,"fork":false,"pushed_at":"2025-04-17T08:58:33.000Z","size":1601,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-17T23:39:06.603Z","etag":null,"topics":["aws","deduplication","node-js","owncloud"],"latest_commit_sha":null,"homepage":"https://tejass-organization-5.gitbook.io/deduper","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hahaanisha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-16T12:29:18.000Z","updated_at":"2025-04-17T09:01:47.000Z","dependencies_parsed_at":"2025-04-16T22:08:45.345Z","dependency_job_id":"080fe713-f05d-4dee-8827-8393bb575ff1","html_url":"https://github.com/hahaanisha/DeDuper","commit_stats":null,"previous_names":["hahaanisha/deduper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hahaanisha/DeDuper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahaanisha%2FDeDuper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahaanisha%2FDeDuper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahaanisha%2FDeDuper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahaanisha%2FDeDuper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hahaanisha","download_url":"https://codeload.github.com/hahaanisha/DeDuper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahaanisha%2FDeDuper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28607100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T16:10:39.856Z","status":"ssl_error","status_checked_at":"2026-01-20T16:10:39.493Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","deduplication","node-js","owncloud"],"created_at":"2025-05-21T05:10:42.154Z","updated_at":"2026-01-20T16:28:12.910Z","avatar_url":"https://github.com/hahaanisha.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FileCloud - Deduplication System\n\nFileCloud is a file storage and management system with built-in deduplication using SHA256 checksums. This project consists of a FastAPI middleware, frontend interface, and Node.js server.\n\n![FileCloud Logo](readme_screenshots/header.png)\n\u003c!-- Add your logo screenshot here --\u003e\n\n## Features\n\n- ✅ File upload with drag-and-drop interface\n- ✅ SHA256-based file deduplication\n- ✅ AWS S3 integration for storage\n- ✅ RESTful API middleware\n- ✅ Modern OwnCloud-like UI\n- ✅ Progress tracking for uploads\n- ✅ User-based file management\n\n## Screenshots\n\n### Frontend\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"readme_screenshots/UI.jpeg\" alt=\"Architecture Diagram\" style=\"width:700px;\" /\u003e\n\u003c/p\u003e\n\u003c!-- Add your dashboard screenshot here --\u003e\n\n## Architecture\n\nThe project consists of three main components:\n\n1. **FastAPI Middleware**: Handles file processing, deduplication, and AWS S3 interactions\n2. **Frontend UI**: OwnCloud-like interface for users to interact with the system\n3. **Node.js Server**: Serves the frontend assets\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"readme_screenshots/architecture.png\" alt=\"Architecture Diagram\" style=\"width:700px;\" /\u003e\n\u003c/p\u003e\n\n\u003c!-- Add your architecture diagram here --\u003e\n\n## Installation \u0026 Setup\n\n### Prerequisites\n\n- Python 3.8+\n- Node.js 14+\n- AWS Account with S3 bucket configured\n- AWS Lambda (optional for additional processing)\n\n### 1. FastAPI Middleware Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/filecloud.git\ncd filecloud/middleware\n\n# Create virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n\n# Install dependencies\npip install fastapi uvicorn python-dotenv requests\n\n# Create .env file\necho \"UPLOAD_API=https://your-aws-lambda-url.amazonaws.com/upload\" \u003e .env\n\n# Run the server\nuvicorn main:app --reload --port 8000\n```\n\n### 2. Frontend Setup\n\n```bash\n# Navigate to frontend directory\ncd ../frontend\n\n# Install dependencies\nnpm install\n\n# Run the development server\nnode server.js\n```\n\nThe frontend will be available at `http://localhost:5501`.\n\n### 3. Configuration\n\nCreate a `.env` file in the middleware directory with the following:\n\n```\nUPLOAD_API=https://your-aws-lambda-url.amazonaws.com/upload\n```\n\n## API Endpoints\n\n### File Upload\n\n```\nPOST /owncloud-upload?userID={userID}\n```\n\n**Parameters:**\n- `userID`: User identifier for file ownership\n\n**Request Body:**\n- Form data with `file` field containing the file to upload\n\n**Response:**\n```json\n{\n  \"fileName\": \"example.jpg\",\n  \"fileHash\": \"8f7d88e901a5ad3a05d8cc0de93313fd76028f076ccd1d416cbf0fdb3e1c10fe\",\n  \"s3Key\": \"user123/example.jpg\",\n  \"deduplicated\": false\n}\n```\n\n## Detailed Documentation\n\nFor more detailed documentation, please visit our [GitBook](https://tejass-organization-5.gitbook.io/deduper).\n\nThe documentation includes:\n- Complete API reference\n- AWS setup guide\n- Advanced configuration options\n- Performance optimization tips\n- Security best practices\n\n## Development\n\n### Running Tests\n\n```bash\n# For middleware tests\ncd middleware\npytest\n\n# For frontend tests\ncd frontend\nnpm test\n```\n\n### Building for Production\n\n```bash\n# Build frontend\ncd frontend\nnpm run build\n\n# Deploy middleware\ncd middleware\n# Follow deployment steps for your hosting platform\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## Collaborators\n\n- [Anisha Shankar](https://github.com/hahaanisha)\n- [Tejas Gadge ](https://github.com/tejasgadge2504)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhahaanisha%2Fdeduper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhahaanisha%2Fdeduper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhahaanisha%2Fdeduper/lists"}