{"id":26132186,"url":"https://github.com/shawonbarman/shipping-data-extractor","last_synced_at":"2025-03-10T23:01:48.133Z","repository":{"id":281711906,"uuid":"946173252","full_name":"ShawonBarman/shipping-data-extractor","owner":"ShawonBarman","description":"AI-powered tool for extracting structured shipping data from various document formats (PDF, Excel, CSV). Features multi-page processing, intelligent field extraction, interactive data table, column management, export options, and an AI chat assistant for data analysis. Perfect for logistics professionals.","archived":false,"fork":false,"pushed_at":"2025-03-10T18:28:26.000Z","size":354,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-10T19:31:15.599Z","etag":null,"topics":["css","flask","html","javascript","openai","python"],"latest_commit_sha":null,"homepage":"https://shipping-data-extractor-app.up.railway.app/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShawonBarman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-10T18:15:34.000Z","updated_at":"2025-03-10T18:28:30.000Z","dependencies_parsed_at":"2025-03-10T19:31:23.758Z","dependency_job_id":"d7c7b19f-f11c-4a85-9e3a-b03115e45ef1","html_url":"https://github.com/ShawonBarman/shipping-data-extractor","commit_stats":null,"previous_names":["shawonbarman/shipping-data-extractor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShawonBarman%2Fshipping-data-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShawonBarman%2Fshipping-data-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShawonBarman%2Fshipping-data-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShawonBarman%2Fshipping-data-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShawonBarman","download_url":"https://codeload.github.com/ShawonBarman/shipping-data-extractor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242940015,"owners_count":20209883,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["css","flask","html","javascript","openai","python"],"created_at":"2025-03-10T23:00:49.859Z","updated_at":"2025-03-10T23:01:48.095Z","avatar_url":"https://github.com/ShawonBarman.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shipping Data Extractor\n\nAI-powered tool for extracting structured shipping data from various document formats (PDF, Excel, CSV). Features multi-page processing, intelligent field extraction, interactive data table, column management, export options, and an AI chat assistant for data analysis.\n\n## Live: https://shipping-data-extractor-app.up.railway.app/\n\n## Features\n\n- **Multi-format Document Support**: Process PDFs, Excel files, CSVs, and images\n- **Intelligent Data Extraction**: AI-powered field recognition for shipping documents\n- **Multi-page Processing**: Full support for multi-page PDFs with page detection\n- **Interactive Data Table**: Sort, filter, and manage shipping data with ease\n- **Column Management**: Customize visible columns and their order\n- **Export Options**: Export to Excel, CSV, or JSON\n- **AI Assistant**: Ask questions about your shipping data\n- **Single-page Application**: Clean, intuitive interface with responsive design\n\n## Tech Stack\n\n- **Frontend**: HTML, CSS, JavaScript\n- **Backend**: Python with Flask\n- **AI Integration**: OpenAI API (GPT-4) for document processing\n- **PDF Processing**: PyPDF2\n- **Data Handling**: Pandas\n- **Styling**: Bootstrap 5, Font Awesome\n\n## Installation\n\n### Prerequisites\n\n- Python 3.8+\n- OpenAI API key\n\n### Setup\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/ShawonBarman/shipping-data-extractor.git\n   cd shipping-data-extractor\n   ```\n\n2. Create and activate a virtual environment:\n   ```bash\n   python -m venv venv\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\n   ```\n\n3. Install dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. Create a `.env` file in the project root with your OpenAI API key:\n   ```\n   OPENAI_API_KEY=your_api_key_here\n   SECRET_KEY=your_flask_secret_key_here\n   ```\n\n5. Run the application:\n   ```bash\n   python app.py\n   ```\n\n6. Open your browser and navigate to `http://127.0.0.1:5000`\n\n## Usage\n\n1. **Upload Documents**: Drag and drop shipping documents onto the upload area\n2. **Process Files**: Click \"Extract Data\" to start the AI processing\n3. **View Results**: Explore the extracted data in the interactive table\n4. **Customize View**: Toggle column visibility or reorder columns as needed\n5. **Export Data**: Export to your preferred format (Excel, CSV, JSON)\n6. **Ask Questions**: Use the AI assistant to analyze your shipping data\n\n## Project Structure\n\n```\nproject_root/\n│\n├── app.py                  # Main Flask application\n│\n├── static/\n│   ├── css/\n│   │   └── styles.css      # Main stylesheet\n│   │\n│   ├── js/\n│   │   └── main.js         # Application JavaScript\n│\n├── templates/\n│   └── index.html          # Single-page application template\n│\n├── uploads/                # Folder for uploaded files\n│\n├── .env                    # Environment variables (not in repo)\n│\n└── requirements.txt        # Python dependencies\n```\n\n## Configuration\n\nKey configuration options:\n\n- `UPLOAD_FOLDER`: Directory where uploaded files are stored\n- `ALLOWED_EXTENSIONS`: File extensions that can be processed\n- `TABLE_FIELDS`: Fields to extract from shipping documents\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- OpenAI for GPT models powering the extraction logic\n- Bootstrap team for responsive UI components\n- Font Awesome for icons\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshawonbarman%2Fshipping-data-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshawonbarman%2Fshipping-data-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshawonbarman%2Fshipping-data-extractor/lists"}