{"id":43171946,"url":"https://github.com/opencitations/oc_download","last_synced_at":"2026-02-01T02:35:40.442Z","repository":{"id":294176114,"uuid":"944428416","full_name":"opencitations/oc_download","owner":"opencitations","description":"This repository contains the DOWNLOAD service for OpenCitations","archived":false,"fork":false,"pushed_at":"2025-12-12T11:32:44.000Z","size":6635,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-12-14T01:26:17.721Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://download.opencitations.net/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opencitations.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-07T10:27:25.000Z","updated_at":"2025-12-12T11:32:18.000Z","dependencies_parsed_at":"2025-07-14T13:03:56.491Z","dependency_job_id":"37db6001-888f-484b-bfe3-ad4aafcf7f30","html_url":"https://github.com/opencitations/oc_download","commit_stats":null,"previous_names":["opencitations/oc_download"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/opencitations/oc_download","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_download","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_download/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_download/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_download/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opencitations","download_url":"https://codeload.github.com/opencitations/oc_download/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_download/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28965430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T02:14:24.993Z","status":"ssl_error","status_checked_at":"2026-02-01T02:13:55.706Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-01T02:35:39.793Z","updated_at":"2026-02-01T02:35:40.434Z","avatar_url":"https://github.com/opencitations.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenCitations DOWNLOAD Service\n\nThis repository contains the DOWNLOAD service for OpenCitations.\n\n### Environment Variables\n\nThe service requires the following environment variables. These values take precedence over the ones defined in `conf.json`:\n\n- `BASE_URL`: Base URL for the DOWNLOAD endpoint\n- `LOG_DIR`: Directory path where log files will be stored\n- `SYNC_ENABLED`: Enable/disable static files synchronization (default: false)\n\nFor instance:\n\n```env\nBASE_URL=download.opencitations.net\nLOG_DIR=/home/dir/log/\nSYNC_ENABLED=true\n```\n\n\u003e **Note**: When running with Docker, environment variables always override the corresponding values in `conf.json`. If an environment variable is not set, the application will fall back to the values defined in `conf.json`.\n\n### Static Files Synchronization\n\nThe application can synchronize static files from a GitHub repository. This configuration is managed in `conf.json`:\n\n```json\n{\n  [...]\n  \"oc_services_templates\": \"https://github.com/opencitations/oc_services_templates\",\n  \"sync\": {\n    \"folders\": [\n      \"static\",\n      \"html-template/common\"\n    ],\n    \"files\": [\n      \"test.txt\"\n    ]\n  }\n}\n```\n\n- `oc_services_templates`: The GitHub repository URL to sync files from\n- `sync.folders`: List of folders to synchronize\n- `sync.files`: List of individual files to synchronize\n\nWhen static sync is enabled (via `--sync-static` or `SYNC_ENABLED=true`), the application will:\n1. Clone the specified repository\n2. Copy the specified folders and files\n3. Keep the local static files up to date\n\n\u003e **Note**: Make sure the specified folders and files exist in the source repository.\n\n## Running Options\n\n\n### Local Development\n\nThis project uses [uv](https://docs.astral.sh/uv/) for dependency management.\n\nFirst, install uv package manager and project dependencies:\n```bash\n# Install uv package manager\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n# Install project dependencies\nuv sync\n```\n\nFor local development and testing, the application uses the built-in web.py HTTP server:\n\nExamples:\n```bash\n# Run with default settings\nuv run download_oc.py\n\n# Run with static sync enabled\nuv run download_oc.py --sync-static\n\n# Run on custom port\nuv run download_oc.py --port 8085\n\n# Run with both options\nuv run download_oc.py --sync-static --port 8085\n```\n\nThe application supports the following command line arguments:\n\n- `--sync-static`: Synchronize static files at startup\n- `--port PORT`: Specify the port to run the application on (default: 8080)\n\n### Production Deployment (Docker)\n\nWhen running in Docker/Kubernetes, the application uses **Gunicorn** as the WSGI HTTP server for better performance and concurrency handling:\n\n- **Server**: Gunicorn with gevent workers\n- **Workers**: 4 concurrent worker processes\n- **Worker Type**: gevent (async) for handling thousands of simultaneous requests\n- **Timeout**: 1200 seconds (to handle long-running SPARQL queries)\n- **Connections per worker**: 400 simultaneous connections\n\nThe Docker container automatically uses Gunicorn and is configured with static sync enabled by default.\n\n\u003e **Note**: The application code automatically detects the execution environment. When run with `uv run download_oc.py`, it uses the built-in web.py server. When run with Gunicorn (as in Docker), it uses the WSGI interface.\nYou can customize the Gunicorn server configuration by modifying the `gunicorn.conf.py` file.\n\n### Dockerfile\n\nYou can change these variables in the Dockerfile:\n\n```dockerfile\n# Base image: Python slim for a lightweight container\nFROM python:3.11-slim\n\n# Define environment variables with default values\n# These can be overridden during container runtime\nENV BASE_URL=\"download.opencitations.net\" \\\n    SYNC_ENABLED=\"true\" \\\n    LOG_DIR=\"/mnt/log_dir/oc_download\"\n\n# Ensure Python output is unbuffered\nENV PYTHONUNBUFFERED=1\n\n# Install system dependencies required for Python package compilation\n# We clean up apt cache after installation to reduce image size\nRUN apt-get update \u0026\u0026 \\\n    apt-get install -y \\\n    git \\\n    python3-dev \\\n    build-essential\n\n# Install uv package manager\nRUN wget -qO- https://astral.sh/uv/install.sh | sh\nENV PATH=\"/root/.cargo/bin:$PATH\"\n\n# Set the working directory for our application\nWORKDIR /website\n\n# Copy the application code\nCOPY . .\n\n# Install Python dependencies using uv\nRUN uv sync --frozen --no-dev\n\n# Expose the port that our service will listen on\nEXPOSE 8080\n\n# Start the application with gunicorn via uv\nCMD [\"uv\", \"run\", \"gunicorn\", \"-c\", \"gunicorn.conf.py\", \"download_oc:application\"]","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencitations%2Foc_download","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopencitations%2Foc_download","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencitations%2Foc_download/lists"}