{"id":15050549,"url":"https://github.com/andrei-demidov/django-sqlite-efs","last_synced_at":"2026-02-12T12:02:19.026Z","repository":{"id":256493481,"uuid":"854332372","full_name":"andrei-demidov/django-sqlite-efs","owner":"andrei-demidov","description":"Django database backend for SQLite on Amazon EFS","archived":false,"fork":false,"pushed_at":"2024-09-15T21:26:41.000Z","size":20,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-31T22:37:10.177Z","etag":null,"topics":["aws-efs","aws-lambda","aws-lambda-python","django","django-database","django-sqlite","django-sqlite3","dynamodb","efs","elasticfilesystem","lambda","serverless","sqlite","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andrei-demidov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-09T01:15:43.000Z","updated_at":"2025-07-09T20:57:35.000Z","dependencies_parsed_at":"2024-09-11T04:14:21.920Z","dependency_job_id":"32820471-e63a-49d2-8a88-d10523a90add","html_url":"https://github.com/andrei-demidov/django-sqlite-efs","commit_stats":null,"previous_names":["efficient-solutions/django-sqlite-efs","andrei-demidov/django-sqlite-efs"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/andrei-demidov/django-sqlite-efs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrei-demidov%2Fdjango-sqlite-efs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrei-demidov%2Fdjango-sqlite-efs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrei-demidov%2Fdjango-sqlite-efs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrei-demidov%2Fdjango-sqlite-efs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andrei-demidov","download_url":"https://codeload.github.com/andrei-demidov/django-sqlite-efs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrei-demidov%2Fdjango-sqlite-efs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29365583,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T08:51:36.827Z","status":"ssl_error","status_checked_at":"2026-02-12T08:51:26.849Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-efs","aws-lambda","aws-lambda-python","django","django-database","django-sqlite","django-sqlite3","dynamodb","efs","elasticfilesystem","lambda","serverless","sqlite","sqlite3"],"created_at":"2024-09-24T21:27:15.303Z","updated_at":"2026-02-12T12:02:18.998Z","avatar_url":"https://github.com/andrei-demidov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Django SQLite EFS Backend\n\n**django_sqlite_efs** is a custom database backend for Django, designed to work with **SQLite** on **AWS Lambda** using **Amazon EFS** (Elastic File System) and **Amazon DynamoDB**. This backend provides a solution to protect SQLite databases from corruption caused by concurrent writes in network-attached storage environments that lack proper file locking mechanisms for SQLite.\n\n## Features\n\n- Implements a distributed locking mechanism using **DynamoDB** to prevent concurrent write access to the SQLite database.\n- Designed for environments like **AWS Lambda** where multiple instances may attempt to access the SQLite database simultaneously.\n- Protects SQLite databases from corruption due to the limitations of EFS's advisory locking.\n- Uses **Amazon DynamoDB** to coordinate database locks and ensure safe write operations.\n\n## Requirements\n\n- **Python**: 3.11 or higher\n- **Django**: 5.1 or higher\n- **SQLite**: No additional installation required (built-in with Python)\n- **AWS Services**:\n  - **AWS Lambda**: For running the code.\n  - **Amazon EFS**: For storing the SQLite database.\n  - **Amazon DynamoDB**: For distributed locking to prevent concurrent writes.\n\n## Demo\n\nA basic Django polls application:\n\n- [Live Demo](https://efficient.solutions/link/qxamw/)\n- [Admin Portal](https://efficient.solutions/link/ffypx/) (read-only)\n  - **Username:** demo\n  - **Password:** djangoserverless\n\n## Installation\n\nTo install `django-sqlite-efs`, simply use pip:\n\n```bash\npip install django-sqlite-efs\n```\n\n## DynamoDB Configuration\n\nCreate a DynamoDB table to support distributed locking. The table should have the following schema:\n\n- **Primary Key**: `pk` (Type: String)\n- **Optional**: Configure expiration for the `expires_at` field for automatic cleanup of expired locks.\n\n### Example DynamoDB Table:\n\n| Attribute Name | Type    |\n|----------------|---------|\n| pk             | String  |\n| lock_id        | String  |\n| expires_at     | Number  |\n\nConfiguring `expires_at` with a TTL (Time-to-Live) policy is recommended for automatic removal of expired locks.\n\n## Configuration\n\nIn your Django project, update the `settings.py` file to use the custom **database backend** provided by `django-sqlite-efs`:\n\n```python\nDATABASES = {\n    'default': {\n        'ENGINE': 'django_sqlite_efs',\n        'NAME': 'path_to_your_sqlite_db_file',\n        \"OPTIONS\": {\n            # `timeout` - number of seconds to wait for lock acquisition.\n            # It must be at least several seconds less than the timeout of\n            # your Lambda function. Default and minimum value is 3.\n            \"timeout\": timeout\n            # Setting `init_commands` is not recommended because it overrides\n            # default commands, which may lead to unexpected behavior.\n        }\n    }\n}\n```\n\nAdditionally, configure the following settings in `settings.py` or as environment variables:\n\n- **SQLITE_LOCK_MAX_ATTEMPTS**: Maximum number of retries for acquiring a lock before raising an error (default: 10).\n- **SQLITE_LOCK_EXPIRATION**: Lock expiration time in seconds (should be at least equal to or greater than the Lambda function's timeout).\n- **SQLITE_LOCK_DYNAMODB_TABLE**: The name of the DynamoDB table used for locking.\n\n### AWS Configuration\n\nEnsure that your AWS credentials are correctly configured via environment variables or IAM roles. The package uses `boto3` to interact with DynamoDB. The Lambda function must have `PutItem` and `DeleteItem` permissions on the DynamoDB table.\n\n## How It Works\n\nSQLite uses file-based locking to prevent concurrent writes, but this is unreliable on **Amazon EFS** because EFS employs advisory locks. Advisory locks do not prevent processes from writing to a locked file if they have adequate permissions.\n\nThe **django_sqlite_efs backend** mitigates this by using **Amazon DynamoDB** to manage database locks:\n\n- For each write operation (e.g., `INSERT`, `UPDATE`, `DELETE`), the backend attempts to acquire a lock in DynamoDB. \n- All write operations lock the database for both reads and writes until the operation completes.\n- If a lock cannot be acquired, the backend retries multiple times using exponential backoff.\n- The lock is released when the write operation completes.\n- Read-only queries (`SELECT`) do not acquire a lock, allowing for concurrent read access without blocking.\n\n## Limitations\n\n1. **Concurrent Writes**: This backend **does not** support concurrent write operations. During any write operation, the database is locked, blocking all reads and writes until the operation completes.\n  \n2. **High Latency**:\n   - **Read Latency**: Even for read-only requests, the latency for a typical Lambda execution with a Django app interacting with the database is over 100-150 ms due to the overhead of interacting with EFS.\n   - **Write Latency**: Write operations have a latency of 300 ms or more during a typical Lambda execution.\n\nThis solution is designed for environments where write operations are infrequent.\n\n## License\n\nThis project is licensed under the **MIT License**. See the [LICENSE](LICENSE) file for more details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrei-demidov%2Fdjango-sqlite-efs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrei-demidov%2Fdjango-sqlite-efs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrei-demidov%2Fdjango-sqlite-efs/lists"}