{"id":22083778,"url":"https://github.com/unlyed/airtable-backups-boilerplate","last_synced_at":"2026-03-01T10:33:05.683Z","repository":{"id":53021601,"uuid":"206376399","full_name":"UnlyEd/airtable-backups-boilerplate","owner":"UnlyEd","description":"Configurable automated backups for Airtable meant to be self-hosted, powered by AWS Lambda/S3 with the Serverless framework","archived":false,"fork":false,"pushed_at":"2021-04-09T09:37:23.000Z","size":354,"stargazers_count":45,"open_issues_count":1,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-01T12:24:09.012Z","etag":null,"topics":["airtable","airtable-api","airtable-backups","aws-lambda","backups"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UnlyEd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"liberapay":"unlyEd","github":["UnlyEd","Vadorequest"]}},"created_at":"2019-09-04T17:37:19.000Z","updated_at":"2025-02-08T02:51:52.000Z","dependencies_parsed_at":"2022-08-26T13:40:46.662Z","dependency_job_id":null,"html_url":"https://github.com/UnlyEd/airtable-backups-boilerplate","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/UnlyEd/airtable-backups-boilerplate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnlyEd%2Fairtable-backups-boilerplate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnlyEd%2Fairtable-backups-boilerplate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnlyEd%2Fairtable-backups-boilerplate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnlyEd%2Fairtable-backups-boilerplate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UnlyEd","download_url":"https://codeload.github.com/UnlyEd/airtable-backups-boilerplate/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnlyEd%2Fairtable-backups-boilerplate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29966827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T09:33:09.965Z","status":"ssl_error","status_checked_at":"2026-03-01T09:25:48.915Z","response_time":124,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airtable","airtable-api","airtable-backups","aws-lambda","backups"],"created_at":"2024-12-01T00:17:38.975Z","updated_at":"2026-03-01T10:33:05.664Z","avatar_url":"https://github.com/UnlyEd.png","language":"JavaScript","funding_links":["https://liberapay.com/unlyEd","https://github.com/sponsors/UnlyEd","https://github.com/sponsors/Vadorequest"],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://unly.org\"\u003e\u003cimg src=\"https://storage.googleapis.com/unly/images/ICON_UNLY.png\" align=\"right\" height=\"20\" alt=\"Unly logo\" title=\"Unly logo\" /\u003e\u003c/a\u003e\n[![Maintainability](https://api.codeclimate.com/v1/badges/a6ff14f16df566d20013/maintainability)](https://codeclimate.com/github/UnlyEd/airtable-backups-boilerplate/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/a6ff14f16df566d20013/test_coverage)](https://codeclimate.com/github/UnlyEd/airtable-backups-boilerplate/test_coverage)\n[![Known Vulnerabilities](https://snyk.io/test/github/UnlyEd/airtable-backups-boilerplate/badge.svg?targetFile=package.json)](https://snyk.io/test/github/UnlyEd/airtable-backups-boilerplate?targetFile=package.json)\n\n# Airtable Backup Boilerplate\n\n\u003e This project is a boilerplate meant to perform backups of Airtable Bases, at a regular interval (scheduled backups, AKA crons).\n\u003e Those backups are performed by AWS Lambda, and stored in an AWS S3 bucket.\n\u003e\n\u003e This project is meant to be hosted on **your own AWS Account**, so you have complete ownership of the project and its configuration.\n\u003e The backups are yours and yours only\n\u003e \n\u003e In order to get started, please fork this project and follow the [\"getting started\" guide](#getting-started).\n\nA **demo of this tool** has been published on **[BuiltOnAir podcast](https://builtonair.com/builtonair-s04e07-ambroise-dhenain-cofounder-of-unly/)**, it's a great resource **to see how it works** beforehand. (*[Starts at 22:50](https://youtu.be/DR7zgsoJkTg?t=1371)*)\n\n---\n\n\u003c!-- toc --\u003e\n\n- [Our recommended AWS configuration](#our-recommended-aws-configuration)\n- [Getting started](#getting-started)\n  * [Local install](#local-install)\n  * [Configuring Airtable](#configuring-airtable)\n    + [Setup Airtable credentials](#setup-airtable-credentials)\n  * [Configuring AWS](#configuring-aws)\n    + [Selecting AWS region](#selecting-aws-region)\n    + [Configuring AWS S3](#configuring-aws-s3)\n  * [Configuring a scheduled backup](#configuring-a-scheduled-backup)\n    + [Testing project locally (mocked data)](#testing-project-locally-mocked-data)\n- [Deploying on AWS](#deploying-on-aws)\n- [Airtable - In depth](#airtable---in-depth)\n  * [A word of caution about Airtable API Key](#a-word-of-caution-about-airtable-api-key)\n  * [We leaked our own Airtable API Key!](#we-leaked-our-own-airtable-api-key)\n- [Error monitoring with Epsagon](#error-monitoring-with-epsagon)\n- [Logs](#logs)\n- [Test](#test)\n- [Release](#release)\n- [FAQ](#faq)\n  * [`Can't find Airtable table Video trackers with provided base`](#cant-find-airtable-table-video-trackers-with-provided-base)\n  * [My deployment worked but no file is added to S3, AKA \"I don't know what's happening on AWS\"](#my-deployment-worked-but-no-file-is-added-to-s3-aka-i-dont-know-whats-happening-on-aws)\n- [Vulnerability disclosure](#vulnerability-disclosure)\n- [Contributors and maintainers](#contributors-and-maintainers)\n- [**[ABOUT UNLY]**](#about-unly-)\n\u003c!-- tocstop --\u003e\n\n## Our recommended AWS configuration\n\n\u003e As best practice, this boilerplate comes built-in with different environments (an \"environment\" is similar to a \"stage\").\n\u003e\n\u003e Each env is completely independent. We recommend having one environment in one dedicated AWS Account as best practice, for complete separation of concerns.\n\u003e But, you can also use the same AWS account, it's entirely up to you.\n\nOur environments are specified in `serverless.yml:custom.envs`.\nWe use the same \"profile\" for both staging and production envs in this boilerplate, for the sake of simplicity.\n\nBut you can use different AWS profile if you wish (that's what we do in our internal fork of this boilerplate).\n\nIf you're not familiar with AWS profiles and alike, [I recommend reading this](https://forum.serverless.com/t/restructuring-aws-proper-way-to-configure-aws-accounts-organisations-and-profiles-when-using-serverless/5009).\n\nP.S: Because we use AWS profiles, we don't directly manipulate AWS SECRET/API keys, they're stored in our `~/.aws/` folder and we don't have to care about them.\nYou may prefer to manage your AWS credentials with environment variables.\n\n---\n\n## Getting started\n\n\u003e First, fork this project, or clone it.\n\n### Local install\n\n```bash\nnvm use # Select the same node version as the one that'll be used by AWS (see .nvmrc) (optional)\nyarn install # Install node modules\n```\n\n### Configuring Airtable\n\n#### Setup Airtable credentials\n\n\u003e First, you need to find your Airtable API KEY, you can find it in your [Airtable account](https://airtable.com/account), or in the API documentation of your Airtable Base. \n\nBecause your Airtable API Key must not be tracked by git, it's meant to be added in a non-tracked file, that depends on the environment you're deploying to:\n\n- Staging environment: `./.env.staging`\n- Production environment: `./.env.production`\n\nCreate both files and add your Airtable API key as `AIRTABLE_TOKEN` (see [.env.test](./.env.test) as example)\n\n\u003e See [Airtable - In depth](#airtable---in-depth) section to learn more about our Airtable's recommendations\n\n### Configuring AWS\n\n#### Selecting AWS region\n\u003e First, you need to know which region you want to use. This region will be used for storing your backups (S3) and that's also where your Lambda are gonna be running.\n\nBy default, the region is `Ireland`, but you can either change the default value in [serverless config](./serverless.yml), or customise the [`deploy` script](package.json) to match the region you want.\n \n#### Configuring AWS S3\n\n##### Creating AWS S3 Bucket\n\u003e You need to **manually** create a bucket [in AWS Console](https://console.aws.amazon.com/s3/home) _(PR welcome to automate this process!)_\n\nThe name of the bucket is [dynamic by default](./serverless.yml), `bucket: ${self:service}-${self:custom.environment}`, which will resolve to either `airtable-backups-demo-staging` or `airtable-backups-demo-production` in our case.\n\n##### Selecting AWS S3 Storage Class (optional)\n\nSelecting the right storage class is a bit complex and won't be covered in this short tutorial. We recommend reading the [official documentation](https://aws.amazon.com/en/s3/storage-classes/).\n\nThe storage class we recommend for storing backups is **`STANDARD_IA`** (AKA \"Standard Infrequent Access\"), because it's the best compromise in terms of cost, accessibility, backup redundancy (multi zones), etc. \n\nBut, each business may have its own preferences. Also, you can optimize cost using automated lifecycle.\n\n##### Configuring AWS S3 Bucket's lifecycle (optional)\n\nYou can configure a lifecycle for your S3 bucket. (you can still configure/change this later on if you change your mind).\n\n- Here is a good tutorial on how to delete a backup after X days, [see this advanced tutorial](https://www.joe0.com/2017/05/24/amazon-s3-how-to-delete-files-older-than-x-days/)\n- You could also optimise cost by automatically moving backups that are older than 30 days to another class storage, such as \"Glacier\", etc.\n- Here is official documentation [here](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)\n\n\u003e We personally use lifecycle to delete files older than 180 days, so that our usage of AWS S3 doesn't increase indefinitely.\n\n### Configuring a scheduled backup\n\nFinally, after going through all the previous steps, you can now finally configure a scheduled backup for an Airtable base!\n\nThe configuration is done in [serverless config](./serverless.yml), at `custom.airtableBackups.events`:\n\n```yaml\n  airtableBackups: # The same lambda is used to configure all backups (each backup is a distinct \"scheduled event\", AKA \"cron\")\n    handler: src/functions/makeAirtableBackup.handler\n    events:\n      - schedule:\n          description: \"Airtable backups for the 'Airtable backups boilerplate' base (demo)\"\n          rate: rate(5 minutes) # TODO Set your own rate : https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html\n          enabled: true\n          input:\n            AIRTABLE_BASE: \"app7nfLmoVHva1Vdv\" # TODO Set your own base id\n            AIRTABLE_TABLES: \"Video tracker;Staff directory;Agencies;Agency contacts;Scenes;Shots;Locations;Props and equipment\" # TODO Set your table names\n            S3_DIRECTORY: \"airtableBackupsBoilerplate/\" # TODO Set the s3 sub-directory you want the backups to be stored in\n            STORAGE_CLASS: 'STANDARD_IA' # Set the storage class to use within those values: \"STANDARD\"|\"REDUCED_REDUNDANCY\"|\"STANDARD_IA\"|\"ONEZONE_IA\"|\"INTELLIGENT_TIERING\"|\"GLACIER\"|\"DEEP_ARCHIVE\" - See https://aws.amazon.com/en/s3/storage-classes/\n```\n\nThe same AWS Lambda (`airtableBackups`) is used to perform all backups. You can schedule several events if you wish, each with its own base, tables, S3 directory and storage class strategy.\n\n#### Testing project locally (mocked data)\n\n\u003e We will store a backup in your AWS S3 bucket for real, but is triggered locally (not on AWS Lambda)\n\n```bash\nyarn invoke:airtableBackups\n```\n\n**Note that this script uses our [mocked data](mocks/test-event.json)**, and not the ones that have been defined in serverless.yml\n\nThe output should be something like this :\n```\nBackup \"airtableBackupsBoilerplate/2020_01_11_17-47-04.json\" successfully uploaded to bucket \"airtable-backups-demo-staging\" (using storage class: \"STANDARD_IA\")\n{\n    \"statusCode\": 200,\n    \"body\": \"Successfully created backup\"\n}\n```\n\n---\n\n## Deploying on AWS\n\n\u003e Before deploying, search for all `TODO` in the code source and resolve them. (serverless.yml)\n\u003e\n\u003e They're mostly there to highlight changes that you should perform before deploying this project on AWS.\n\n```bash\nyarn deploy # Deploy on staging environment\nNODE_ENV=production yarn deploy # Deploys to production\n```\n\n---\n\n## Airtable - In depth\n\n### A word of caution about Airtable API Key\n\n\u003e Airtable API Key security **sucks**, in our opinion.\n\nThey use the same API key for all bases, and you can only have one. \nTherefore, if you API Key leaks, you'll have to invalidate it in your [Airtable account](https://airtable.com/account), which will break all API integration for all bases at once.\n\nA better way for them to secure API keys would have been to allow us to create more API keys (and name them), so we could have used one different API key per base and per environment.\nThis would have been better in case we need to invalidate one key, it'd have a much smaller damage radius. But unfortunately, that's not the case.\n\n\u003e Be extra cautious about not leaking your Airtable API Key anywhere (like on github, for instance)\n\n### We leaked our own Airtable API Key!\n\nNote that for the sake of simplicity, we leaked our own Airtable API Key in the `.env.test` and `mocks/test-event.json` files, but we created a separated Airtable Account unrelated to our business, meant to be used by this boilerplate only.\n\nA more elegant solution would have been to not use environment variables to store the `AIRTABLE_TOKEN`, such as KMS or similar.\n\n---\n\n## Error monitoring with Epsagon\n\nWe use Epsagon in this boilerplate to monitor errors and lambda invoke. It's very handy for faster debugging.\n\nYou can set up your own credentials [in serverless.yml](./serverless.yml):\n```yaml\ncustom:\n  epsagon:\n    token: '' # TODO Set your Epsagon token - Won't be applied if not provided\n```\n\nThis is completely optional and opt-in. You're opt-out by default.\n\n\u003e If you decide to use it, make sure to configure Epasagon with Slack (or similar) to be notified about staging/production errors.\n\n---\n\n## Logs\n\n- **airtableBackups** function:\n```bash\nyarn logs:airtableBackups\n```\n\n- **status** function:\n```bash\nyarn logs:status\n```\n\nSimilar to reading the logs from the AWS Console\n\n---\n\n## Test\n\n```\nyarn test\nyarn test:coverage\n```\n\n---\n\n## Release\n\nWill prompt version to release, run tests, commit/push commit + tag\n\n```\nyarn release\n```\n\n\n\u003e Check the [./package.json](./package.json) file to see what other utility scripts are available\n\n--\n\n## FAQ\n\n### `Can't find Airtable table Video trackers with provided base`\n\n\u003e Make sure all your `AIRTABLE_TOKEN`, `AIRTABLE_BASE` and `AIRTABLE_TABLES` are correct. \n\nIf the base doesn't exist or if the table doesn't exist within that base then you'll get this error.\n\nSame thing if the airtable token is incorrect. It makes it harder to debug a misconfiguration, but that's how Airtable's API works...\n\n### My deployment worked but no file is added to S3, AKA \"I don't know what's happening on AWS\"\n\n\u003e Make sure to first test your backup configuration with a **fast rate**, like `rate: rate(2 minutes)`\n\n[Watch quick video about how to debug using Epsagon](https://youtu.be/KYdIS82lwlI)\n\nAlways use a fast rate when testing things out, that way you have a fast feedback about what's working or not. \nDon't use `rate: rate(1 day)` before trying your configuration on AWS first, for instance.\n\n**Also, when you deploy your scheduled backup, AWS Lambda won't be triggered immediately. It will actually wait before triggering for the first time.**\n\n\u003e i.e: `rate: rate(1 day)` will not be triggered before 24h after deploying\n\n# Vulnerability disclosure\n\n[See our policy](https://github.com/UnlyEd/Unly).\n\n---\n\n# Contributors and maintainers\n\nThis project is being maintained by:\n- [Unly] Ambroise Dhenain ([Vadorequest](https://github.com/vadorequest)) **(active)**\n- [Contributor] Hugo Martin ([Demmonius](https://github.com/Demmonius)) **(active)**\n\n---\n\n# **[ABOUT UNLY]** \u003ca href=\"https://unly.org\"\u003e\u003cimg src=\"https://storage.googleapis.com/unly/images/ICON_UNLY.png\" height=\"40\" align=\"right\" alt=\"Unly logo\" title=\"Unly logo\" /\u003e\u003c/a\u003e\n\n\u003e [Unly](https://unly.org) is a socially responsible company, fighting inequality and facilitating access to higher education. \n\u003e Unly is committed to making education more inclusive, through responsible funding for students. \n\nWe provide technological solutions to help students find the necessary funding for their studies. \n\nWe proudly participate in many TechForGood initiatives. To support and learn more about our actions to make education accessible, visit : \n- https://twitter.com/UnlyEd\n- https://www.facebook.com/UnlyEd/\n- https://www.linkedin.com/company/unly\n- [Interested to work with us?](https://jobs.zenploy.io/unly/about)\n\nTech tips and tricks from our CTO on our [Medium page](https://medium.com/unly-org/tech/home)!\n\n#TECHFORGOOD #EDUCATIONFORALL\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funlyed%2Fairtable-backups-boilerplate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funlyed%2Fairtable-backups-boilerplate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funlyed%2Fairtable-backups-boilerplate/lists"}