{"id":23500422,"url":"https://github.com/eegli/spotify-history","last_synced_at":"2025-10-29T21:17:13.451Z","repository":{"id":38207447,"uuid":"351146636","full_name":"eegli/spotify-history","owner":"eegli","description":" Spotify scrobbing project with Google Drive backup support.","archived":false,"fork":false,"pushed_at":"2023-06-16T09:20:50.000Z","size":2413,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-15T18:50:53.766Z","etag":null,"topics":["aws","dynamodb","google","googleapis","lambda","scrobbing","serverless","spotify","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eegli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-24T16:18:02.000Z","updated_at":"2024-10-22T05:55:21.000Z","dependencies_parsed_at":"2025-04-15T18:53:52.613Z","dependency_job_id":null,"html_url":"https://github.com/eegli/spotify-history","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eegli/spotify-history","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eegli%2Fspotify-history","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eegli%2Fspotify-history/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eegli%2Fspotify-history/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eegli%2Fspotify-history/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eegli","download_url":"https://codeload.github.com/eegli/spotify-history/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eegli%2Fspotify-history/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281699514,"owners_count":26546332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-29T02:00:06.901Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","dynamodb","google","googleapis","lambda","scrobbing","serverless","spotify","typescript"],"created_at":"2024-12-25T06:44:15.661Z","updated_at":"2025-10-29T21:17:13.418Z","avatar_url":"https://github.com/eegli.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spotify History\n\nA simple Spotify scrobbler. Gets your listening history from Spotify, saves it to a database and creates a weekly backup in Google Drive.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/eegli/6a89dcb0ccbc9b64058e7c6a426c6ccb/raw/spotify_history_coverage.json\" alt=\"Coverage\"\u003e\n \u003cimg src=\"https://img.shields.io/github/languages/top/eegli/spotify-history?label=TypeScript\" alt=\"Languages\" /\u003e\n \u003cimg src=\"https://img.shields.io/depfu/eegli/spotify-history?label=Dependencies\" alt=\"Dependencies\" /\u003e\n\u003c/p\u003e\n\n## Features\n\n- [Free of charge\\*](#about-billing) - uses the AWS free tier and the Google Drive API, which is free to use\n- Utilities to get refresh tokens from both Google and Spotify\n- Easily customizable\n- Listening history export to Google Drive\n\n## Motivation\n\nSpotify's API only exposes the [last 50 songs you've listened to](https://support.spotify.com/us/article/listening-history/).\n\nThis project seeks to provide an easy and free solution to saving your Spotify listening history in an accessible place (Google Drive) where you can retrieve and analyze it quickly.\n\nOther than that, you can of course use everything here as a starting point/guideline to create something else with Spotify, AWS, Google Drive and Serverless.\n\n## Before you start\n\n- This project makes use of two AWS Lambda functions, one for getting your history from Spotify and one for creating backups in Google Drive.\n\n- Unlike Last.FM, Spotify apparently counts as song as _listened to_ when you listen to it for [\"over 30 seconds\"](https://artists.spotify.com/help/article/how-we-count-streams). The exact behaviour of how Spotify counts a song as _listened to_ is not clear to me, but it seems like 30 seconds are the minimum.\n\n- By default, the history Lambda (scrobbler) is **scheduled to get the history from Spotify at an hourly interval**. With this interval, most \"regular\" users who listen through a song will have their full listening history captured. Assuming a very low average song duration of ~2 minutes would mean that one could listen to max. 30 songs per hour. As Spotify keeps track of the last 50 songs you've listened to, this interval would cover the entire hour. However, you may change the schedule.\n\n- By default, the backup Lambda is **scheduled to run weekly** at the start of the week (Monday at 12:30 a.m.). A week is defined according to the [ISO 8610 standard](https://en.wikipedia.org/wiki/ISO_8601#Week_dates) and thus starts on Monday.\n\n- By default, **items in the database expire after 1 month** since they have already been backed up and are not needed anymore.\n\n- You might want to adjust the region in `serverless.yml - provider.region` if you don't live near Frankfurt (default is `eu-central-1`). [Available regions](https://docs.aws.amazon.com/general/latest/gr/rande.html).\n\nYou can customize the backup, schedules, item expiration and much more. [Customization guide](#customization).\n\n## Requirements\n\n- An AWS account\n- A Spotify account\n- `serverless \u003e= 3`\n- `node \u003e= v14.17.4`\n- Docker (optional)\n\n## Getting started\n\n1. Fork and/or clone this repository and install dependencies:\n\n```bash\ngit clone git@github.com:eegli/spotify-history.git\n\ncd spotify-history\n\nyarn\n```\n\n2. **Spotify setup** - [Create a Spotify application](https://developer.spotify.com/dashboard/applications) - app status \"development\" is fine - and set the redirect URL to `http://localhost:3000`.\n3. In the root directory, create a folder named `.secrets` (notice the dot!)\n4. Create a file named `credentials_spotify.json` and copy the template below. Insert your client id and client secret. Your Spotify secrets file should look like this:\n\n```json\n{\n  \"clientId\": \"\u003cyour-client-id\u003e\",\n  \"clientSecret\": \"\u003cyour-client-secret\u003e\"\n}\n```\n\n5. **Google Drive setup** - [Follow the quickstart guide](https://developers.google.com/drive/api/v3/quickstart/nodejs) to create a Google Cloud project and enable the Drive API. When asked to configure the consent screen, your publishing status should be _testing_. You will need to manually add the Google account who's drive you want to use under \"Test users\". In the end, you should be prompted to download your OAuth client credentials for your newly created desktop client as a JSON file.\n6. Download the credentials file, rename it to `credentials_google.json` and put it in the `.secrets` folder. It should look like this:\n\n```json\n{\n  \"installed\": {\n    \"client_id\": \"blablabla\",\n    \"project_id\": \"spotify-history-32as4\",\n    \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n    \"token_uri\": \"https://oauth2.googleapis.com/token\",\n    \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n    \"client_secret\": \"blablabla\",\n    \"redirect_uris\": [\"urn:ietf:wg:oauth:2.0:oob\", \"http://localhost\"]\n  }\n}\n```\n\nAlmost done!\n\n7.  Run the following command and follow the steps. This will create a `token_spotify.json` file in the `.secrets` folder containing your long-lived Spotify refresh token. **KEEP THIS FILE SECURE!**\n\n```bash\nnpm run token:spotify\n```\n\n8.  Run the following command and follow the steps. This will create a `token_google.json` file in the `.secrets` folder containing your long-lived Google Drive refresh token. **KEEP THIS FILE SECURE!**\n\n```bash\nnpm run token:google\n```\n\n9. Done!\n\n## Deploying the environments\n\nThis project includes both a staging and production environment. By default, **the schedules are only enabled in production** in order to save quota. The **staging version** is meant to be deployed but **invoked manually only**.\nIf you wish to enable the schedule on staging as well, change `serverless.yml`:\n\n```yml\ncustom:\n  scheduleEnabled:\n    prod: true\n    stg: true # Schedule enabled on staging\n```\n\n**Keep in mind that this will double the calls made to Lambda and DynamoDB!**\n\nIn order to deploy the production version, run:\n\n```bash\nnpm run prod:deploy\n```\n\nYou can deploy the staging version as well:\n\n```bash\n# Deploy everything\nnpm run stg:deploy\n\n# Deploy functions only\nnpm run stg:deploy:history\nnpm run stg:deploy:backup\n```\n\nAgain, the staging functions are **NOT scheduled** by default as they are meant to be invoked manually:\n\n```bash\n# Get history from Spotify and save to DynamoDB\nnpm run stg:invoke:history\n\n# Create backup in Google Drive\nnpm run stg:invoke:backup\n```\n\n## Logging\n\nTo check the logs of your Lambda functions, either go to the AWS CloudWatch dashboard or retrieve them in your console.\n\nExample: Getting the logs for production in the last 24h\n\n```bash\nsls logs -f spotify-history -s prod --startTime 1d\nsls logs -f spotify-history-backup -s prod --startTime 1d\n```\n\n[More info about logging with Serverless](https://www.serverless.com/framework/docs/providers/aws/cli-reference/logs/).\n\n## Customization\n\n### Changing history properties\n\nBy default, these are the song properties that are saved to the database (and backup):\n\n```ts\ninterface DynamoHistoryElement {\n  name: string;\n  id: string;\n  playedAt: string;\n}\n```\n\nIf you want to save other song properties, simply change this interface in `src/config/types.ts` and TypeScript will show you where you'll need to make adjustments. Obviously, it makes sense to at least store the timestamp of when the song was played (`playedAt`) and its id (`id`).\n\n### Changing item expiration in the database\n\nBy default, items in DynamoDB are set to expire after 1 month. If you wish to disable this, set the TTL specification in `serverless.yml` to `false` (or remove the implementation altogether for a cleaner codebase).\n\n```yml\nTimeToLiveSpecification:\n  AttributeName: 'expire_at'\n  Enabled: false\n```\n\nIf you want to specify a different TTL, change the `dynamoExpireAfter` default in `src/config/defaults.ts`\n\n### Changing the backup schedule\n\nIf you want to change the backup schedule, e.g. running it daily or monthly, you'll need to adjust the cron expression in `serverles.yml`. Here are some resources regarding cron jobs.\n\n- [AWS EventBridge schedule expressions (cron)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html)\n- [Cron expression tester](https://crontab.cronhub.io/)\n\n**⚠️ Keep in mind that the backup schedule, item expiration and time range to retrieve the items for the backup are logically connected! ⚠️**\n\nIf you change the backup schedule, you'll also need to change the time range of the backup and, most likely, the item TTL as shown above.\n\n```ts\n// Example: Include history from last month\n\nconst defaults: Readonly\u003cDefaults\u003e = {\n  dynamoExpireAfter: [2, 'months'], // Extend the expiration date\n  backupRange: [1, 'month'], // Extend the backup range\n  ...\n};\n```\n\n### Changing the backup folder\n\nUpdate the stage and production folder names in `src/config/defaults.ts`.\n\nNote that, for security reasons, the backup handler only has access to folders and files it has created itself (see [OAuth 2.0 scopes](https://developers.google.com/drive/api/v2/about-auth) and `scripts/google.ts`). For simplicity, the backup folder is created at the root of your Google Drive.\n\n## Development and Testing\n\n### Running DynamoDB locally\n\nFor local development and testing the db integration, AWS's official DynamoDB Docker image can be run along with another image that provides a nice GUI for inspecting the tables and items.\n\n1. Start the containers (DynamoDB and GUI):\n\n```bash\nnpm run dynamo:start\n```\n\n2. Migrate the table and seed:\n\n```bash\n\nnpm run dynamo:migrate\n```\n\n3. If you want to check if everything has been setup correctly, visit http://localhost:8001/\n\n4. Invoke locally:\n\n   Note that `npm run local:backup` will, despite its naming, **still** hit the Google Drive API but save the content in a folder separate from `stg` and `prod` (`local`).\n\n```bash\n# Gets the history and saves it to local DynamoDB\nnpm run local:history\n\n# Backs up the history to Google Drive\nnpm run local:backup\n```\n\n## Good to know\n\nThe core of this project uses [AWS DynamoDB Data Mapper](https://github.com/awslabs/dynamodb-data-mapper-js). Unfortunately, this package does not seem to be actively maintaned and is only compatible with the AWS SDK v2. By default, [the AWS SDK v2 is included in the Lambda runtime environment](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html), but not the modular version 3. Those are the reasons why it is currently not possible to upgrade this project to use the modular AWS SDK.\n\n## Resources\n\n- [Mocking TS method overloads with Jest](https://javascript.plainenglish.io/mocking-ts-method-overloads-with-jest-e9c3d3f1ce0c)\n- [Amazon DynamoDB DataMapper For JavaScript](https://github.com/awslabs/dynamodb-data-mapper-js)\n- [Amazon DynamoDB DataMapper Annotations](https://github.com/awslabs/dynamodb-data-mapper-js/tree/master/packages/dynamodb-data-mapper-annotations)\n- [Using the DynamoDB Document Client](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/dynamodb-example-document-client.html)\n- [Serverless DynamoDB Local](https://www.npmjs.com/package/serverless-dynamodb-local)\n- [TypeScript: adjusting types in reduce function with an async callback](https://dev.to/pedrohasantiago/typescript-adjusting-types-in-reduce-function-with-an-async-callback-2kc8)\n\n## About billing\n\n\\* Serverless uses S3 to store the code of the deployed functions. Technically, S3 is not free. It costs [a fraction of a $](https://aws.amazon.com/s3/pricing/?nc=sn\u0026loc=4) per GB, but a deployment takes up so little space, you most likely won't be billed. A full month of testing \"cost\" me 0.01\\$ and I was not billed. Be aware that, if you change the schedules, this project may not be \"free\" anymore!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feegli%2Fspotify-history","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feegli%2Fspotify-history","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feegli%2Fspotify-history/lists"}