{"id":21591107,"url":"https://github.com/wwbrannon/audio-scraper","last_synced_at":"2025-07-17T00:30:57.468Z","repository":{"id":263922170,"uuid":"743834189","full_name":"mit-ccc/audio-scraper","owner":"mit-ccc","description":"Audio stream scraper","archived":false,"fork":false,"pushed_at":"2024-11-21T01:36:15.000Z","size":259,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T22:41:09.955Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-ccc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-16T04:53:11.000Z","updated_at":"2024-12-11T21:00:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"1d263c6f-d8cc-4bc1-9743-aceed296aa3d","html_url":"https://github.com/mit-ccc/audio-scraper","commit_stats":null,"previous_names":["wwbrannon/audio-scraper","mit-ccc/audio-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mit-ccc/audio-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-ccc%2Faudio-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-ccc%2Faudio-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-ccc%2Faudio-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-ccc%2Faudio-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-ccc","download_url":"https://codeload.github.com/mit-ccc/audio-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-ccc%2Faudio-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265554756,"owners_count":23787278,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-24T16:22:52.423Z","updated_at":"2025-07-17T00:30:57.462Z","avatar_url":"https://github.com/mit-ccc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Audio scraper\n---\n\nThis application ingests audio from online streams or files, saves it to either\nan S3 bucket or a local folder, and transcribes it. Various parts of\nthis ingestion process are configurable: the bucket, the number of workers, the\ntranscription quality and whether to use GPUs, and more. The output transcripts\ninclude not only transcribed text but also diarized speaker IDs and word-level\ntime alignments. (Transcription is via OpenAI's Whisper models and the excellent\n[WhisperX](https://github.com/m-bain/whisperX) package.)\n\nThere's support for many kinds of online streams, including direct aac/mp3/etc\nstreams and playlists like .pls and .m3u. There's also support for webscrape\nstreams where direct or playlist URLs have to be extracted from a page. Out of\nthe box, only iHeartRadio streams are webscrape-able; if you want others you'll\nhave to write a bit of code that says how to get stream URLs from the page.\n\nThe app is packaged with Kubernetes for both local use via minikube and an easy\nmigration path to a larger deployment on a K8s cluster.\n\n### Setup\nTo get started, you need to do a few things:\n* For local use, install [docker](https://docs.docker.com/engine/install/) and\n  [minikube](https://minikube.sigs.k8s.io/docs/start/).\n* Go to Huggingface and accept the access conditions for certain gated models.\n  They're available for free, but you have to agree to share your contact info\n  to indicate you use them. (This helps the developers apply for grants.)\n    * [pyannote/speaker-diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1)\n    * [pyannote/segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0)\n* Create `secrets.env`, a simple key=value list of environment variables, one\n  per line, with the following keys:\n    * `POSTGRES_PASSWORD`, which is the password used to access the coordinating\n      database container. Pick whatever you like.\n    * `HF_TOKEN`, a [Huggingface access token](https://huggingface.co/docs/hub/en/security-tokens)\n      that allows the app to pull the gated models you signed up for above.\n    * If audio and transcripts are written to S3, you also need to provide\n      `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.\n* Create the list of audio sources to ingest. This is a tab-separated file\n  that needs to be at the path `images/postgres/data/source-data.csv`. (See\n  `images/postgres/data/station-data-example.csv` for an example.) It should\n  contain 6 columns (in this order):\n    * `source_id`, an arbitrary integer ID you can assign;\n    * `name`, an arbitrary text name for the source (e.g., for a radio station,\n      its callsign is a good choice).\n    * `stream_url`, the URL to scrape;\n    * `auto_ingest`, whether to start ingesting this source immediately on\n      startup. If `False`, you'll have to manually run SQL against the database\n      container to start ingesting this source.\n    * `lang`, the two-letter ISO code for the language of the audio.\n      If null, the language will be autodetected, but errors in language\n      detection may reduce transcription quality.\n    * `retry_on_close`, whether to reopen the connection if the server closes it\n      as complete. Some ongoing streams are misconfigured and will do this even\n      if there's more audio available on reconnecting.\n* Edit some settings:\n  * The configuration map at the top of `deploy.yaml` should have your preferred\n    settings, especially the storage location to write audio and transcripts to.\n  * The number of replicas for the ingest and transcribe deployments in\n    `deploy.yaml` may need to be increased or decreased, depending on how many\n    sources you're ingesting.\n  * Finally, for local use you may need to experiment with the `cpus` and\n    `memory` arguments that define the minikube cluster resource allocations.\n    The requested and maximum CPU/memory usage for each container are marked in\n    the `deploy.yaml` file so that you can calculate how much your deployments\n    will need.\n\n## Usage\nTo start up locally on one machine, you can use the provided Makefile and\ndeploy.yaml. Run `make start` to create a minikube cluster, and then `make` or\nequivalently `make up` to deploy the application to it. You can monitor it by\nexamining the files it writes, or using Kubernetes' `kubectl` command. Minikube\nalso supports the Kubernetes dashboard for a graphical interface -- see the\nMakefile. You'll need to allow a few minutes on startup for the application to\nbegin working; the transcribe containers in particular can take as long as 10\nminutes to load their models and start processing audio. (Note that if you have\ntoo few transcribe containers for the sources being ingested, you can run out of\ndisk space!)\n\nFor deployment to a larger, preexisting Kubernetes cluster, you'll need to make\nsome changes to the manifest and shouldn't need the Makefile. If you need to do\nthis, you already either know your way around devops or have access to someone\nwho does, so we won't say more about it here. Note that there is a possibly\nhelpful script provided at `bin/ecr.sh` to build the container images and push\nthem to Amazon ECR repositories in case you're using EKS.\n\n## Live editing of sources\nTo edit the set of sources marked for ingest once the app is running, there are\ntwo things you can do:\n1. Stop the app with `make down`, edit the set of sources (or change their\n   `auto_ingest` flags) and redeploy, or\n2. Run a bit of SQL. First, connect to the PostgreSQL database container. Load\n   source information into the data.sources table in the same format as in the\n   CSV file, and insert corresponding rows into the ingest.jobs table. If you\n   want to stop ingesting something that's already defined in data.sources, just\n   delete the appropriate ingest.jobs row. Jobs which are no longer present will\n   stop being ingested, and new ones will be picked up by a worker (if any are\n   free - you may need to scale out the ingest and/or transcribe deployments).\n\n## Output format\nThe ingest workers will transcode their input audio to WAV format before saving\nit, for uniformity's sake. Audio is saved in chunks of the same duration, rather\nthan the same input file size, with the default being 30 seconds. Audio files\nare removed by default after being transcribed, to save space and reduce costs,\nbut you can keep them by setting the `REMOVE_AUDIO` configuration in\n`deploy.yaml` to 'false'.\n\nThe transcribe workers use [WhisperX](https://github.com/m-bain/whisperX) and\nalso perform diarization and word-level alignment. The output `.json.gz` files\nhave diarized speaker IDs and word-level timestamps, as well as the transcribed\ntext.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwwbrannon%2Faudio-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwwbrannon%2Faudio-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwwbrannon%2Faudio-scraper/lists"}