{"id":25314915,"url":"https://github.com/lanterndata/pg","last_synced_at":"2025-10-28T19:30:47.450Z","repository":{"id":265777192,"uuid":"728865736","full_name":"lanterndata/pg","owner":"lanterndata","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-30T21:25:11.000Z","size":706,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-02T18:41:34.515Z","etag":null,"topics":["mailing-list","postgres","postgresql","search"],"latest_commit_sha":null,"homepage":"https://pg.lantern.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lanterndata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-07T21:40:28.000Z","updated_at":"2024-11-30T21:25:14.000Z","dependencies_parsed_at":"2024-11-30T22:45:15.176Z","dependency_job_id":null,"html_url":"https://github.com/lanterndata/pg","commit_stats":null,"previous_names":["lanterndata/pg"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanterndata%2Fpg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanterndata%2Fpg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanterndata%2Fpg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanterndata%2Fpg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lanterndata","download_url":"https://codeload.github.com/lanterndata/pg/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238701688,"owners_count":19516204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mailing-list","postgres","postgresql","search"],"created_at":"2025-02-13T17:39:05.937Z","updated_at":"2025-10-28T19:30:47.007Z","avatar_url":"https://github.com/lanterndata.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PG Mailing List\n\nThis project provides scripts for retrieving Postgres mailing list archives, populating a Postgres database with them, and rendering the archives with NextJS.\n\n## Getting started\n\n- Run `git clone --recurse-submodules lanterndata/pg`\n\n## Populate data\n\n- Use `./scripts/scrape.sh` to download the mailing list archives and populate a Postgres database. Note that `DATABASE_URL` should be set in the environment.\n- Use `pgsql-lists-offline` to download the mailing list archives to a local `data` directory. Note the dependencies on `parallel` and `curl`. See more below.\n- Use `yarn scrape` to process the archives and populate a Postgres database. Note that `DATABASE_URL` should be set in the environment.\n\n## Create migration\n\n```bash\nyarn dbmate new \u003cmigration-name\u003e\n```\n\n## Run migrations\n\nWith `DATABASE_URL` set in the environment:\n\n```bash\nyarn dbmate up\n```\n\n## Notes for local development\n\nThe `body_dense_vector` column in `messages` was added and populated using the Lantern dashboard, and is not in the migration files.\n\n## pgsql-lists-offline\n\nSource: \u003chttps://github.com/wsdookadr/pgsql-lists-offline\u003e\n\nDownloads mailing list archives for PostgreSQL, PostGIS and pgRouting.\n\nThe mailing list archives that will be downloaded are in the [mbox format](https://en.wikipedia.org/wiki/Mbox) and can be read\nusing any mbox-aware email client (for example [mutt](http://www.mutt.org/) ).\n\n### Dependencies\n\n```sh\nsudo apt-get install parallel curl\n```\n\n### Usage\n\n```sh\n./pgsql-lists-offline.sh -l\n```\n\nTo download a list (for example pgsql-general):\n\n```sh\n./pgsql-lists-offline.sh -g pgsql-general\n```\n\nTo download a list for a particular month:\n\n```sh\n./pgsql-lists-offline.sh -g pgsql-general -m 202403\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flanterndata%2Fpg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flanterndata%2Fpg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flanterndata%2Fpg/lists"}