{"id":24429205,"url":"https://github.com/peterk/munin-indexer","last_synced_at":"2025-04-12T11:21:58.603Z","repository":{"id":41947310,"uuid":"146926063","full_name":"peterk/munin-indexer","owner":"peterk","description":"A social media open post web archiving tool","archived":false,"fork":false,"pushed_at":"2025-03-07T07:57:07.000Z","size":1955,"stargazers_count":25,"open_issues_count":6,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T06:11:17.464Z","etag":null,"topics":["archiving","high-fidelity-preservation","preservation","webarchiving"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peterk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-31T17:59:35.000Z","updated_at":"2025-03-07T07:57:04.000Z","dependencies_parsed_at":"2024-12-07T07:22:20.732Z","dependency_job_id":"9d3eb79a-e82f-456c-a571-17a145b702ad","html_url":"https://github.com/peterk/munin-indexer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterk%2Fmunin-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterk%2Fmunin-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterk%2Fmunin-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterk%2Fmunin-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peterk","download_url":"https://codeload.github.com/peterk/munin-indexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248558134,"owners_count":21124224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiving","high-fidelity-preservation","preservation","webarchiving"],"created_at":"2025-01-20T13:33:49.084Z","updated_at":"2025-04-12T11:21:58.579Z","avatar_url":"https://github.com/peterk.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Munin - a social media archiver\n\nThis tool will monitor open Facebook, Instagram and VKontakte account seeds for new posts and archive those posts. Posts are archived in the WARC file format using the excellent Squidwarc package. A playback tool and a simple dashboard is available to monitor collections.\n\n\u003cimg src=\"https://user-images.githubusercontent.com/19284/49699663-3e337b80-fbd4-11e8-8282-035ea7f219ba.png\" alt=\"Munin dashboard screenshot\"\u003e\n\n# System overview\n\nMunin builds on great software by other people. Indexing of post items is done in [snscrape](https://github.com/JustAnotherArchivist/snscrape). Archiving of individual pages is done with [Squidwarc](https://n0tan3rd.github.io/Squidwarc/). Playback of WARC files is enabled by [pywb](https://pywb.readthedocs.io/en/latest/).\n\n\u003cimg src=\"https://user-images.githubusercontent.com/19284/50910651-8392d500-142e-11e9-9133-8766249c09b8.png\" alt=\"System overview - a Django application manages seeds and post URL:s in a PostgreSQL database. A queue for indexing finds more post URLs for the seeds. A queue for archiving makes sure post URLs are archived.\"/\u003e\n\n# Install\n\n1. To run you need to [install Docker](https://docs.docker.com/get-docker/) and Docker Compose. It has only been tested on Linux and Mac OSX currently and the instructions below are for those platforms. Make sure you have git installed.\n\n2. Clone this repository\n\n`$ git clone https://github.com/peterk/munin-indexer`\n\n3. Enter the directory and create an empty data directory for postgres\n\n`$ cd munin-indexer`\n`$ mkdir data`\n\n4. Set up environment variables\n\nRename the `example_env_file` to `env_file` and update it with your settings. You should change the time zone (TZ) to match your location ([see the list of time zone names here](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)). \n\nStart everything:\n\n`$ docker-compose up -d`\n\nThe first time the application starts it can take a while (several minutes) before the application becomes available. You can monitor progress by watching the docker logs.\n\nSet up a superuser when the application is up (it will ask you for details to create an administrator):\n\n`$ docker-compose exec web python manage.py createsuperuser`\n\nLogin to the admin dashboard with the newly created superuser at http://0.0.0.0:4444/admin\n\nStart by adding your first Collection item in the admin interface. Then add one or more seed URLs to the collection (e.g. https://www.facebook.com/visitberlin/). You can bulk add multiple seeds (one per line) fron the dashboard.\n\nAfter a couple of minutes the crawler should have discovered public posts and archived them. You can monitor the dashboard for new items added to the collection. Clicking the play icon will open the archived page. All archived pages are available for playback from http://0.0.0.0:4445/munin/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterk%2Fmunin-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeterk%2Fmunin-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterk%2Fmunin-indexer/lists"}