{"id":13930014,"url":"https://github.com/zulip/zulip-archive","last_synced_at":"2025-04-09T05:11:42.663Z","repository":{"id":38858501,"uuid":"194659145","full_name":"zulip/zulip-archive","owner":"zulip","description":"Generate a static HTML archive of messages in any combination of streams in a Zulip organization.","archived":false,"fork":false,"pushed_at":"2025-02-24T19:31:28.000Z","size":160,"stargazers_count":78,"open_issues_count":18,"forks_count":45,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-01T11:12:31.705Z","etag":null,"topics":["github-actions","zulip"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zulip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":"zulip","patreon":"zulip","open_collective":"zulip"}},"created_at":"2019-07-01T11:24:06.000Z","updated_at":"2025-03-09T22:32:50.000Z","dependencies_parsed_at":"2023-11-10T20:27:51.116Z","dependency_job_id":"73380618-6bf2-4c0e-b603-624a062d8272","html_url":"https://github.com/zulip/zulip-archive","commit_stats":{"total_commits":112,"total_committers":18,"mean_commits":6.222222222222222,"dds":0.6964285714285714,"last_synced_commit":"697debb7eedcf2656e3da6197e84b008ea6c475a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zulip%2Fzulip-archive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zulip%2Fzulip-archive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zulip%2Fzulip-archive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zulip%2Fzulip-archive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zulip","download_url":"https://codeload.github.com/zulip/zulip-archive/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247980844,"owners_count":21027808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-actions","zulip"],"created_at":"2024-08-07T18:02:40.767Z","updated_at":"2025-04-09T05:11:42.644Z","avatar_url":"https://github.com/zulip.png","language":"Python","funding_links":["https://github.com/sponsors/zulip","https://patreon.com/zulip","https://opencollective.com/zulip"],"categories":["others"],"sub_categories":[],"readme":"# Zulip HTML archive\n\n[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nGenerates an HTML archive of a configured set of streams within a\n[Zulip](https://zulip.com) organization. It is common to archive all [public](https://zulip.com/help/stream-permissions) or [web-public](https://zulip.com/help/public-access-option) streams.\n\nExample: [Lean Prover\narchive](https://leanprover-community.github.io/archive/).\n\n`zulip-archive` works by downloading Zulip message history via the\nAPI, storing it in JSON files, maintaining its local archive with\nincremental updates, and turning those JSON files into the HTML\narchive.\n\nThis archive tool is often used in addition to enabling the [public access option](https://zulip.com/help/public-access-option) for your organization, which lets administrators configure selected streams to be web-public. Web-public streams can be viewed by anyone on the Internet without creating an account in your organization. The public access option does not yet support search engine indexing, which makes this archive tool a good option if it's important for your organization's chat history to appear in search results. It is easy to configure `zulip-archive` to automatically archive all web-public streams in your organization.\n\n### Contents\n* [Running zulip-archive as a GitHub action](#running-zulip-archive-as-a-github-action)\n* [Running zulip-archive without GitHub actions](#running-zulip-archive-without-github-actions)\n* [Why archive](#why-archive)\n* [Contributing and future plans](#contributing-and-future-plans)\n\n## Running zulip-archive as a GitHub action\n\nRunning `zulip-archive` as a GitHub action is easiest way to get up and running. The action will periodically sync a GitHub repository with the latest messages, and publish the archive website using GitHub pages. Follow the steps below to set up a `zulip-archive` GitHub action in a few minutes.\n\n### Step 1 - Create a repository for running the action\n\nIt's best to use a dedicated repository for running the action. You can create a new repository at https://github.com/new/.\n\n### Step 2 - Generate credentials\n\nThe GitHub action requires a Zulip API key in order to run. The key is used for fetching messages in public streams in your Zulip organization. It is strongly recommended that you [create a bot](https://zulip.com/help/add-a-bot-or-integration) and use its zuliprc, rather than using your personal zuliprc.\n\n### Step 3 - Store credentials as secrets in the repository\n\nThe credentials for your bot need to be stored in the repository as secrets, so that the action can access them during run time. You can create secrets in your repository at `https://github.com/\u003cusername\u003e/\u003crepo-name\u003e/settings/secrets`, where `\u003cusername\u003e` is your GitHub username, and `\u003crepo-name\u003e` is the name of the repository you are using.\n\nYou will need to create the following secret. Use the credentials generated in the above step as the value of each secret.\n\n|Secret name   | Value                                                |\n|--------------|------------------------------------------------------|\n|zuliprc       | The file content of the zuliprc obtained from step 2 |\n\n### Step 4 - Enable GitHub Pages or set up base URL\n\nGo to `https://github.com/\u003cusername\u003e/\u003crepo-name\u003e/settings/pages`, select `main` (or a branch of your choosing), and `/` as the folder. Save the changes. The base URL of the generated site will be resolved to GitHub Pages, i.e., `https://\u003cusername\u003e.github.io/\u003crepo-name\u003e` or the configured custom domain name.\n\nAlternatively, you can configure the `base_url` option to populate the base URL. This option could be useful in situation when you are not using GitHub Pages.\n\n### Step 5 - Configure the streams you want to index\n\nYou will need to configure which streams will be indexed by `zulip-archive` by creating a `streams.yaml` file in the repository you are using for the GitHub action. As a starting point, you can make a copy of the default configuration file: `cp default_streams.yaml streams.yaml`\n\nTo index all the [web-public streams](https://zulip.com/help/public-access-option) in your organization, set the following as the content of your `streams.yaml` file.\n\n```yaml\nincluded:\n  - 'web-public:*'\n```\n\nTo index all the [public streams](https://zulip.com/help/stream-permissions), set the following as the content of your `streams.yaml` file. Note that public streams include all web-public streams.\n\n```yaml\nincluded:\n  - '*'\n```\n\nYou can exclude specific public streams by placing them under the `excluded` key.\n\n```yaml\nincluded:\n  - '*'\n\nexcluded:\n  - general\n  - development help\n```\n\nAlternatively, you can specify only the streams that you want to index.\n\n```yaml\nincluded:\n  - python\n  - data structures\n  - javascript\n```\n\n### Step 6 - Enable the zulip-archive action\n\nEnable the action by creating a file called `.github/workflows/main.yaml`:\n\n#### Sample `main.yaml` file\n\n```yaml\non:\n  schedule:\n   - cron: '*/20 * * * *'\n\njobs:\n  publish_archive_job:\n    runs-on: ubuntu-latest\n    name: A job to publish zulip-archive in GitHub pages\n    steps:\n    - name: Checkout\n      uses: actions/checkout@v3\n    - name: Run archive\n      id: archive\n      uses: zulip/zulip-archive@master\n      with:\n        zuliprc: ${{ secrets.ZULIPRC }}\n        # Using the GitHub Token that is provided automatically by GitHub Actions\n        # (no setup needed).\n        github_token: ${{ secrets.GITHUB_TOKEN }}\n        delete_history: true\n        archive_branch: main\n```\n\n#### Configure run frequency\n\nThe above file tells GitHub to run the `zulip-archive` action every 20 minutes. You can [adjust](https://en.wikipedia.org/wiki/Cron) the `cron` key to modify the schedule as you feel appropriate.\n\nIf you Zulip organization history is very large (not the case for most users), it is recommended that you initially increase the time between runs to an hour or longer (e.g., `'0 * * * *'`). This is is because the initial archive run that fetches the messages for the first time will take a long time, and you don't want the second cron job to start before the first run is completed. After the initial run, you can shorten the cron job period as desired.\n\n#### Configure `delete_history` option\n\nIf you are running frequent updates with a busy Zulip organization,\nthe Git repository that you use to run the action will grow very\nquickly. In this situation, it is recommended that you set the `delete_history` option to\n`true`. This will overwrite the Git _history_ in the repository, but\nkeep all the _content_. If you are using the repository for more than\njust the Zulip archive (not recommended), you may want to set the `delete_history` flag to `false`, but be\nwarned that the repository size may explode.\n\n### Step 7 - Verify that everything works\n\nFinally, verify that everything is working as expected. You can track the status of the action by visiting `https://github.com/\u003cgithub-username\u003e/\u003crepo-name\u003e/actions`. Once the initial run is completed, you should be able to visit the archive by opening the link provided at the end of the action run log. The link will generally be of the form `\u003cgithub-username\u003e.github.io/\u003crepo-name\u003e`, or `\u003cyour-personal-domain\u003e/\u003crepo-name\u003e` if you have configured your own personal domain to point to GitHub pages.\n\nIf you configure `base_url` option, you can track the status of the action by visiting the URL instead.\n\n## Running zulip-archive without GitHub actions\n\nFor most users, running `zulip-archive` as GitHub actions should be good enough. If you want to run `zulip-archive` in your own server or do something else, see the [instructions](instructions.md) docs. The [hosting docs](hosting.md) also offer a few suggestions for good ways to host the output of this tool.\n\n## Why archive?\n\nThe best place to participate actively in a Zulip community is an app\nthat tracks unread messages, formats messages properly, and is\ndesigned for efficient interaction.  However, there are several use\ncases where this HTML archive tool is a valuable complement to the\nZulip apps:\n\n* A public HTML archive can be indexed by search engines and doesn't\n  require authentication to access.  For open source projects and\n  other open communities, this provides a convenient search and\n  browsing experience for users who may not want to sign up an account\n  just to find previous answers to common questions.\n\n* It's common to set up Zulip instances for one-time events such as\n  conferences, retreats, or hackathons.  Once the event ends, you may\n  want to shut down the Zulip instance for operational convenience,\n  but still want an archive of the communications.\n\n* You may also decide to shut down a Zulip instance, whether to move\n  to another communication tool, to deduplicate instances, or because\n  your organization is shutting down.  You can always [export your\n  Zulip data](https://zulip.com/help/export-your-organization),\n  but the other tool may not be able to import it.  In such a case,\n  you can use this archive tool to keep the old conversations\n  accessible. (Contrast this to scenarios where your provider locks\n  you in to a solution, even when folks are dissatisfied with the\n  tool, because they own the data.)\n\n* You may also want to publish your conversations outside of Zulip for\n  branding reasons or to integrate with other data.  You can modify\n  the tools here as needed for that.  You own your own data.\n\n\n## Contributing and future plans\n\nFeedback, issues, and pull requests are encouraged!  Our goal is for\nthis project to support the needs of any community looking for an HTML\narchive of their Zulip organization's history, through just\nconfiguration changes.  So please report even minor inconveniences,\neither via a GitHub issue or by posting in the\n[#integrations](https://chat.zulip.org/#narrow/stream/127-integrations/) stream\nin the [Zulip development community](https://zulip.com/development-community/).\n\nThis project is licensed under the MIT license.\n\nAuthor: [Robert Y. Lewis](https://robertylewis.com/) ([@robertylewis](https://github.com/robertylewis))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzulip%2Fzulip-archive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzulip%2Fzulip-archive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzulip%2Fzulip-archive/lists"}