{"id":48653872,"url":"https://github.com/peterxcli/ozone-s3-compatibility","last_synced_at":"2026-05-06T21:02:16.386Z","repository":{"id":348544183,"uuid":"1198035911","full_name":"peterxcli/ozone-s3-compatibility","owner":"peterxcli","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-03T02:58:24.000Z","size":1531,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-03T04:29:08.151Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://peterxcli.github.io/ozone-s3-compatibility/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peterxcli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-01T04:22:45.000Z","updated_at":"2026-05-03T02:58:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/peterxcli/ozone-s3-compatibility","commit_stats":null,"previous_names":["peterxcli/ozone-s3-compatibility"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/peterxcli/ozone-s3-compatibility","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterxcli%2Fozone-s3-compatibility","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterxcli%2Fozone-s3-compatibility/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterxcli%2Fozone-s3-compatibility/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterxcli%2Fozone-s3-compatibility/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peterxcli","download_url":"https://codeload.github.com/peterxcli/ozone-s3-compatibility/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterxcli%2Fozone-s3-compatibility/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32711965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-06T19:35:05.142Z","status":"ssl_error","status_checked_at":"2026-05-06T19:35:03.996Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-10T08:58:40.955Z","updated_at":"2026-05-06T21:02:16.380Z","avatar_url":"https://github.com/peterxcli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ozone S3 Compatibility Nightly\n\nThis repo builds Apache Ozone from source on GitHub Actions, starts the packaged compose cluster, runs `ceph/s3-tests` and `minio/mint`, and publishes a historical compatibility report to GitHub Pages. The default `s3-tests` run excludes cases marked `fails_on_aws`, because the target is AWS S3 compatibility rather than RGW-specific behavior.\n\nThe report keeps:\n\n- Daily suite-level compatibility trends for `s3-tests` and `mint`\n- Feature-level trend charts at the top of the page\n- Sticky top navigation for Latest Run, Topline Trends, and Archived Runs\n- The latest run shown as a full report\n- Every archived run shown inline with its suite details\n- A GitHub repo link in the published header\n- Run-scope labels so smoke or subset publishes are distinguishable from full nightlies\n- Comment-triggered Ozone PR checks that compare a PR run with the latest published main run without adding the PR run to Pages history\n\n## Flow\n\nEach nightly run does this:\n\n1. Clone Ozone and reset to the requested ref, default `master`\n2. Build the Ozone dist package\n3. Start the packaged compose cluster from the built artifact\n4. Stage `s3-tests` from the pinned repo submodule into `.work`, patch it, then run it\n5. Stage `mint` from the pinned repo submodule into `.work`, patch it, build the required SDK/tool image payload, then run it against the same cluster\n6. Normalize both outputs into one run JSON\n7. Merge the new run into historical data and rebuild the static site\n8. Force-push the rebuilt site and run data to `gh-pages`\n\nThe PR comment flow is separate. A comment listener on the Ozone PR sends `repository_dispatch` to this repo, `.github/workflows/ozone-pr-s3-compatibility.yml` runs against the PR head branch, compares `out/pr-run/run.json` with the latest published main run under `gh-pages/data/runs`, and posts the markdown comparison back to the PR. The Actions run title includes the PR number and dispatched short commit when the comment forwarder sends it, and the comparison is also written to the run summary. It uploads the PR run as an Actions artifact only; it does not write the PR run to `gh-pages/data/runs`.\n\n## Repo Layout\n\n- `.github/workflows/nightly.yml`: scheduled workflow and manual/`act` entrypoint\n- `.github/workflows/ozone-pr-s3-compatibility.yml`: repository-dispatch/manual workflow for Ozone PR comment-triggered checks\n- `.github/workflows/refresh-pages-ui.yml`: manual workflow that builds the Vue frontend and updates only the published UI assets on `gh-pages`\n- `.agents/skills/ozone-s3-compat-failure-fixer`: repo-carried agent skill for inspecting PR compatibility artifacts from an Ozone checkout\n- [`scripts/run-nightly.sh`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/scripts/run-nightly.sh): orchestration for clone/build/start/run\n- [`scripts/normalize_run.py`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/scripts/normalize_run.py): converts raw outputs into a report-friendly JSON model\n- [`scripts/build_pages.py`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/scripts/build_pages.py): rebuilds the static Pages site from historical run JSON files\n- [`scripts/compare_runs.py`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/scripts/compare_runs.py): writes the Ozone PR comparison markdown against the latest published main run\n- [`site/src/App.vue`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/site/src/App.vue) and [`site/src/components`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/site/src/components): Vue 3 frontend source\n- [`site/vite.config.js`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/site/vite.config.js) and [`site/package.json`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/site/package.json): frontend build config\n\n## GitHub Setup\n\n1. Create the repo.\n2. Push `main`.\n3. In GitHub Pages settings, set the source to the `gh-pages` branch root.\n4. Leave the workflow permissions at the repository default, or allow `contents: write`.\n5. For Ozone PR comment posting, add `OZONE_PR_COMMENT_TOKEN` with permission to create comments on the Ozone repository.\n\nThe workflow handles branch creation itself if `gh-pages` does not exist yet.\nIf you only want to publish frontend changes without rebuilding run history, trigger `refresh-pages-ui`. It builds the Vue app, updates the published UI files on `gh-pages`, and leaves `data/` untouched.\nFor `/s3-compat` comments on Ozone PRs, install a comment forwarder in the Ozone repository or a GitHub App that sends `repository_dispatch` to this repo. See [`docs/ozone-pr-comment-bot.md`](/Users/lixucheng/Documents/small-project/ozone-s3-compatibility/docs/ozone-pr-comment-bot.md).\nFor agent-assisted fixing from an Ozone checkout, install or reference the bundled `ozone-s3-compat-failure-fixer` skill and use it to download the PR artifact, summarize failing cases, inspect raw logs, and guide the Ozone-side fix.\n\n## Publish Paths\n\n- `nightly.yml`: runs Ozone, `s3-tests`, and Mint, normalizes a new run, rebuilds the full Pages output, and publishes both UI and `data/` to `gh-pages`\n- `ozone-pr-s3-compatibility.yml`: runs Ozone PR branches on demand, compares with the latest published main run, comments on the Ozone PR, and uploads artifacts without publishing `data/`\n- `refresh-pages-ui.yml`: refreshes the published UI while keeping existing `data/` intact, and regenerates `social-preview.svg` from the latest published run so the OG image stays current\n\n## Local Run\n\nInitialize the suite submodules first:\n\n```bash\ngit submodule update --init --recursive\n```\n\nYou can then run the orchestration script directly against the pinned submodules.\n\n```bash\nexport OZONE_REPO=/Users/lixucheng/Documents/oss/apache/ozone\nexport S3_TESTS_ARGS='s3tests/functional/test_s3.py::test_bucket_list_empty'\nexport MINT_TARGETS='healthcheck awscli'\nexport OUTPUT_ROOT=\"$PWD/out/run\"\n\nbash scripts/run-nightly.sh\nnpm --prefix site ci\nnpm --prefix site run build\npython3 scripts/build_pages.py --output-dir out/pages --new-run out/run/run.json\n```\n\nIf you want to stage from different local clones instead of the pinned submodules, set these overrides before `bash scripts/run-nightly.sh`:\n\n```bash\nexport S3_TESTS_SOURCE=/tmp/ozone-compat-src/s3-tests\nexport MINT_SOURCE=/tmp/ozone-compat-src/mint\n```\n\nOpen `out/pages/index.html` in a local web server after that.\n\nSet `S3_TESTS_MARK_EXPR=''` if you need to disable the default AWS marker filter for a one-off run.\n\n## Using `act`\n\nThe workflow exposes `workflow_dispatch` inputs specifically so `nektos/act` can run a smaller smoke job locally.\n\nRecommended first pass:\n\n```bash\ngit submodule update --init --recursive\n\n./scripts/build-act-runner.sh\n\nact workflow_dispatch \\\n  -W .github/workflows/nightly.yml \\\n  -e .github/act/nightly-event.json \\\n  --secret GITHUB_TOKEN=\"$(gh auth token)\"\n```\n\nNotes:\n\n- `.actrc` maps `ubuntu-latest` and `ubuntu-24.04` to a local runner image with Docker, Java, Python, Maven, and rsync ready to go.\n- `./scripts/build-act-runner.sh` builds the local `ozone-s3-compatibility/act-runner:latest` image used by `.actrc`.\n- `actions/checkout` now initializes the `s3-tests` and `mint` submodules before the nightly job stages patched copies into `.work`.\n- The full nightly path is heavy. Start with a narrow `s3-tests` selector and a small `mint_targets` list.\n- When `mint_targets` is set, the local Mint image build now installs only those selected SDK/tool targets by default.\n- Publishing is disabled in the sample `act` event file. Turn on `publish_pages` only when you actually want to push `gh-pages`.\n- Set `ACT_RUNNER_PLATFORM` before building if you need to force a non-default Docker platform for the local runner image.\n\nIf you want to override the build-time subset independently of the runtime Mint selection, set `MINT_BUILD_TARGETS`. Leaving it unset makes the build follow `MINT_TARGETS`; setting it to an empty string forces the full Mint image build.\n\n## Compatibility Rate\n\nThe site uses:\n\n`compatibility_rate = passed / (passed + failed + errored)`\n\nSkipped and `NA` results are tracked separately and excluded from the rate.\n\n## Current Tradeoffs\n\n- `s3-tests` history stores non-passing case detail instead of every passing case. That keeps the archive small enough for long-term Pages history while still keeping feature summaries for every run.\n- `mint` stores full case detail because its output is compact.\n- The default workflow uses one datanode to keep GitHub-hosted runners practical. If you want multi-node coverage, raise `ozone_datanodes` in `workflow_dispatch` or the workflow env.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterxcli%2Fozone-s3-compatibility","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeterxcli%2Fozone-s3-compatibility","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterxcli%2Fozone-s3-compatibility/lists"}