{"id":13515116,"url":"https://github.com/hbish/smex","last_synced_at":"2025-07-10T17:31:35.259Z","repository":{"id":93060616,"uuid":"260383975","full_name":"hbish/smex","owner":"hbish","description":"A blazing fast CLI application that processes sitemaps in golang.","archived":false,"fork":false,"pushed_at":"2020-07-21T15:20:59.000Z","size":98,"stargazers_count":22,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-04-05T15:35:20.978Z","etag":null,"topics":["cli","cross-platform","csv","extraction","go-cli","golang","golang-library","json","seotools","sitemap","sitemap-extractor","sitemap-parser"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hbish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-05-01T05:06:28.000Z","updated_at":"2023-08-01T22:36:12.000Z","dependencies_parsed_at":"2024-01-31T23:57:38.399Z","dependency_job_id":"d65b02a5-dae8-4943-a2ba-e3d213f777e1","html_url":"https://github.com/hbish/smex","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/hbish/smex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hbish%2Fsmex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hbish%2Fsmex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hbish%2Fsmex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hbish%2Fsmex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hbish","download_url":"https://codeload.github.com/hbish/smex/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hbish%2Fsmex/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264619028,"owners_count":23638394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","cross-platform","csv","extraction","go-cli","golang","golang-library","json","seotools","sitemap","sitemap-extractor","sitemap-parser"],"created_at":"2024-08-01T05:01:06.458Z","updated_at":"2025-07-10T17:31:32.492Z","avatar_url":"https://github.com/hbish.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# smex\n\nA blazing fast CLI application that processes sitemaps in Go Lang. \n\n---\n\n[![GoDoc](https://godoc.org/github.com/hbish.smex?status.svg)](https://godoc.org/github.com/hbish/smex)\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fhbish%2Fsmex.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fhbish%2Fsmex?ref=badge_shield)\n![](https://img.shields.io/badge/license-MIT-blue.svg)\n[![smex](https://circleci.com/gh/hbish/smex.svg?style=shield)](https://circleci.com/gh/hbish/smex)\n[![Go Report Card](https://goreportcard.com/badge/github.com/hbish/smex)](https://goreportcard.com/report/github.com/hbish/smex)\n[![Coverage Status](https://coveralls.io/repos/github/hbish/smex/badge.svg?branch=develop)](https://coveralls.io/github/hbish/smex?branch=develop)\n\nSmex is short for **S**ite**M**ap **EX**trator. It can handle various operations related to sitemaps with more being \nbuild over time. Smex has grown from the frustration of migrating multiple websites in the past and handling changes in \nURL semantics and massive amount of assets like images.  \n\nSmex supports: \n- [x] extraction of urls\n- [x] process local/remote sitemaps\n- [x] output to csv/json\n- [x] pattern matching on urls\n- [x] extraction of images\n- [x] extraction of video\n- [x] extraction of news\n- [ ] basic auth to retrieve sitemap\n- [ ] url status checking\n- [ ] support sitemap indexes\n- [ ] support for sitemaps with multiple languages\n\nNote: smex is not a sitemap validator and would not check the validity of sitemaps against the xsd. It will try to parse\nthe sitemaps on best effort.\n\n[![asciicast](https://asciinema.org/a/327587.svg)](https://asciinema.org/a/327587)\n\n## Usages\n\n### Extract\n\nTo extract information from sitemap\n\nThe following command extracts only the urls from yoast's post sitemaps and prints to stdout\n\n`smex extract https://yoast.com/post-sitemap.xml --remote --loc`\n\nYou can also perform the extraction at a sitemap locally\n\n`smex extract ~/Download/sitemap.xml`\n\nBy default, smex write to stdout. Due to the limited space only important fields are displayed. If you want a full extract \nof the sitemap, then add `--format` or `-f` flag to write to a file, currently smex only support csv \u0026 json.\n\n`smex extract https://yoast.com/post-sitemap.xml --remote --format csv`\n\nTo filter the URLs you can supply a valid regex pattern using `--pattern` or `-p` flag\n\n`smex extract https://yoast.com/post-sitemap.xml --remote  --pattern \".*seo.*\" --format csv`\n\nThe `--output` or `-o` flag will change the filename (no extension required), this is defaulted to `smex-output.(csv|json)`\n\n### Check\n\nTo check the status of the pages\n\n__TODO: this feature has not yet being implemented__\n\n### Help\n\nTo get help simply run `smex` without any commands and flags.\n\n## Installation\n\n### Using Go\n\n`go get -u github.com/hbish/smex`\n\n### Using Homebrew\n\n`brew install hbish/taps/smex`\n\n### Binary\n\nCurrently cross-compiled for:\n\n- Mac (64 bit)\n- Linux (32/64 bit)\n- Windows (32/64 bit)\n\nLatest versions can be downloaded via [Releases](https://github.com/hbish/smex/releases).\n\n## Documentation\n\nFor package documentation please check on [pkg.go.dev](https://pkg.go.dev/github.com/hbish/smex).\n\n## Contribute\n\n### Getting the code\n\nClone the repo\n\n`git clone git@github.com:hbish/smex.git`\n\nInitialise local environment and install commit hook\n\n`make init`\n\nRunning the source\n\n`go run github.com/hbish/smex [command] [flags...]`\n\nRunning tests\n\n`make test`\n\nMake the changes, the linter is set up to run when you commit your code, if it passes feel free to submit a PR!\n\n## Last Thing\n\nSmex is my first stab at building with Go. If you have any feedback, comments or notice any bugs, I'd be more than happy \nto consider them through the github issue tracker or better yet send me a pull request! \n\n\n\n## License\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fhbish%2Fsmex.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fhbish%2Fsmex?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhbish%2Fsmex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhbish%2Fsmex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhbish%2Fsmex/lists"}