{"id":19298259,"url":"https://github.com/casbin/google-groups-crawler","last_synced_at":"2025-06-23T05:10:29.561Z","repository":{"id":49283418,"uuid":"361180454","full_name":"casbin/google-groups-crawler","owner":"casbin","description":"This tool is for legitimate use. We internally use it for synchronizing posts between Casnode forum and our own Casbin Google Group","archived":false,"fork":false,"pushed_at":"2021-06-20T12:52:58.000Z","size":47,"stargazers_count":5,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-09T13:27:39.623Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/casbin/casnode","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/casbin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-24T14:17:13.000Z","updated_at":"2025-01-16T11:53:27.000Z","dependencies_parsed_at":"2022-09-26T16:41:48.849Z","dependency_job_id":null,"html_url":"https://github.com/casbin/google-groups-crawler","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/casbin/google-groups-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casbin%2Fgoogle-groups-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casbin%2Fgoogle-groups-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casbin%2Fgoogle-groups-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casbin%2Fgoogle-groups-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/casbin","download_url":"https://codeload.github.com/casbin/google-groups-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casbin%2Fgoogle-groups-crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260918435,"owners_count":23082570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T23:07:25.288Z","updated_at":"2025-06-23T05:10:29.511Z","avatar_url":"https://github.com/casbin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google Groups Crawler\n\n## Installation\n\n``` go\nimport crawler \"github.com/casbin/google-groups-crawler\"\n```\n\n## Usage\n\nWe must get an instance of `GoogleGroup` first:\n\n``` go\ngroup := crawler.NewGoogleGroup(groupName string, cookie ...string)\n```\n\nThe second parameter `cookie` is optional. Google group won't tell you email address of all repliers until you logged\nin, so you need to fill the parameter with a logged-in user's cookie. (Of course, this user must be a member of the\ngroup)\n\nIt is OK to leave `cookie` blank, code still works. But `AuthorEmail` in `GoogleGroupMessage` will be empty. If you do need `cookie` to access emails of repliers, please follow these steps:\n\n- open Google Chrome (or another browser) and login\n- Navigate to [Google Group](https://groups.google.com/), select the group you want to craw\n- Press F12, and select `network`\n- Select a conversation (any conversation in this group is OK)\n- Select the first item in the request list\n- Select `Headers`\n- In `Request Headers`, right click `cookie`, and copy the value\n- Fill the parameter `cookie` with what you copied\n\n### Get all conversations of the group\n\n- For some special reasons, you cannot access Google Groups in some area. You can set up a http proxy, and fill the parameter `http.Client` with it. If you can access Google Groups directly, then you can just fill the parameter like the example code.\n- this function returns an array of `GoogleGroupConversation`\n\n``` go\nconversations := group.GetConversations(http.Client{})\n```\n\n### Get all messages of the conversation\n\n- `conversation` is an instance of `GoogleGroupConversation`\n- parameter `http.Client` is the same effect as above\n- this function returns an array of `GoogleGroupMessage`\n\n```go\nmessages := conversation.GetAllMessages(http.Client{}, removeGmailQuote)\n```\n\n## Data Structure\n\n```go\ntype GoogleGroup struct {\n    GroupName string\n    Cookie    string\n}\n\ntype GoogleGroupConversation struct {\n    Title     string\n    Id        string\n    GroupName string\n    Time      float64\n    Cookie    string\n}\n\ntype GoogleGroupMessage struct {\n    Author      string\n    AuthorEmail string\n    Content     string\n    Time        float64\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasbin%2Fgoogle-groups-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcasbin%2Fgoogle-groups-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasbin%2Fgoogle-groups-crawler/lists"}