{"id":16193519,"url":"https://github.com/pyramation/nats-queue-issue-43","last_synced_at":"2025-10-11T11:38:19.299Z","repository":{"id":66351086,"uuid":"153695200","full_name":"pyramation/nats-queue-issue-43","owner":"pyramation","description":null,"archived":false,"fork":false,"pushed_at":"2018-10-18T22:35:30.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-07T15:18:54.062Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyramation.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-18T22:29:04.000Z","updated_at":"2018-10-18T22:35:31.000Z","dependencies_parsed_at":"2023-02-24T18:00:54.339Z","dependency_job_id":null,"html_url":"https://github.com/pyramation/nats-queue-issue-43","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pyramation/nats-queue-issue-43","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyramation%2Fnats-queue-issue-43","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyramation%2Fnats-queue-issue-43/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyramation%2Fnats-queue-issue-43/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyramation%2Fnats-queue-issue-43/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyramation","download_url":"https://codeload.github.com/pyramation/nats-queue-issue-43/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyramation%2Fnats-queue-issue-43/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007035,"owners_count":26084227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T08:15:13.483Z","updated_at":"2025-10-11T11:38:19.284Z","avatar_url":"https://github.com/pyramation.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"Code for issue https://github.com/openfaas/nats-queue-worker/issues/43\n\n# steps to reproduce\n\n### environment \n\n* Using docker for mac kubernetes\n\n* using `--gateway 127.0.0.1:31112` external, and `gateway.openfaas:8080` for internal. \n\n* I'm also not using a Always pull policy, so you may need to look inside of `scripts/` and adjust to a prefix and publish an image and update the `stack.yml` to get this to run\n\n### Build the functions and deploy them\n\n```\nfaas remove -f stack.yml --gateway 127.0.0.1:31112\nfaas build -f stack.yml\nfaas deploy -f stack.yml --gateway 127.0.0.1:31112\n```\n\n### log queue worker\n\nOpen another shell, and log the queue worker\n\n```\n./scripts/log-queue-worker.sh\n```\n\n### Crash\n\nGo back to the first shell and run the script that should crash the `queue-worker`:\n\n```\n./scripts/crash.sh\n```\n\n** potentially try running the crash script twice if it doesn't work the first time\n\n### Un-crash the worker\n\nDepending on your system, the deploy could have worked. For mine, it seems as though there is a lag to removal (probably k8s Termination), so the call to `deploy` returns a 500 code. If that's the case, wait a while, then manually `deploy` again to get functions up, and the nats queue should go back up.\n\n```\nfaas deploy -f stack.yml --gateway 127.0.0.1:31112\n```\n\nAnd you'll have to tail the queue-worker again, you should eventually see it go back up.\n\n```\n./scripts/log-queue-worker.sh\n```\n\n### Teardown\n\n```\nfaas remove -f stack.yml --gateway 127.0.0.1:31112\n```\n\n## notes on crash\n\nDuring the `remove` nothing bad happens, however, right around the time the 2nd `deploy` happens, this happens (probably because it may still be removing?)\n\n```\nUnexpected status: 500, message: object is being deleted: deployments.extensions \"bar\" already exists\n\nDeploying: foo.\n\nUnexpected status: 500, message: object is being deleted: deployments.extensions \"foo\" already exists\n```\n\nMy gut tells me it's something due to the lag of k8s when the deployment is attempted, but the removal has not completed. Or potentially some parts of the k8s are setup but others are not.\n\n## full log from crash script\n\n```\n$ ./scripts/02-crash.sh\ncalling async callback 1 (deployed)\ncalling async callback 2 (deployed)\ncalling async callback 3 (deployed)\ncalling async callback 4 (deployed)\ncalling async callback 5 (deployed)\ncalling async callback 6 (deployed)\ncalling async callback 7 (deployed)\ncalling async callback 8 (deployed)\ncalling async callback 9 (deployed)\ncalling async callback 10 (deployed)\nDeleting: foo.\nRemoving old function.\nDeleting: bar.\nRemoving old function.\ncalling async callback 1 (removing)\ncalling async callback 2 (removing)\ncalling async callback 3 (removing)\ncalling async callback 4 (removing)\ncalling async callback 5 (removing)\ncalling async callback 6 (removing)\ncalling async callback 7 (removing)\ncalling async callback 8 (removing)\ncalling async callback 9 (removing)\ncalling async callback 10 (removing)\ncalling async callback 11 (removing)\ncalling async callback 12 (removing)\ncalling async callback 13 (removing)\ncalling async callback 14 (removing)\ncalling async callback 15 (removing)\nDeploying: bar.\n\nUnexpected status: 500, message: object is being deleted: deployments.extensions \"bar\" already exists\n\nDeploying: foo.\n\nUnexpected status: 500, message: object is being deleted: deployments.extensions \"foo\" already exists\n\nFunction 'bar' failed to deploy with status code: 500\nFunction 'foo' failed to deploy with status code: 500\ncalling async callback 1 (redeploying)\ncalling async callback 2 (redeploying)\ncalling async callback 3 (redeploying)\ncalling async callback 4 (redeploying)\ncalling async callback 5 (redeploying)\ncalling async callback 6 (redeploying)\ncalling async callback 7 (redeploying)\ncalling async callback 8 (redeploying)\ncalling async callback 9 (redeploying)\ncalling async callback 10 (redeploying)\nDeploying: foo.\n\nDeployed. 202 Accepted.\nURL: http://127.0.0.1:31112/function/foo\n\nDeploying: bar.\n\nDeployed. 202 Accepted.\nURL: http://127.0.0.1:31112/function/bar\n\ncalling async callback 1 (redeploying)\ncalling async callback 2 (redeploying)\ncalling async callback 3 (redeploying)\ncalling async callback 4 (redeploying)\ncalling async callback 5 (redeploying)\ncalling async callback 6 (redeploying)\ncalling async callback 7 (redeploying)\ncalling async callback 8 (redeploying)\ncalling async callback 9 (redeploying)\ncalling async callback 10 (redeploying)\n```\n\n### log from queue-worker\n\n```\nCallback to: http://gateway.openfaas:8080/function/bar\nPosted result: 200\nError with AddBasicAuth : Unable to read basic auth: invalid SecretMountPath specified for reading secrets\nPosting report - 200\n[#11] Received on [faas-request]: 'sequence:132 subject:\"faas-request\" data:\"{\\\"Header\\\":{\\\"Accept\\\":[\\\"*/*\\\"],\\\"Content-Length\\\":[\\\"52\\\"],\\\"Content-Type\\\":[\\\"application/x-www-form-urlencoded\\\"],\\\"User-Agent\\\":[\\\"curl/7.54.0\\\"],\\\"X-Call-Id\\\":[\\\"de92184d-105d-4260-9eb8-496ece782baa\\\"],\\\"X-Callback-Url\\\":[\\\"http://gateway.openfaas:8080/function/bar\\\"],\\\"X-Start-Time\\\":[\\\"1539900637321942500\\\"]},\\\"Host\\\":\\\"localhost:31112\\\",\\\"Body\\\":\\\"eyJ3b3JrZXJJZCI6Indvcmtlci0wLjkxNzY3OTgwODM4Njk2MTciLCAiam9iSWQiOjEyfQ==\\\",\\\"Method\\\":\\\"POST\\\",\\\"Path\\\":\\\"\\\",\\\"QueryString\\\":\\\"\\\",\\\"Function\\\":\\\"foo\\\",\\\"CallbackUrl\\\":{\\\"Scheme\\\":\\\"http\\\",\\\"Opaque\\\":\\\"\\\",\\\"User\\\":null,\\\"Host\\\":\\\"gateway.openfaas:8080\\\",\\\"Path\\\":\\\"/function/bar\\\",\\\"RawPath\\\":\\\"\\\",\\\"ForceQuery\\\":false,\\\"RawQuery\\\":\\\"\\\",\\\"Fragment\\\":\\\"\\\"}}\" timestamp:1539900637324999700 '\nRequest for foo.\nWrote 0 Bytes\nCallback to: http://gateway.openfaas:8080/function/bar\n200 OK\nPosted result: 200\nError with AddBasicAuth : Unable to read basic auth: invalid SecretMountPath specified for reading secrets\nPosting report - 200\nRequest for foo.\n[#12] Received on [faas-request]: 'sequence:133 subject:\"faas-request\" data:\"{\\\"Header\\\":{\\\"Accept\\\":[\\\"*/*\\\"],\\\"Content-Length\\\":[\\\"52\\\"],\\\"Content-Type\\\":[\\\"application/x-www-form-urlencoded\\\"],\\\"User-Agent\\\":[\\\"curl/7.54.0\\\"],\\\"X-Call-Id\\\":[\\\"f3d9d551-c093-44c2-bc3d-14a035de1e0a\\\"],\\\"X-Callback-Url\\\":[\\\"http://gateway.openfaas:8080/function/bar\\\"],\\\"X-Start-Time\\\":[\\\"1539900638759159000\\\"]},\\\"Host\\\":\\\"localhost:31112\\\",\\\"Body\\\":\\\"eyJ3b3JrZXJJZCI6Indvcmtlci0wLjkxNzY3OTgwODM4Njk2MTciLCAiam9iSWQiOjEyfQ==\\\",\\\"Method\\\":\\\"POST\\\",\\\"Path\\\":\\\"\\\",\\\"QueryString\\\":\\\"\\\",\\\"Function\\\":\\\"foo\\\",\\\"CallbackUrl\\\":{\\\"Scheme\\\":\\\"http\\\",\\\"Opaque\\\":\\\"\\\",\\\"User\\\":null,\\\"Host\\\":\\\"gateway.openfaas:8080\\\",\\\"Path\\\":\\\"/function/bar\\\",\\\"RawPath\\\":\\\"\\\",\\\"ForceQuery\\\":false,\\\"RawQuery\\\":\\\"\\\",\\\"Fragment\\\":\\\"\\\"}}\" timestamp:1539900638767709600 '\nPost http://foo.openfaas-fn.svc.cluster.local:8080/: dial tcp 10.105.147.103:8080: i/o timeout\nCallback to: http://gateway.openfaas:8080/function/bar\npanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x69b878]\n\ngoroutine 19 [running]:\nmain.postResult(0xc4200719e0, 0x0, 0x0, 0x0, 0x0, 0xc420302000, 0x29, 0xc42025b980, 0x24, 0x0, ...)\n        /go/src/github.com/openfaas/nats-queue-worker/main.go:239 +0xb8\nmain.main.func1(0xc4202ec360)\n        /go/src/github.com/openfaas/nats-queue-worker/main.go:122 +0x8ba\ngithub.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats-streaming.(*conn).processMsg(0xc4200f6380, 0xc420120870)\n        /go/src/github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats-streaming/stan.go:751 +0x26f\ngithub.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats-streaming.(*conn).(github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats-streaming.processMsg)-fm(0xc420120870)\n        /go/src/github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats-streaming/sub.go:228 +0x34\ngithub.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats.(*Conn).waitForMsgs(0xc4200ac500, 0xc42015c240)\n        /go/src/github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats/nats.go:1652 +0x24a\ncreated by github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats.(*Conn).subscribe\n        /go/src/github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/go-nats/nats.go:2374 +0x4de\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyramation%2Fnats-queue-issue-43","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyramation%2Fnats-queue-issue-43","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyramation%2Fnats-queue-issue-43/lists"}