{"id":13413937,"url":"https://github.com/pingcap/failpoint","last_synced_at":"2025-05-14T12:11:51.086Z","repository":{"id":39615868,"uuid":"179026050","full_name":"pingcap/failpoint","owner":"pingcap","description":"An implementation of failpoints for Golang.","archived":false,"fork":false,"pushed_at":"2024-05-28T01:13:02.000Z","size":311,"stargazers_count":847,"open_issues_count":7,"forks_count":63,"subscribers_count":111,"default_branch":"master","last_synced_at":"2025-04-10T11:02:03.259Z","etag":null,"topics":["failpoint","failure-injection","fault-injection","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pingcap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-02T07:48:18.000Z","updated_at":"2025-04-03T03:06:01.000Z","dependencies_parsed_at":"2024-05-28T04:56:48.306Z","dependency_job_id":null,"html_url":"https://github.com/pingcap/failpoint","commit_stats":{"total_commits":86,"total_committers":21,"mean_commits":4.095238095238095,"dds":0.7209302325581395,"last_synced_commit":"2eaa32854a6cece9be893bf4e3605c18586e9d6a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ffailpoint","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ffailpoint/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ffailpoint/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ffailpoint/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pingcap","download_url":"https://codeload.github.com/pingcap/failpoint/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254140768,"owners_count":22021220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["failpoint","failure-injection","fault-injection","golang"],"created_at":"2024-07-30T20:01:53.155Z","updated_at":"2025-05-14T12:11:46.072Z","avatar_url":"https://github.com/pingcap.png","language":"Go","readme":"# failpoint\n[![LICENSE](https://img.shields.io/github/license/pingcap/failpoint.svg)](https://github.com/pingcap/failpoint/blob/master/LICENSE)\n[![Language](https://img.shields.io/badge/Language-Go-blue.svg)](https://golang.org/)\n[![Go Report Card](https://goreportcard.com/badge/github.com/pingcap/failpoint)](https://goreportcard.com/report/github.com/pingcap/failpoint)\n[![Build Status](https://github.com/pingcap/failpoint/actions/workflows/suite.yml/badge.svg?branch=master)](https://github.com/pingcap/failpoint/actions/workflows/suite.yml?query=event%3Apush+branch%3Amaster)\n[![Coverage Status](https://codecov.io/gh/pingcap/failpoint/branch/master/graph/badge.svg)](https://codecov.io/gh/pingcap/failpoint)\n[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)  \n\nAn implementation of [failpoints][failpoint] for Golang. Fail points are used to add code points where errors may be injected in a user controlled fashion. Fail point is a code snippet that is only executed when the corresponding failpoint is active.\n\n[failpoint]: http://www.freebsd.org/cgi/man.cgi?query=fail\n\n## Quick Start (use `failpoint-ctl`)\n\n1.  Build `failpoint-ctl` from source\n\n    ``` bash\n    git clone https://github.com/pingcap/failpoint.git\n    cd failpoint\n    make\n    ls bin/failpoint-ctl\n    ```\n\n2.  Inject failpoints to your program, eg:\n\n    ``` go\n    package main\n\n    import \"github.com/pingcap/failpoint\"\n\n    func main() {\n        failpoint.Inject(\"testPanic\", func() {\n            panic(\"failpoint triggerd\")\n        })\n    }\n    ```\n\n3.  Transfrom your code with `failpoint-ctl enable`\n\n4.  Build with `go build`\n\n5.  Enable failpoints with `GO_FAILPOINTS` environment variable\n\n    ``` bash\n    GO_FAILPOINTS=\"main/testPanic=return(true)\" ./your-program\n    ```\n\n    Note: `GO_FAILPOINTS` does not work with `InjectCall` type of marker.\n\n6.  If you use `go run` to run the test, don't forget to add the generated `binding__failpoint_binding__.go` in your command, like:\n\n    ```bash\n    GO_FAILPOINTS=\"main/testPanic=return(true)\" go run your-program.go binding__failpoint_binding__.go\n    ```\n\n## Quick Start (use `failpoint-toolexec`)\n\n1.  Build `failpoint-toolexec` from source\n\n    ``` bash\n    git clone https://github.com/pingcap/failpoint.git\n    cd failpoint\n    make\n    ls bin/failpoint-toolexec\n    ```\n\n2.  Inject failpoints to your program, eg:\n\n    ``` go\n    package main\n\n    import \"github.com/pingcap/failpoint\"\n\n    func main() {\n        failpoint.Inject(\"testPanic\", func() {\n            panic(\"failpoint triggerd\")\n        })\n    }\n    ```\n\n3.  Use a separate build cache to avoid mixing caches without `failpoint-toolexec`, and build\n\n    `GOCACHE=/tmp/failpoint-cache go build -toolexec path/to/failpoint-toolexec`\n\n4.  Enable failpoints with `GO_FAILPOINTS` environment variable\n\n    ``` bash\n    GO_FAILPOINTS=\"main/testPanic=return(true)\" ./your-program\n    ```\n\n5.  You can also use `go run` or `go test`, like:\n\n    ```bash\n    GOCACHE=/tmp/failpoint-cache GO_FAILPOINTS=\"main/testPanic=return(true)\" go run -toolexec path/to/failpoint-toolexec your-program.go\n    ```\n\n## Design principles\n\n- Define failpoint in valid Golang code, not comments or anything else\n- Failpoint does not have any extra cost\n\n    - Will not take effect on regular logic\n    - Will not cause regular code performance regression\n    - Failpoint code will not appear in the final binary\n\n- Failpoint routine is writable/readable and should be checked by a compiler\n- Generated code by failpoint definition is easy to read\n- Keep the line numbers same with the injecting codes(easier to debug)\n- Support parallel tests with context.Context\n\n## Key concepts\n\n- Failpoint\n\n    Faillpoint is a code snippet that is only executed when the corresponding failpoint is active.\n    The closure will never be executed if `failpoint.Disable(\"failpoint-name-for-demo\")` is executed.\n\n    ```go\n    var outerVar = \"declare in outer scope\"\n    failpoint.Inject(\"failpoint-name-for-demo\", func(val failpoint.Value) {\n        fmt.Println(\"unit-test\", val, outerVar)\n    })\n    ```\n\n- Marker functions\n\n    - It is just an empty function\n\n        - To hint the rewriter to rewrite with an equality statement\n        - To receive some parameters as the rewrite rule\n        - It will be inline in the compiling time and emit nothing to binary (zero cost)\n        - The variables in external scope can be accessed in closure by capturing, and the converted code is still legal\n        because all the captured-variables location in outer scope of IF statement.\n\n    - It is easy to write/read \n    - Introduce a compiler check for failpoints which cannot compile in the regular mode if failpoint code is invalid\n\n- Marker funtion list\n\n    - `func Inject(fpname string, fpblock func(val Value)) {}`\n    - `func InjectContext(fpname string, ctx context.Context, fpblock func(val Value)) {}`\n    - `func InjectCall(fpname string, args ...any) {}`\n    - `func Break(label ...string) {}`\n    - `func Goto(label string) {}`\n    - `func Continue(label ...string) {}`\n    - `func Fallthrough() {}`\n    - `func Return(results ...interface{}) {}`\n    - `func Label(label string) {}`\n\n- Supported failpoint environment variable\n\n    failpoint can be enabled by export environment variables with the following patten, which is quite similar to [freebsd failpoint SYSCTL VARIABLES](https://www.freebsd.org/cgi/man.cgi?query=fail)\n\n    Note: `InjectCall` cannot be enabled by environment variables.\n\n    ```regexp\n    [\u003cpercent\u003e%][\u003ccount\u003e*]\u003ctype\u003e[(args...)][-\u003e\u003cmore terms\u003e]\n    ```\n\n    The \u003ctype\u003e argument specifies which action to take; it can be one of:\n\n    - off: Take no action (does not trigger failpoint code)\n    - return: Trigger failpoint with specified argument\n    - sleep: Sleep the specified number of milliseconds\n    - panic: Panic\n    - break: Execute gdb and break into debugger\n    - print: Print failpoint path for inject variable\n    - pause: Pause will pause until the failpoint is disabled\n\n## How to inject a failpoint to your program\n\n- You can call `failpoint.Inject` to inject a failpoint to the call site, where `failpoint-name` is\nused to trigger the failpoint and `failpoint-closure` will be expanded as the body of the IF statement.\n\n    ```go\n    failpoint.Inject(\"failpoint-name\", func(val failpoint.Value) {\n        failpoint.Return(\"unit-test\", val)\n    })\n    ```\n\n    The converted code looks like:\n\n    ```go\n    if val, _err_ := failpoint.Eval(_curpkg_(\"failpoint-name\")); _err_ == nil {\n        return \"unit-test\", val\n    }\n    ```\n\n- `failpoint.Value` is the value that passes by `failpoint.Enable(\"failpoint-name\", \"return(5)\")`\nwhich can be ignored.\n\n    ```go\n    failpoint.Inject(\"failpoint-name\", func(_ failpoint.Value) {\n        fmt.Println(\"unit-test\")\n    })\n    ```\n\n    OR\n\n    ```go\n    failpoint.Inject(\"failpoint-name\", func() {\n        fmt.Println(\"unit-test\")\n    })\n    ```\n\n    And the converted code looks like:\n\n    ```go\n    if _, _err_ := failpoint.Eval(_curpkg_(\"failpoint-name\")); _err_ == nil {\n        fmt.Println(\"unit-test\")\n    }\n    ```\n\n- Also, the failpoint closure can be a function which takes `context.Context`. You can\ndo some customized things with `context.Context` like controlling whether a failpoint is\nactive in parallel tests or other cases. For example,\n\n    ```go\n    failpoint.InjectContext(ctx, \"failpoint-name\", func(val failpoint.Value) {\n        fmt.Println(\"unit-test\", val)\n    })\n    ```\n\n    The converted code looks like:\n\n    ```go\n    if val, _err_ := failpoint.EvalContext(ctx, _curpkg_(\"failpoint-name\")); _err_ == nil {\n        fmt.Println(\"unit-test\", val)\n    }\n    ```\n\n- You can ignore `context.Context`, and this will generate the same code as above non-context version. For example,\n\n    ```go\n    failpoint.InjectContext(nil, \"failpoint-name\", func(val failpoint.Value) {\n        fmt.Println(\"unit-test\", val)\n    })\n    ```\n\n    Becomes\n\n    ```go\n    if val, _err_ := failpoint.EvalContext(nil, _curpkg_(\"failpoint-name\")); _err_ == nil {\n        fmt.Println(\"unit-test\", val)\n    }\n    ```\n\n- You can use `failpoint.InjectCall` to inject a function call, this type of marker can only be enabled using `failpoint.EnableCall` and it must be called in the same process as the `InjectCall` call site. Using this marker, you can avoid failpoint code pollute you source code. See [examples](./examples/injectcall/inject_call.go).\n\n- You can control a failpoint by failpoint.WithHook\n\n    ```go\n    func (s *dmlSuite) TestCRUDParallel() {\n        sctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {\n            return ctx.Value(fpname) != nil // Determine by ctx key\n        })\n        insertFailpoints = map[string]struct{} {\n            \"insert-record-fp\": {},\n            \"insert-index-fp\": {},\n            \"on-duplicate-fp\": {},\n        }\n        ictx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {\n            _, found := insertFailpoints[fpname] // Only enables some failpoints.\n            return found\n        })\n        deleteFailpoints = map[string]struct{} {\n            \"tikv-is-busy-fp\": {},\n            \"fetch-tso-timeout\": {},\n        }\n        dctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {\n            _, found := deleteFailpoints[fpname] // Only disables failpoints. \n            return !found\n        })\n        // other DML parallel test cases.\n        s.RunParallel(buildSelectTests(sctx))\n        s.RunParallel(buildInsertTests(ictx))\n        s.RunParallel(buildDeleteTests(dctx))\n    }\n    ```\n\n- If you use a failpoint in the loop context, maybe you will use other marker functions.\n\n    ```go\n    failpoint.Label(\"outer\")\n    for i := 0; i \u003c 100; i++ {\n        inner:\n            for j := 0; j \u003c 1000; j++ {\n                switch rand.Intn(j) + i {\n                case j / 5:\n                    failpoint.Break()\n                case j / 7:\n                    failpoint.Continue(\"outer\")\n                case j / 9:\n                    failpoint.Fallthrough()\n                case j / 10:\n                    failpoint.Goto(\"outer\")\n                default:\n                    failpoint.Inject(\"failpoint-name\", func(val failpoint.Value) {\n                        fmt.Println(\"unit-test\", val.(int))\n                        if val == j/11 {\n                            failpoint.Break(\"inner\")\n                        } else {\n                            failpoint.Goto(\"outer\")\n                        }\n                    })\n            }\n        }\n    }\n    ```\n\n    The above code block will generate the following code:\n\n    ```go\n    outer:\n        for i := 0; i \u003c 100; i++ {\n        inner:\n            for j := 0; j \u003c 1000; j++ {\n                switch rand.Intn(j) + i {\n                case j / 5:\n                    break\n                case j / 7:\n                    continue outer\n                case j / 9:\n                    fallthrough\n                case j / 10:\n                    goto outer\n                default:\n                    if val, _err_ := failpoint.Eval(_curpkg_(\"failpoint-name\")); _err_ == nil {\n                        fmt.Println(\"unit-test\", val.(int))\n                        if val == j/11 {\n                            break inner\n                        } else {\n                            goto outer\n                        }\n                    }\n                }\n            }\n        }\n    ```\n\n- You may doubt why we do not use `label`, `break`, `continue`, and `fallthrough` directly\ninstead of using failpoint marker functions. \n\n    - Any unused symbol like an ident or a label is not permitted in Golang. It will be invalid if some\n    label is only used in the failpoint closure. For example,\n    \n        ```go\n        label1: // compiler error: unused label1\n            failpoint.Inject(\"failpoint-name\", func(val failpoint.Value) {\n                if val.(int) == 1000 {\n                    goto label1 // illegal to use goto here\n                }\n                fmt.Println(\"unit-test\", val)\n            })\n        ```\n\n    - `break` and `continue` can only be used in the loop context, which is not legal in the Golang code \n    if we use them in closure directly.\n\n### Some complicated failpoints demo\n\n- Inject a failpoint to the IF INITIAL statement or CONDITIONAL expression\n\n    ```go\n    if a, b := func() {\n        failpoint.Inject(\"failpoint-name\", func(val failpoint.Value) {\n            fmt.Println(\"unit-test\", val)\n        })\n    }, func() int { return rand.Intn(200) }(); b \u003e func() int {\n        failpoint.Inject(\"failpoint-name\", func(val failpoint.Value) int {\n            return val.(int)\n        })\n        return rand.Intn(3000)\n    }() \u0026\u0026 b \u003c func() int {\n        failpoint.Inject(\"failpoint-name-2\", func(val failpoint.Value) {\n            return rand.Intn(val.(int))\n        })\n        return rand.Intn(6000)\n    }() {\n        a()\n        failpoint.Inject(\"failpoint-name-3\", func(val failpoint.Value) {\n            fmt.Println(\"unit-test\", val)\n        })\n    }\n    ```\n\n    The above code block will generate something like this:\n\n    ```go\n    if a, b := func() {\n        if val, _err_ := failpoint.Eval(_curpkg_(\"failpoint-name\")); _err_ == nil {\n            fmt.Println(\"unit-test\", val)\n        }\n    }, func() int { return rand.Intn(200) }(); b \u003e func() int {\n        if val, _err_ := failpoint.Eval(_curpkg_(\"failpoint-name\")); _err_ == nil {\n            return val.(int)\n        }\n        return rand.Intn(3000)\n    }() \u0026\u0026 b \u003c func() int {\n        if val, ok := failpoint.Eval(_curpkg_(\"failpoint-name-2\")); ok {\n            return rand.Intn(val.(int))\n        }\n        return rand.Intn(6000)\n    }() {\n        a()\n        if val, ok := failpoint.Eval(_curpkg_(\"failpoint-name-3\")); ok {\n            fmt.Println(\"unit-test\", val)\n        }\n    }\n    ```\n\n- Inject a failpoint to the SELECT statement to make it block one CASE if the failpoint is active\n\n    ```go\n    func (s *StoreService) ExecuteStoreTask() {\n        select {\n        case \u003c-func() chan *StoreTask {\n            failpoint.Inject(\"priority-fp\", func(_ failpoint.Value) {\n                return make(chan *StoreTask)\n            })\n            return s.priorityHighCh\n        }():\n            fmt.Println(\"execute high priority task\")\n\n        case \u003c- s.priorityNormalCh:\n            fmt.Println(\"execute normal priority task\")\n\n        case \u003c- s.priorityLowCh:\n            fmt.Println(\"execute normal low task\")\n        }\n    }\n    ```\n\n    The above code block will generate something like this:\n\n    ```go\n    func (s *StoreService) ExecuteStoreTask() {\n        select {\n        case \u003c-func() chan *StoreTask {\n            if _, ok := failpoint.Eval(_curpkg_(\"priority-fp\")); ok {\n                return make(chan *StoreTask)\n            })\n            return s.priorityHighCh\n        }():\n            fmt.Println(\"execute high priority task\")\n\n        case \u003c- s.priorityNormalCh:\n            fmt.Println(\"execute normal priority task\")\n\n        case \u003c- s.priorityLowCh:\n            fmt.Println(\"execute normal low task\")\n        }\n    }\n    ```\n\n- Inject a failpoint to dynamically extend SWITCH CASE arms\n\n    ```go\n    switch opType := operator.Type(); {\n    case opType == \"balance-leader\":\n        fmt.Println(\"create balance leader steps\")\n\n    case opType == \"balance-region\":\n        fmt.Println(\"create balance region steps\")\n\n    case opType == \"scatter-region\":\n        fmt.Println(\"create scatter region steps\")\n\n    case func() bool {\n        failpoint.Inject(\"dynamic-op-type\", func(val failpoint.Value) bool {\n            return strings.Contains(val.(string), opType)\n        })\n        return false\n    }():\n        fmt.Println(\"do something\")\n\n    default:\n        panic(\"unsupported operator type\")\n    }\n    ```\n\n    The above code block will generate something like this:\n\n    ```go\n    switch opType := operator.Type(); {\n    case opType == \"balance-leader\":\n        fmt.Println(\"create balance leader steps\")\n\n    case opType == \"balance-region\":\n        fmt.Println(\"create balance region steps\")\n\n    case opType == \"scatter-region\":\n        fmt.Println(\"create scatter region steps\")\n\n    case func() bool {\n        if val, ok := failpoint.Eval(_curpkg_(\"dynamic-op-type\")); ok {\n            return strings.Contains(val.(string), opType)\n        }\n        return false\n    }():\n        fmt.Println(\"do something\")\n\n    default:\n        panic(\"unsupported operator type\")\n    }\n    ```\n\n- More complicated failpoints\n\n    - There are more complicated failpoint sites that can be injected to\n        - for the loop INITIAL statement, CONDITIONAL expression and POST statement\n        - for the RANGE statement\n        - SWITCH INITIAL statement\n        - …\n    - Anywhere you can call a function\n\n## Failpoint name best practice\n\nAs you see above, `_curpkg_` will automatically wrap the original failpoint name in `failpoint.Eval` call.\nYou can think of `_curpkg_` as a macro that automatically prepends the current package path to the failpoint name. For example,\n\n```go\npackage ddl // which parent package is `github.com/pingcap/tidb`\n\nfunc demo() {\n\t// _curpkg_(\"the-original-failpoint-name\") will be expanded as `github.com/pingcap/tidb/ddl/the-original-failpoint-name`\n\tif val, ok := failpoint.Eval(_curpkg_(\"the-original-failpoint-name\")); ok {...}\n}\n```\n\nYou do not need to care about `_curpkg_` in your application. It is automatically generated after running `failpoint-ctl enable`\nand is deleted with `failpoint-ctl disable`.\n\nBecause all failpoints in a package share the same namespace, we need to be careful to\navoid name conflict. There are some recommended naming rules to improve this situation.\n\n- Keep name unique in current subpackage\n- Use a self-explanatory name for the failpoint\n    \n    You can enable failpoints by environment variables\n    ```shell\n    GO_FAILPOINTS=\"github.com/pingcap/tidb/ddl/renameTableErr=return(100);github.com/pingcap/tidb/planner/core/illegalPushDown=return(true);github.com/pingcap/pd/server/schedulers/balanceLeaderFailed=return(true)\"\n    ```\n    \n## Implementation details\n\n1. Define a group of marker functions\n2. Parse imports and prune a source file which does not import a failpoint\n3. Traverse AST to find marker function calls\n4. Marker function calls will be rewritten with an IF statement, which calls `failpoint.Eval` to determine whether a\nfailpoint is active and executes failpoint code if the failpoint is enabled\n\n![rewrite-demo](./media/rewrite-demo.png)\n\n## Acknowledgments\n\n- Thanks [gofail](https://github.com/etcd-io/gofail) to provide initial implementation.\n","funding_links":[],"categories":["测试","Testing","Go","测试相关","Template Engines","Selenium and browser control tools","测试相关`测试库和测试数据集生成库`","Fail injection"],"sub_categories":["HTTP客户端","Fail injection","HTTP Clients","查询语"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpingcap%2Ffailpoint","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpingcap%2Ffailpoint","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpingcap%2Ffailpoint/lists"}