pingcap/failpoint

An implementation of failpoints for Golang.

failpoint

An implementation of failpoints for Golang. Fail points are used to add code points where errors may be injected in a user controlled fashion. Fail point is a code snippet that is only executed when the corresponding failpoint is active.

Quick Start (use failpoint-ctl)

1. Build failpoint-ctl from source

   git clone https://github.com/pingcap/failpoint.git
   cd failpoint
   make
   ls bin/failpoint-ctl

2. Inject failpoints to your program, eg:

   package main
   
   import "github.com/pingcap/failpoint"
   
   func main() {
       failpoint.Inject("testPanic", func() {
           panic("failpoint triggerd")
       })
   }

3. Transfrom your code with failpoint-ctl enable

4. Build with go build

5. Enable failpoints with GO_FAILPOINTS environment variable

   GO_FAILPOINTS="main/testPanic=return(true)" ./your-program

6. If you use go run to run the test, don't forget to add the generated binding__failpoint_binding__.go in your command, like:

   GO_FAILPOINTS="main/testPanic=return(true)" go run your-program.go binding__failpoint_binding__.go

Quick Start (use failpoint-toolexec)

1. Build failpoint-toolexec from source

   git clone https://github.com/pingcap/failpoint.git
   cd failpoint
   make
   ls bin/failpoint-toolexec

2. Inject failpoints to your program, eg:

   package main
   
   import "github.com/pingcap/failpoint"
   
   func main() {
       failpoint.Inject("testPanic", func() {
           panic("failpoint triggerd")
       })
   }

3. Use a separate build cache to avoid mixing caches without failpoint-toolexec, and build

   GOCACHE=/tmp/failpoint-cache go build -toolexec path/to/failpoint-toolexec

4. Enable failpoints with GO_FAILPOINTS environment variable

   GO_FAILPOINTS="main/testPanic=return(true)" ./your-program

5. You can also use go run or go test, like:

   GOCACHE=/tmp/failpoint-cache GO_FAILPOINTS="main/testPanic=return(true)" go run -toolexec path/to/failpoint-toolexec your-program.go

Design principles

• Define failpoint in valid Golang code, not comments or anything else

• Failpoint does not have any extra cost

  • Will not take effect on regular logic
  • Will not cause regular code performance regression
  • Failpoint code will not appear in the final binary

• Failpoint routine is writable/readable and should be checked by a compiler

• Generated code by failpoint definition is easy to read

• Keep the line numbers same with the injecting codes(easier to debug)

• Support parallel tests with context.Context

Key concepts

• Failpoint

  Faillpoint is a code snippet that is only executed when the corresponding failpoint is active. The closure will never be executed if failpoint.Disable("failpoint-name-for-demo") is executed.

  var outerVar = "declare in outer scope"
  failpoint.Inject("failpoint-name-for-demo", func(val failpoint.Value) {
      fmt.Println("unit-test", val, outerVar)
  })

• Marker functions

  • It is just an empty function

    • To hint the rewriter to rewrite with an equality statement
    • To receive some parameters as the rewrite rule
    • It will be inline in the compiling time and emit nothing to binary (zero cost)
    • The variables in external scope can be accessed in closure by capturing, and the converted code is still legal because all the captured-variables location in outer scope of IF statement.

  • It is easy to write/read

  • Introduce a compiler check for failpoints which cannot compile in the regular mode if failpoint code is invalid

• Marker funtion list

  • func Inject(fpname string, fpblock func(val Value)) {}
  • func InjectContext(fpname string, ctx context.Context, fpblock func(val Value)) {}
  • func Break(label ...string) {}
  • func Goto(label string) {}
  • func Continue(label ...string) {}
  • func Fallthrough() {}
  • func Return(results ...interface{}) {}
  • func Label(label string) {}

• Supported failpoint environment variable

  failpoint can be enabled by export environment variables with the following patten, which is quite similar to freebsd failpoint SYSCTL VARIABLES

  [<percent>%][<count>*]<type>[(args...)][-><more terms>]

  The argument specifies which action to take; it can be one of:

  • off: Take no action (does not trigger failpoint code)
  • return: Trigger failpoint with specified argument
  • sleep: Sleep the specified number of milliseconds
  • panic: Panic
  • break: Execute gdb and break into debugger
  • print: Print failpoint path for inject variable
  • pause: Pause will pause until the failpoint is disabled

How to inject a failpoint to your program

• You can call failpoint.Inject to inject a failpoint to the call site, where failpoint-name is used to trigger the failpoint and failpoint-closure will be expanded as the body of the IF statement.

  failpoint.Inject("failpoint-name", func(val failpoint.Value) {
      failpoint.Return("unit-test", val)
  })

  The converted code looks like:

  if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
      return "unit-test", val
  }

• failpoint.Value is the value that passes by failpoint.Enable("failpoint-name", "return(5)") which can be ignored.

  failpoint.Inject("failpoint-name", func(_ failpoint.Value) {
      fmt.Println("unit-test")
  })

  OR

  failpoint.Inject("failpoint-name", func() {
      fmt.Println("unit-test")
  })

  And the converted code looks like:

  if _, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
      fmt.Println("unit-test")
  }

• Also, the failpoint closure can be a function which takes context.Context. You can do some customized things with context.Context like controlling whether a failpoint is active in parallel tests or other cases. For example,

  failpoint.InjectContext(ctx, "failpoint-name", func(val failpoint.Value) {
      fmt.Println("unit-test", val)
  })

  The converted code looks like:

  if val, _err_ := failpoint.EvalContext(ctx, _curpkg_("failpoint-name")); _err_ == nil {
      fmt.Println("unit-test", val)
  }

• You can ignore context.Context, and this will generate the same code as above non-context version. For example,

  failpoint.InjectContext(nil, "failpoint-name", func(val failpoint.Value) {
      fmt.Println("unit-test", val)
  })

  Becomes

  if val, _err_ := failpoint.EvalContext(nil, _curpkg_("failpoint-name")); _err_ == nil {
      fmt.Println("unit-test", val)
  }

• You can control a failpoint by failpoint.WithHook

  func (s *dmlSuite) TestCRUDParallel() {
      sctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
          return ctx.Value(fpname) != nil // Determine by ctx key
      })
      insertFailpoints = map[string]struct{} {
          "insert-record-fp": {},
          "insert-index-fp": {},
          "on-duplicate-fp": {},
      }
      ictx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
          _, found := insertFailpoints[fpname] // Only enables some failpoints.
          return found
      })
      deleteFailpoints = map[string]struct{} {
          "tikv-is-busy-fp": {},
          "fetch-tso-timeout": {},
      }
      dctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
          _, found := deleteFailpoints[fpname] // Only disables failpoints. 
          return !found
      })
      // other DML parallel test cases.
      s.RunParallel(buildSelectTests(sctx))
      s.RunParallel(buildInsertTests(ictx))
      s.RunParallel(buildDeleteTests(dctx))
  }

• If you use a failpoint in the loop context, maybe you will use other marker functions.

  failpoint.Label("outer")
  for i := 0; i < 100; i++ {
      inner:
          for j := 0; j < 1000; j++ {
              switch rand.Intn(j) + i {
              case j / 5:
                  failpoint.Break()
              case j / 7:
                  failpoint.Continue("outer")
              case j / 9:
                  failpoint.Fallthrough()
              case j / 10:
                  failpoint.Goto("outer")
              default:
                  failpoint.Inject("failpoint-name", func(val failpoint.Value) {
                      fmt.Println("unit-test", val.(int))
                      if val == j/11 {
                          failpoint.Break("inner")
                      } else {
                          failpoint.Goto("outer")
                      }
                  })
          }
      }
  }

  The above code block will generate the following code:

  outer:
      for i := 0; i < 100; i++ {
      inner:
          for j := 0; j < 1000; j++ {
              switch rand.Intn(j) + i {
              case j / 5:
                  break
              case j / 7:
                  continue outer
              case j / 9:
                  fallthrough
              case j / 10:
                  goto outer
              default:
                  if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
                      fmt.Println("unit-test", val.(int))
                      if val == j/11 {
                          break inner
                      } else {
                          goto outer
                      }
                  }
              }
          }
      }

• You may doubt why we do not use label, break, continue, and fallthrough directly instead of using failpoint marker functions.

  • Any unused symbol like an ident or a label is not permitted in Golang. It will be invalid if some label is only used in the failpoint closure. For example,

    label1: // compiler error: unused label1
        failpoint.Inject("failpoint-name", func(val failpoint.Value) {
            if val.(int) == 1000 {
                goto label1 // illegal to use goto here
            }
            fmt.Println("unit-test", val)
        })

  • break and continue can only be used in the loop context, which is not legal in the Golang code if we use them in closure directly.

Some complicated failpoints demo

• Inject a failpoint to the IF INITIAL statement or CONDITIONAL expression

  if a, b := func() {
      failpoint.Inject("failpoint-name", func(val failpoint.Value) {
          fmt.Println("unit-test", val)
      })
  }, func() int { return rand.Intn(200) }(); b > func() int {
      failpoint.Inject("failpoint-name", func(val failpoint.Value) int {
          return val.(int)
      })
      return rand.Intn(3000)
  }() && b < func() int {
      failpoint.Inject("failpoint-name-2", func(val failpoint.Value) {
          return rand.Intn(val.(int))
      })
      return rand.Intn(6000)
  }() {
      a()
      failpoint.Inject("failpoint-name-3", func(val failpoint.Value) {
          fmt.Println("unit-test", val)
      })
  }

  The above code block will generate something like this:

  if a, b := func() {
      if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
          fmt.Println("unit-test", val)
      }
  }, func() int { return rand.Intn(200) }(); b > func() int {
      if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
          return val.(int)
      }
      return rand.Intn(3000)
  }() && b < func() int {
      if val, ok := failpoint.Eval(_curpkg_("failpoint-name-2")); ok {
          return rand.Intn(val.(int))
      }
      return rand.Intn(6000)
  }() {
      a()
      if val, ok := failpoint.Eval(_curpkg_("failpoint-name-3")); ok {
          fmt.Println("unit-test", val)
      }
  }

• Inject a failpoint to the SELECT statement to make it block one CASE if the failpoint is active

  func (s *StoreService) ExecuteStoreTask() {
      select {
      case <-func() chan *StoreTask {
          failpoint.Inject("priority-fp", func(_ failpoint.Value) {
              return make(chan *StoreTask)
          })
          return s.priorityHighCh
      }():
          fmt.Println("execute high priority task")
  
      case <- s.priorityNormalCh:
          fmt.Println("execute normal priority task")
  
      case <- s.priorityLowCh:
          fmt.Println("execute normal low task")
      }
  }

  The above code block will generate something like this:

  func (s *StoreService) ExecuteStoreTask() {
      select {
      case <-func() chan *StoreTask {
          if _, ok := failpoint.Eval(_curpkg_("priority-fp")); ok {
              return make(chan *StoreTask)
          })
          return s.priorityHighCh
      }():
          fmt.Println("execute high priority task")
  
      case <- s.priorityNormalCh:
          fmt.Println("execute normal priority task")
  
      case <- s.priorityLowCh:
          fmt.Println("execute normal low task")
      }
  }

• Inject a failpoint to dynamically extend SWITCH CASE arms

  switch opType := operator.Type(); {
  case opType == "balance-leader":
      fmt.Println("create balance leader steps")
  
  case opType == "balance-region":
      fmt.Println("create balance region steps")
  
  case opType == "scatter-region":
      fmt.Println("create scatter region steps")
  
  case func() bool {
      failpoint.Inject("dynamic-op-type", func(val failpoint.Value) bool {
          return strings.Contains(val.(string), opType)
      })
      return false
  }():
      fmt.Println("do something")
  
  default:
      panic("unsupported operator type")
  }

  The above code block will generate something like this:

  switch opType := operator.Type(); {
  case opType == "balance-leader":
      fmt.Println("create balance leader steps")
  
  case opType == "balance-region":
      fmt.Println("create balance region steps")
  
  case opType == "scatter-region":
      fmt.Println("create scatter region steps")
  
  case func() bool {
      if val, ok := failpoint.Eval(_curpkg_("dynamic-op-type")); ok {
          return strings.Contains(val.(string), opType)
      }
      return false
  }():
      fmt.Println("do something")
  
  default:
      panic("unsupported operator type")
  }

• More complicated failpoints

  • There are more complicated failpoint sites that can be injected to
    • for the loop INITIAL statement, CONDITIONAL expression and POST statement
    • for the RANGE statement
    • SWITCH INITIAL statement
    • …
  • Anywhere you can call a function

Failpoint name best practice

As you see above, _curpkg_ will automatically wrap the original failpoint name in failpoint.Eval call. You can think of _curpkg_ as a macro that automatically prepends the current package path to the failpoint name. For example,

package ddl // which parent package is `github.com/pingcap/tidb`

func demo() {
	// _curpkg_("the-original-failpoint-name") will be expanded as `github.com/pingcap/tidb/ddl/the-original-failpoint-name`
	if val, ok := failpoint.Eval(_curpkg_("the-original-failpoint-name")); ok {...}
}

You do not need to care about _curpkg_ in your application. It is automatically generated after running failpoint-ctl enable and is deleted with failpoint-ctl disable.

Because all failpoints in a package share the same namespace, we need to be careful to avoid name conflict. There are some recommended naming rules to improve this situation.

• Keep name unique in current subpackage

• Use a self-explanatory name for the failpoint

  You can enable failpoints by environment variables

  GO_FAILPOINTS="github.com/pingcap/tidb/ddl/renameTableErr=return(100);github.com/pingcap/tidb/planner/core/illegalPushDown=return(true);github.com/pingcap/pd/server/schedulers/balanceLeaderFailed=return(true)"

Implementation details

1. Define a group of marker functions
2. Parse imports and prune a source file which does not import a failpoint
3. Traverse AST to find marker function calls
4. Marker function calls will be rewritten with an IF statement, which calls failpoint.Eval to determine whether a failpoint is active and executes failpoint code if the failpoint is enabled

Acknowledgments

• Thanks gofail to provide initial implementation.

failpoint's Issues

Is there any demo or userguide for failpoint?

I cannot have any idea that how to use failpoint. I tried write demo like this:

 var outerVar = "declare in outer scope"

err := failpoint.Enable("failpoint-name", "return")
if err != nil {
	fmt.Println(err)
	return
}

failpoint.Inject("failpoint-name", func(val failpoint.Value) {
	fmt.Println("unit-test", val, outerVar)
})

There isn't have any output. How could I use it in the right way?

failpoint doesn't work if the marker function is defined in the main package

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

For example, we have a file named main.go:

package main

import (

    "github.com/pingcap/failpoint"
)

func F1(name string) string {

    failpoint.Inject("failpoint-name", nil)

    return "hello " + name
}

func main() {
    failpoint.Enable("failpoint-name", "panic")
    F1("ttt")
}

After transfrom the code with failpoint-ctl enable, the generated binding__failpoint_binding__.go looks like this:

package main

import "reflect"

type __failpointBindingType struct {pkgpath string}
var __failpointBindingCache = &__failpointBindingType{}

func init() {
    __failpointBindingCache.pkgpath = reflect.TypeOf(__failpointBindingType{}).PkgPath()
}
func _curpkg_(name string) string {
    return  __failpointBindingCache.pkgpath + "/" + name
}

You may notice that binding__failpoint_binding__.go is also in package main, that means binding__failpoint_binding__.go could not be found by main.go.

Run go run main.go then, you will find the following error:

 ./main.go:8:17: undefined: _curpkg_

2. What did you expect to see?

program panics due to failpoint

3. What did you see instead?

 ./main.go:8:17: undefined: _curpkg_

4. Versions of the failpoint

$ ./bin/failpoint-ctl -V
ReleaseVersion bf45ab2
BuildTS 2019-05-15 09:48:44
GitHash bf45ab2
GitBranch master
GoVersion go version go1.12 darwin/amd64

Why do we call `fmt.Println` in `failpoints.Eval`

failpoint/failpoints.go

Lines 272 to 278 in 7acb0f0

	func Eval(failpath string) (Value, error) {
	val, err := failpoints.Eval(failpath)
	if err, ok := errors.Cause(err).(FpError); !ok && err != nil {
	fmt.Println(err)
	}
	return val, err
	}

There seems no reason we do it. Either return an error or log at a proper level is reasonable.

Failpoint doesn't work when Inject function defined in a closure argument

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

	f := func(g func() error) {}
	f(func() error {
		failpoint.Inject("XXX", func() {
			failpoint.Return(errors.New("mock error"))
		})
		return nil
	})

2. What did you expect to see?
   After enabled failpoint,failpoint.Inject can be changed to failpoint.Eval.

3. What did you see instead?

Nothing changed

4. Versions of the failpoint

   • failpoint-ctl version (run failpoint-ctl -V):
     The version comes from tidb's go.mod

     v0.0.0-20200702092429-9f69995143ce
     

failpoint.Disable should be idempotent

For example, I may register a defer block to disable failpoint to ensure that it's triggered no matter intermediate error occurs, while I'm going to disable the failpoint on the fly of the code flow early to switch behavior.

Then, one failpoint can be disabled twice and with current implementation, this causes failpoint: failpoint is disabled error.

func foo() {
    failpoint.Enable(...)
    defer failpoint.Disable(...)
    // ... error may occur in this block, thus I need a defer block above
    failpoint.Disable(...)
    // failpoint has been disabled, but it will be disable again when executing the defer block
}

I'd like to make failpoint.Disable idempotent.

What do you think @lonng @kennytm?

type assert OK print error message

 // mockStmtCommitError
 // gofail: var mockStmtCommitError bool
 // if mockStmtCommitError && mockStmtCommitErrorOnce {
 //  mockStmtCommitErrorOnce = false
 //  return kv.ErrRetryable
 // }

gofail.Enable("github.com/pingcap/tidb/session/mockStmtCommitError", return(true))

第二次进来的时候会打印错误信息

Why the type of failpoint.Value can't be string

the definition in the source code about Value's type in "github.com/pingcap/failpoint/failpoint.go"

	// Value represents value that retrieved from failpoint terms.
	// It can be used as following types:
	// 1. val.(int)      // GO_FAILPOINTS="failpoint-name=return(1)"
	// 2. val.(string)   // GO_FAILPOINTS="failpoint-name=return('1')"
	// 3. val.(bool)     // GO_FAILPOINTS="failpoint-name=return(true)"
	Value interface{}

but in my code

failpoint.Inject("demoPanic", func(val failpoint.Value) {
		fmt.Println(val)
		fmt.Println(val.(string))
}

and start my progrom with cmd:
GO_FAILPOINTS="mypackagename/demoPanic=return('100')" ./beego-backend
An exception was thrown
[panic.go:522] Handler crashed with error interface conversion: failpoint.Value is bool, not string ···

How to inject faults to a program at running time with failpoint

Question

The REDAME file teaches me how to use failpoint when staring a program.

However, it does not tell about injecting faults to a program at running time.

For example,

First I start a go program :

./test

After 5 mins, I set the ENV GO_FAILPOINTS="main/testPanic=return(true)"

The fault cannot be injected to the program.

How can I inject faults to a program after running the progarm?

Create Release Binary amd and Darwin

Feature Request

Is your feature request related to a problem? Please describe:

Yes

Describe the feature you'd like:

Release tags and binary for failpoint.

Describe alternatives you've considered:

Create git tags and release binary for both linux and mac builds? Its easier for download the builds directly from github and include in CI/CD pipelines as apposed to clone the entire repo and make builds.

Teachability, Documentation, Adoption, Optimization:

Users wanting to include failopint in their build pipelines , could directly curl/wget the tar files from github and use failpoint-ctl enable in their repository. Its much simpler that way , instead of cloning the entire repo and making builds manually.

failpoint can not use a constant-string as marker function name

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   failpoint fail to rewrite the following circumstance:

package main

import (

    "github.com/pingcap/failpoint"
)
const (
   Marker = "failpoint-name"
)
func F1(name string) string {

    failpoint.Inject(Marker, nil)

    return "hello " + name
}

2. What did you expect to see?
   This file can be correctly rewritten by failpoint-ctl tool

3. What did you see instead?
   rewrite failed.

/bin/sh: Rewrite: command not found
make: *** [failpoint-enable] Error 127

4. Versions of the failpoint

   • failpoint-ctl version (run failpoint-ctl -V):

ReleaseVersion bf45ab2
BuildTS 2019-05-15 09:48:44
GitHash bf45ab2
GitBranch master
GoVersion go version go1.12 darwin/amd64

Add a "DO NOT EDIT" comment to failpoint-enabled source code

Feature Request

Is your feature request related to a problem? Please describe:

Although failpoint-ctl disable will automatically apply diff when the original source is changed, this doesn't always work, and it would be frustrating to be stuck in a state where failpoint-ctl disable cannot proceed and you've got some changes hard to be extracted.

Most editors supporting Go recognize the generated source indicator and warns when you try to edit the file, thus preventing the need to merge diffs in the first place.

Describe the feature you'd like:

In the source code generated by failpoint-ctl enable (all of the rewritten *.go, *__failpoint_binding__.go, and *.go__failpoint_stash__), insert this comment on first line:

// Code generated by failpoint DO NOT EDIT.

Remove this line after failpoint-ctl disable.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Optimization:

go test failed

Bug Report

go get github.com/pingcap/failpoint/
cd failpoint
go test .

/V/d/g/s/g/p/failpoint (master|✔) [2] $ go test .
failed to parse "invalid" past "invalid"
failed to enable "failpoint-name=invalid" (failpoint: could not parse terms)

----------------------------------------------------------------------
FAIL: http_test.go:47: httpSuite.TestServeHTTP

http_test.go:124:
    c.Assert(err, IsNil)
... value *url.Error = &url.Error{Op:"Get", URL:"http://127.0.0.1:23389/failpoint-env1", Err:(*net.OpError)(0xc000172050)} ("Get http://127.0.0.1:23389/failpoint-env1: dial tcp 127.0.0.1:23389: connect: connection refused")

failed to parse "invalid" past "invalid"
failed to enable "runtime-test-2=invalid" (failpoint: could not parse terms)

----------------------------------------------------------------------
FAIL: runtime_test.go:21: runtimeSuite.TestRuntime

runtime_test.go:133:
    c.Assert(ok, IsTrue)
... obtained bool = false

OOPS: 1 passed, 2 FAILED
--- FAIL: TestFailpoint (1.00s)
failed to parse "invalid" past "invalid"
failed to enable "failpoint-name=invalid" (failpoint: could not parse terms)

----------------------------------------------------------------------
FAIL: http_test.go:47: httpSuite.TestServeHTTP

http_test.go:124:
    c.Assert(err, IsNil)
... value *url.Error = &url.Error{Op:"Get", URL:"http://127.0.0.1:23389/failpoint-env1", Err:(*net.OpError)(0xc000172d70)} ("Get http://127.0.0.1:23389/failpoint-env1: dial tcp 127.0.0.1:23389: connect: connection refused")

failed to parse "invalid" past "invalid"
failed to enable "runtime-test-2=invalid" (failpoint: could not parse terms)

----------------------------------------------------------------------
FAIL: runtime_test.go:21: runtimeSuite.TestRuntime

runtime_test.go:133:
    c.Assert(ok, IsTrue)
... obtained bool = false

OOPS: 1 passed, 2 FAILED
--- FAIL: TestHttp (1.00s)
failed to parse "invalid" past "invalid"
failed to enable "failpoint-name=invalid" (failpoint: could not parse terms)

----------------------------------------------------------------------
FAIL: http_test.go:47: httpSuite.TestServeHTTP

http_test.go:124:
    c.Assert(err, IsNil)
... value *url.Error = &url.Error{Op:"Get", URL:"http://127.0.0.1:23389/failpoint-env1", Err:(*net.OpError)(0xc000146500)} ("Get http://127.0.0.1:23389/failpoint-env1: dial tcp 127.0.0.1:23389: connect: connection refused")

failed to parse "invalid" past "invalid"
failed to enable "runtime-test-2=invalid" (failpoint: could not parse terms)

----------------------------------------------------------------------
FAIL: runtime_test.go:21: runtimeSuite.TestRuntime

runtime_test.go:133:
    c.Assert(ok, IsTrue)
... obtained bool = false

OOPS: 1 passed, 2 FAILED
--- FAIL: TestNewRestorer (1.01s)
FAIL
FAIL	github.com/pingcap/failpoint	3.030s
/V/d/g/s/g/p/failpoint (master|✔) [1] $

failpoint can be actived if it is injected with in a function clusure

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   demo code:

f := func(g func() error) {}
f(func() error {
  failpoint.Inject("with-in-a-clusure", func() {
    failpoint.Return(errors.New("mock err"))
  })
  return nil
})

exec failpoint-ctl enable, the failpoint.Inject remains unchanged.

2. What did you expect to see?

3. What did you see instead?

4. Versions of the failpoint

Failpoints are getting executed while running tests using `go test -v`

Bug Report

1. What did you do? If possible, provide a recipe for reproducing the error.

I have the following setup (all in the same directory):
go.mod

module example.com/modv1

go 1.20

require (
	github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c
	github.com/stretchr/testify v1.8.4
)

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/pingcap/errors v0.11.4 // indirect
	github.com/pmezard/go-difflib v1.0.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
)

samplestruct.go

package main

import (
	"log"

	"github.com/pingcap/failpoint"
)

type SampleStruct struct {
	cnt int
}

func (s *SampleStruct) increment() {
	if _, _err_ := failpoint.Eval(_curpkg_("testPanic")); _err_ == nil {
		log.Printf("[SampleStruct] SUCCESS!!! Executed testPanic FP: The count for s is %v", s.cnt)
	}
	s.cnt++
}

func (s *SampleStruct) decrement() {
	s.cnt--
}

func (s *SampleStruct) incrementFP() string {
	log.Printf("[SampleStruct] Executing incrementFP with old cnt %v", s.cnt)
	s.cnt++
	log.Printf("[SampleStruct] Executing incrementFP with new cnt %v", s.cnt)
	if _, _err_ := failpoint.Eval(_curpkg_("doubleIncrement")); _err_ == nil {
		s.cnt++
		log.Printf("[SampleStruct] SUCCESS!!! Executed doubleIncrement FP. The count for s is %v", s.cnt)
	}
	return "single"
}

main.go

package main

import (
	"fmt"
	"log"
	"os"
)

func main() {
	log.Printf(fmt.Sprintf("[main] This is main function being executed in an executable at path %v", os.Args[0]))
	//os.Setenv("Go_FAILPOINTS", "main/doubleIncrement=return(true);main/testPanic=return(true)")
	log.Printf(fmt.Sprintf("[main] After FP code in main function"))

	sampleStruct := SampleStruct{cnt: 1}
	log.Printf("[main] sampleStruct cnt before increment is %v", sampleStruct.cnt)
	sampleStruct.increment()
	log.Printf("[main] sampleStruct cnt after increment is %v", sampleStruct.cnt)
	log.Printf("[main] sampleStruct cnt before double increment is %v", sampleStruct.cnt)
	sampleStruct.incrementFP()
	log.Printf("[main] sampleStruct cnt after double increment is %v", sampleStruct.cnt)
}

sample_test.go

package main

import (
	"log"
	"testing"
	"os/exec"
	"os"
	"fmt"
	"bytes"
	"github.com/stretchr/testify/assert"
)

func TestFailpoint(t *testing.T) {
	log.Printf("[TestFailpoint] Executing TestFailpoint")
	mySampleStruct := SampleStruct{cnt: 2}
	assert.Equal(t, mySampleStruct.cnt, 2)

	// fpsToEnable := map[string]struct{}{
	// 	"doubleIncrement": {},
	// }

	// cmd := exec.Command("export", "GO_FAILPOINTS='main/doubleIncrement=return(true);main/testPanic=return(true)'")
	// if err := cmd.Run(); err != nil {
	// 	log.Fatal(fmt.Sprintf("Encountered error while setting up the GO_FAILPOINTS environment variable. Error: %v", err))
	// } else {
	// 	log.Printf("Successfully set up the GO_FAILPOINTS environment variable")
	// }

	os.Setenv("GO_FAILPOINTS","'main/doubleIncrement=return(true);main/testPanic=return(true)'")
	log.Printf("[TestFailpoint] Successfully set up the GO_FAILPOINTS environment variable")

	cmd := exec.Command("./modv1")
	var out bytes.Buffer
	var stderr bytes.Buffer
	cmd.Stdout = &out
	cmd.Stderr = &stderr
	if err := cmd.Start(); err != nil {
		log.Printf(fmt.Sprintf("[TestFailpoint] Error while starting modv1 executable. Error: %v", err))
		log.Printf("[TestFailpoint] Command error is: " + stderr.String())
	} else {
		log.Printf("[TestFailpoint] Successfully started modv1 executable")
		log.Printf("[TestFailpoint] Result: " + out.String() + ",Error: " + stderr.String())
	}

	if err := cmd.Wait(); err != nil {
		log.Printf(fmt.Sprintf("[TestFailpoint] Error while waiting for modv1 executable. Error: %v", err))
		log.Printf("[TestFailpoint] Command error is: " + stderr.String())
	} else {
		log.Printf("[TestFailpoint] Successfully waited for modv1 executable")
		log.Printf("[TestFailpoint] Result: " + out.String() + ",Error: " + stderr.String())
	}

	//failpoint.Enable("doubleIncrement", "return(1)")
	log.Printf("[TestFailpoint] Calling increment")
	mySampleStruct.increment()
	log.Printf("[TestFailpoint] Called increment")
	log.Printf("[TestFailpoint] Calling incrementFP")
	result := mySampleStruct.incrementFP()
	log.Printf("[TestFailpoint] Called incrementFP")
	assert.Equal(t, mySampleStruct.cnt, 4)
	assert.Equal(t, result, "single")

	os.Unsetenv("GO_FAILPOINTS")
	log.Printf("[TestFailpoint] Successfully unset the GO_FAILPOINTS environment variable")
}

2. What did you expect to see?
   When I run go build and set the environment variable export GO_FAILPOINTS=main/doubleIncrement=return(true);main/testPanic=return(true) and then execute the ./modv1 executable I see that the failpoints in samplestruct.go are hit and the failpoint closure is executed. I expected same behavior when I run go test -v i.e. when the TestFailpoint test is executed.

3. What did you see instead?
   I did not see the failpoints in samplestruct.go getting executed while running the test TestFalipoint. Is this expected behavior? Is the failpoint code executed only in the compiled binary modv1 and not during test execution even though the environment variable GO_FAILPOINTS is kept around (and additionally also set/unset during) test execution?

4. Versions of the failpoint

$ ./failpoint-ctl -V
ReleaseVersion ed9079f
BuildTS 2023-09-01 02:53:58
GitHash ed9079f47f3761cbbeadaded49d48e3264c23427
GitBranch fp-testing
GoVersion go version go1.21.0 darwin/arm64

failpoint in anonymous function can not be enable

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

package testpkg

import (
	"fmt"
	
	"github.com/pingcap/failpoint"
)

func test() {
	func() {
		failpoint.Inject("testPoint", func() {
			fmt.Println()
		})
	}()
}

then failpoint-ctl enable~

2. What did you expect to see?

func test() {
	func() {
		if _, ok := failpoint.Eval(_curpkg_("testPoint")); ok {
			fmt.Println()
		}
	}()
}

3. What did you see instead?

func test() {
	func() {
		failpoint.Inject("testPoint", func() {
			fmt.Println()
		})
	}()
}

4. Versions of the failpoint

failpoint-ctl version (run failpoint-ctl -V):

ReleaseVersion None
BuildTS None
GitHash None
GitBranch None
GoVersion None

newest tidb master 

Failpoints.Disable not remove failpath from fps.reg

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   I'm trying to iterate over failpoint.List() to obtain all the enabled failpoint keys. However, I'm also retrieving deprecated keys from the list.
2. What did you expect to see?
   After calling failpoint.disable(key), the key should no longer be present in fps.reg
3. What did you see instead?
   Now, even after I disable a failpoint key, I can still retrieve it once it has been enabled.4. Versions of the failpoint

   • failpoint-ctl version (run failpoint-ctl -V):

     (paste failpoint-ctl version here)
     ReleaseVersion None
     BuildTS None
     GitHash None
     GitBranch None
     GoVersion None
     

Register an action func to be called when a failpoint is evaluated

Feature Request

Is your feature request related to a problem? Please describe:
When writing a unit test, I want to know if a failpoint is evaluated and the evaluated value.

Describe the feature you'd like:
When enabling a failpoint, pass in a function callback func(val interface{}). Afterwards, whenever the failpoint is evaluated, failpoint is responsible to call callback.

Describe alternatives you've considered:
Allow user to register a function to replace term. Every time the failpoint is evaluated, it calls the function to get the value.

Teachability, Documentation, Adoption, Optimization:

failpoint not activated

I write this in foo.go:

func pingCapFail() (string, failpoint.Value) {
	failpoint.Inject("failpoint-name", func(val failpoint.Value) {
		failpoint.Return("unit-test", val)
	})
	return "success", nil
}

and i run failpoint-ctl enable , code becomes this;

func pingCapFail() (string, failpoint.Value) {
	if val, ok := failpoint.Eval(_curpkg_("failpoint-name")); ok {
		return "unit-test", val
	}
	return "success", nil
}

then in foo_test.go ,i write my test code like this:

func Test_pingCapFail(t *testing.T) {
	failpoint.Enable("failpoint-name", "return(5)")
	got, got1 := pingCapFail()
	if got != "unit-test" {
		t.Errorf("pingCapFail() got = %v, want %v", got, "unit-test")
	}
	if !reflect.DeepEqual(got1, 5) {
		t.Errorf("pingCapFail() got1 = %v, want %v", got1, 5)
	}
}

and it fails, and obviously the failpoint is not activated, can someone tell me why?