checks that 95% of entire table satisfy multiple conditions over different columns

[baljijit]

checks that 95% of entire table satisfy multiple conditions over different columns

[SirWerto]

Hello @baljijit

You can define 2 checks with a threshold of 95% each one

val verificationResult = VerificationSuite()
  .onData(data)
  .addCheck(
    Check(CheckLevel.Error, "unit testing my data")
    .haveCompleteness(Array(date_x, date_y), _ >= 0.95)
    .isContainedIn("typology", Array("aaaa", "bbbb"), _ >= 0.95))
  .run()

moreover, it does not verify that 95% of the rows fulfill the 2 conditions , for that you will need to create a column on your own and apply a check to it.
E.g.

df.withColumn("combined_checks", 'date_x.notNull && 'date_y.notNull && 'typology.contains("aaaa"))

I don't really know if you can define assertions based on a combination of other assertions.