named_regex_patterns should actually do something.

[jpcampb2]

named_regex_patterns should actually do something.

[jpcampb2]

@abegong: Looking at this with some mixed feelings.

I just created a PR to do it as it stands now, but my concern is that those regexs might get too closely identified with GE and/or be hard to version--certainly many of them could be better than they are now, and we'll want to be able to improve that without breaking peoples' expectations later on.

Adding a GE version to an expectation config may address that.

[abegong]

Instead of including named regexes as first-class citizens, we're going to make it simple to include named regexes (and other variables) in an expectation_config "meta" object.

Proposed API:

Suppose meta looks like this:

{
  "__version__" : "0.2.3",
  "regexes" : {
    "namelike" : "^[A-Z][a-z]+$",
    "leading_or_trailing_whitespace" : "^\W.*?\W$",
    "double_vowels" : [
      "aa", "ee", "ii", "oo", "uu", "yy"
    ]
  }
}

...then these are all valid function calls:

expect_column_values_to_match_regex("my_column", {"$meta_var" : "regexes.namelike"})
expect_column_values_to_not_match_regex("my_column", {"$meta_var" : "regexes.leading_or_trailing_whitespace"})
expect_column_values_to_match_regex_list("my_column", {"$meta_var" : "regexes.double_vowels"}, "any")

#Match against 'aa'
expect_column_values_to_match_regex("my_column", {"$meta_var" : "regexes.double_vowels.0"}) 

#Match against 'aa' or 'yy'
expect_column_values_to_match_regex_list("my_column", [{"$meta_var" : "regexes.double_vowels.0"}, {"$meta_var" : "regexes.double_vowels.5"}], "any")

At runtime, each $meta_var expression is evaluated by traversing the meta objects tree. This syntax is roughly equivalent to mongodb syntax for querying nested objects. (Mongo also supports matching on multiple criteria, which we don't need here.)

In the config, these will create entries like:

{
  "expectation_type" : "expect_column_values_to_match_regex",
  "kwargs" : {
    "column" : "my_column", 
    "regex" : {"$meta_value" : "regexes.namelike"}
  }
}

{
  "expectation_type" : "expect_column_values_to_not_match_regex",
  "kwargs" : {
    "column" : "my_column", 
    "regex" : {"$meta_value" : "regexes.leading_or_trailing_whitespace"}
  }
}

{
  "expectation_type" : "expect_column_values_to_match_regex_list",
  "kwargs" : {
    "column" : "my_column", 
    "regex_list" : {"$meta_value" : "regexes.double_vowels"},
    "match_on" : "any"
  }
}

{
  "expectation_type" : "expect_column_values_to_match_regex",
  "kwargs" : {
    "column" : "my_column", 
    "regex" : {"$meta_value" : "regexes.double_vowels.0"},
  }
}


{
  "expectation_type" : "expect_column_values_to_match_regex_list",
  "kwargs" : {
    "column" : "my_column", 
    "regex_list" : [
      {"$meta_value" : "regexes.double_vowels.0"},
      {"$meta_value" : "regexes.double_vowels.5"}
    ],
    "match_on" : "any"
  }
}

Note: all of these examples are given in terms of regexes, but this pattern can be applied to virtually any expectation argument.

Therefore, I think the most straightforward way to implement this proposal is by adding logic to @expectation which will iterate over all args and kwargs to find instances of {"$meta_var": ...} and perform the appropriate substitution.

[jpcampb2]

Thanks for capturing this discussion! One small tweak: rather than __version__ what about ge_version: that avoid confusion about name mangling (since it's a dict key not a class variable anyway) and also confusion about whether it's a version of the config (it's not) or a version of the ge library (it is).

[abegong]

How about we remove all ambiguity and go with great_expectations.__version__?

That's explicit enough that a developer encountering a config for the first time (with no background in great_expectations at all) could still Google and figure it out.

[jpcampb2]

Sounds good to me.

[jpcampb2]

Confirming a discussion we had here:

• we want to build a mechanism for making explicit when the meta object access will be explicit and when it will be implicit.