ReBulk
ReBulk is a python library that performs advanced searches in strings that would be hard to implement using re module or String methods only.
It includes some features like Patterns
, Match
, Rule
that allows developers to build a
custom and complex string matcher using a readable and extendable API.
This project is hosted on GitHub: https://github.com/Toilal/rebulk
Install
$ pip install rebulk
Usage
Regular expression, string and function based patterns are declared in a Rebulk
object. It use a fluent API to
chain string
, regex
, and functional
methods to define various patterns types.
>>> from rebulk import Rebulk
>>> bulk = Rebulk().string('brown').regex(r'qu\w+').functional(lambda s: (20, 25))
When Rebulk
object is fully configured, you can call matches
method with an input string to retrieve all
Match
objects found by registered pattern.
>>> bulk.matches("The quick brown fox jumps over the lazy dog")
[<brown:(10, 15)>, <quick:(4, 9)>, <jumps:(20, 25)>]
If multiple Match
objects are found at the same position, only the longer one is kept.
>>> bulk = Rebulk().string('lakers').string('la')
>>> bulk.matches("the lakers are from la")
[<lakers:(4, 10)>, <la:(20, 22)>]
String Patterns
String patterns are based on str.find method to find matches, but returns all matches in the string. ignore_case
can be enabled to ignore case.
>>> Rebulk().string('la').matches("lalalilala")
[<la:(0, 2)>, <la:(2, 4)>, <la:(6, 8)>, <la:(8, 10)>]
>>> Rebulk().string('la').matches("LalAlilAla")
[<la:(8, 10)>]
>>> Rebulk().string('la', ignore_case=True).matches("LalAlilAla")
[<La:(0, 2)>, <lA:(2, 4)>, <lA:(6, 8)>, <la:(8, 10)>]
You can define several patterns with a single string
method call.
>>> Rebulk().string('Winter', 'coming').matches("Winter is coming...")
[<Winter:(0, 6)>, <coming:(10, 16)>]
Regular Expression Patterns
Regular Expression patterns are based on a compiled regular expression. re.finditer method is used to find matches.
If regex module is available, it will be used by rebulk instead of default re module.
>>> Rebulk().regex(r'l\w').matches("lolita")
[<lo:(0, 2)>, <li:(2, 4)>]
You can define several patterns with a single regex
method call.
>>> Rebulk().regex(r'Wint\wr', 'com\w{3}').matches("Winter is coming...")
[<Winter:(0, 6)>, <coming:(10, 16)>]
All keyword arguments from re.compile are supported.
>>> import re # import required for flags constant
>>> Rebulk().regex('L[A-Z]KERS', flags=re.IGNORECASE) \
... .matches("The LaKeRs are from La")
[<LaKeRs:(4, 10)>]
>>> Rebulk().regex('L[A-Z]', 'L[A-Z]KERS', flags=re.IGNORECASE) \
... .matches("The LaKeRs are from La")
[<La:(20, 22)>, <LaKeRs:(4, 10)>]
>>> Rebulk().regex(('L[A-Z]', re.IGNORECASE), ('L[a-z]KeRs')) \
... .matches("The LaKeRs are from La")
[<La:(20, 22)>, <LaKeRs:(4, 10)>]
If regex module is available, it automatically supports repeated captures.
>>> # If regex module is available, repeated_captures is True by default.
>>> matches = Rebulk().regex(r'(\d+)(?:-(\d+))+').matches("01-02-03-04")
>>> matches[0].children # doctest:+SKIP
[<01:(0, 2)>, <02:(3, 5)>, <03:(6, 8)>, <04:(9, 11)>]
>>> # If regex module is not available, or if repeated_captures is forced to False.
>>> matches = Rebulk().regex(r'(\d+)(?:-(\d+))+', repeated_captures=False) \
... .matches("01-02-03-04")
>>> matches[0].children
[<01:(0, 2)>, <04:(9, 11)>]
abbreviations
Defined as a list of 2-tuple, each tuple is an abbreviation. It simply replace
tuple[0]
withtuple[1]
in the expression.>>> Rebulk().regex(r'Custom-separators', abbreviations=[("-", "[\W_]+")])\ ... .matches("Custom_separators using-abbreviations") [<Custom_separators:(0, 17)>]
Functional Patterns
Functional Patterns are based on the evaluation of a function.
The function should have the same parameters as Rebulk.matches
method, that is the input string,
and must return at least start index and end index of the Match
object.
>>> def func(string):
... index = string.find('?')
... if index > -1:
... return 0, index - 11
>>> Rebulk().functional(func).matches("Why do simple ? Forget about it ...")
[<Why:(0, 3)>]
You can also return a dict of keywords arguments for Match
object.
You can define several patterns with a single functional
method call, and function used can return multiple
matches.
Patterns parameters
All patterns have options that can be given as keyword arguments.
validator
Function to validate
Match
value given by the pattern. Can also be adict
, to usevalidator
with pattern named with key.>>> def check_leap_year(match): ... return int(match.value) in [1980, 1984, 1988] >>> matches = Rebulk().regex(r'\d{4}', validator=check_leap_year) \ ... .matches("In year 1982 ...") >>> len(matches) 0 >>> matches = Rebulk().regex(r'\d{4}', validator=check_leap_year) \ ... .matches("In year 1984 ...") >>> len(matches) 1
Some base validator functions are available in rebulk.validators
module. Most of those functions have to be
configured using functools.partial
to map them to function accepting a single match
argument.
formatter
Function to convert
Match
value given by the pattern. Can also be adict
, to useformatter
with matches named with key.>>> def year_formatter(value): ... return int(value) >>> matches = Rebulk().regex(r'\d{4}', formatter=year_formatter) \ ... .matches("In year 1982 ...") >>> isinstance(matches[0].value, int) True
name
The name of the pattern. It is automatically passed to
Match
objects generated by this pattern.tags
A list of string that qualifies this pattern.
value
Override value property for generated
Match
objects.validate_all
By default, validator is called for returned
Match
objects only. Enable this option to validate them all, parent and children included.format_all
By default, formatter is called for returned
Match
values only. Enable this option to format them all, parent and children included.disabled
A
function(context)
to disable the pattern if returningTrue
.children
If
True
, all childrenMatch
objects will be retrieved instead of a single parentMatch
object.private
If
True
,Match
objects generated from this pattern are available internally only. They will be removed at the end ofRebulk.matches
method call.private_parent
Force parent matches to be returned and flag them as private.
private_children
Force children matches to be returned and flag them as private.
marker
If
true
,Match
objects generated from this pattern will be markers matches instead of standard matches. They won't be included inMatches
sequence, but will be available inMatches.markers
sequence (seeMarkers
section).
Match
A Match
object is the result created by a registered pattern.
It has a value
property defined, and position indices are available through start
, end
and span
properties.
In some case, it contains children Match
objects in children
property, and each child Match
object
reference its parent in parent
property. Also, a name
property can be defined for the match.
If groups are defined in a Regular Expression pattern, each group match will be converted to a
single Match
object. If a group has a name defined ((?P<name>group)
), it is set as name
property in a child
Match
object. The whole regexp match (re.group(0)
) will be converted to the main Match
object,
and all subgroups (1, 2, ... n) will be converted to children
matches of the main Match
object.
>>> matches = Rebulk() \
... .regex(r"One, (?P<one>\w+), Two, (?P<two>\w+), Three, (?P<three>\w+)") \
... .matches("Zero, 0, One, 1, Two, 2, Three, 3, Four, 4")
>>> matches
[<One, 1, Two, 2, Three, 3:(9, 33)>]
>>> for child in matches[0].children:
... '%s = %s' % (child.name, child.value)
'one = 1'
'two = 2'
'three = 3'
It's possible to retrieve only children by using children
parameters. You can also customize the way structure
is generated with every
, private_parent
and private_children
parameters.
>>> matches = Rebulk() \
... .regex(r"One, (?P<one>\w+), Two, (?P<two>\w+), Three, (?P<three>\w+)", children=True) \
... .matches("Zero, 0, One, 1, Two, 2, Three, 3, Four, 4")
>>> matches
[<1:(14, 15)+name=one>, <2:(22, 23)+name=two>, <3:(32, 33)+name=three>]
Match object has the following properties that can be given to Pattern objects
formatter
Function to convert
Match
value given by the pattern. Can also be adict
, to useformatter
with matches named with key.>>> def year_formatter(value): ... return int(value) >>> matches = Rebulk().regex(r'\d{4}', formatter=year_formatter) \ ... .matches("In year 1982 ...") >>> isinstance(matches[0].value, int) True
format_all
By default, formatter is called for returned
Match
values only. Enable this option to format them all, parent and children included.conflict_solver
A
function(match, conflicting_match)
used to solve conflict. Returned object will be removed from matches byConflictSolver
default rule. If__default__
string is returned, it will fallback to default behavior keeping longer match.
Matches
A Matches
object holds the result of Rebulk.matches
method call. It's a sequence of Match
objects and
it behaves like a list.
All methods accepts a predicate
function to filter Match
objects using a callable, and an index
int to
retrieve a single element from default returned matches.
It has the following additional methods and properties on it.
starting(index, predicate=None, index=None)
Retrieves a list of
Match
objects that starts at given index.ending(index, predicate=None, index=None)
Retrieves a list of
Match
objects that ends at given index.previous(match, predicate=None, index=None)
Retrieves a list of
Match
objects that are previous and nearest to match.next(match, predicate=None, index=None)
Retrieves a list of
Match
objects that are next and nearest to match.tagged(tag, predicate=None, index=None)
Retrieves a list of
Match
objects that have the given tag defined.named(name, predicate=None, index=None)
Retrieves a list of
Match
objects that have the given name.range(start=0, end=None, predicate=None, index=None)
Retrieves a list of
Match
objects for given range, sorted from start to end.holes(start=0, end=None, formatter=None, ignore=None, predicate=None, index=None)
Retrieves a list of hole
Match
objects for given range. A hole match is created for each range where no match is available.conflicting(match, predicate=None, index=None)
Retrieves a list of
Match
objects that conflicts with given match.chain_before(self, position, seps, start=0, predicate=None, index=None)
:Retrieves a list of chained matches, before position, matching predicate and separated by characters from seps only.
chain_after(self, position, seps, end=None, predicate=None, index=None)
:Retrieves a list of chained matches, after position, matching predicate and separated by characters from seps only.
at_match(match, predicate=None, index=None)
Retrieves a list of
Match
objects at the same position as match.at_span(span, predicate=None, index=None)
Retrieves a list of
Match
objects from given (start, end) tuple.at_index(pos, predicate=None, index=None)
Retrieves a list of
Match
objects from given position.names
Retrieves a sequence of all
Match.name
properties.tags
Retrieves a sequence of all
Match.tags
properties.to_dict(details=False, implicit=False)
Convert to an ordered dict, with
Match.name
as key andMatch.value
as value.It's a subclass of OrderedDict, that contains a
matches
property which is a dict withMatch.name
as key and list ofMatch
objects as value.If
implicit
isTrue
and distinct values are found for the same name, value will be converted as list. IfFalse
, first value only will be kept and values lists can be retrieved withvalues_list
which is a dict withMatch.name
as key and list ofMatch.value
as value.If
details
is True,Match.value
objects are replaced with completeMatch
object.markers
A custom
Matches
sequences specialized formarkers
matches (see below)
Markers
If you have defined some patterns with markers
property, then Matches.markers
points to a special Matches
sequence that contains only markers
matches. This sequence supports all methods from Matches
.
Markers matches are not intended to be used in final result, but can be used to implement a Rule
.
Rules
Rules are a convenient and readable way to implement advanced conditional logic involving several Match
objects.
When a rule is triggered, it can perform an action on Matches
object, like filtering out, adding additional tags or
renaming.
Rules are implemented by extending the abstract Rule
class. They are registered using Rebulk.rule
method by
giving either a Rule
instance, a Rule
class or a module containing Rule classes
only.
For a rule to be triggered, Rule.when
method must return True
, or a non empty list of Match
objects, or any other truthy object. When triggered, Rule.then
method is called to perform the action with
when_response
parameter defined as the response of Rule.when
call.
Instead of implementing Rule.then
method, you can define consequence
class property with a Consequence classe
or instance, like RemoveMatch
, RenameMatch
or AppendMatch
. You can also use a list of consequence when
required : when_response
must then be iterable, and elements of this iterable will be given to each consequence in
the same order.
When many rules are registered, it can be useful to set priority
class variable to define a priority integer
between all rule executions (higher priorities will be executed first). You can also define dependency
to declare
another Rule class as dependency for the current rule, meaning that it will be executed before.
For all rules with the same priority
value, when
is called before, and then
is called after all.
>>> from rebulk import Rule, RemoveMatch
>>> class FirstOnlyRule(Rule):
... consequence = RemoveMatch
...
... def when(self, matches, context):
... grabbed = matches.named("grabbed", 0)
... if grabbed and matches.previous(grabbed):
... return grabbed
>>> rebulk = Rebulk()
>>> rebulk.regex("This match(.*?)grabbed", name="grabbed")
<...Rebulk object ...>
>>> rebulk.regex("if it's(.*?)first match", private=True)
<...Rebulk object at ...>
>>> rebulk.rules(FirstOnlyRule)
<...Rebulk object at ...>
>>> rebulk.matches("This match is grabbed only if it's the first match")
[<This match is grabbed:(0, 21)+name=grabbed>]
>>> rebulk.matches("if it's NOT the first match, This match is NOT grabbed")
[]