robinl/fuzzymatcher

Record linking package that fuzzy matches two Python pandas dataframes using sqlite3 fts4

fuzzymatcher

Note: fuzzymatcher is no longer actively maintained. Please see splink for a more accurate, scalable and performant solution

A Python package that allows the user to fuzzy match two pandas dataframes based on one or more common fields.

Fuzzymatches uses sqlite3's Full Text Search to find potential matches.

It then uses probabilistic record linkage to score matches.

Finally it outputs a list of the matches it has found and associated score.

Installation

pip install fuzzymatcher

Note that you will need a build of sqlite which includes FTS4. This seems to be widely included by default, but otherwise see here.

Usage

See examples.ipynb for examples of usage and the output.

You can run these examples interactively here.

Simple example

Suppose you have a table called df_left which looks like this:

id	ons_name
0	Darlington
1	Monmouthshire
2	Havering
3	Knowsley
4	Charnwood
...	etc.

And you want to link it to a table df_right that looks like this:

id	os_name
0	Darlington (B)
1	Havering London Boro
2	Sir Fynwy - Monmouthshire
3	Knowsley District (B)
4	Charnwood District (B)
...	etc.

You can write:

import fuzzymatcher
fuzzymatcher.fuzzy_left_join(df_left, df_right, left_on = "ons_name", right_on = "os_name")

And you'll get:

best_match_score	ons_name	os_name
0.178449	Darlington	Darlington (B)
0.133371	Monmouthshire	Sir Fynwy - Monmouthshire
0.102473	Havering	Havering London Boro
0.155775	Knowsley	Knowsley District (B)
0.155775	Charnwood	Charnwood District (B)
...	etc.	etc.

fuzzymatcher's Issues

python3 TypeError: join() takes no keyword arguments

I tried it out with python3 and got some error. I know, that you are working with that other python, but if you switch:

File "/home/s/.local/lib/python3.6/site-packages/fuzzymatcher/init.py", line 24, in link_table
m.match_all()
File "/home/s/.local/lib/python3.6/site-packages/fuzzymatcher/matcher.py", line 39, in match_all
self.data_preprocessor.preprocess()
File "/home/s/.local/lib/python3.6/site-packages/fuzzymatcher/data_preprocessor_default.py", line 41, in preprocess
self._concat_all_fields(self.matcher.df_left, left_cols)
File "/home/s/.local/lib/python3.6/site-packages/fuzzymatcher/data_preprocessor_default.py", line 58, in _concat_all_fields
df['_concat_all'] = df[cols].apply(' '.join, axis=1)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/series.py", line 2510, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas/_libs/src/inference.pyx", line 1521, in pandas._libs.lib.map_infer
File "/usr/local/lib/python3.6/dist-packages/pandas/core/series.py", line 2497, in
f = lambda x: func(x, *args, **kwds)
TypeError: join() takes no keyword arguments

But you would find it by yourself for sure.

OperationalError: No Such Module:fts4

Hello,
I have tried this with Anaconda 32 Bit version. This worked perfectly. But due to performance issues, I uninstalled 32 Bit and Installed 64 Bit Version. When I try to execute my code, I get this error. Please assist me on this.

• 1. Will this only work with 32 Bit version Anaconda?
• 2. When I tried to install "FuzzyMatcher" for the 64 bit, it says version issue and cannot be installed. So I tried to manually keep the fuzzymatcher 32 bit installed files in the 64 bit site-libraries. The the above error occurs.

Please help me with this. any thanks in advance

Readme should explain meaning of scores

On what scale are the matches scored?

I noticed with fuzzymatcher.fuzzy_left_join my best_match_scoreranges from -0.7 to + 1.15.

What is the highest possible score in this case? Can it go higher than 1.15?

Usually for fuzzy matching I would have a cutoff of around 0.8 or 0.9., which is on a scale of 0 to 1.

Improve efficiency of search

There are a number of things here:

• Optimise search strategy based on average number of tokens in records
• Optimise search strategy by improving way in which FTS search tokens are chosen

Cosine/levenstein

One output of link_table should probably be an additional cosine or similarity levenstein score, to provide a comparator to the probabalistic approach

URGENT:How to use Fuzzymatcher for one dataset

Hi all,
Is there a way to identify duplicates in one file? The manual talks about comparing two data files, however, I only need to look for duplicates in one single file.
I hope someone has a solution.
Thanks in advance!

Random ideas

Should a contraction count as a match.
e.g. Will, William
J, James
etc

Don't depend on python-levenstein

See here - https://stackoverflow.com/a/47629587/1779128

Probabilities

Should be based on token frequency in the rigth dataset, not both.

This allows us to easily spot tokens in the left dataset which never exist in the right one.

These should be dropped prior to searching.

Type check to make sure everything works if 'on' is a string, not a list

Don't include potential matches if too many potential matches are found

fts5 considers unrecognized column options to be errors

First, thank you, you just saved me a ton of time.

I recently installed and tried fuzzymatcher. I encountered an error running the code in the article https://pbpython.com/record-linking.html. At first, I was getting a OperationalError: no such module: fts4, but following some advice in another issue, I replaced all my SQLite dlls with the latest.

However, I then began getting a Unrecognized column option error. After reading this - https://www.sqlite.org/fts5.html - and seeing it mention FTS5 considers column options to be errors, I suspected (I haven't confirmed) that the latest SQLite windows dll included FTS5, so in data_getter_sqlite.py I changed:

USING fts4({} TEXT, _concat_all TEXT, _concat_all_alternatives TEXT);"""format(matcher.right_id_col)

to

USING fts5({}, _concat_all, _concat_all_alternatives);""".format(matcher.right_id_col)

and it worked for me.

I suspect it is related to my fresh install of everything on windows? Sharing in case it helps others.

Dealing with 'punishment' of non-matching tokens

If you have two unusual names robin and robyn which you think are a match, both will be very unusual and a bit punishment will be applied

before applying punishment check that they're not misspellings of each other

Using RapidFuzz instead FuzzyWuzzy

Hello, if I am not wrong, fuzzymatcher is using FuzzyWuzzy?
If so, did you consider replacing it with RapidFuzz, which is claiming to provide same functionality, but being much faster?
See here: https://github.com/maxbachmann/RapidFuzz

Regards,
Zlatko

New Zero Division Error Issue

Several months ago I was often using Fuzzymatcher and never ran into a problem. However, now when I (and a colleague on a completely different OS) try to fuzzymatch two data frames I'm getting a ZeroDivisionError (below). I've read through the responses to a similar problem a few years ago (issue 42 below), and they suggest incorporating a "except (ValueError, ZeroDivisionError)" into line 44 of the tokencomparison.py file. The problem is that file already has the ZeroDivisionError built in, which suggests that this ZeroDivisionError is a new one with a different provenance. Any suggestions for how to fix this without breaking the package would be helpful.

fuzzymatched=fuzzymatcher.fuzzy_left_join(dataframe1, dataframe2, 'address', 'address')
Traceback (most recent call last):

File "", line 1, in
fuzzymatched=fuzzymatcher.fuzzy_left_join(dataframe1, dataframe2, 'address', 'address')

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/init.py", line 41, in fuzzy_left_join
m.match_all()

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/matcher.py", line 92, in match_all
self.link_table = self._match_processed_data()

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/matcher.py", line 136, in _match_processed_data
this_record.find_and_score_potential_matches()

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/record.py", line 76, in find_and_score_potential_matches
self.matcher.data_getter.get_potential_match_ids_from_record(self)

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/data_getter_sqlite.py", line 93, in get_potential_match_ids_from_record
self._search_specific_to_general_single(token_list, rec_left)

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/data_getter_sqlite.py", line 119, in _search_specific_to_general_single
self._add_matches_to_potential_matches(new_matches, rec_left)

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/data_getter_sqlite.py", line 167, in _add_matches_to_potential_matches
scored_potential_match = self.matcher.scorer.score_match(rec_left.record_id, right_id)

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/scorer_default.py", line 57, in score_match
p = self._field_to_prob(f_left, record_left, record_right)

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/scorer_default.py", line 78, in _field_to_prob
prob_unmatching2 = self._get_prob_unmatching(unmatching_tokens_right, tokens_left, field_right, field_left)

File "/Users/stephencranney/opt/anaconda3/lib/python3.7/site-packages/fuzzymatcher/scorer_default.py", line 107, in _get_prob_unmatching
return 1/prob

ZeroDivisionError: float division by zero

Profile code

What's causing the greatest inefficiencies?

Add run time statistics to .txt test log files

Processing time high on large datasets

Taking long time for processing 50000 rows.Any chances to reduce it?

Improve simple example

Scorer

In the code which reduces the score for tokens which don't match, do not include dmetaphone tokens

OperationalError: malformed MATCH expression

Using fuzzy_left_join, I came across the above error while matching addresses.

It seems to relate to the following string: '16 Orchard Court, Near Stepaside Park, Stepaside, Dublin, ireland'

(note: I swapped out the street name for another for privacy reasons, but string length and structure is the same)

Code which I ran:

import fuzzymatcher
import pandas as pd

df_left = contacts_unique_addresses
df_right = register
left_on = ["match_string"]
right_on = ["match_string"]

matched = fuzzymatcher.fuzzy_left_join(df_left, df_right, left_on, right_on)

And the error. Is it possible that my string is too long?

OperationalError                          Traceback (most recent call last)
<ipython-input-78-6bd0c667558c> in <module>
      7 right_on = ["match_string"]
      8 
----> 9 matched = fuzzymatcher.fuzzy_left_join(df_left, df_right, left_on, right_on)

~\Anaconda3\lib\site-packages\fuzzymatcher\__init__.py in fuzzy_left_join(df_left, df_right, left_on, right_on, left_id_col, right_id_col)
     39     m = Matcher(dp, dg, s)
     40     m.add_data(df_left, df_right, left_on, right_on,  left_id_col, right_id_col)
---> 41     m.match_all()
     42 
     43     return m.get_left_join_table()

~\Anaconda3\lib\site-packages\fuzzymatcher\matcher.py in match_all(self)
     90 
     91         # Get a table that contains only the matches, scores and ids
---> 92         self.link_table = self._match_processed_data()
     93 
     94     def get_formatted_link_table(self):

~\Anaconda3\lib\site-packages\fuzzymatcher\matcher.py in _match_processed_data(self)
    134                 log.debug(str_template.format(counter, (counter/total)*100, diff.minutes, diff.seconds))
    135 
--> 136             this_record.find_and_score_potential_matches()
    137             link_table_list.extend(this_record.get_link_table_rows())
    138 

~\Anaconda3\lib\site-packages\fuzzymatcher\record.py in find_and_score_potential_matches(self)
     74     def find_and_score_potential_matches(self):
     75         # Each left_record has a list of left_record ids
---> 76         self.matcher.data_getter.get_potential_match_ids_from_record(self)
     77 
     78     def get_link_table_rows(self):

~\Anaconda3\lib\site-packages\fuzzymatcher\data_getter_sqlite.py in get_potential_match_ids_from_record(self, rec_left)
     91 
     92         for token_list in token_lists:
---> 93             self._search_specific_to_general_single(token_list, rec_left)
     94             if not self._found_enough_matches(rec_left):
     95                 self._search_specific_to_general_band(token_list, rec_left)

~\Anaconda3\lib\site-packages\fuzzymatcher\data_getter_sqlite.py in _search_specific_to_general_single(self, token_list, rec_left)
    115         for i in range(len(token_list)):
    116             sub_tokens = token_list[i:]
--> 117             new_matches = self._tokens_to_matches(tuple(sub_tokens))
    118 
    119             self._add_matches_to_potential_matches(new_matches, rec_left)

~\Anaconda3\lib\site-packages\fuzzymatcher\data_getter_sqlite.py in _tokens_to_matches(self, tokens, misspelling)
    190 
    191         cur = self.con.cursor()
--> 192         cur.execute(sql)
    193         results = cur.fetchall()
    194 

OperationalError: malformed MATCH expression: [NEAR IRELAND STEPASIDE STEPASIDE ORCHARD 16 COURT PARK DUBLIN]

Deduping a single df

I'm looking to do this on one df (and then on more than 2). Is there a way to do this that I've missed in the docs?

Improve serach

Initial search takes combinations of tokens from the original tokens without dmetaphones

If this doesn't work, take combination of tokens from the dmetaphones.

Don't combine the two.

Division By Zero in def is_mispelling

Hey,

I've been using your lib on 0.0.1 and just updated recently (I had to hack some of the SQLite fts keywords and will fix that up again) but I've come across a problem:

You get a div zero error in tokencomparison.py -> def is_mispelling(self, token1, token2)

Here are the values of the vars in that function when it throws:

float division by zero
token1: 0
token2: 2
mis_t1: []
mis_t2: []
common: []

I know you're comparing distance for string tokens, but what is the logic behind numeric values? Whats the logic behind determining if two numbers are misspellings? (even ignoring the 0 value)

Even if you swap the max( ) / min ( ) to min ( ) / max ( ) and take the inverse you'll still get 0 for 0 values.

Maybe an absolute difference is better but that stuffs you up when there are addition errors (e.g. 1 typo to 10)

Maybe edit distance is still best used here?

As an aside, thanks for making this library; it's saved me some time so far :)

Can fuzzymatcher support unicode characters？

Hi,

Can fuzzymatcher support unicode characters？For example, é.

Thanks!

User to specify some columns on which *must* match for the join to be valid

...and not score tokens which match in those groups.

This is equivalent to matching 'within groups of records'

This is essentially binning.

`best_match_score` equals `nan`

Good day, @RobinL.

I am matching a dataframe with names and one combination gives me nan. Any suggestions?

Thanks!

ZeroDivisionError: float division by zero

I've been trying to match 2 dfs but get the following error.

File "fuzzyLookup.py", line 18, in <module>
    linked_table = link_table(app_annie_apps, our_apps, aa_apps_on, our_apps_on)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\__init__.py", line 24, in link_table
    m.match_all()
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\matcher.py", line 92, in match_all
    self.link_table = self._match_processed_data()
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\matcher.py", line 136, in _match_processed_data
    this_record.find_and_score_potential_matches()
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\record.py", line 68, in find_and_score_potential_matches
    self.matcher.data_getter.get_potential_match_ids_from_record(self)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\data_getter_sqlite.py", line 93, in get_potential_match_ids_from_record
    self._search_specific_to_general_single(token_list, rec_left)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\data_getter_sqlite.py", line 119, in _search_specific_to_general_single
    self._add_matches_to_potential_matches(new_matches, rec_left)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\data_getter_sqlite.py", line 167, in _add_matches_to_potential_matches
    scored_potential_match = self.matcher.scorer.score_match(rec_left.record_id, right_id)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\scorer_default.py", line 57, in score_match
    p = self._field_to_prob(f_left, record_left, record_right)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\scorer_default.py", line 77, in _field_to_prob
    prob_unmatching1 = self._get_prob_unmatching(unmatching_tokens_left, tokens_right, field_right, field_left)
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\scorer_default.py", line 100, in _get_prob_unmatching
    if not self._is_misspelling_of_one(umt, record_tokens):
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\scorer_default.py", line 111, in _is_misspelling_of_one
    if self.matcher.token_comparison.is_mispelling(token, t):
  File "C:\Users\bhamehul\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-packages\fuzzymatcher\tokencomparison.py", line 38, in is_mispelling
    if max(t1f, t2f)/min(t1f, t2f) < self.number_fuzz_threshold:
ZeroDivisionError: float division by zero

Improve documentation

Punctuation bug

OperationalError: malformed MATCH expression: [HAMMERSMITH FULHAM AND]

Improve scoring mechanism

There are a number of ways which may improve scoring:

• Reduce score for terms which do not match, by an amount proportional to their uniqueness/prob
• Increase score for terms which almost match, but by a reduced amount proportional to the closeness of match (cosine/levenshtein etc.)
• Could anything in fuzzywuzzy be useful for scoring?

Properly split out __concat_all and __alternative_tokens

This will speed up issues around dmetaphones.

This should be done in the FTS table.

Link table fails on NaN

The link table function will fail and throw a Type error if there are any NaNs in the matching columns.

TypeError: ('sequence item 0: expected str instance, float found', 'occurred at index 10')

ImportError: cannot import name lru_cache

Hello,
I tried importing fuzzymatcher via import fuzzymatcher, but I get the following error:

ImportError: cannot import name lru_cache

I tried uninstalling and re-installing lru_cache with no change. I am using Python 2.7.

Thanks!

Add more unit tests

Add processing time to performance metrics

Unable to install package

Ran pip install fuzzymatcher, but I keep getting the errow below:

creating build\temp.win-amd64-3.9\Release\Levenshtein
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\schoo\appdata\local\programs\python\python39\include -Ic:\users\schoo\appdata\local\programs\python\python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30037\include /TcLevenshtein/_levenshtein.c /Fobuild\temp.win-amd64-3.9\Release\Levenshtein/_levenshtein.obj
_levenshtein.c
c:\users\schoo\appdata\local\programs\python\python39\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\cl.exe' failed with exit code 2

I've reinstalled my Windows SDK, C++, and python version, but nothing seems to work.

Allow user to choose to score full cartesian product

OperationalError: malformed MATCH expression

What I'm experiencing is when the term 'NOT' is within the string(s) of columns we are trying to join, I'm getting back 'OperationalError: malformed MATCH expression' . My feeling is because there is a SQL dependency for reserved words, example below ought to show error.

The below code snipped ought to reproduce error.

x1=pd.DataFrame(['xyz NOT', 'YES', '425255'], columns=['X'])
x2=pd.DataFrame(['NOT', 'OK', '42525511'], columns=['X'])

fuzzy_left_join(x1,x2, left_on='X', right_on='X')

Question on string length / parsing before fuzzy matching.

Under the Performance section of the example notebook, I ran the matcher using:

on = ["first_name", "surname", "dob", "city"]

lt = fuzzymatcher.link_table(df_left, df_right, on, on)

We can then measure the performance with link_table_percentage_correct(link_table) and it was in the region of 70%.

Then there is a following section where it is shown how performance is increased by creating initials for the names and combining first_name and surname. Using link_table_percentage_correct(link_table) we can see that accuracy has increased: 'Percent matches correct: 82.0%'

I then combined all strings into one, using:

df_left['merged'] = df_left.first_name +' '+df_left.surname+' '+df_left.dob+' '+df_left.city+' '+df_left.email

df_right['merged'] = df_right.first_name +' '+df_right.surname+' '+df_right.dob+' '+df_right.city+' '+df_right.email

And then ran the script again, getting the following result:

'Percent matches correct: 97.9%'

It seems the best result is from combining all the strings together.

So the question is, should I be doing this with every dataset? I am currently working with addresses. I was going to parse every bit of the address, but now I may just keep it as one string.

However, unlike the dataset in the example I have no easy way to tell which match is correct.

Error when running on Windows: sqlite3.OperationalError: no such module: fts4

I am on a Windows 10 machine using Anaconda with fuzzymatcher installed from PyPI:

$ conda list | grep fuzzy
fuzzymatcher              0.0.5                    pypi_0    pypi
fuzzywuzzy                0.18.0                   pypi_0    pypi

When I run a Python script that uses fuzzymatcher I get the following error:

$ python match_addresses.py -c C:/home/data/good_addresses.csv -u C:/home/data/unmatched_addresses.csv
Traceback (most recent call last):
  File "match_addresses.py", line 65, in <module>
    match()
  File "C:\home\miniconda3\envs\canada\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\home\miniconda3\envs\canada\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\home\miniconda3\envs\canada\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\home\miniconda3\envs\canada\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "match_addresses.py", line 56, in match
    matched_results = fuzzymatcher.fuzzy_left_join(unmatched_df,
  File "C:\home\miniconda3\envs\canada\lib\site-packages\fuzzymatcher\__init__.py", line 41, in fuzzy_left_join
    m.match_all()
  File "C:\home\miniconda3\envs\canada\lib\site-packages\fuzzymatcher\matcher.py", line 89, in match_all
    self.data_getter.add_data(self)
  File "C:\home\miniconda3\envs\canada\lib\site-packages\fuzzymatcher\data_getter_sqlite.py", line 60, in add_data
    con.execute(sql)
sqlite3.OperationalError: no such module: fts4

I have installed sqlite_fts4 but this hasn't helped.

$ conda list | grep fts4
sqlite-fts4               0.5.2                    pypi_0    pypi

I've even uninstalled fuzzymatcher and the reinstalled it fresh, thinking that maybe it finds the fts4 goodies it needs when installing but no luck, same error when I try again after doing that.

I do not have the same error when I run this code in the Windows Subsystem for Linux (Ubuntu 20.04) so I think this is a Windows-specific issue.

Thanks in advance for any suggestions on how to resolve this issue. In any event thanks to all the developers of this package, it's quite useful!

Cartesian and normal datagetter produce different results due to dmetaphone problems

dmetaphone only computed for df right with the cartesian dg

Need better performance metrics

Probably write a separate .ipynb for this.

Need to serach through the various parameters to see what are the optimal combinations

scorer_default.py 107 return 1/prob ZeroDivisionError: float division by zero

fuzzymatcher\scorer_default.py in _get_prob_unmatching(self, unmatching_tokens, record_tokens, field_right, field_left)
105
106 prob = Scorer._adjust_prob_towards_one(prob)
--> 107 return 1/prob
108
109 def _is_misspelling_of_one(self, token, token_list):

ZeroDivisionError: float division by zero