akkornel/syncrepl

Python LDAP Syncrepl client

What is This?

syncrepl_client is a Python module that makes LDAP Syncrepl easy to use.

LDAP Syncrepl allows you to keep up-to-date with an LDAP server, effectively in real time, even without LDAP administrator credentials.

If your LDAP directory is used as the source of truth (or a delegate for the soource of truth), this keeps you informed when something changes. Callbacks—which you write—are triggered by this code when something happens. You can then take appropriate action, such as by inserting into a queue or sending a message over a bus.

What is Syncrepl?

Syncrepl (as described in RFC 4533) is a standard which allows an LDAP server to keep clients in sync with itself. The clients keep track of a "cookie", an opaque string that the server uses to know how far behind the client is. The LDAP server then "refreshes" the client by sending details of new & changed entries, as well as information on which entries have been deleted. After the refresh is complete, the client is able to keep a long-running connection open to the server, and receive notice as soon as a change happens on the server.

Syncrepl is what OpenLDAP uses to implement replication, but the client does not have to be an OpenLDAP server. In fact, because Syncrepl is layered on top of an ordinary LDAP search, regular LDAP clients—even those with limited access—are able to use Syncrepl to be efficiently notified as soon as the results of their search has changed. This includes notification on:

• New entries that match your search filter.
• Entries being deleted.
• Entries, which used to match, no longer matching. This is essentially the same as deletion (at least, it is when you are using a search filter).
• Existing entries having their attributes or DN changed.

The entries you see, and the changes made to them, are based on the intersection of four things.

1. The entries currently in the directory.
2. Your access rights, as determined by your bind DN.
3. Your search filter.
4. Your list of requested attributes.

Thanks to the Syncrepl protocol, you don't have to worry about all of the above. The LDAP server handles the work of figuring out what you can see.

Requirements

syncrepl_client has four major requirements:

• Python 2.7, or Python 3.3+.

  If you use Python 2.7 or 3.3, you will also need enum34.

  If you plan on doing "refresh and persist" operations (which run for a long time), your Python should support threads.

• An appropriate Python LDAP library:

  • For Python 2.7, python-ldap 99 or later is needed.
  • For Python 3, pyldap 2.4.37 or later is needed.

  Older versions may be supported. Read more in patches.

• The pyasn1 module, at least version 0.2.2, and less than version 0.3.1.
• A fast data store, large enough to store a copy of all the LDAP data received, and a corresponding amount of RAM.

• An LDAP server which supports RFC 4533, and which is keeping track of changes.

  In the case of OpenLDAP, this means following the instructions in Section 18.3.1 of the Admin Guide.

Lots more details are available in the Requirements page.

How to Use

Although you'll still need to do a fair bit of coding (mainly in Step 1), syncrepl_client is (intentionally) pretty easy to use! Over the life of your code's execution, you should do these four things:

1. Create a class which implements the methods defined in BaseCallback This is how you are notified of changes from the LDAP server.
2. In your main code, import syncrepl_client and instantiate a new Syncrepl object. The instantiation will handle the connection and the search setup.
3. Call poll until it returns False. If you're running single-threaded, set the timeout parameter to some positive, non-zero value. Call please_stop when you want to safely shut down, and then resume calling poll until it returns False.
4. Call unbind. You're done!

Lots more details are available in the Requirements page, and see syncrepl-client (which setup.py and pip installs as a script) for a simple example.

Copyright and License

The contents of this repository are copywrited according to the contents of the AUTHORS file.

The code is made available under the BSD 3-Clause License.

Other code is made available under the Creative Commons CC0 Public Domain Dedication.

Documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International Public License (the CC BY-SA License). Code contained within documentation is made available under both the BSD 3-Clause License, and the CC BY-SA License.

To identify the license for any particular file, refer to the contents of the file.

The text of the BSD 3-Clause License is reproduced in the file LICENSE.md. The text of the other licenses may be found in the file LICENSE_others.md. Note that all three licenses are equally important, but are kept in a separate files to aid GitHub's irepository license-detection mechanisms.

syncrepl's Issues

LDAP URL with multiple attributes throws exception

If an LDAP URL is used which requests multiple attributes, an exception will be thrown.

For example, the following command will throw: syncrepl-client --persist ldaps://directory-master.stanford.edu:636/cn=people,dc=stanford,dc=edu?suSunetID,suPrivilegeGroup?sub?uid=ljlgeek?bindname=GSSAPI karl

The output which appears is:

Data files will be stored here: karl
Refresh-and-persist mode will be used
CLIENT SETUP START
Traceback (most recent call last):
  File "/Users/akkornel/Library/Python/3.6/bin/syncrepl-client", line 92, in <module>
    else SyncreplMode.REFRESH_ONLY
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client/__init__.py", line 397, in __init__
    SimpleLDAPObject.__init__(self, ldap_url.unparse(), **kwargs)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/pyldap-2.4.37-py3.6-macosx-10.12-x86_64.egg/ldap/ldapobject.py", line 87, in __init__
    self._l = ldap.functions._ldap_function_call(ldap._ldap_module_lock,_ldap.initialize,uri)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/pyldap-2.4.37-py3.6-macosx-10.12-x86_64.egg/ldap/functions.py", line 66, in _ldap_function_call
    result = func(*args,**kwargs)
ldap.LDAPError: (2, 'No such file or directory')
Exception ignored in: <bound method Syncrepl.__del__ of <syncrepl_client.Syncrepl object at 0x108f873c8>>
Traceback (most recent call last):
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client/__init__.py", line 502, in __del__
    return self.unbind()
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client/__init__.py", line 480, in unbind
    unbind_result = SimpleLDAPObject.unbind(self)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/pyldap-2.4.37-py3.6-macosx-10.12-x86_64.egg/ldap/ldapobject.py", line 852, in unbind
    return self.unbind_ext(None,None)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/pyldap-2.4.37-py3.6-macosx-10.12-x86_64.egg/ldap/ldapobject.py", line 831, in unbind_ext
    res = self._ldap_call(self._l.unbind_ext,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/pyldap-2.4.37-py3.6-macosx-10.12-x86_64.egg/ldap/ldapobject.py", line 329, in __getattr__
    self.__class__.__name__,repr(name)
AttributeError: Syncrepl has no attribute '_l'

Some dict operations are missing from ItemList

Our ItemList class (constructed inside syncrepl_refreshdone) is missing some of the dict methods, like items.

The shelve goes BOOM

379,239 partial records. The BDB file was about 550 MB on-disk.

DN0a626bf0:~/git/wglurp akkornel(p)$ ~/Library/Python/3.6/bin/wglurp-ldap -f
Added program-startup handler.  Log to stderr until early-start is complete.
Disabling log propagation.
[61362] [logging.py:42 (<module>)] INFO: stanford_wglurp (wglurp-ldap) version 0.0.1 early startup...
[61362] [logging.py:75 (<module>)] DEBUG: Installing custom last-resort exception-handler hook.
[61362] [config.py:76 (<module>)] DEBUG: Setting configuration defaults.
[61362] [config.py:113 (<module>)] DEBUG: Reading configuration files...
[61362] [config.py:59 (find_config_files)] DEBUG: Will load config files in the following order: ['/opt/local/Library/Frameworks/Python.framework/Versions/3.6/etc/wglurp.conf', '/Users/akkornel/.wglurp.conf']
[61362] [config.py:145 (<module>)] DEBUG: Valdating read configuration...
[61362] [logging.py:105 (<module>)] DEBUG: Configured log target is /Users/akkornel/git/wglurp/log
[61362] [logging.py:201 (<module>)] INFO: Will log to file at path /Users/akkornel/git/wglurp/log.
[61362] [logging.py:212 (<module>)] INFO: Running in foreground.  Logs will now go to stdout.  (The next entry will log to both stdout and stderr.)
[61362] [logging.py:221 (<module>)] INFO: Program-startup logging to stderr will now end.
[61362] [logging.py:221 (<module>)] INFO: Program-startup logging to stderr will now end.
[61362] [logging.py:229 (<module>)] INFO: Welcome to stanford_wglurp (wglurp-ldap) version 0.0.1!
[61362] [logging.py:234 (<module>)] INFO: Logging threshold changed to INFO
[61362] [ldap.py:354 (main)] INFO: Metrics will write to "/Users/akkornel/git/wglurp/metrics/ldap"
[61362] [ldap.py:370 (main)] INFO: LDAP URL is ldaps://ldap-master.stanford.edu:636/cn%3Dpeople%2Cdc%3Dstanford%2Cdc%3Dedu?suRegID,uid,suPrivilegeGroup?sub?%28objectClass%3D%2A%29?bindname=GSSAPI
[61362] [ldap.py:55 (bind_complete)] INFO: LDAP bind complete!  We are "dn:uid=akkornel,cn=accounts,dc=stanford,dc=edu"
[61362] [ldap.py:426 (main)] INFO: LDAP client thread #123145468792832 launched!
[61362] [ldap.py:438 (main)] INFO: LDAP metrics thread #123145474048000 launched!
HASH: Out of overflow pages.  Increase page size
Exception in thread LDAP client:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client-0.95.1-py3.6.egg/syncrepl_client/__init__.py", line 696, in run
    poll_result = self.poll()
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client-0.95.1-py3.6.egg/syncrepl_client/__init__.py", line 604, in poll
    all=1, timeout=3)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/ldap/syncrepl.py", line 385, in syncrepl_poll
    self.syncrepl_entry(dn, attrs, c.entryUUID)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client-0.95.1-py3.6.egg/syncrepl_client/__init__.py", line 1099, in syncrepl_entry
    self.__uuid_attrs[uuid] = attrs
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shelve.py", line 125, in __setitem__
    self.dict[key.encode(self.keyencoding)] = f.getvalue()
_dbm.error: cannot add item to database

HASH: Out of overflow pages.  Increase page size
[61362] [logging.py:63 (handle_exception)] CRITICAL: UNCAUGHT EXCEPTION!
[61362] [logging.py:69 (handle_exception)] CRITICAL: _dbm.error: cannot add item to database
[61362] [logging.py:69 (handle_exception)] CRITICAL: TRACEBACK (failing call last):
[61362] [logging.py:69 (handle_exception)] CRITICAL:   File "/Users/akkornel/Library/Python/3.6/bin/wglurp-ldap", line 11, in <module>
[61362] [logging.py:69 (handle_exception)] CRITICAL:     load_entry_point('stanford-wglurp', 'console_scripts', 'wglurp-ldap')()
[61362] [logging.py:69 (handle_exception)] CRITICAL:   File "/Users/akkornel/git/wglurp/stanford_wglurp/ldap.py", line 463, in main
[61362] [logging.py:69 (handle_exception)] CRITICAL:     client.unbind()
[61362] [logging.py:69 (handle_exception)] CRITICAL:   File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/syncrepl_client-0.95.1-py3.6.egg/syncrepl_client/__init__.py", line 497, in unbind
[61362] [logging.py:69 (handle_exception)] CRITICAL:     self.__uuid_attrs.close()
[61362] [logging.py:69 (handle_exception)] CRITICAL:   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shelve.py", line 144, in close
[61362] [logging.py:69 (handle_exception)] CRITICAL:     self.sync()
[61362] [logging.py:69 (handle_exception)] CRITICAL:   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shelve.py", line 168, in sync
[61362] [logging.py:69 (handle_exception)] CRITICAL:     self[key] = entry
[61362] [logging.py:69 (handle_exception)] CRITICAL:   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shelve.py", line 125, in __setitem__
[61362] [logging.py:69 (handle_exception)] CRITICAL:     self.dict[key.encode(self.keyencoding)] = f.getvalue()

demo.py, referenced in README, is not present in master

Though I found it in the v.90 tag.

Before wiping data store, extract URL

In some situations, we need to wipe the data store. If that happens, extract the URL first. If the client didn't provide a URL, then we'll use what we extracted.

Consistency break when full sync includes an entry's deletion and re-creation

Pondering the code, I've discovered a way to break consistency between our data store, and the client's. It will happen if a full sync is needed, and that involves an entry being deleted, then re-created with the same DN.

I haven't seen this happen yet, but I know it will.

Here's the (theoretical) sequence of what might happen:

1. At some point in the past, you were in sync. But right now you're not connected.
2. On the LDAP server, an entry (which you have in your local cache) is deleted.
3. On the LDAP server, a new entry is created, and the entry has the same DN as the entry from 2.
4. You connect, but your cookie is so old that the LDAP begins a full sync.
5. The Syncrepl instance gets a call to syncrepl_entry where uuid is new to us, but dn is one we've seen before.
6. Eventually the Syncrepl instance gets a call to syncrepl_present with no UUIDs, and with refreshDeletes set to False. That tells us "All the UUIDs you have, which we haven't already said are present, are gone," and it triggers us to delete the old UUID. That triggers a call to syncrepl_delete.

That's how things look to the Syncrepl class. This is actually OK, because our data stores (the UUID-to-DN map and the UUID-to-attributes map) are both keyed on UUID, so there is no conflict between these two entries.

However, the callbacks are not keyed on UUID, the callbacks are keyed on DN! So, here's the sequence of callbacks which would be triggered in the above scenario:

1. A bind_complete callback, triggered by Step 4.
2. A record_add callback, triggered by Step 5. This will be confusing to the client, as they've already added the DN previously (as per Step 1).
3. A record_delete callback, triggered by Step 6. This will completely break consistency, as our client deletes what is probably still a valid entry.

At this point, our data store and the client's data store will be out of sync. The only way synchronization would come back into sync automatically would be if:

• A future reconnect causes another full re-sync.
• The entry is deleted and re-added on the LDAP server.
• The entry is modified on the LDAP server.

Either way, it will still be bad!

pyldap 3.0 incompatibility

If pyldap 3.0.0post1 is installed instead of 2.4.37, refresh-and-persist will bail out at the end of the initial syncwith the following traceback:

Exception in thread ldapwatch:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/data/web/nextcloud/venv/lib/python3.5/site-packages/syncrepl_client/__init__.py", line 801, in run
    poll_result = self.poll()
  File "/data/web/nextcloud/venv/lib/python3.5/site-packages/syncrepl_client/__init__.py", line 702, in poll
    all=1, timeout=3)
  File "/data/web/nextcloud/venv/lib/python3.5/site-packages/ldap/syncrepl.py", line 448, in syncrepl_poll
    sim = SyncInfoMessage(resp)
  File "/data/web/nextcloud/venv/lib/python3.5/site-packages/ldap/syncrepl.py", line 320, in __init__
    if comp is not None and comp.hasValue():
AttributeError: 'RefreshDelete' object has no attribute 'hasValue'

All records eventually deleted by syncrepl when running in REFRESH_ONLY

I noticed an oddity with syncrepl_client when using REFRESH_ONLY mode. This issue does not present when I use REFRESH_AND_PERSIST so I'll admit I may not understand how the LDAP Synchronize spec is supposed to behave with REFRESH_ONLY.

After running my script several times over a span of 10 minutes, an entry gets flagged as deleted by syncrepl_client even though it still exists on the server. That entry will not be added again by the snycrepl_client until I modify it on the server. The cookie value on the execution that triggers the delete matches the cookie value from the previous execution so I'm perplexed as to why
snycrepl_client think it needs to delete something? Is this the expected behavior of REFRESH_ONLY?

I'm testing with FreeIPA, Directory 389, running inside docker (https://hub.docker.com/r/freeipa/freeipa-server/).

LDAP_SCOPE = SUBTREE
LDAP_FILTER_STR = "(objectclass=organizationalperson)"
LDAP_DN = "cn=users,cn=accounts,dc=example,dc=com"

Thanks

Debug callback

There is a callback for debug, but it's not being used for much. Should it be kept or removed?

gracefully handle sizelimit / do a paged initial sync ?

If the amount of records in the server is above olcSizeLimit, during the initial sync at some point syncrepl will blow:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/landry/scratch/py-syncrepl/syncrepl_client/__init__.py", line 801, in run
    poll_result = self.poll()
  File "/home/landry/scratch/py-syncrepl/syncrepl_client/__init__.py", line 702, in poll
    all=1, timeout=3)
  File "/usr/lib/python3/dist-packages/ldap/syncrepl.py", line 352, in syncrepl_poll
    add_intermediates=1, add_ctrls=1, all = 0
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 687, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 263, in _ldap_call
    result = func(*args,**kwargs)
ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'}

There should probably be a way to either gracefully handle that case, or do a paged sync request defaulting to 500 objects (which is the openldap default in most configs..)

Docs cleanup

Do another pass of all the existing docs.

Explicitly set encoding when calling open

The long dashes in README.rst raise a UnicodeDecodeError when building the project on Python 3.5.2. Unfortunately python 3.5.2 is the default python 3 version in Ubuntu 16.04 LTS.

root@ba3058541282:/src/syncrepl# pip3 wheel .
Processing /src/syncrepl
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-1v94ry17-build/setup.py", line 103, in <module>
        long_description = readme(),
      File "/tmp/pip-1v94ry17-build/setup.py", line 95, in readme
        return file.read()
      File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1706: ordinal not in range(128)

Specifying the encoding when opening the file fixes the issue

def readme():
    with open('README.rst', encoding='utf8') as file:
        return file.read()

Set up Travis and Coveralls

We should set up Travis to auto-run tests an our supported Pythons.

At the same time, Coveralls should be set up to monitor coverage data generated by Travis.

As part of this, badges would be added to the README.

Depends on #4.

Control-flow bug found

There's apparently a control-flow defect somewhere in the library code.

https://scan.coverity.com/projects/akkornel-syncrepl/view_defects will have the results, once Coverity has signed off on the project.

Data store consistency check should include Python version

Besides checking our library version, also check the Python version. That means we have to store it in the map.

Configure Coverity scan

Coverity can scan Python code, so we should have it scan ours!

If there's a badge, add it at this time.

Related to #9, in that we can link Travis and Coverity.

py2/py3 conflict on ItemIter class

Trying this code on debian stretch with python 3.5 and 2.7 with the following libs installed:

ii  python-ldap                           2.4.28-0.1                     amd64        LDAP interface module for Python
ii  python3-ldap3                         1.2.2-1                        all          Pure Python LDAP client library
ii  python3-pyldap                        2.4.25.1-2                     amd64        implements an LDAP client - Python 3.x

it seems theres a conflict on the ItemIter class, when running as py2:

  File "/home/landry/scratch/py-syncrepl/syncrepl_client/__init__.py", line 947, in __iter__
    return ItemIter(self.__syncrepl_list)
TypeError: Can't instantiate abstract class ItemIter with abstract methods next

but if i rename the method from __next__ to next then running as py3 is broken the other way:

  File "/home/landry/scratch/py-syncrepl/syncrepl_client/__init__.py", line 947, in __iter__
    return ItemIter(self.__syncrepl_list)
TypeError: Can't instantiate abstract class ItemIter with abstract methods __next__

how... fun.

Test suite

Even though this is just a wrapper on top of python-ldap (or pyldap), it would be nice to have a proper test suite for all this stuff.

@akkornel has been writing a generic set of syncrepl tests for the python-ldap project. Maybe take those, along with python-ldap's test harness, and implement that here. NOTE: We'd have to check license compatibility if we take the test harness code. The syncrepl tests were written by @akkornel, so that's safe to take.

Clean up demo script

Fairly self-explanatory. It needs comments and documentation.

syncrepl_delete TypeError

Is it safe to swap %d to %s?

  File "/usr/local/lib/python3.5/dist-packages/syncrepl_client/__init__.py", line 702, in poll
    all=1, timeout=3)
  File "/usr/local/lib/python3.5/dist-packages/ldap/syncrepl.py", line 417, in syncrepl_poll
    self.syncrepl_delete(sim.syncIdSet['syncUUIDs'])
  File "/usr/local/lib/python3.5/dist-packages/syncrepl_client/__init__.py", line 1012, in syncrepl_delete
    'does not exist!' % (uuid,)
TypeError: %d format: a number is required, not str

Badges!

Our README needs more badges! I'm thinking:

• Something to show Python supported versions.
• A link to PyPi.
• Maybe a Stanford and/or SRCC badge?

Travis and Coveralls ones would be added as part of #9. If a Coverity badge exists, it'll be added as part of #10.

Any other ones?

Match patch for ldap.TIMEOUT bug affecting python-ldap

python-ldap has a bug related to the handling of ldap.TIMEOUT. Until that bug is fixed, syncrepl_client won't really work on python-ldap.

Rather than waiting for the bug to be fixed, provide a patch that people can apply themselves.

This is a problem that I haven't been able to look into as much, so I might not target it at Version 1.

Make patch for UUID bug affecting pyldap

It seems like it's gonna be a while before the UUID issue gets fixed in the pyldap repository. So, let's create a patch that users can use to fix the problem themselves.

BaseCallback.record_add method signature does not match documentation

The docstring for BaseCallback.record_add indicates it should accept a cursor parameter but cursor is missing from the method signature. LoggingCallback.record_add does have cursor in the method signature.

Create custom exceptions

A number of custom exceptions need to be created and documented.

NameError in `record_rename` callback function

I got the following error when record_rename callback function is called:

Exception in thread Thread-1 (run):
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/root/.pyenv/versions/3.10.10/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/syncrepl_client-0.95.1-py3.10.egg/syncrepl_client/__init__.py", line 696, in run
  File "/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/syncrepl_client-0.95.1-py3.10.egg/syncrepl_client/__init__.py", line 603, in poll
  File "/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/ldap/syncrepl.py", line 383, in syncrepl_poll
    self.syncrepl_entry(dn, attrs, c.entryUUID)
  File "/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/syncrepl_client-0.95.1-py3.10.egg/syncrepl_client/__init__.py", line 1074, in syncrepl_entry
NameError: name '_Syncrepl__uuid_dn_map' is not defined

The error exists in the v.95.1 version. Looks like a self prefix is missing in del self.__dn_uuid_map[__uuid_dn_map[uuid]] in line 1074

pyasn1 0.3.1 causes syncrepl-client to return no records

When running with pyasn1==0.3.1 (the latest available), syncrepl-client appears to work, but it doesn't show any records.

Example:

python3 /usr/local/bin/syncrepl-client ldaps://ldap-uatmaster.stanford.edu:636/cn=people,dc=stanford,dc=edu?uid,suprivilegegroup?base?uid=ljlgeek?bindname=GSSAPI uat-syncs
Data files will be stored here: uat-syncs
Refresh-only mode will be used
CLIENT SETUP START
BIND COMPLETE!
    WE ARE: dn:uid=ljlgeek,cn=accounts,dc=stanford,dc=edu
CLIENT SETUP COMPLETE!
CLIENT LOOP START
COOKIE CHANGED: rid=000,sid=064,csn=20170621020008.854439Z#000000#000#000000;20170816225047.962399Z#000000#064#000000;20170816125603.382427Z#000000#065#000000
REFRESH COMPLETE!
BEGIN DIRECTORY CONTENTS:
END DIRECTORY CONTENTS
CLIENT LOOP END
    Loop Result: False
CLIENT LOOP COMPLETE!
CLIENT EXIT START
CLIENT EXIT COMPLETE!

In the above example, one record should've been returned.

Credit to @ljl-geek for finding this.

Call refresh_done even in refresh-only mode

This is meant to make things easier for the client. It would be nice if the refresh_done callback can universally mean "The refresh is done", even in refresh-only mode.

Modifying multiple records at a time do not trigger all callbacks

I have a FreeIPA server and I'm adding users and adding those users to groups.

FreeIPA does 3 things:

1. Creates the user.
2. Updates the group member attribute with the user dn.
3. Updates the user memberOf attribute with the group dn.

SyncRepl only sees 1 and 2. If I restart my client it will then see the "missing" event 3.

The scary part is IF I create another user without restarting the client, event 3 is permanently lost because syncrepl has moved to a newer cookie value even though it did not handle all of the events.

My best guess is that 2 and 3 are happening so quickly that syncrepl isn't handling it for some reason.

I'm running with REFRESH_AND_PERSIST and have confirmed this behavior with the logging callback.

Any advice?

Dependency for pyasn1<0.3.1 is no longer needed

The workaround for issue #18 is no longer needed. Python-ldap has a compatiblity patch for newer pyasn1 versions since 10 months:

python-ldap/python-ldap@dab08df

Could you please remove the dependency rule that requires pyasn1<0.3.1?

Greetings
Jan

syncrepl_delete is not defined

i know this code is more or less abandoned, but here's an exception for posterity:

File "/tmp/venv/lib/python3.5/site-packages/syncrepl_client/__init__.py", line 1301, in
    syncrepl_delete(old_uuid)
NameError: name 'syncrepl_delete' is not defined

I think it should be self.syncrepl_delete(old_uuid) in https://github.com/akkornel/syncrepl/blob/master/syncrepl_client/__init__.py#L1301

pyldap deprecated and merged back into python-ldap

The pyldap module has been merged back into python-ldap and deprecated. This should be reflected in the docs. Currently the docs are saying to use pyldap for python3, but it should now only say to use python-ldap.

Install demo script along with other stuff

The setup script should be able to install the demo script, along with the rest of syncrepl-client.

Depends on #6.

clone() should set synchronous & journal_mode

Doing some testing from multiple threads, other threads (that got an sqlite connection using client.db()) dont see the recent writes on the sqlite database.

To get a consistent behaviour, i had to add:

newbie.__db.execute('PRAGMA synchronous = NORMAL')
newbie.__db.execute('PRAGMA journal_mode = WAL')

in https://github.com/akkornel/syncrepl/blob/master/syncrepl_client/db.py#L179 - maybe that was just forgotten after 2b3ed43 ?

Missing scope in LDAP URL confuses upstream code

When we provide an LDAP URL that doesn't include a scope, we get a weird error from inside the LDAP code.

blargh:~ akkornel(p)$ ~/Library/Python/3.6/bin/syncrepl-client --nodir --persist 'ldaps://ldap-uatmaster.stanford.edu:636/cn=people,dc=stanford,dc=edu?suRegID,uid,suPrivilegeGroup??(&(uid=*)(suPrivilegeGroup=research-computing:*))?bindname=GSSAPI' zkarl
Data files will be stored here: zkarl
Refresh-and-persist mode will be used
CLIENT SETUP START
BIND COMPLETE!
	WE ARE: dn:uid=akkornel,cn=accounts,dc=stanford,dc=edu
Traceback (most recent call last):
  File "/Users/akkornel/Library/Python/3.6/bin/syncrepl-client", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/akkornel/git/syncrepl/syncrepl-client", line 117, in <module>
    else 0
  File "/Users/akkornel/git/syncrepl/syncrepl_client/__init__.py", line 425, in __init__
    attrlist=ldap_url.attrs
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/ldap/syncrepl.py", line 340, in syncrepl_search
    return self.search_ext(base, scope, **search_args)
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/ldap/ldapobject.py", line 791, in search_ext
    timeout,sizelimit,
  File "/Users/akkornel/Library/Python/3.6/lib/python/site-packages/ldap/ldapobject.py", line 294, in _ldap_call
    result = func(*args,**kwargs)
TypeError: an integer is required (got type NoneType)

Besides dealing with it, we should throw if the scope is missing from the LDAP URL.