andrewdodd/django-nsync

A simple way to keep your Django Model data 'n-sync with N external systems.

django-nsync

Django NSync provides a simple way to keep your Django Model data 'n-sync with N external systems.

Features

Includes:

• Synchronise models with data from external systems
  • Create, update or delete model objects
  • Modify relational fields
  • Allow multiple systems to modify the same model object
• CSV file support out of the box
  • Ships with commands to process a single CSV file or multiple CSV files
• No need for more code
  • Nsync does not require you to inherit from special classes, add 'model mapping' objects or really define anything in Python

Not-included:

• Export (to CSV or anything else for that matter)
  • There are other packages that can do this
  • Why do you want the data out? Isn't that what your application is for? ;-)
• Admin integration
  • There isn't much to this package, if you want to add the models to your admin pages it is probably better if you do it (that's what I've done in my use case)

Not-yet included:

• Other file formats out of the box
  • Love it or hate it, CSV is ubiquitous and simple (its limitations also force simplicity)
  • The CSV handling part is separated from the true NSync part, so feel free to write your own lyrics-from-wav-file importer.
• Intricate data format handling
  • E.g. parsing date times etc
  • This can be side-stepped by creating @property annotated handlers though (see the examples from more info)

Documentation

The full documentation is at https://django-nsync.readthedocs.org.

Credits

Tools used in rendering this package:

• Cookiecutter Used to create the initial repot
• cookiecutter-pypackage Used by Cookiecutter to create the initial repo

For helping me make sense of the python pacakging world (and the bad practices codified in some of the tools/blogs out there):

• Hynek Schlawack Whose blog posts on packaging Python apps etc were indispensible
• Ionel Cristian Maries (sorry, too lazy for unicode) Whose blog post on python packaging was also indispensible

django-nsync's Issues

Excessive deletions

It appears that the changes for #1 caused objects to constantly be deleted and immediately recreated with the same data. However, because CASCADE is the default on_delete behaviour, this will cause related objects to also be deleted.

Update the documentation

Update docs for:

• New match_on post-fix feature
• Many-to-many relations

Add better options for match_on

I need to be able to match on OR relationships too.

In particular, if there are 2 attributes in the data to be synchronised but only one is used for the match_on, there is a possibility you will see DB integrity errors (as you try to create/update an object to have the same value for a unique field).

'' value has an invalid date format. It must be in YYYY-MM-DD format.

The field is set as not mandatory:
PubDate = models.DateField(blank=True, null=True)

But when I try to import a row with an empty cell I get the error:
'' value has an invalid date format. It must be in YYYY-MM-DD format.

Import files programmatically

Can't find in te documentation how to import files programmatically.

FileField is not supported

It is not possible to import a file using a FileField

Allow multiple match fields

Currently the 'match_field_name' only allows the 'selection' of the target model object through a single attribute/field. This should be extended to allow many fields to be used, similar to the related_to links.

Implementation suggestion: The fields should be separated by a character (e.g. space) that is illegal in Django attribute/field names, but preferably not a comma, tab, semi-colon or any other common delimiter of files (i.e. so it doesn't conflict with CSV / TSV etc).

Update fails if match fields have changed

If Nsync is 'updating' objects but their 'match fields' change, it does not succeed (and if both create & update actions are specified, it will create 2 objects).

A common occurrence of this is if the sync data is being produced from a database and an in-row update occurs which changes the match fields but leaves the 'external key' (i.e. an SQL 'UPDATE ... WHERE ...' statement).

E.g. A person table might look like this:

ID (a DB sequence)	Employee Number	Name
10123	EMP001	Andrew Dodd

This could be used to produce an Nsync input CSV like this:

external_key	action_flags	match_on	employee_number	name
10123	cu*	employee_number	EMP001	Andrew Dodd

This would result in an "Andrew Dodd, EMP001" Person object being created and/or updated with an ExternalKeyMapping object holding the '10123' id and a link to Andrew.

If Andrew became a contractor instead of an employee, perhaps the table could be updated to look like this:

ID (a DB sequence)	Employee Number	Name
10123	CONT999	Andrew Dodd

This would then produce an Nsync input CSV like this:

external_key	action_flags	match_on	employee_number	name
10123	cu*	employee_number	CONT999	Andrew Dodd

At the moment, Nsync will create a new Andrew Dodd object (in fact, there will be 2 Andrew Dodds, each with a different employee id). This is despite the 'external key' remaining unchanged.

Nsync should use the ExternalKeyMapping object if it is available instead of relying on the 'match fields'.

match_on referential fields

I can't find in the documentation how to match a record on an referential field.

Gregorio

Feedback on progress of the import

Hi,
I have some big imports to do (programmatically) and it would be very useful to have a function that gives the feedback on progress of the import.

Is there something that can help me?

Gregorio

['“” value has an invalid date format. It must be in YYYY-MM-DD format.']

Add Many-to-many support

Need to add many-to-many support for related fields (essential for managing grouping that is optional).