iwoca/django-deep-collector

A tool to collect every object related to a given object.

django-deep-collector's Issues

PyPI upload failed

Travis' deploy pipelines are failing https://travis-ci.org/github/iwoca/django-deep-collector/jobs/765482723

Probably some credentials or configuration issue

How to handle big dump?

Hello,

First of all, thanks for this project, it looks very promising. Sorry if my question seems stupid, I'm not a really a developer.

I'm facing an issue while try to collect a lot of data. My server quickly run out of memory and the process get killed.
I'm wondering if it would be technically possible with the current code architecture too, at some point, save the model data to file "on the fly" without breaking the whole logic? If yes, where would be the best place to do so?

Thanks for your work and your answer.

Regards

DeWaRs.

support django >= 2.0 ?

Hi,

Does this package still not support Django 2.0 ?
I have following error with my environment.

>>> ip = Ipaadress.objects.get(id=1)
>>> ip
<Ipaddress: 192.168.1.1>
>>> collector = RelatedObjectsCollector()
>>> collector.collect(ip)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/kobayashi/.virtualenvs/foo/lib/python3.6/site-packages/deep_collector/core.py", line 268, in collect
    children = self._collect(parent, obj)
  File "/Users/kobayashi/.virtualenvs/foo/lib/python3.6/site-packages/deep_collector/core.py", line 323, in _collect
    self.post_collect(obj)
  File "/Users/kobayashi/.virtualenvs/foo/lib/python3.6/site-packages/deep_collector/core.py", line 347, in post_collect
    if isinstance(field, ForeignKey) and not field.unique and field.rel.to == type(self.root_obj):
AttributeError: 'ForeignKey' object has no attribute 'rel'

Here is my setup.

$ pip freeze |grep Django
Django==2.0.5

$ pip freeze |grep deep-collector
django-deep-collector==0.4.2

Removing debug logs

Debug logs are still in main code:

• Replace it with a report logic,
• Remove all traces of ASCII art for displaying purpose. It can be done, but only from report data, not in the core logic.

Django 3

With django 3, the lib is not working anymore. Is there a chance for an upgrade?

The issue is related with from django.utils import six

get_fields() compatibility change have introduced a bug

The changes I introduced to remove deprecation warnings in a875cde have introduced a bug - when running the tests in a project that depends on django-deep-collector I get the following error:

  File "/home/pete/.venvs/test/local/lib/python2.7/site-packages/deep_collector/core.py", line 141, in clean_by_fields
    field_accessor = get_field_fn(field)
  File "/home/pete/.venvs/iwoca-django/local/lib/python2.7/site-packages/deep_collector/core.py", line 411, in <lambda>
    lambda x: x.get_accessor_name(), self.EXCLUDE_RELATED_FIELDS)
AttributeError: 'OneToOneField' object has no attribute 'get_accessor_name'

This happens with 0.4.1 but not 0.3.1 or 0.4.0.

ALLOWS_SAME_TYPE_AS_ROOT_COLLECT and indirect relations

Hi!
I noticed the wrong logic of ALLOWS_SAME_TYPE_AS_ROOT_COLLECT options.
For example we have Users (id, name), Organization (id, name, owner) and Organization_has_user(id, userId, groupId, date_added)
Given Organization owned by User (1, SomeUser)
1 | Some_Title | 1
And Organization_has_user has for example 3 rows
1 | 1 | 1 | 2018-04-26
2 | 2 | 1 | 2018-04-26
3 | 3 | 1 | 2018-04-26
Let's collect the data for User.objects.get(id=1).
When we reach Organization and then Organization_has_user relations it notices relations to User object (same type as root) I see warning about the same type as root objects, and this bit of code

        if not self.ALLOWS_SAME_TYPE_AS_ROOT_COLLECT:
            for field in self.get_local_fields(obj):
                if isinstance(field, ForeignKey) and not field.unique and field.rel.to == type(self.root_obj):
                    setattr(obj, field.name, self.root_obj)

Substitutes all user relations with self.root_obj (our given user with id=1) overriding actual relations so in

collector.get_json_serialized_objects()

We will see all Organization_has_user instances related to User(id=1) which isn't really correct.

{
  "model": "organizationhasuser",
  "pk": 2,
  "fields": {
     "id": 2
     "user": 1
  }

Is it actually a bug or an expected behaviour?
Is it maybe worth checking not for the same type but for the same object (by pk or something else)
As a quickest solution I've just set the ALLOWS_SAME_TYPE_AS_ROOT_COLLECT to True.

How to collect only dependent objects?

Thanks for creating this project.
It works really well.

Let me briefly explain my use case: Before deleting an object A I want to show a preview of all dependent objects that will be affected by the deletion (depending on the on_delete parameter).
In terms of the README, I am interested in all 'related' fields but not the 'direct' fields.

I have an existing code base with a large number of classes that makes excluding all individual fields via EXCLUDE_DIRECT_FIELDS cumbersome.
Currently, I am patching the get_local_objs() method in https://github.com/iwoca/django-deep-collector/blob/0.6.0/deep_collector/core.py#L368 to always return an empty list [].

Is there a way to disable collecting objects via 'direct' fields?
Is this a valid use case of this project or can you recommend another solution?

Collecting Generic Foreign Key

Hi,

Apparently DDC does not collect generic foreign key :/

Is it something you plan to support ? :)

Deprecated warning for get_all_related_objects and get_all_related_m2m_objects_with_model

Warnings generated in Django 1.9 to clean for 1.10:

.../django-deep-collector/deep_collector/core.py:408: RemovedInDjango110Warning: 'get_all_related_objects is an unofficial API that has been deprecated. You may be able to replace it with 'get_fields()'
  return self.clean_by_fields(obj, obj._meta.get_all_related_objects(),

.../django-deep-collector/deep_collector/core.py:412: RemovedInDjango110Warning: 'get_all_related_m2m_objects_with_model is an unofficial API that has been deprecated. You may be able to replace it with 'get_fields()'
  return self.clean_by_fields(obj, obj._meta.get_all_related_m2m_objects_with_model(),

Error raised while collecting objects

AttributeError
'NoneType' object has no attribute '_meta'

django/core/handlers/base.py in get_response at line 109
  collector.collect(user)
deep_collector/utils.py in collect at line 263
  children = self._collect(parent, obj)
deep_collector/utils.py in _collect at line 282
  local_objs = self.get_local_objs(obj)
deep_collector/utils.py in get_local_objs at line 337
  for field in self.get_local_fields(obj):
deep_collector/utils.py in get_local_fields at line 317
  concrete_model = obj._meta.concrete_model

Django 1.9 compatibility

There is an issue with Django 1.9, probably simple to fix

from django.utils.deprecation import RemovedInDjango19Warning
ImportError: cannot import name RemovedInDjango19Warning

AttributeError: 'RelatedObjectsCollector' object has no attribute 'get_all_related_objects'

No matter what branch I try, this error pops up on Django version 1.9.4:

---

$] python manage.py shell

from deep_collector.core import RelatedObjectsCollector
from django.contrib.auth.models import User

user = User.objects.all()[0]
collector = RelatedObjectsCollector()
collector.collect(user)

:385: RemovedInDjango110Warning: 'get_all_related_objects is an unofficial API that has been deprecated. You may be able to replace it with 'get_fields()'
  return self.clean_by_fields(obj, obj._meta.get_all_related_objects(),

:389: RemovedInDjango110Warning: 'get_all_related_m2m_objects_with_model is an unofficial API that has been deprecated. You may be able to replace it with 'get_fields()'
  return self.clean_by_fields(obj, obj._meta.get_all_related_m2m_objects_with_model(),

related_objects = collector.get_all_related_objects()

AttributeError: 'RelatedObjectsCollector' object has no attribute 'get_all_related_objects'

---

Last but not least that I tried was 'develop' branch, as well as 'django_19_support':

pip install -e git+https://github.com/iwoca/django-deep-collector.git@develop#egg=django-deep-collector