Giter Site home page Giter Site logo

django-fixture-magic's Introduction

https://travis-ci.org/davedash/django-fixture-magic.svg?branch=master

Requirements

This package requires:

  • Python 2.7, 3.6
  • Django 1.8 - 2.1

Installation

You can get fixture-magic from pypi with:

pip install django-fixture-magic

The development version can be installed with:

pip install -e git://github.com/davedash/django-fixture-magic#egg=fixture-magic

For use in python3 install the following:

pip install future

fixture-magic adds two commands to manage.py therefore you should add it to your INSTALLED_APPS in settings.py:

INSTALLED_APPS = (
    ...
    'fixture_magic',
    ...
)

Usage

There are four commands. dump_object returns the json representation of a specific object as well as all its dependencies (as defined by ForeignKeys):

./manage.py dump_object APP.MODEL PK1 PK2 PK3 ... > my_new_fixture.json

Or:

./manage.py dump_object APP.MODEL --query '{"pk__in": [PK1, PK2, PK3]}' > my_new_fixture.json

Or you can get all objects with all dependencies by passing an asterisk:

./manage.py dump_object APP.MODEL '*' > my_new_fixture.json

You can now safely load my_new_fixture.json in a test without foreign key i errors.

By default, fixture magic will dump related fixtures to your model in your fixture. This can be disabled by passing the option --no-follow to dump_object. This is useful if your target database is already partially setup. Here is and example default output of dump_object:

./manage.py dump_object APP.Book
[
  {
      "model": "APP.Author",
      "fields": {
          "pk": 5,
          "name": "Mark Twain",
      }
  },
  {
      "model": "APP.Book",
      "fields": {
          "pk": 2,
          "title": "Tom Sawyer",
          "author": 5
      }
  }
]

Running with the --no-follow options yields:

./manage.py dump_object APP.Book --no-follow
[
  {
      "model": "APP.Book",
      "fields": {
          "pk": 2,
          "title": "Tom Sawyer",
          "author": 5
      }
  }
]
Note:The above example assumes that an Author with an ID of 5 exists in the target database.

The second command is merge_fixtures. This command takes several fixture files and does a simple de-dupe operation (based on model and pk) and returns a clean json file. This is helpful if you have multiple json fixtures that might have redundant data:

./manage.py merge_fixtures fixture1.json fixture2.json fixture3.json ... \
> all_my_fixtures.json

The third command is reorder_fixtures. This command takes a single file and several model names (in app.model format as they are specified in fixture files). This reorders your fixtures so the models you specifiy first show up in the fixture first. This is helpful if you tend to get foreign-key errors when loading models:

./manage.py reorder_fixtures fixture.json APP1.MODEL1 APP2.MODEL2 ... \
> ordered_fixture.json

Unspecified models will be appended to the end.

The fourth command is custom_dump. This reads a setting CUSTOM_DUMPS:

## Fixture Magic
CUSTOM_DUMPS = {
    'addon': {  # Initiate dump with: ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are the attributes/methods of the model that we wish to dump.
            'current_version',
            'current_version.files.all.0',
        ],
        'order': ('app1.model1', 'app2.model2',),
        'order_cond': {'app1.model1': lambda x: 1 if x.get('fields').get('parent_model1') else 0,
                        'app2.model2': lambda x: -1 * x.get('pk')},
    }
}

It runs the equivalent of dump_object on the dependents (which in turn pick up the primary object). The JSON dumps are then merged together. Very handy for dumping multi-dependent objects. dependents, order and order_cond are optional.

dependents: Defines additional properties/methods to dump the return values of. Magic will convert "current_version.files.all.0" to object.current_version.files.all()[0]

order: Specify an order in which objects should be dumped based on their model class. In the above example, all app1.model1 objects will preceed any app2.model2 objects, which will preceed any objects of any other model class.

order_cond: Specify an order to dump objects of one or more particular model classes. In the above example, all app1.model1 objects with a truthy self.parent_model1 attribute will come after any other app1.model1 object that does not have a truthy value for this attribute. A sort operation is called on the list of all objects of that model type, with the value associated with a model name being passed to the sort function as the key keyword argument. Keep in mind that the model objects will have already been serialized to a dictionary object prior to the sort operation.

django-fixture-magic's People

Contributors

andyzickler avatar athom09 avatar atten avatar carljm avatar chrfr avatar davedash avatar davitovmasyan avatar dineshs91 avatar duduklein avatar fourk avatar germanoguerrini avatar kevingrahamfoster avatar koryd-bixly avatar miigotu avatar momirza avatar nektor211 avatar orientalperil avatar pratyushmittal avatar punchagan avatar pykler avatar sobolevn avatar tony avatar troyharvey avatar xrmx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-fixture-magic's Issues

Issue while dumping a single object

I was trying to get a sample fixture for an object and got the following:

python manage.py dump_object "collector.TwitterData" 492784

/usr/local/lib/python2.7/dist-packages/fixture_magic/management/commands/dump_object.py:9: RemovedInDjango19Warning: The utilities in django.db.models.loading are deprecated in favor of the new application loading system.
from django.db.models import loading

CommandError: object_class must be provided in the following format: app_name.model_name
Try calling dump_object with --help argument or use the following arguments:
<[--kitchensink | -k] [--natural] [--query] object_class id [id ...]>

I'm using django 1.8 for this project

merge_fixtures should write to stdout

Just a consistency issue: dump_objects writes to stdout, while merge_fixtures doesn't.

I want to use the commands from functions, so capturing stdout is easy:

    with open("my_app/fixtures/test_fixture.json", 'w+') as f:
        call_command('dump_object', 'my_app.MyModel *list_of_some_pks, '-k', '-n', stdout=f)
        f.readlines()

Whereas trying to merge fixtures the same way fails (or I fail to capture the output):

with open("my_app/fixtures/test_fixtures_combined.json", 'w+') as f:
        call_command(
            "merge_fixtures",
            "my_app/fixtures/test_fixture_1.json",
            "my_app/fixtures/test_fixture_2.json",
            stdout=f
        )
        f.readlines()

Is this intended behaviour?

Unable to dump BinaryField

Error:

root@hzdocker2 ~/odi # docker run -i  -v /root/odi:/code jimmy927/odi-gh:master python3.9 manage.py  dump_object core.profilejsonbinary  --query '{"person_id": 1415}'
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
  File "/code/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/dist-packages/fixture_magic/management/commands/dump_object.py", line 144, in handle
    self.stdout.write(serialize(options.get('format', 'json'),
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/base.py", line 107, in serialize
    self.handle_field(obj, field)
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/python.py", line 49, in handle_field
    self._current[field.name] = self._value_from_field(obj, field)
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/python.py", line 46, in _value_from_field
    return value if is_protected_type(value) else field.value_to_string(obj)
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/fields/__init__.py", line 2388, in value_to_string
    return b64encode(self.value_from_object(obj)).decode('ascii')
  File "/usr/lib/python3.9/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'dict'
Waiting up to 5 seconds.
Sent all pending logs.
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Please update PyPI

Please bump the version number and make a new release to PyPI so the newest changes are retrievable.

Also README.rst says:

./manage.py dump_object APP.MODEL * > my_new_fixture.json

This should be written with quotes around the asterisk:

./manage.py dump_object APP.MODEL '*' > my_new_fixture.json

dump_object fails to parse command arguments correctly

The dump_object command is broken for me with Python 2.7, Django 1.9 on Windows 7 x64.

It falsely reads in the last argument as the model argument. Example:

$ python manage.py dump_object auth.User 1

will result in an options dict (see https://github.com/davedash/django-fixture-magic/blob/master/fixture_magic/management/commands/dump_object.py#L48) like:

{'ids': [],
 'kitchensink': False,
 'model': '1',
 'natural': False,
 'no_color': False,
 'pythonpath': None,
 'query': None,
 'settings': None,
 'traceback': False,
 'verbosity': 1}

As you can see, ids is empty and model got the id instead. It is always filled with the last argument for some reason. Maybe the parser behaviour changed from Python 2.6 to 2.7? I will try to find the reason.

A quick fix for me was to make the model and id arguments optional so I could do:

$ python manage.py dump_object --model=auth.User --ids=1

ImportError: cannot import name loading

File "/usr/local/lib/python2.7/dist-packages/fixture_magic/management/commands/dump_object.py", line 6, in
from django.db.models import loading
ImportError: cannot import name loading

I am getting the above error. I just upgraded to django 1.9.2 and I think the problem is with django. I tried importing loading it in the shell and got the same error.

Custom dump nested deps

Currently if you have multiple levels of dependencies, custom_dump command only goes one level deep through straight foreign keys and not reverse foreign keys. It goes from the parent to get the children, but will only get grandchild if it can access it directly from the parent.

It would be nice to be able to use something like

CUSTOM_DUMPS = {
    'addon': {  # Initiate dump with: ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are the attributes/methods of the model that we wish to dump.
            'current_version',
            'current_version.files.all.0',
            {
                'primary': 'previous_versions.all',
                'dependents': ['files.all']
            }
        ],
        'order': ('app1.model1', 'app2.model2',),
        'order_cond': {'app1.model1': lambda x: 1 if x.get('fields').get('parent_model1') else 0,
                        'app2.model2': lambda x: -1 * x.get('pk')},
    }
}

reorder_json does not ensure order

The reorder_json is very useful but python dict are not ordered by default or at least in python 2.7 (cf.: https://github.com/davedash/django-fixture-magic/blob/master/fixture_magic/utils.py#L7).

ipdb> bucket
{'tagging.tag': [], 'account.profile': [], 'account.user': [], 'document.document': [], 'sharing.sharing': []}
ipdb> models
['account.user', 'account.profile', 'tagging.tag', 'document.document', 'sharing.sharing']

May be I'm wrong but at this state the method is useless, or works sometimes. We should may be use OrderedDict for that: https://docs.python.org/2/library/collections.html#collections.OrderedDict

dump_object args parsing problem

Django==1.8.18
Running ./manage.py dump_object declarations.section 25976, getting error:

CommandError: object_class must be provided in the following format: app_name.model_name
Try calling dump_object with --help argument or use the following arguments:
 <[--kitchensink | -k] [--natural] [--query] object_class id [id ...]>

Investigated problem, and solved by defining named --ids argument (dump_object.py:30):

        parser.add_argument('--ids', dest='ids', default=None, nargs='*',
                            help='Use a list of ids e.g. 0 1 2 3')

1146, "Table 'db.table_name' doesn't exist" - error

I created a fixture using dump_object with --kitchensink option.
python manage.py dump_object --kitchensink tours.tour ids 400 > tours/fixtures/tours.json

I got the tours.json file. Now when I run tests, with the below command
./manage.py test -n It says the child_table_name does not exist

How to resolve this?

Could not dump objects until I modified dump_object.py as per comment

Everything I tried would yield the following error, even on internal Django models:

CommandError: object_class must be provided in the following format: app_name.model_name
Try calling dump_object with --help argument or use the following arguments:
 <[--kitchensink | -k] [--natural] [--query] object_class id [id ...]>

Then I tried the fix from this comment on closed issue #45 and after that I was able to dump objects successfully.

I don't understand what that args line is doing. I could submit a PR removing it but it would be good to know why it is there first.

Django version is 1.9.1.

Is this project still being developed?

I'm currently making some changes to meet my requirements. I have at present hard coded things, but was thinking about building it up to be configurable/have tests etc, but if the project is no longer being developed maintained, it wouldn't really be worth it. So is there an intent to keep developing?

merge_fixtures is having some problem parsing parameters with Python 3

Using the following Python and Django

Python 3.4.6
Django 1.10.7

When I run the merge_fixtures command

python manage.py merge_fixtures foo.json bar.json baz.json > all_fixtures.json

It shows the following error message:

usage: manage.py merge_fixtures [-h] [--version] [-v {0,1,2,3}]
                                [--settings SETTINGS]
                                [--pythonpath PYTHONPATH] [--traceback]
                                [--no-color]
manage.py merge_fixtures: error: unrecognized arguments: foo.json bar.json baz.json

I looked into it a little bit and it seems to be unhappy in parsing the command line parameters. I think it is unhappy here:

        parser = self.create_parser(argv[0], argv[1])
        options = parser.parse_args(argv[2:])

But I am a bit unfamiliar with what is expected here and didn't have time to fully investigate the issue. Do you have a hint where I should be looking?

Cannot install version from PyPi

Trying to install django-fixture-magic from pypi results in an error:

Downloading/unpacking django-fixture-magic (from -r requirements.txt (line 15))
  Downloading django-fixture-magic-0.0.2.macosx-10.6-i386.tar.gz
  Running setup.py egg_info for package django-fixture-magic
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
    IOError: [Errno 2] No such file or directory: '/home/evan/python-envs/plat/build/django-fixture-magic/setup.py'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

IOError: [Errno 2] No such file or directory: '/home/evan/python-envs/plat/build/django-fixture-magic/setup.py'

Installing direct from github using -e git works.

The machine is running python 2.5 on Debian Lenny.

dump_object fails on '*'

The dump_object command fails when receiving '*', succeeds when receiving individual IDs.

$ ./manage dump_object service.model '*'
/service/venv/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "django/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/service/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/service/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/service/venv/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/service/venv/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/service/venv/lib/python3.6/site-packages/fixture_magic/management/commands/dump_object.py", line 117, in handle
    add_to_serialize_list(objs)
UnboundLocalError: local variable 'objs' referenced before assignment

Python 3.6.3
Django==2.1
django-fixture-magic==0.1.3

Update PyPi

Howdy,
Looks like it has been a while since PyPi has been updated (May 2013). Can you update with the latest version so I can get access to the --query argument? Thanks!

New django adds a label arg, changes arguments to fixture magic

In zamboni we recently upgraded django:

 $ git submodule status src/django
 40e28df9e19bcf80525010df44fea9eeffec54b4 src/django (1.2.1-1774-g40e28df)

After that fixture magic did not work with this command:

./manage.py dump_object --kitchensink zadmin.ValidationResult 793

It displayed an error about trying to load 793 as a model and finally I realized that app label was getting passed in as an arg by default. This command got it to work:

./manage.py dump_object --kitchensink zadmin zadmin.ValidationResult 793

(the app label had to be doubled)

dump_object does not follow object relations

Environment: Django 3.0.6

In my case the app does not follow relations when exporting data with dump_object

Simplified models:

class Member(BaseModel):

    email = models.EmailField(unique=True)
class Subscription(BaseModel):

    member = models.ForeignKey(Member, on_delete=models.CASCADE)

When I execute python manage.py dump_object members.member --query '{"pk__in": [1]} for a Member with several Subscriptions I only get the Member model dumped:

[
{
    "model": "accounts.member",
    "pk": 1,
    "fields": {
        "email": "[email protected]",
    }
}
]

Pip installs wrong version from PyPi

Pip does find 0.1.4:

$ pip search django-fixture-magic | grep django-fixture-magic
django-fixture-magic (0.1.4)                - A few extra management tools to handle fixtures.

But when I try to install pinned:

$ pip install django-fixture-magic==0.1.4
Collecting django-fixture-magic==0.1.4
  Could not find a version that satisfies the requirement django-fixture-magic==0.1.4 (from versions: 0.0.2.macosx-10.6-i386, 0.1.4.macosx-10.13-x86_64, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.7, 0.0.8, 0.1.0, 0.1.1, 0.1.2, 0.1.3)
No matching distribution found for django-fixture-magic==0.1.4

And unpinned:

$ pip install django-fixture-magic
Collecting django-fixture-magic
Installing collected packages: django-fixture-magic
Successfully installed django-fixture-magic-0.1.3

I don't know why this happens, but it happens in any environment that I've tried. Can you fix this? Is there anything that I can do to help?

unexpected keyword argument 'use_natural_foreign_keys'

I added its entry(django-fixtur-magic) in "MY" requirements.txt file and also added 'fixture_magic' to the list of INSTALLED_APPS in MY application's settings.py.

Then I ran the command
./manage.py dump_object hmi.odeskmenutask 1901783 > test_fixture.json.

1901783 is its primary key.
This causes an error like this:
TypeError: init() got an unexpected keyword argument 'use_natural_foreign_keys'.

query using the call_command

Could you advise how to use the dump_object command with query programatically using the call_command ?

Consider this
from django.core.management import call_command
q = UploadedFile.objects.filter(user=user).values('pk')
call_command('dump_object', 'validate.UploadedFile', '--no-follow', query=''{"pk__in": ' + str(list(q)) + '}'')

Do sent seem to work ? Any solutions ?

error: no such option: --query in latest pip version of django-fixture-magic

It seems django-fixture-magic is not updated after last master release. Version 0.0.5 doesn't support --query option. Here is console output:

(.virtualenv)alx@x1:~/sermonis-server$ pip install django-fixture-magic --upgrade
Downloading/unpacking django-fixture-magic from https://pypi.python.org/packages/source/d/django-fixture-magic/django-fixture-magic-0.0.5.tar.gz#md5=23ed02c964513d2a6fe5235ca49d8855
  Downloading django-fixture-magic-0.0.5.tar.gz
  Running setup.py egg_info for package django-fixture-magic

Installing collected packages: django-fixture-magic
  Found existing installation: django-fixture-magic 0.0.4
    Uninstalling django-fixture-magic:
      Successfully uninstalled django-fixture-magic
  Running setup.py install for django-fixture-magic

Successfully installed django-fixture-magic
Cleaning up...
(.virtualenv)alx@x1:~/sermonis-server$ ./manage.py dump_object sermonis.dictionary.Translation --query '{"pk__lt":100}' > 100translations.json
Usage: ./manage.py dump_object [options] <[--kitchensink | -k] object_class id1 [id2 [...]]>

Dump specific objects from the database into JSON that you can use in a fixture

./manage.py: error: no such option: --query

merge_fixtures is having some problem parsing parameters

python:2.7.5
version:0.1.3
errror:

usage: manage.py merge_fixtures [-h] [--version] [-v {0,1,2,3}]
                                [--settings SETTINGS]
                                [--pythonpath PYTHONPATH] [--traceback]
                                [--no-color]
manage.py merge_fixtures: error: unrecognized arguments:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.