trixterfilm/trackteroid

Declarative, object-oriented wrapper for Ftrack queries. Powerful functional-style interactions with resulting collections.

trackteroid's Issues

shortest path is not always being taken for schema relationships

Problem Statement

When parsing relationships from the schema it is not always considering the shortest path.

Expected Behavior

Instead of going through a redirection of fields resulting in the same collection it should consider the shortest possible path. Additional logic might be required to decide on one relationship path incase the pathlength is identical for multiple possible relationships.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main
• Ftrack Python API version: Any (unrelated)
• Ftrack Server version: Any (unrelated)

Reproducible Steps

This should reveal the problem:

from trackteroid import *

Query(Shot).get_first()
print(Shot.relationship[Note])

print(Query(TypedContext).by_name(ObjectType, "Shot"))

print(Query(TypedContext).by_name(Project, "name"))

Workaround

A different relationship can be provided within the configurable RELATIONSHIP_RESOLVER value that would override the relationship inferred from the schema.

`EntityCollection.children` property should not fetch

Problem Statement

Accessing the children attribute is producing a query to update children.object_type.name. This is a leftover from the past and should be removed.

Expected Behavior

Accessing children should not perform queries as this might be redundant and brings performance costs.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main
• Ftrack Python API version: -
• Ftrack Server version: -

Setup the "Publish Python Package" Github action

Goal

Setup the "Publish Python Package" Github action.

Motivation

When doing a new release via Github we should automatically publish it as a Python package to PyPI.

create entity classes for all available declarations

Goal

In order to use projections like this Query(AssetVersion).get_first(projections=[SequenceComponent.name]) we need to have them implemented as actual entity subclass.

Motivation

Currently, there is a difference between a ForwardDeclare and an Entity subclass as they don't provide the same features. Similar to what we are doing for TypedContext subclasses already we should automatically create Entity subclasses for all available declarations automatically.

Considerations

An alternative approach to consider would be to introduce a factory system to provide custom types or to extend existing types. But this doesn't exist currently.

Risks

There are no obvious risks.

user config workflow should be more clear

Problem Statement

According to the docs, two files have to be created - trackteroid_user_config.py and trackteroid_relationships.json, but it does not state where to put them and where to import/call them. The RELATIONSHIPS_RESOLVER is being called in two places - trackteroid.query.__init__ and trackteroid.query.query`, which is confusing - one of them should be removed.
With a bit of digging, all this information can be concluded from the code but that is time consuming and an inconvenience for the user.

Expected Behavior

To avoid the user having to find where to place the user files and how to hook them in without diving into the code, the user files could be included in an empty form / containing the default RELATIONSHIPS_RESOLVER from configuration.py (where it can be removed). The RELATIONSHIPS_RESOLVER is then created in one place only and is called only once. The docs can then point to these files directly and all the user has to do is copy/paste the example and all is working.

relative terminator is referring to wrong type

Problem Statement

This code resolves the projections incorrectly and fails:

from trackteroid import *

empty_collection = Query(Asset).by_name("ThisDoesntExist123").get_first()
entity_collection = Query(Asset).get_first(projections=[ComponentLocation.resource_identifier])

for collection in [empty_collection, entity_collection]:
    result = collection.ComponentLocation.resource_identifier
    if result:
        print("Result: ", result)
    else:
        print("NO Result", result)

    result = collection.versions.get(ComponentLocation).resource_identifier
    if result:
        print("Result: ", result)
    else:
        print("NO Result", result)

Expected Behavior

This code shouldn't fail and correctly resolve the underlying relationship.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main/v0.1.0rc4
• Ftrack Python API version: 2.5.0
• Ftrack Server version: -

The problem is that the underlying _get_relatives function still refers to the original collection entity type which is Asset instead of referring to AssetVersion retrieved via the versions attribute.

__setattr__ unable to retrieve source if attribute produces an EmptyCollection

Problem Statement

Setting an attribute like this fails if the attribute that needs to be updated produced an EmptyCollection:

from trackteroid import *

collection1 = Query(AssetVersion).get_all(limit=1)
collection2 = Query(AssetVersion).get_all(limit=1, offset=1)

collection1.uses_versions = collection2

print(
    f"collection1 id: {collection1.id[0]}\n"
    f"collection2 id: {collection2.id[0]}\n"
    f"collection1 uses ids: {collection1.uses_versions}\n"
    f"collection2 used in ids: {collection2.used_in_versions}\n"
)

Expected Behavior

This example should work.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main/v0.1.0rc3
• Ftrack Python API version: 2.5.0
• Ftrack Server version: -

Reproducible Steps

See the example which leads to the following error:

... line 577, in __setattr__
    key, collection = attribute_value._source
ValueError: not enough values to unpack (expected 2, got 1)

`KeyValueMappedCollectionProxy` attribute access always returning "metadata" attribute value

Problem Statement

When accessing an attribute that holds a KeyValueMappedCollectionProxy type currently the value of the metadata is being returned.

Expected Behavior

The actual attribute value should be returned and not necessarily metadata.

Context

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main/v0.1.0rc3
• Ftrack Python API version: 2.5.0
• Ftrack Server version: -

Reproducible Steps

from trackteroid import *

print(Query(Sequence).get_all(projections=["custom_attributes", "metadata"]).custom_attributes)

Workaround

Accessing the actual data via wrapped ftrack entities.

Removal of e-Tugra root certificate

Problem Statement

fix the vulnerability detected in requirements.txt
https://github.com/TrixterFilm/trackteroid/security/dependabot
 

validate `RELATIONSHIPS_RESOLVER` value

Goal

Trackteroid should validate if the returned value of the RELATIONSHIPS_RESOLVER follows the expected schema.
This should cover

• datatypes check
• included "entities" entry (also for overrides)
• included "name" entry for overrides

Motivation

The idea is to make the configuration process of the API less error-prone and easier to set up.

Create a tox environment for testing

Goal

To create a tox environment on the project to enable easy testing of the project (implemented by #10 )

Motivation

Testing with multiple environments can be very inconvenient. Luckily, projects like tox make it much more convenient.

unable to source user configuration

Problem Statement

Sourcing a user configuration doesn't work at the moment as importlib.load_source doesn't exist anymore in Python >=3.7

Expected Behavior

As user configuration should be sourced and the provided overrides be applied.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main
• Ftrack Python API version: unrelated
• Ftrack Server version: unrelated

Reproducible Steps

Provide a user configuration file via TRACKTEROID_CONFIGURATION envvar and import trackteroid.

Workaround

No workaround.

`children`, `parent`, `descendants` and `ancestors` should coerce to `TypedContext`

Goal

When retrieving collections using children, parent, descendants, and ancestors, the resulting collection should be a TypedContext collection, and the type coercion should be performed automatically.

Motivation

Currently, the type of the collection is retrieved by inspecting the first element in the collection. As with the mentioned attributes, there s no guarantee it will produce a collection of the same type we have to coerce it to its actual parent type which is TypedContext.

Risks

This is breaking change and requires filtering all the time.

Collection attribute empty if accessed before query

Problem Statement

When accessing a collection attribute before it's been queried, that attribute becomes inaccessible. We have tracked it down to a bug in the Ftrack API, but because trackteroid internally uses the Ftrack API, this issue is also affecting it. This issue is to come up with a workaround for it while Ftrack gets it fixed.

Expected Behavior

Even if the initial collection entity has no data, after querying that field it should be available.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: 3.11
• Trackteroid version: master
• Ftrack Python API version: v2.5.0
• Ftrack Server version: 4.12.0.current.16.4248

Reproducible Steps

The pure ftrack reproducible code:

import ftrack_api

session = ftrack_api.Session(auto_populate=False)

# We query an Asset on ftrack that has at least one version
asset = session.query("Asset where versions.id like '%'").first()
# Print the list of versions. Because we haven't included the "versions" projection, this will
# expectedly print an empty list. This line is what triggers the bug: When we access asset["versions"],
# in AbstractCollectionAttribute.get_value:L437 there's a piece of code that checks whether the value is
# NOT_SET and in that case it sets it's local value to None, which here translates to an empty collection.
# The rationale of this is stated on a comment on said piece of the source code, the users should be able
# to use the same collection API whether there's any data or not, this desired behaviour is understandable,
# but unfortunately we have unknowingly set the local value of the entity's ftrack attribute storage to an
# empty collection without recording the operation, meaning that now the local value is favored over the
# remote value and will always be returned even if later on we populate the remote value
print(list(asset["versions"]))
# Now we query that same asset but this time we want to fetch the "versions" field.
asset = session.query(f"select versions from Asset where id is {asset['id']}").one()
# As mentioned before, this will now perform a lookup of the ftrack attribute storage dict, find a local value
# and think that that local value is the correct one, not being able to distinguish between an actual user created
# local value and the placeholder collection the API has set automatically. While the versions are properly contained
# in the remote attribute storage key, the empty collection from the local will be returned here.
print(list(asset["versions"]))

# Now to properly visualize the bug, we can create a new session, devoid of any local overrides and run the same
# query again. This time, because there's no local value already set, and we are actually retrieving that data we
# can see how the asset now returns its actual list of versions.
session = ftrack_api.Session(auto_populate=False)
asset = session.query(f"select versions from Asset where id is {asset['id']}").one()
print(list(asset["versions"]))

And by extension, this also happens on trackteroid:

from trackteroid import Query, Asset, AssetVersion, SESSION

asset_collection = Query(Asset).by_id(AssetVersion, "%").get_first()
print(asset_collection.versions.id)

asset_collection = Query(Asset).by_id(AssetVersion, "%").get_first(projections=["versions"])
print(asset_collection.versions.id)

SESSION.reconnect()
asset_collection = Query(Asset).by_id(AssetVersion, "%").get_first(projections=["versions"])
print(asset_collection.versions.id)

Workaround

Using the logic implemented in Ftrack's __getitem__ method within the entities triggers the issue, but individually getting the local or remote values of the internal does not. Therefore a good workaround would be to ensure that when accessing the internal Ftrack entities, instead of using entity[attribute] or entity.get(attribute) we do something like this:

attribute = entity.attributes.get(item)

local_value = attribute.get_local_value(entity)
remote_value = attribute.get_remote_value(entity)

if local_value is not ftrack_api.symbol.NOT_SET:
    return local_value

return remote_value

unable to set attribute if attribute is relationship shortcut

Problem Statement

Setting attributes won't work if the attribute is a relationship shortcut. This results in a KeyError.

Expected Behavior

This code will work and the task will be set accordingly

from trackteroid import (
    AssetVersion,
    Task,
    Query
)

av_collection = Query(AssetVersion).get_first(projections=[Task])
task_collection = Query(Task).get_first()

av_collection.Task = task_collection

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main
• Ftrack Python API version: 2.5.0
• Ftrack Server version: unrelated

Reproducible Steps

See expected behavior.

Workaround

Use the full relationship attribute access.

defer session initialization

Goal

Initialize a session when actually needed and not on module import.

Motivation

Currently, the SESSION singleton will be created when importing trackteroid which is not ideal as it results in an ImportError if a connection can not be established. Instead, the connection attempt should be made when actually needed.

Considerations

As our Session class already acts as a delegate for the Ftrack Python API Session we might be able to track and establish a connection on any self._session attribute access.

Port and rework tests

Goal

Add tests.

Motivation

The proprietary predecessor of Trackteroid had good test coverage. We have to identify if those tests can be ported and potentially rework them, but obviously, we need tests.

Considerations

We want to consider going fully with pytest instead of unittest.

tests failing

Problem Statement

two of the tests are failing, namely
test_session.py -> test_get_cached_collections
test_authoring.py -> test_create_note

• Win/Linux
• Python 3.7
• v0.1.0rc5
• 2.5.0
• 4.12.0

`EntityCollection.map(predicate)` should produce a list

Problem Statement

EntityCollection.map(predicate) produces a generator and not a list anymore.

Expected Behavior

EntityCollection.map(predicate) should produce a list and not a generator object, as this was the intended behavior and correlates to the current behavior of the group_and_map method.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main

Reproducible Steps

1. Produce a collection via query
2. print the result of collection.map(lambda _: True).

refactor ad-hoc entity class sourcing

Goal

Get rid of sourcing entity classes like this getattr(importlib.import_module("..entities", __name__), entity_type.__name__).

Motivation

We have more and more use cases where we need to retrieve the actual entity class and this is an ugly approach.

Considerations

We could potentially expose a proper type map on the session instance that allows convenient access to entity classes.

add a `apply` method on EntityCollection

Goal

Add a apply(predicate, attribute_name=None) method on EntityCollection. This method should be responsible for assigning a generated value to a given attribute on all collection items or the collection items of the caller directly.

Motivation

Single-entity collections can not be assigned to a target collection that contains multiple entities. The same is true for primitive data types. This leads to the situation that if you want to assign a single value or single entity collection to multiple receivers you'd have to loop as we don't want to change this explicit concept.

Nevertheless, the current expected user approach:

for single_collection in multiple_collection:
    single_collection.some_attr =  single_collection.another_attr[0] + "_edited"

could be streamlined towards:

multiple_collection.apply(lambda c: c.another_attr[0] + "_edited", "some_attr")

This new method should also be able to put a generated value directly onto the caller collection when no attribute is provided.

assetversion_collection.Task.Status.apply(lambda avc: status_collection)

allow direct SCHEMA import

Goal

As a user, I want to be able to directly import the SCHEMA object from Trackteroid as I can do with Query and SESSION.

Example:

from trackteroid import (
    AssetVersion,
    Query,
    Sequence,
    SCHEMA
)

Query(AssetVersion, schema=SCHEMA.custom).by_name(Sequence, "foobar").get_all()

Motivation

This ensures consistency with the imports of all other relevant classes and objects.

`uses_versions` and `used_in_versions` not updated bidirectionally

Problem Statement

Updating used_in_versions should be reflected in uses_versions of the assigned collection and vise versa.

Expected Behavior

Currently, the operation is recorded properly and can be updated on the server when committing but it is not reflected within the local cache.

Context

• OS: Any
• Python version: 3.x
• Trackteroid version: main
• Ftrack Python API version: 2.5.0
• Ftrack Server version: -

Reproducible Steps

This code can be used to reproduce the issue:

from trackteroid import *

collection1 = Query(AssetVersion).get_all(limit=1, projections=["used_in_versions", "uses_versions"])
collection2 = Query(AssetVersion).get_all(limit=1, offset=1, projections=["used_in_versions", "uses_versions"])

print(
    f"collection1 id: {collection1.id[0]}\n"
    f"collection2 id: {collection2.id[0]}\n"
    f"collection1 uses ids: {collection1.uses_versions.id}\n"
    f"collection2 used in ids: {collection2.used_in_versions.id}\n"
)
# output: 
# collection1 id: 029a25ce-8b60-11eb-bdb7-c2ffbce28b68
# collection2 id: 082158fc-6591-11ed-a73a-92ba0fc0dc3d
# collection1 uses ids: ['1fe0e8ae-6596-11ed-a73a-92ba0fc0dc3d']
# collection2 used in ids: EmptyCollection[AssetVersion]

collection1.uses_versions = collection2

print(
    f"collection1 id: {collection1.id[0]}\n"
    f"collection2 id: {collection2.id[0]}\n"
    f"collection1 uses ids: {collection1.uses_versions.id}\n"
    f"collection2 used in ids: {collection2.used_in_versions.id}\n"
)
# output: 
# collection1 id: 029a25ce-8b60-11eb-bdb7-c2ffbce28b68
# collection2 id: 082158fc-6591-11ed-a73a-92ba0fc0dc3d
# collection1 uses ids: ['082158fc-6591-11ed-a73a-92ba0fc0dc3d']
# collection2 used in ids: EmptyCollection[AssetVersion]

Workaround

To work with properly reflected data committing and re-querying should reflect the attribute changes.

prevent certain operations on collections when using different sessions and/or schemas

Goal

Certain operations should be prevented between collections if those are using different session objects and/or schemas.

Motivation

As each session maintains its own cache we need to prevent the creation of collections that are based on different sessions. The same is true for different user-provided schemas, as these are driving the attribute resolution.

The following methods need to consider a check:

• __setattr__ (also used by apply)
• union
• difference
• symmetric_difference
• intersection

configurable default projections

Goal

Make default projections configurable.

Motivation

Currently, entities provide a list of default projections, hardcoded for each implemented entity. This is not necessarily reflecting a user's requirements or expectations, so it should be configurable.

Considerations

It could be considered to use Ftrack's factory mechanism to override default projections and not provide any additional default projections within Trackteroid.

Risks

If the API is in use and default projections would change for codebases that already expect and rely on certain of them this might be a breaking change and code can break.

linking inputs/outputs not reflected in local attribute

Problem Statement

While linking properly creates Link objects and committing to the database works fine the changes aren't reflected on the local attributes incoming_links, and outgoing_links.

Expected Behavior

Extending and removing links should be reflected in the beforementioned attributes on the collection.

Context

Provide any relevant background information or context that may help in understanding the issue.

• OS: Any
• Python version: Any 3.x
• Trackteroid version: main/v0.1.0rc3
• Ftrack Python API version: 2.5.0
• Ftrack Server version: -

Reproducible Steps

from trackteroid import (
    Query,
    AssetBuild,
    SESSION
)

assetbuild_collection = Query(AssetBuild).get_all(
    limit=3,
    projections=[
        "name",
        "incoming_links.from",
        "incoming_links.from.name",
        "outgoing_links.to",
        "outgoing_links.to.name"
    ]
)
print(f"Link to add: {assetbuild_collection[0].name}")
print(f"Currently linked: {getattr(assetbuild_collection[1].incoming_links, 'from').name}")
print(f"Local attribute holds: {getattr(assetbuild_collection[1].link_inputs(assetbuild_collection[0]).incoming_links, 'from').name}")

Workaround

Retrieving the newly created link objects can be done by committing the changes first and querying again.