Selective Sync

[getavalon]

Selective Sync

[mottosso]

With #31 and #25 in place, this can now move forward.

At the moment, the artist can browse and start work on a remote project via the Launcher. The hierarchy will be created in an identical fashion to others working at other locations on the same project.

Currently, attempting to load an asset not currently available locally, it simply fails.

What should happen, is a dialog or button with the option to download this and dependent assets.

Implementation

There are two challenges involved here.

1. How does a server broadcast files?
2. How does a client receive them?

Server

We need some way of a producer of content, like the Mindbender studio, to enable others to download files from it. The same goes for when another studio or freelancer produces content that the Mindbender studio needs to use.

Spontaneously, I'd imagine a simple file-server like Flask. Each party can advertise themselves via their public IP.

{
  "type": "version",
  "locations": ["24.51.234.1:21011"],
}

Where the first location is where the file was first produced. Anyone downloading the file, assuming he is also capable of broadcasting files, then appends to this list.

1. Mindbender Gothenburg produces /hulk/Bruce/modelDefault/1/ma
2. Mindbender Brazil downloads this file
3. The file is now available from two locations.

{
  "type": "version",
  "locations": ["24.51.234.1:21011", "194.51.2.1:21011"],
}

For this to happen, there needs to be an actively running server where anyone is producing content. The spontaneous solution to this is to imbue the Launcher with this ability. So long as the Launcher is running, others are able to download files from this location.

This would work well for one-off freelancing gigs getting set-up quickly.

For more established locations, we need something more robust.

1. A dedicated domain, e.g. gb.mindbender.com and bz.mindbender.com
2. A dedicated server

{
  "type": "version",
  "locations": ["gb.mindbender.com", "bz.mindbender.com"],
}

Opening up a computer towards the internet also has security concerns, so it's important that the transfer protocol remain safe and unable to infect the computer in ways other than serving specific files. The standard HTTPS format should suffice for data in transit, and exposing read-only access per location should cover remaining gaps in security.

Client

An artist should somehow be able to indicate to a server that he wishes to download an asset.

Naively, if the button to do so is located in the Loader then the Loader should be able to manage downloading it. However, this approach isn't sustainable and prone to errors.

1. What should happen when the user closes the Loader window, and the download isn't yet finished?
2. If we let it sit in the background of the application, such as Maya, what should happen when the user closes Maya?
3. If the user has multiple Loader windows or applications open, downloads would happen concurrently and slow each other down, limiting our ability to manage downloads.

A perhaps less naive option is to delegate this responsibility to the Launcher.

1. A download is initiated
2. The request is sent to the Launcher
3. The launcher downloads the requested files
4. The launcher provides a status window on what files are in progress and are about to be downloaded.

Working files

The same mechanism could potentially be used to download artist files, we'll take a look at that as soon as published files are done.

Notes

In theory, simple stuff. But not having seen it elsewhere makes me a little cautious about how simple it really is. Luckily, it's a workflow easily complemented by simply sending the files across manually so if anything breaks or works unexpectedly, there is another way.

Determining whether or not to offer an option to download things should depend on files being physically available on disk. So if it has already been sent via e.g. Skype or SyncThing, the option would simply not be offered and the artist could work as per usual.

[BigRoy]

In theory, simple stuff. But not having seen it elsewhere makes me a little cautious about how simple it really is.

Don't Shotgun and Ftrack offer similar solutions? For example Locations in ftrack. Commenting this purely as a reference.

[mottosso]

Yes, that's a great reference. I think Shotgun has something along these lines as well.

[mottosso]

Hm, couldn't find anything official about Shotgun, apart from a blog post about storing everything in a single cloud-based location.

[mottosso]

Here's how I imagine this working from a user's perspective.

Initially, I think I'll let the Loader GUI be responsible for both downloading and visualising progress. This would enable a quicker prototype to be built and for us to evaluate whether the workflow fits, before getting into the land of interprocess communication and concurrency.

[mottosso]

Technically, each version is available in the Loader regardless of there being any files thanks to the live connection to the Asset Database. Each version would have an associated path, such as:

{root}/hulk/assets/Bruce/publish/modelDefault/v004

It would then be up to the file hosting service, e.g. files.mindbender.com to receive a request for this path and transmit whatever it finds there.

The state of the button in the Loader, whether it already exists locally or not, would depend on whether this path existed locally. The option to actually click it would depend on whether the files did exist at files.mindbender.com or any of the other advertised locations, such as another.server.com or files.aws.com.

In a multi-download situation, such as downloading entire assets, this is where we could potentially distribute the request to more than one location at a time, such as getting some from Brazil and some from Gothenburg. Alternatively, files could be fetched from the closest location geographically, or from the location which is currently under the least amount of load.

[mottosso]

Possible scenarios.

1. Click project, Make Available Offline
   • For each contained asset
   • For each contained subset
   • Download all representations of latest version
2. Click asset, Make Available Offline
   • For each contained subset
   • Download all representations of latest version
3. Click subset, Make Available Offline
   • Download all representations of latest version
4. Click version, Make Available Offline
   • Download all representations of selected version
5. Click representations, Make Available Offline
   • Download just the one representation

A little sceptical about (1) and whether it's actually useful in practice; who would ever need to download an entire project?

(5) is marginal, but useful for performance or spacial concerns.

Source files

What about transferring work-in-progress files?

There could be a second toggle to indicate whether one would want to include the source files for the given version. The files transferred would go beyond just copying the /work folder and include the source file as determined the publishing origin.

[mottosso]

Source Relationships

When downloading the source file for a given version, it'll become important to also include any additional files used within that source files.

Before

{
  # ... other version-related data
  "source": "{root}/hulk/assets/Bruce/work/maya/scenes/my_lookdev_v001.ma"
}

After

{
  # ... other version-related data
  "source": {
    "path": "{root}/hulk/assets/Bruce/work/maya/scenes/my_lookdev_v001.ma",
    "externals": [
      {
        # Include information on how the file is used internally
      	"type": "reference",

        # Maintain {root} directive
      	"path": "{root}/hulk/assets/Bruce/publish/modelDefault/v004/modelDefault.ma"
      },
      {
      	"type": "file",
      	"path": "{root}/hulk/assets/Bruce/publish/textureDefault/v002/textureDefault.png/body.png"
      },
    ]
  }
}

A Collector could take on the responsibility of sifting through a given working file and detect any nodes making any references to external files. Then, when making available offline, these files can simply be included in the download.

The {root} directive is maintained, the user should in fact be warned if this directive is not included because if not then the file might not be downloadable. For example if the file is located on a user's desktop.

[mottosso]

Reference on how MPC deals with multi-site sync.

• Sync at MPC on Lightning Thief

[mottosso]

With Paperspace and the cloud behind us, it's time to implement proper support for cross-site file synchronisation.

Here's where we left off.

Here's what will need to happen.

1. Re-instate the make-available-offline button in the new Loader
2. Establish an on-premise file server for remote artists to download from
3. Implement remote publish-to-server into the integrator

[mottosso]

Implemented in #266

Some things left to consider; what happens when you download an asset that depends on another asset, such as a rig that depends on a model? Ideally they would both be pulled. There is dependency tracking in place, it would primarily be a matter of exploiting this through this feature.