This issue attempts to give a compact overview of protocol issues I ran into while implementing the ios-sdk.
Info endpoints
I found the number of endpoints to hit before being able to start a session in the iOS app unnecessarily long:
.well-known/openid-configuration to detect OIDCstatus.php to detect redirections, version and maintenance modeuser.php to retrieve the current usercapabilities.php to retrieve the current set of capabilities
A single endpoint for fetching all of this information in one request would simplify this greatly. such an endpoint could be passed a list of info segments to include, like f.ex.: info.php?include=status,user,capabilities (or an extended status.php with that functionality).
Thumbnails
Clients currently have to request a thumbnail for a file to determine if a thumbnail is available. In a directory with many files for whose file types no thumbnails are available, this generates a lot of unnecessary requests.
What would help:
- a list of mime types (provided through f.ex.
capabilities) for which the server supports thumbnail generation
- the DAV endpoint providing this info for every item with a custom tag (f.ex.
oc:thumbnail-available)
The latter approach would have the benefit of keeping the logic for which item thumbnails are available in the server.
Related issue: owncloud/core#31267
Unnecessary large PROPFIND responses
PROPFINDs typically return two d:propstat tags for every item.
First, the part with information on the item:
<d:response>
<d:href>/remote.php/dav/files/admin/</d:href>
<d:propstat>
<d:prop>
<d:resourcetype>
<d:collection/>
</d:resourcetype>
<d:getlastmodified>Fri, 23 Feb 2018 11:52:05 GMT</d:getlastmodified>
<d:getetag>"5a9000658388d"</d:getetag>
<d:quota-available-bytes>-3</d:quota-available-bytes>
<d:quota-used-bytes>5812174</d:quota-used-bytes>
<oc:size>5812174</oc:size>
<oc:id>00000009ocre5kavbk8j</oc:id>
<oc:permissions>RDNVCK</oc:permissions>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>Then, second, the part with requested information not available for the item:
<d:propstat>
<d:prop>
<d:creationdate/>
<d:getcontentlength/>
<d:displayname/>
<d:getcontenttype/>
</d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>… and finally the closing tag:
The second part (on information not available) consumes bandwidth, CPU cycles, memory and power - all of which are very valuable on a mobile device - but provides no benefit at all.
It'd therefore be great to be able to omit the second/useless part. And if it needs to be there to conform with the WebDAV standard, provide an option to omit it.
WebDAV
Related to the aforementioned issue: I like and respect WebDAV (a lot!), but where it really falls short is in the expression of information in a compact format. Above XML, after all, just contains this information:
location: /remote.php/dav/files/admin/
type: collection
last-modified: Fri, 23 Feb 2018 11:52:05 GMT
Etag: "5a9000658388d"
quota-available: -3
quota-used: 5812174
size: 5812174
fileID: 00000009ocre5kavbk8j
permissions: RDNVCK
Even in HTTP-header-esque notation, it's already a lot less bytes. Now imagine what a binary format could achieve ([VID] is a VarInt encoding block type + length, [VIN] is a VarInt encoding a number):
[VID]/remote.php/dav/files/admin/[VID][VIN][VID][VIN][VID]"5a9000658388d"[VID][VIN][VID][VIN][VID][VIN][VID]00000009ocre5kavbk8j[VID][VIN]
Therefore, when working on a new architecture, I believe it could be good future-proofing to move away from WebDAV internally - and isolate everything related to WebDAV into a backend that performs the conversion between the native internals and WebDAV requests and responses.
With protocol-agnostic internals, a new backend providing access via new, more compact request and response formats would not be far away.
Sharing API
There's currently no way to detect changes to shares without requesting and comparing them in entirety.
It'd be great to have an ETag returned for Sharing API responses that only changes if the requested shares in the response change. And then, also support forIf-None-Match.
Change detection
In order to find changed items, it is currently necessary to check the ETag of the root folder, if it changed, fetch its contents, then find items with changed ETags among it … and repeat this until all changes have been discovered.
Some ideas on how this could be improved:
Sync Anchor
Delivered as part of the PROPFIND response. That SyncAnchor could then be used to make a subsequent request for a list of changes that occurred since that PROPFIND response. If a SyncAnchor is too old, a respective error would be returned.
Event subscription
Clients could subscribe to change events. Every change on the server would then generate an event and put it in the client's event queue.
While the client is connected, events could be delivered immediately (push).
If the client can't consume the events as fast as they are generated (f.ex. because it's not connected or only through a slow connection), and the number of events in the queue surpasses a limit (f.ex. 1000 events - or a time threshold), they get dropped and replaced with a single reload event, which would tell the client to perform old-style change detection by traversing the tree.
Alternatively, the subscription could expire at that point, and the client receive an error response when trying to resume the connection to the event subscription endpoint.
Metadata change propagation
Currently, not all changes to an item that are relevant to clients lead to a new ETag. Examples include a change in sharing of oc:favorite status.
For this reason, the new iOS app currently has to:
- always retrieve a full
PROPFIND (depth 1) when the user navigates to a directory, to make sure sharing and favorite status of the items are up-to-date when presented to the user. The toll on bandwidth/CPU/memory/battery can be significant for folders with a lot of items. If all relevant changes were propagated to the ETag, a depth 0 PROPFIND would be sufficient to detect changes.
- poll the server for a list of favorites when the user enters a view presenting a list of all of them.
- poll the server for a full (=> also see Sharing API above) list of shares on a regular basis to keep file.
I understand that the ETag may, by definition and meaning, not be the right vehicle for propagating these changes. A MTag (or any other name) for tracking and propagating these metadata changes could also solve this issue.
File IDs / Local IDs
Implementing offline support in the new iOS app and SDK required finding a way to efficiently track an item from local generation to upload to file on the server.
Since the oc:id (File ID) of an item is generated by the server, it can't be known beforehand - and therefore can't be used at the beginning of the item's lifecycle.
The solution implemented in the new iOS SDK was to generate a unique ID locally and track the item by Local ID. This works well, but involves quite a bit of complexity to ensure that, once a file is on the server, it always get's the same Local ID attached.
A great simplification would be if it was possible for clients to provide a Client ID, which gets stored alongside the File ID and doesn't change for the lifetime of the item on the server.
And then, the option to have all APIs (especially PROPFIND and sharing) also include that Client ID in addition to the existing File ID in responses.
Client-generated IDs would be prefixed with an app-specific prefix, f.ex. ios:[UUID goes here].
In cases, where no Client ID was provided for an item, its Item ID would be returned instead.
Moved files
Moved files present a challenge as they often first get noticed by clients as missing from the folder they were previously in. Only to reappear in a different folder while changes are discovered.
In consequence, the client needs to keep the missing item around until it has finished change discovery and can't f.ex. remove local copies immediately.
A way to perform a PROPFIND on a File ID to determine its current whereabouts and status would be great to make this more efficient.
Atomic operations
Item operations – like uploads or folder creation – only return rudimentary information on the item operated upon.
While that information is sufficient to form subsequent requests, it's not sufficient for clients to generate a full-fledged local representation of that item with all metadata.
In consequence, after performing item operations, it's often necessary to perform a subsequent PROPFIND on the item to retrieve the entire set of metadata.
This is not a big issue, but costs (some) performance and there's - in theory - a small window between the operation finishing and the PROPFIND during which another client could replace, move or delete the item again.
If the response contained more metadata (requested f.ex. through a new HTTP request header listing the tags requested), the operation would be entirely atomic.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent