Language packs updating... ?

[yahniselsts]

Language packs updating... ?

[YahnisElsts]

There are no definite plans for that at the moment. I don't think I'll need that feature myself, and I suspect it would be a niche project at best.

That said, when I get a moment, I'll take a look at how WP retrieves those updates. Maybe there's an easy way to adapt the existing update checker code to language pack updates.

Also, it might be an interesting project to refactor plugin-update-checker into something that can handle all three: plugins, themes and language packs. The update mechanisms for plugins and themes are similar enough that you could reuse a lot of the same code (e.g. as a common parent class for both update checkers).

[DavidAnderson684]

I'll definitely be using this mechanism, so I'll let you know what I find. Over 50% of the UpdraftPlus zip size is language packs, and they take up 11Mb unzipped. It causes various issues, mostly with people who are near their web hosting account disk space limit when they update the plugin - WordPress handles problems with copying files in an update really badly, and it pushes support requests to us every time we release an update.

We'll probably wait until at least an RC that has this in, though, on the assumption that it may still be in a state of flux.

[DavidAnderson684]

I've done a little investigation... it seems to be fairly straightforward. It's the same site transient as used for plugin updates - 'update_plugins'. Since whatever WP-version introduced downloadable translations, there's been an extra array entry, with key 'translations'. The format of that array is like this:

    [translations] => Array
        (
            [0] => Array
                (
                    [type] => plugin
                    [slug] => akismet
                    [language] => pt_PT
                    [version] => 3.1
                    [updated] => 2015-04-22 23:22:42
                    [package] => https://downloads.wordpress.org/translation/plugin/akismet/3.1/pt_PT.zip
                    [autoupdate] => 1
                )

            [1] => Array
                (
                    [type] => plugin
                    [slug] => updraftplus
                    [language] => pt_PT
                    [version] => 1.11.20
                    [updated] => 2015-12-21 14:28:36
                    [package] => https://downloads.wordpress.org/translation/plugin/updraftplus/1.11.20/pt_PT.zip
                    [autoupdate] => 1
                )

        )

So, the whole site transient, on my testing site, looks like this:

stdClass Object
(
    [last_checked] => 1453889329
    [response] => Array
        (
            [akismet/akismet.php] => stdClass Object
                (
                    [id] => 15
                    [slug] => akismet
                    [plugin] => akismet/akismet.php
                    [new_version] => 3.1.7
                    [url] => https://wordpress.org/plugins/akismet/
                    [package] => https://downloads.wordpress.org/plugin/akismet.3.1.7.zip
                )

        )

    [translations] => Array
        (
            [0] => Array
                (
                    [type] => plugin
                    [slug] => akismet
                    [language] => pt_PT
                    [version] => 3.1
                    [updated] => 2015-04-22 23:22:42
                    [package] => https://downloads.wordpress.org/translation/plugin/akismet/3.1/pt_PT.zip
                    [autoupdate] => 1
                )

            [1] => Array
                (
                    [type] => plugin
                    [slug] => updraftplus
                    [language] => pt_PT
                    [version] => 1.11.20
                    [updated] => 2015-12-21 14:28:36
                    [package] => https://downloads.wordpress.org/translation/plugin/updraftplus/1.11.20/pt_PT.zip
                    [autoupdate] => 1
                )

        )

    [no_update] => Array
        (
            [updraftplus/updraftplus.php] => stdClass Object
                (
                    [id] => 31679
                    [slug] => updraftplus
                    [plugin] => updraftplus/updraftplus.php
                    [new_version] => 1.11.23
                    [url] => https://wordpress.org/plugins/updraftplus/
                    [package] => https://downloads.wordpress.org/plugin/updraftplus.1.11.23.zip
                )

            [wordpress-beta-tester/wp-beta-tester.php] => stdClass Object
                (
                    [id] => 8693
                    [slug] => wordpress-beta-tester
                    [plugin] => wordpress-beta-tester/wp-beta-tester.php
                    [new_version] => 1.0
                    [url] => https://wordpress.org/plugins/wordpress-beta-tester/
                    [package] => https://downloads.wordpress.org/plugin/wordpress-beta-tester.1.0.zip
                )

            [wp-crontrol/wp-crontrol.php] => stdClass Object
                (
                    [id] => 1426
                    [slug] => wp-crontrol
                    [plugin] => wp-crontrol/wp-crontrol.php
                    [new_version] => 1.3
                    [url] => https://wordpress.org/plugins/wp-crontrol/
                    [package] => https://downloads.wordpress.org/plugin/wp-crontrol.1.3.zip
                )

        )

)

Thus, to over-write the translation pack download URL, I just need to hook the site transient, and fiddle with ->['translations'][$index]->package.

Since it's part of the same transient that plugin-update-checker is adjusting, it seems natural for plugin-update-checker to provide a method to help with that - ?

[YahnisElsts]

I'm not opposed to the idea, but I'm still not sure when/if I'll have time for this. If we're going to support other types of updates - language packs, themes, GitHub updates, someone asked for BitBucket support as well... - it would be a good idea to refactor the whole thing first to avoid lots of code duplication. That would take quite a while.

[DavidAnderson684]

Hi Yahnis,

In case it helps (either you or anyone else looking to implement this), today I've finally got around to doing this.

Here's what I did:

1. Got the server to include a 'translations' item, with data in the required format (as above). (Of course, you also need to put the zip referred to containing your translations on the server, in the format that WP uses - which means, it contains a .mo and a .po at the top level, with filenames in the format (slug)-(locale).[mp]o ).

2. In the client, used the puc_retain_fields filter in PluginUpdateChecker to retain this field.

3. Add my own filter to site_transient_update_plugins, to run after the one in PluginUpdateChecker, and used that to inject the translations data into the result, with this hopefully-clear-enough method:

public function possibly_inject_translations($updates) {

        $slug = $this->slug;

        $checker = is_object($this->plug_updatechecker) ? get_site_option($this->plug_updatechecker->optionName, null) : null;

        if (!isset($checker->update->translations) || !is_array($checker->update->translations)) return $updates;

        foreach ($checker->update->translations as $translation) {
            $use_key = false;
            if (isset($updates->translations) && is_array($updates->translations)) {
                foreach ($updates->translations as $k => $translation) {
                    if ($translation['type'] == 'plugin' && $translation['slug'] == $slug) $use_key = $k;
                }
            }

            $translation_array = (array)$translation;
            $translation_array['type'] = 'plugin';
            $translation_array['slug'] = $slug;

            if (false !== $use_key) {
                $updates->translations[$use_key] = $translation_array;
            } else {
                $updates->translations[] = $translation_array;
            }

        }

        return $updates;
}

[YahnisElsts]

Interesting. I have a few questions about your implementation:

• How do you decide which translations the server should include? Does it just push all available translations every time, or is there some more sophisticated logic? As far as I can tell, WordPress doesn't do any client-side filtering of translation updates, so returning all available translations would make it treat them as "new" updates every time.
• How are you dealing with translation versioning?
• What's going on with the $use_key bit and overwriting preexisting updates? Does your plugin use the same slug/text-domain as a plugin in the wordpress.org repository?
• What if there are multiple existing entries with that slug? It looks like the current implementation would only catch one of them.

[DavidAnderson684]

How do you decide which translations the server should include? Does it just push all available translations every time, or is there some more sophisticated logic? As far as I can tell, WordPress doesn't do any client-side filtering of translation updates, so returning all available translations would make it treat them as "new" updates every time.

Indeed... we just discovered that issue yesterday... I'll get back to you when I've fixed it!!

How are you dealing with translation versioning?

We have a system on the server that writes out new .zip files, only if something has changed in the underlying .po/.mo files. The version numbers are then generated on the server based on the timestamp of the zip file (and hence will only update if the zip is changed).

What's going on with the $use_key bit and overwriting preexisting updates? Does your plugin use the same slug/text-domain as a plugin in the wordpress.org repository?

Yes - we're stuck with that, unfortunately (alternative is to attempt to transition zillions of users to a different slug for either the free or paid version). We prevent version clashes between free/paid versions via a version scheme that prevents paid users ever getting offered an update from wordpress.org (even if inactive).

But, the code was intended to be general. You never know when wordpress.org are going to add a plugin to their directory that's the same as yours, that may interfere. I added a ticket in Trac for repository identifiers, and there's some interest; even a patch - but no action for years, sadly.

What if there are multiple existing entries with that slug? It looks like the current implementation would only catch one of them.

I think I should amend that fragment to check the locale as well. Presumably there can't be multiple updates that match slug+locale.

[DavidAnderson684]

How do you decide which translations the server should include? Does it just push all available translations every time, or is there some more sophisticated logic?

I didn't completely answer this. For deciding which translations to actually include... the client request indicates the currently active locale, and we only inform of any available update for that.

[YahnisElsts]

You never know when wordpress.org are going to add a plugin to their directory that's the same as yours, that may interfere.

Yes, that's a risk. Maybe the simplest solution would be to just remove all existing entries that match your slug and type, then add your translations afterwards. No need to overwrite specific keys.

For deciding which translations to actually include... the client request indicates the currently active locale, and we only inform of any available update for that.

Do you think that would also work as a general solution? It doesn't look like WordPress itself does it this way. I've definitely seen the wordpress.org update API return multiple translations for the same plugin, though I'm not sure how it decides which ones to include.

[YahnisElsts]

Also, how about sites using WPML or something similar? They might need several different translations.

[DavidAnderson684]

Hmmm... so.... when the request is fired off to the wordpress.org API ( POST to /plugins/update-check/1.1/ ), the POST payload includes (amongst the other stuff) two fields, locale and translations. The values for these are JSON encoded. locale is an array, listing the installed locales; translations is an object, with entries for each slug.

Example for locale:

["es_ES","fr_FR"]

Example (after JSON-decoding) for translations:

stdClass Object
(
    [akismet] => stdClass Object
        (
            [es_ES] => stdClass Object
                (
                    [POT-Creation-Date] => 
                    [PO-Revision-Date] => 2015-11-05 16:34:43 0000
                    [Project-Id-Version] => Stable (latest release)
                    [X-Generator] => GlotPress/1.0-alpha-1100
                )

            [fr_FR] => stdClass Object
                (
                    [POT-Creation-Date] => 
                    [PO-Revision-Date] => 2016-01-05 07:31:33 0000
                    [Project-Id-Version] => Stable (latest release)
                    [X-Generator] => GlotPress/1.1.0-alpha
                )

        )
)

So, it seems that the way that WP is handling versions is that a) on the client side, WP processes the .po file to get the header, and b) that's compared with the same data on the server side. I suppose that PO-Revision-Date is the only one it's actually using.

wp_get_installed_translations() in wp-includes/l10n.php is the function that extracts that info. That's called by wp_update_plugins() in wp-includes/update.php. I think, then, that the client side of the updater needs to do the same - and my server side needs to process that info.

[DavidAnderson684]

The fix suggested in my previous comment seems to be working. The client side now advises of a) its current locale and b) what locales are installed, and their PO-Revision-Date values. The server side now only advises of updates that are later than those installed (based on analysing the same header). This all appears to be working, based on my tests + snooping of the HTTP conversation.

By the way - because the checker class uses wp_remote_get(), I have to be careful what info to add to the request (via the addQueryArgFilter() method), to avoid it getting too long - too much useless info on un-interesting PO file headers, on a site with WPML and zillions of installed translations, could perhaps get near the limit. WP core uses POST for its update checks. Perhaps your checker should either do, or at least optionally allow, this too?

[YahnisElsts]

Another option would be to include all translations but filter them client-side (i.e. in the update checker).

Advantages:

• Simpler server logic.
• No need to add extra query arguments or switch to POST (= better backwards compatibility).
• All "which translations do you need?" logic in one place instead of split up between the server and update checker.
• ( If it gets implemented in PUC proper: ) Fits the general approach of being flexible about where update metadata comes from. Technically, you can even use a static JSON file.The current PUC implementation doesn't require any server-side logic.

Disadvantages:

• Some bandwidth wasted by sending metadata for translations that will not get used. Quick estimate: 200 - 300 bytes per translation per request.
• Slight performance hit for clients due to extra logic; probably insignificant. Disk access like scanning the language directory and reading PO/MO metadata will probably take much longer than filtering the list.

Personally, I'm leaning towards this approach.

[DavidAnderson684]

I was thinking of that. Of course, there's no reason why it can't be filtered on the client anyway, regardless of what the server sends. I decided to have my server only send requested updates, because that's how wordpress.org works and I need a quick solution so decided I should be conservative and avoid doing something new and then later finding out there was a reason for the wordpress.org behaviour. Though, if the client-side filters, it's hard to see that there could be any unintended side effects.

[YahnisElsts]

I've implemented something similar to your solution in this branch and I'd be interested to hear your feedback.

Personally, one thing that concerns me is whether the PluginUpdate class is really the right place for the $translations array. You can update translations without updating the plugin itself, and vice versa. It seems like it should be a separate thing. However, if it's not part of PluginUpdate then I don't see a clean way to get it from requestInfo to checkForUpdates. Any ideas?

[DavidAnderson684]

I agree that it seems strange to be putting in language updates in a plugin updater class... however, that also appears to be what is happening with WordPress core / wordpress.org - the translation update request + reply is riding on top of the plugin update request. As such, it seems to be the 'right' place for you to do it, following their lead.

[YahnisElsts]

however, that also appears to be what is happening with WordPress core / wordpress.org - the translation update request + reply is riding on top of the plugin update request.

Yes, but not quite. WordPress does put that info in the same request/reply, but it's still conceptually separated from plugin updates. Abstractly speaking, WP stores plugin and translation updates as two branches of the same tree. The update_plugins transient is structured like this:

{
   last_checked: timestamp,
   checked: { plugin versions },
   response: { available plugin updates },
   translations: { available translation updates },
   no_update: { plugins without updates }
}

A PluginUpdate is equivalent to an individual entry of the response array. Translations are not part of that entry; WP has a separate array for that.

To store translation updates the same way that WordPress does, PUC would have to put them in a separate property of the object maintained by setUpdateState. I'd actually like that approach better, but as I mentioned before, I don't see a good way to get translation updates from requestInfo/requestUpdate without passing them via PluginUpdate.

[DavidAnderson684]

I see. It seems all pretty much the same thing to me; though WP puts them separately in the update_plugins transient, they are, of course, not entirely independent bits of data. So, it seems to me to not matter that much how you conceive of the object that you end up storing, as they can both be justified.

[DavidAnderson684]

P.S., though, I think you've got a better feel for it than me, with your closer knowledge of your code!

[YahnisElsts]

I wasn't able to come up with anything significantly better in a week, so I guess we'll go with the current implementation.

Let me know if you have any additional feedback about the code. If there's no objection, I'll merge it.

[YahnisElsts]

The feature branch has been merged into master.