The api.keyman.com from keymanapp

Removing a model or keyboard does not remove the info from disk

Because the model_info and keyboard_info folders are not purged during the database build, old model_info and keyboard_info files can be left behind. These need to be purged as part of the database build before unzipping the new files.

.keyman-touch-layout files appear to have no json schema (should be on api.keyman.com)

bug: `Warning: mkdir(): File exists` on database build

Except it isn't fixed.

Can't for the life of me see how the folder is being created (nothing else creates it, no other threads or processes):

      if(is_dir($this->models_path)) {
        deleteDir($this->models_path) || fail("Unable to remove folder " . $this->models_path);
      }
      if(!is_dir($this->models_path)) {
        mkdir($this->models_path, 0777, true) || fail("Unable to create folder " . $this->models_path);
      }

Failing on mkdir with Warning: mkdir(): File exists

Originally posted by @mcdurdin in #135 (comment)

purge api.keyman.com cloudflare cache on database build events

https://api.cloudflare.com/#zone-purge-files-by-cache-tags-or-host

This will enable refreshed api data to be available more promptly on deployments and allow us to extend the cache lifetime.

Add timing report to hooks/keyboard-build-success

Because this takes a while, it would be helpful for the report to giving details on what takes longest for future analysis.

Sample run currently:

Downloading https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
Downloading https://downloads.keyman.com/data/keyboard_info.zip
Downloading https://downloads.keyman.com/data/model_info.zip
Triggered database build for keymanapp/keyboards and keymanapp/lexical-models
=============================================================================

Running D:\home\site\wwwroot\tools\db\build/search.sql
Running D:\home\site\wwwroot\tools\db\build/search-queries.sql
Running D:\home\site\wwwroot\tools\db\build/model-queries.sql
Running D:\home\site\wwwroot\tools\db\build/legacy-queries.sql
Running D:\home\site\wwwroot/.data/language-subtag-registry.sql
Running D:\home\site\wwwroot/.data/iso639-3.sql
Running D:\home\site\wwwroot/.data/iso639-3-name-index.sql
Running D:\home\site\wwwroot/.data/ethnologue_language_codes.sql
Running D:\home\site\wwwroot/.data/ethnologue_country_codes.sql
Running D:\home\site\wwwroot/.data/ethnologue_language_index.sql
Running D:\home\site\wwwroot/.data/keyboards.sql
Running D:\home\site\wwwroot/.data/models.sql
Running D:\home\site\wwwroot\tools\db\build/search-prepare-data.sql

[api.keyman.com] languages search not handling script tag?

The query https://keyman.com/keyboards?q=lana returns

Keyboards matching 'lana'

Boonkit Unicde

Languages matching Lana:

Thai, Northern (Lanatai)

but following the languages link, it's not finding nod (The Boonkit keyboard is now under nod-Lana)

Add linux platform to keyboard sql queries

See
build_keyboard_script.php
search-queries.sql

chore: disable ARRAffinity cookie

Recently saw this message in the developer console during testing:

A cookie associated with a cross-site resource at http://api.keyman.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.

We'll probably need to make some adjustments corresponding to this.

Bookmarklet listings are missing keyboards

Looking at the current state of https://keyman.com/bookmarklet/:

Given that we support a number of Khmer keyboards (as an example), why isn't this listing offering any of them?

bug: keyboard catalog returning `[]` in some circumstances?

I'm not sure the circumstances, but from the Sentry logs it appears there's times the Cloud catalog queries return [] and the Keyman Engine JSON parsing code throws an exception treating it as a JSONObject.

I think I'd like to verify the root cause of this on api.keyman.com as well -- and make sure that there are not circumstances where this is happening, wrongly.

Originally posted by @mcdurdin in keymanapp/keyman#7611 (comment)

package-version: Clarifying results for legacy keyboards

I was investigating keymanapp/keyman#3134 to see if I could migrate legacy deprecated keyboards with the package-version endpoint. (e.g. replace us and european2 keyboards with the latest sil_euro_latin.kmp)

The following queries return "kmp" links to a directory (kmp doesn't exist):
https://api.keyman.com/package-version?platform=android&keyboard=european2

{
    "keyboards": {
        "european2": {
            "version": "1.6",
            "kmp": "https://downloads.keyman.com/keyboards/european2/1.6/"
        }
    }
}

https://api.keyman.com/package-version?platform=android&keyboard=us

{
    "keyboards": {
        "us": {
            "version": "1.0",
            "kmp": "https://downloads.keyman.com/keyboards/us/1.0/"
        }
    }
}

A couple questions:

Should the endpoint return "error": "not found" or "error": "not available for <platform>"?
Is it up to the apps to determine the kmp file isn't available?

[api.keyman.com] Deprecate languages[] array in keyboard_info spec

In the current keyboard_info spec,

The languages field can be an array or object.

For #183, the keyboard_info will need to include display names. In order to do this:

We should deprecate the option for languages to be an array
Update the KeyboardLanguageInfo Object spec in api.keyman.com to add the fields:

Member	Required?	Notes
displayName
languageName
scriptName
regionName

Update the spec on help.keyman.com
Also need kmcomp to migrate existing .keyboard_info files (convert arrays to objects) and generate the new fields if not provided

Examples:
(replacing fr-ca with an example not involving suppressed script)

"dnj-Latn-LR" : {
  "displayName" : "Dan (Latin, Liberia)",
  "languageName": "Dan",
  "scriptName": "Latin",
  "regionName": "Liberia" }

"und-fonipa" : {
  "displayName": "IPA",
  "languageName": "Undetermined" }

chore: See if we can build database faster with bcp

Also worth checking bcp to see if we can insert data more quickly. We don't need fk constraints except for test purposes (it's a read only database!)

Originally posted by @mcdurdin in #114 (comment)

https://docs.microsoft.com/en-us/sql/relational-databases/import-export/use-unicode-character-format-to-import-or-export-data-sql-server?view=sql-server-ver15
https://docs.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver15
https://docs.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-ver15

Consider showing localized names for languages in keyboard search

(Extracted from #87)

Add query for keyboard and lexical model updates

In 14.0, the mobile apps are going away from using the cloud keyboard catalogs.
api.keyman.com/package-version should be designed so that apps can query for keyboard and lexical model updates (pass keyboard IDs and lexical model IDs for parameters).

The return should include latest versions and download links.

Draft of update spec

Error received attempting to upload crash report

[Window Title]
Keyman Developer

[Content]
Unable to successfully upload problem details: <br />
<b>Notice</b>:  Undefined index: api_keyman_com_github_user in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>14</b><br />
<br />
<b>Notice</b>:  Undefined index: api_keyman_com_github_repo in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>15</b><br />
<br />
<b>Notice</b>:  Undefined property: stdClass::$number in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>123</b><br />
<br />
<b>Notice</b>:  Undefined property: stdClass::$id in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>132</b><br />
Failed:

{
  "message": "Not Found",
  "documentation_url": "https://developer.github.com/v3"
}



[OK]

chore: database build needs to be async

Right now, the database build is a synchronous step at the end of the keyboard build and deploy, with a ping to a webhook endpoint. The process can be slow, particularly the import into SQL Server. This needs to be an async process with a log file reported ... somewhere so that we don't hit application or web server timeouts.

bug: Cloud API returns for KMW: find closest BCP-47 match

In reference to keymanapp/keyman#2785:

KeymanWeb's bookmarklet is currently experiencing issues with the API-returned keyboard stub. On KMW 13.0.105, https://api.keyman.com/cloud/4.0/keyboards?jsonp=keyman.register&languageidtype=bcp47&version=13.0&keyboardid=sil_uganda_tanzania@ikx&timerid=3 returns:

keyman.register({"options":{"context":"keyboard","dateFormat":"standard","device":"any","keyboardBaseUri":"https://s.keyman.com/keyboard/","fontBaseUri":"https://s.keyman.com/font/deploy/","keyboardid":"sil_uganda_tanzania@ikx","keyboardVersion":"current"},"keyboard":[{"id":"sil_uganda_tanzania","name":"Uganda-Tanzania Bantu (SIL)","filename":"sil_uganda_tanzania/1.1.1/sil_uganda_tanzania-1.1.1.js","version":"1.1.1","source":"https://github.com/keymanapp/keyboards/tree/master/release/sil/sil_uganda_tanzania","devices":{"phone":2,"tablet":2,"desktop":2},"languages":[]}],"timerid":"3"});

Note this tidbit at the very end: "languages":[]. As ikx isn't directly matching anything in the Cloud API's database, no languages are returned... which causes KMW to not register the keyboard.

Performance improvements for keyboard search (and related)

Setup internal links for back-end API calls to use internal.api.keyman.com (non-proxied form of api.keyman.com); these avoid trips outside the datacenter for PHP calls to the backend. Requires also DNS setup.
- although this sounds like a good idea, I am actually thinking that #67 makes it unnecessary and probably a performance cost, because we won't be able to make use of caching. So for now, won't implement.
Verify that all appropriate indexes are in place (especially check short searches such as 'r', 'ra'). (Fix: #111)
Loading /keyboards does a redirect to q=&page=1 which is kinda unnecessary! (Fix: keymanapp/keyman.com#173)

Updating keyboards database should use rotation

Currently, the keyboard database update takes a few seconds, which we can live with in most cases. However, we do run into lock contention, which is breaking things.

For example:
https://build.palaso.org/viewLog.html?buildId=196661&buildTypeId=Keyboards_Upload&tab=buildLog&_focus=13357

Results of this are broken keyboard search, broken keyboard downloads, broken kmw toolbar etc. So nasty.

For greater stability, I think we should update our keyboard database build to use rotation: e.g. build to keyboards_1 and at the very end, update a global database parameter specifying the version of the keyboard database to use. If we do this carefully, we should avoid any downtime at all.

Duplicates in search results

When I do

https://api.keyman.com/search?q=l:*

it's what shows for Portuguese.

This is what the Portuguese data looks like:

  {
        "id": "pt",
        "name": "Portuguese",
        "keyboards": [
            "basic_kbdbr",
            "basic_kbdbr",
            "basic_kbdpo",
            "brazilian_portuguese",
            "brazilian_portuguese",
            "european",
            "european2",
            "european2",
            "kbdbr",
            "kbdpo",
            "malar_braille",
            "portuguese",
            "portuguese_abnt",
            "portuguese_abnt",
            "portuguese_abnt2",
            "portuguese_abnt2",
            "sil_euro_latin",
            "transazerty"
        ]
    },

There are several duplicates there. I could improve my import routine to handle it, but it seems like ideally it should be fixed in the Keyman data, unless I'm misunderstanding something.

legacy40.php is throwing errors in the server logs

[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$ban-bali in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$kaw-bali in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$ar-dz in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$aii-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$bjf-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$bhn-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$cld-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$hrt-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$tmr-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$kqd-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$lhs-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$arc-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice:  Undefined property: stdClass::$oar-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
...

[api.keyman.com] Update keyboard listing for use with .kmp on iOS / Android

Goes with keymanapp/keyboards#337

Add linux platform to version query

For https://api.keyman.com/version/linux

Update APIs to return keyman.com/go/package/download[/type]/... style URLs

Pattern: https://keyman.com/go/package/download[/type]/package_id?[version=package_version][&platform=platform][&tier=tier][&bcp47=bcp47][&update=0|1]

This pattern will be returned from package download APIs on api.keyman.com:

http://api.keyman.com.local/package-version?keyboard=khmer_angkor,bar&keyboard=baz&model=zoo,nrc.en.mtnt&platform=windows
http://api.keyman.com.local/windows/14.0/update?package_khmer_angkor=&tier=alpha&version=14.0

Also, lexical model support for packages. See type parameter above, any lexical model APIs below:

http://api.keyman.com.local/model?q=nrc.en.mtnt

Extracted from https://docs.google.com/document/d/1rhgMeJlCdXCi6ohPb_CuyZd0PZMoSzMqGpv1A8cMFHY/edit?ts=5f17e421#heading=h.mio0p3wdzdye

Depends on keymanapp/keyman.com#155.

bug: stop words prevent some keyboards searches from returning results

One thing I noticed: if I search for the keyboard on keyman, there are zero search results if I include more than one word.
Let me show you what I mean.
The search results for just the first word: https://keyman.com/keyboards?q=aleph
The search results for the first two words: https://keyman.com/keyboards?q=aleph%20with
Can this be fixed?

This turns out to be an oddity in SQL Server Full Text Search which we haven't struck with other keyboards. 'With' is a 'stop word', and so it gets ignored in a phrase search. This does feel a little broken. It's possible to disable stop words; we'll think about doing that.

Originally posted by @mcdurdin in keymanapp/keyboards#2148 (comment)

TODO list for keyboard search

FUTURE:

UPDATE DOCUMENTATION FOR keyboard search on help.keyman.com: #70 (comment) -- mixing and matching filters, docs needed (FIX: keymanapp/help.keyman.com#206)
Show local names for languages (somewhere!) (Will be fixed in #108)

TODO:

===========================================================================================================================

DONE:

Additional Notes: Notes on api.keyman.com changes for langtag consumption
- example keyboard: burushaski_girminas, khw-latn "Khowar (Latin)"
- ខ្មែរ finds zero results but ខ្មែ finds 7...
- Show a 'popular keyboards' list for the empty search -- this can also be the search engine jumping-off point.
- "Show obsolete keyboards" needs an indication of the change of status ("Hide obsolete keyboards") and needs to be outdented. Also
  needs some thought with paginated results.
- Too many pages leads to overwhelming number of page links at bottom (e.g. s:latin)
- http://api.keyman.com.local/search?q=l gives 500
- el_dinka appears to have non-canonical bcp47 codes -- search finds it no trouble.
- Show list of associated languages+scripts+countries in keyboard deatils (and related keyboards?)
- For in-app download links, include information on searched language code (if available), for default language install (#1456)
- Match fields in json should be integer or float where possible, not string! (and update schema accordingly)
- schema for match type should be restrictive to actual types used
- Search "spa" vs "spanish" -- the weighting could be better. Similar "ger" vs "german". (probably need length-based match weight override)
- REFACTOR: region vs country
- REFACTOR: code vs id vs tag
- Pagination
- Need to give more detail on failed links (and make it easier to find in logs, so tweak the broken link search a node wrapper)
- Searches for keyboard ids should work
- Phrases are not working yet (need to split into either a phrase search or separate words)
- Searches for bcp47 tags, scripts, regions should work
  - need to highlight these on keyman.com (incl. keyboard_id)

FAIL: http://api.keyman.com.local/search/2.0?f=1&q=l:%, c:%, etc.

PHP Fatal error:  Uncaught PDOException: SQLSTATE[IMSSP]: The active result for the query contains no fields. in C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php:236
Stack trace:
#0 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php(236): PDOStatement->fetchAll()
#1 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php(77): KeyboardSearch->GetSearchQueries()
#2 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php(73): KeyboardSearch->WriteSearchResults()
#3 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.php(50): KeyboardSearch->GetSearchMatches()
#4 {main}
  thrown in C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php on line 236

Default search should return a FLAT LIST of KEYBOARDS ONLY with highlights. e.g. 'Thai' should return keyboards with 'Thai' in the name, in a language name, or in the country associated with the language.
Search results must be weighted (summed?)
a) match of primary language name 1.0
b) match of alternate language name 0.3
c) match of keyboard name or id 1.0
d) match of script name 1.0
e) match of country name 0.5
f) match on term in description 0.5
g) match quality (whole word match = 1.0, down to 0.1 for further distance? as a multiplicand)
select * from t_langtag_name inner join containstable(t_langtag_name, name, 'isabout (thai weight (1.0), "thai*" weight (0.5))') as KEY_TBL ON t_langtag_name._id = KEY_TBL.[KEY] order by [RaNK] desc
5 / 5 = 1.0
4 / 5 = 0.8
1 / 5 = 0.2
NOTE: final weighting is different but ... let's see how it goes
Can also specify a search:
?q=l:<term> search for keyboards that support a language, by name (does not check id)
?q=l:id:<id> search for keyboards that support a language, by bcp 47 id
?q=c:<term> search for keyboards that support languages within a country
?q=c::id:<id> search for keyboards that support languages within a country, by iso 3166 id
?q=s:<term> search for keyboards that support a script
?q=s:id:<id> search for keyboards that support a script by script id
?q=id:<id> search for keyboards that match the id
?q=legacy:<id> search for keyboards that match the legacy id, only one returned!
Should be able to specify alternate names? Searches should match on NFKD with diacritics stripped.

api.keyman.com#48 -- database rotation

  <br />
  <b>Warning</b>:  sizeof(): Parameter must be an array or an object that implements Countable in <b>C:\Projects\keyman\sites\api.keyman.com\script\search\search.php</b> on line <b>202</b><br />
  <br />
  <b>Warning</b>:  sizeof(): Parameter must be an array or an object that implements Countable in <b>C:\Projects\keyman\sites\api.keyman.com\script\search\search.php</b> on line <b>209</b><br />

when searching for http://api.keyman.com.local/search?q=k:thai

Highlight found term in results

Process:

Add langtags.json to website database so it is available for rewrite. Don't touch APIs at this stage. Deploy this.
Rewrite search.php to support the details above, basing off langtags.json.

Consume langtags.json instead of other data sources

langtags.json is the replacement for alltags and should contain enough information to allow us not to import most of the other data sources we currently reference.

Note also this from @mhosken:

We now have a staging for langtags.json, using the same &staging=1 (or &staging=true).

I have added two new fields to the langtags.json specifically to help Keyman. The "windows" field gives a form of the tag that is appropriate for use on Windows where the only script field that can be removed is for languages with an appropriates suppress-script field in the IANA registry. I have also added that flag as a "suppress" field.

These changes are simple additions and therefore cause a minimal version bump.

Other changes in the staging data set is the addition of new iso639 codes recently approved.

I am waiting on a new set of Ethnologue data before we are really ready to release. So I'm not ready to flick the switch quite yet. But this is certainly ready for testing.

Note also: this yet to be released public document from MS:

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/70feba9f-294e-491e-b6eb-56532684c37f?redirectedfrom=MSDN
which puts the whole addition of a windows field into question.

bug: test failing with SQL Server failing to download

See actions/runner-images#3493

Ref https://github.com/keymanapp/api.keyman.com/pull/153/checks?check_run_id=2714098977

https://keyman-staging.com/keyboards/nw_iranian_latin should show "(Latin)" script for Gilaki language because the default script for Gilaki is Arabic.

https://keyman-staging.com/keyboards/nw_iranian_latin should show "(Latin)" script for Gilaki language because the default script for Gilaki is Arabic.

Originally posted by @mcdurdin in #87 (comment)

Lower-case Language IDs for /model and legacy endpoints?

After #20 and #21, api.keyman.com/cloud/4.0 now returns lower-cased language IDs
( e.g. from str-Latn to now str-latn).

Should /model and legacy APIs also be updated to lower-case language IDs?

Otherwise, we get into a situation where the apps don't associate lexical models with a keyboard language.

bug: version api should not return new version for kmw until s.keyman.com is ready

bug(android): Unexpected installed language

Describe the bug

In Keyman for Android, I clicked on Settings/ Installed Languages and searched for "German". I installed "EuroLatin (SIL) (German language)". On the "Select languages for EuroLatin (SIL)" I didn't select any language, but clicked Install.

I would have expected that this adds EuroLatin for German, but instead it gets installed for Plautdietsch.

If I select German on the "Select languages for EuroLatin (SIL)" page before clicking Install, it installs it for German, but additionally for Plautdietsch.

Keyman for Android:

Device: Realme X3 SuperZoom
OS: Android 10
Keyman version 14.0.270

legacy40.php is still giving some errors internally

[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice:  Undefined index: devanagari_inscript in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning:  Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice:  Undefined index: kyrghyz in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning:  Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice:  Undefined index: urdu in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning:  Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice:  Undefined index: uzbek in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning:  Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319

bug(bookmarklet): use full BCP-47 code, not partial

Since #45 is an issue, it'd probably be best for us to also ensure that Bookmarklets use the full, available BCP-47 code in their calls so that any issues with "closest match" scenarios may be minimized.

For example, noting the bookmarklet calling https://api.keyman.com/cloud/4.0/keyboards?jsonp=keyman.register&languageidtype=bcp47&version=13.0&keyboardid=sil_uganda_tanzania@ikx&timerid=3, it really should be using @ikx-latn, not @ikx - the former's the one actually in the cloud API database.

bug: keyboard_info has wrong last update date

Currently, the last update data for keyboards is always the last publish date for the whole database. This is not right and should actually match the last time the keyboard is changed in the repository; probably we can use the git logs for this:

git log -n 1 --format="format:%cI" -- release/gff/gff_amharic

This probably needs to be done in the .keyboard_info build process though.

Cloud keyboard searches do not ignore deprecated keyboards

Builds on #5. At present, searching for a keyboard in the database with the legacy searches does not exclude deprecated keyboards. It should.

Deprecated keyboards are never returned by cloud api even if specifically requested

it looks like that keyboard isn’t being returned; looks like a recent adjustment to our keyboard API may have been a bit overzealous. @Marc_Keyman, it looks like deprecated keyboards are now never returned by the API, even if a deprecated keyboard is specifically requested. While this is probably fine for general, language-code only requests, this breaks pages that rely on specific keyboards once they are deprecated.

Seems the appropriate resolution here is to not return deprecated keyboards if generic searches are made (query: do we have a way of detecting whether the search is coming from the Keyman app as opposed to websites? would need to check this).

Discussion: https://community.software.sil.org/t/keymanweb-keyboard-is-not-funcitoning-php/3088/7

Future tweaks to keyboard search 2.0

Two things we could improve in keyboard search. Neither of them are high priority.

Search c:id:in gives disordered weighting due to multi-language keyboards which get exaggerated weights.
-- could use max(weight) instead of sum(weight) in f_keyboard_search_results but that reduces most searches to
a very simplistic popularity index. Needs discussion.
Can we merge with the f_keyboard_search patterns sp_keyboard_Search_by_legacy_id, by_id, by_language_bcp47_tag?

bug(search): Wrong bcp47 code when searching for "German" with embedded search

When using the URL https://keyman.com/go/linux/17.0/download-keyboards and searching for "German", the first result ("EuroLatin (SIL)(German language)") has the bcp47 code pdt instead of de.

Things work with the regular search page (https://keyman.com/keyboards?q=german).

See keymanapp/keyman#9114.

[cloud] Keyboard preference order appears to be incorrect

A language-code lookup of 'km' now default-returns Khmer (NiDA) ( basic_kbdkni) instead of Khmer Angkor ( khmer_angkor), and 'und-fonipa' returns the deprecated ipauni11 instead of sil_ipa keyboard. I doubt it's intentional, but wonder what severity we'd call that.

IPA: https://api.keyman.com/cloud/4.0/keyboards?jsonp=keyman.register&languageidtype=bcp47&version=13.0.300&keyboardid=@und-fonipa&timerid=16

Khmer: https://api.keyman.com/cloud/4.0/keyboards?jsonp=keyman.register&languageidtype=bcp47&version=13.0.300&keyboardid=@km&timerid=18

Arose from KeymanWeb testing in keymanapp/keyman#2528.

bug: Search for `el` gives stats for 31 results but only returns 12 unless obsolete=1 is set

http://keyman.com.local/keyboards/languages/el?page=2&obsolete=0

vs

http://keyman.com.local/keyboards/languages/el?page=2&obsolete=1

The stats need to exclude obsolete keyboards on this search.

online update check is spamming server logs with missing keyboards

If a keyboard is not present, it's okay to return 400 from the endpoint. But the onlineupdate check should avoid writing a warning to the log!

[25-Nov-2019 02:50:36 America/Los_Angeles] PHP Warning:  file_get_contents(https://api.keyman.com/keyboard/pyidaungsutwo): failed to open stream: HTTP request failed! HTTP/1.1 400 Fail: Keyboard not found
 in D:\home\site\wwwroot\tools\onlineupdate.php on line 116

[api.keyman.com] Update deprecation flags in database to make querying faster and simpler

Currently, we mark deprecation with the deprecates relation. This gives us one direction. However, this requires a secondary query to determine whether a keyboard is deprecated, which is not ideal for searches. We should build the corresponding relation for each keyboard as a deprecatedBy relation.

Then we need to update the search and cloud APIs to flag deprecated keyboards.

This work comes out of keymanapp/keyboards#374.

bug: error building ISO639-3 data

[02/25/2021 20:58:27 > fc5cb9: INFO] Running D:\home\site\wwwroot/.data/iso639-3.sql
[02/25/2021 20:58:30 > fc5cb9: INFO] Array
[02/25/2021 20:58:30 > fc5cb9: INFO] (
[02/25/2021 20:58:30 > fc5cb9: INFO]     [0] => 21S01
[02/25/2021 20:58:30 > fc5cb9: INFO]     [1] => 110
[02/25/2021 20:58:30 > fc5cb9: INFO]     [2] => [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]There are fewer columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement.
[02/25/2021 20:58:30 > fc5cb9: INFO] )
[02/25/2021 20:58:30 > fc5cb9: INFO] Failure: PDOException: SQLSTATE[21S01]: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]There are fewer columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement. in D:\home\site\wwwroot\tools\db\build\build.inc.php:148
[02/25/2021 20:58:30 > fc5cb9: INFO] Stack trace:
[02/25/2021 20:58:30 > fc5cb9: INFO] #0 D:\home\site\wwwroot\tools\db\build\build.inc.php(148): PDO->exec()
[02/25/2021 20:58:30 > fc5cb9: INFO] #1 D:\home\site\wwwroot\tools\db\build\build.inc.php(67): BuildDatabaseClass->sqlrun()
[02/25/2021 20:58:30 > fc5cb9: INFO] #2 D:\home\site\wwwroot\tools\db\build\build_cli.php(30): BuildDatabaseClass->BuildDatabase()
[02/25/2021 20:58:30 > fc5cb9: INFO] #3 {main}
[02/25/2021 20:58:30 > fc5cb9: INFO]

There are fewer columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement.

This suggests the format of the ISO639-3 data has changed.

This error is preventing the build from completing.

Finish keyboard install page (aka universal link infrastructure) for macOS,Android,iOS

Finish keyboard install page (aka universal link infrastructure) for:

Originally posted by @mcdurdin in #87 (comment)

Cloud Services API not listing some keyboards

It is observed that some keyboards are not included in returned JSON data of API call - https://api.keyman.com/cloud/4.0/keyboards

Examples :-

bug: Keyboard Search for "sil ipa" (sans quotes) returns 0 results

sil ipa returns no results
ipa sil returns a single result - the expected sil_ipa keyboard.

The search should be able to handle searches with words out of order like this.

gff_musnad keyboard name appearing as (?????)

The gff_musnad keyboard was created in keymanapp/keyboards#1089
The keyboard_info description has GFF Musnad (𐩣𐩯𐩬𐩵)

But https://api.keyman.com/keyboard/gff_musnad lists the keyboard name as
GFF Musnad (????) and that's how it appears on https://keyman.com/keyboards/gff_musnad

chore: Simple api to return current database build and tier

This could go on api.keyman.com/_status and return something like this:

{
"host": "api.keyman.com",
"tier": "TIER_PRODUCTION",
"schema": "k0",
"database_build_date": "2020-12-01T14:03:15Z",
"database_build_state": "READY"
}

Where database_build_state can be "READY", "MUST_REBUILD", "BUILDING". If "BUILDING", maybe it will give a "database_build_start_time"

schema can be "k0", "k1", "production_k0", "production_k1".

keymanapp / api.keyman.com Goto Github PK

api.keyman.com's People

Contributors

Stargazers

Watchers

Forkers

api.keyman.com's Issues

Keyboards matching 'lana'

Languages matching Lana:

Recommend Projects

Recommend Topics

Recommend Org