keymanapp / api.keyman.com Goto Github PK
View Code? Open in Web Editor NEWhttps://api.keyman.com/ source
https://api.keyman.com/ source
Because the model_info and keyboard_info folders are not purged during the database build, old model_info and keyboard_info files can be left behind. These need to be purged as part of the database build before unzipping the new files.
Except it isn't fixed.
Can't for the life of me see how the folder is being created (nothing else creates it, no other threads or processes):
if(is_dir($this->models_path)) {
deleteDir($this->models_path) || fail("Unable to remove folder " . $this->models_path);
}
if(!is_dir($this->models_path)) {
mkdir($this->models_path, 0777, true) || fail("Unable to create folder " . $this->models_path);
}
Failing on mkdir
with Warning: mkdir(): File exists
Originally posted by @mcdurdin in #135 (comment)
https://api.cloudflare.com/#zone-purge-files-by-cache-tags-or-host
This will enable refreshed api data to be available more promptly on deployments and allow us to extend the cache lifetime.
Because this takes a while, it would be helpful for the report to giving details on what takes longest for future analysis.
Sample run currently:
Downloading https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
Downloading https://downloads.keyman.com/data/keyboard_info.zip
Downloading https://downloads.keyman.com/data/model_info.zip
Triggered database build for keymanapp/keyboards and keymanapp/lexical-models
=============================================================================
Running D:\home\site\wwwroot\tools\db\build/search.sql
Running D:\home\site\wwwroot\tools\db\build/search-queries.sql
Running D:\home\site\wwwroot\tools\db\build/model-queries.sql
Running D:\home\site\wwwroot\tools\db\build/legacy-queries.sql
Running D:\home\site\wwwroot/.data/language-subtag-registry.sql
Running D:\home\site\wwwroot/.data/iso639-3.sql
Running D:\home\site\wwwroot/.data/iso639-3-name-index.sql
Running D:\home\site\wwwroot/.data/ethnologue_language_codes.sql
Running D:\home\site\wwwroot/.data/ethnologue_country_codes.sql
Running D:\home\site\wwwroot/.data/ethnologue_language_index.sql
Running D:\home\site\wwwroot/.data/keyboards.sql
Running D:\home\site\wwwroot/.data/models.sql
Running D:\home\site\wwwroot\tools\db\build/search-prepare-data.sql
The query https://keyman.com/keyboards?q=lana returns
but following the languages link, it's not finding nod
(The Boonkit keyboard is now under nod-Lana
)
See
build_keyboard_script.php
search-queries.sql
Recently saw this message in the developer console during testing:
A cookie associated with a cross-site resource at http://api.keyman.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
We'll probably need to make some adjustments corresponding to this.
Looking at the current state of https://keyman.com/bookmarklet/:
Given that we support a number of Khmer keyboards (as an example), why isn't this listing offering any of them?
I'm not sure the circumstances, but from the Sentry logs it appears there's times the Cloud catalog queries return
[]
and the Keyman Engine JSON parsing code throws an exception treating it as a JSONObject.
I think I'd like to verify the root cause of this on api.keyman.com as well -- and make sure that there are not circumstances where this is happening, wrongly.
Originally posted by @mcdurdin in keymanapp/keyman#7611 (comment)
I was investigating keymanapp/keyman#3134 to see if I could migrate legacy deprecated keyboards with the package-version endpoint. (e.g. replace us
and european2
keyboards with the latest sil_euro_latin.kmp
)
The following queries return "kmp" links to a directory (kmp doesn't exist):
https://api.keyman.com/package-version?platform=android&keyboard=european2
{
"keyboards": {
"european2": {
"version": "1.6",
"kmp": "https://downloads.keyman.com/keyboards/european2/1.6/"
}
}
}
https://api.keyman.com/package-version?platform=android&keyboard=us
{
"keyboards": {
"us": {
"version": "1.0",
"kmp": "https://downloads.keyman.com/keyboards/us/1.0/"
}
}
}
A couple questions:
In the current keyboard_info spec,
The languages
field can be an array or object.
For #183, the keyboard_info will need to include display names. In order to do this:
Member | Required? | Notes |
---|---|---|
displayName | ||
languageName | ||
scriptName | ||
regionName |
Examples:
(replacing fr-ca
with an example not involving suppressed script)
"dnj-Latn-LR" : {
"displayName" : "Dan (Latin, Liberia)",
"languageName": "Dan",
"scriptName": "Latin",
"regionName": "Liberia" }
"und-fonipa" : {
"displayName": "IPA",
"languageName": "Undetermined" }
Also worth checking bcp to see if we can insert data more quickly. We don't need fk constraints except for test purposes (it's a read only database!)
Originally posted by @mcdurdin in #114 (comment)
https://docs.microsoft.com/en-us/sql/relational-databases/import-export/use-unicode-character-format-to-import-or-export-data-sql-server?view=sql-server-ver15
https://docs.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver15
https://docs.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-ver15
(Extracted from #87)
In 14.0, the mobile apps are going away from using the cloud keyboard catalogs.
api.keyman.com/package-version should be designed so that apps can query for keyboard and lexical model updates (pass keyboard IDs and lexical model IDs for parameters).
The return should include latest versions and download links.
Draft of update spec
[Window Title]
Keyman Developer
[Content]
Unable to successfully upload problem details: <br />
<b>Notice</b>: Undefined index: api_keyman_com_github_user in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>14</b><br />
<br />
<b>Notice</b>: Undefined index: api_keyman_com_github_repo in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>15</b><br />
<br />
<b>Notice</b>: Undefined property: stdClass::$number in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>123</b><br />
<br />
<b>Notice</b>: Undefined property: stdClass::$id in <b>C:\Projects\keyman\sites\api.keyman.com\script\desktop\10.0\exception\index.php</b> on line <b>132</b><br />
Failed:
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/v3"
}
[OK]
Right now, the database build is a synchronous step at the end of the keyboard build and deploy, with a ping to a webhook endpoint. The process can be slow, particularly the import into SQL Server. This needs to be an async process with a log file reported ... somewhere so that we don't hit application or web server timeouts.
In reference to keymanapp/keyman#2785:
KeymanWeb's bookmarklet is currently experiencing issues with the API-returned keyboard stub. On KMW 13.0.105, https://api.keyman.com/cloud/4.0/keyboards?jsonp=keyman.register&languageidtype=bcp47&version=13.0&keyboardid=sil_uganda_tanzania@ikx&timerid=3 returns:
keyman.register({"options":{"context":"keyboard","dateFormat":"standard","device":"any","keyboardBaseUri":"https://s.keyman.com/keyboard/","fontBaseUri":"https://s.keyman.com/font/deploy/","keyboardid":"sil_uganda_tanzania@ikx","keyboardVersion":"current"},"keyboard":[{"id":"sil_uganda_tanzania","name":"Uganda-Tanzania Bantu (SIL)","filename":"sil_uganda_tanzania/1.1.1/sil_uganda_tanzania-1.1.1.js","version":"1.1.1","source":"https://github.com/keymanapp/keyboards/tree/master/release/sil/sil_uganda_tanzania","devices":{"phone":2,"tablet":2,"desktop":2},"languages":[]}],"timerid":"3"});
Note this tidbit at the very end: "languages":[]
. As ikx
isn't directly matching anything in the Cloud API's database, no languages are returned... which causes KMW to not register the keyboard.
Setup internal links for back-end API calls to use internal.api.keyman.com (non-proxied form of api.keyman.com); these avoid trips outside the datacenter for PHP calls to the backend. Requires also DNS setup.
Verify that all appropriate indexes are in place (especially check short searches such as 'r', 'ra'). (Fix: #111)
Loading /keyboards does a redirect to q=&page=1
which is kinda unnecessary! (Fix: keymanapp/keyman.com#173)
Currently, the keyboard database update takes a few seconds, which we can live with in most cases. However, we do run into lock contention, which is breaking things.
For example:
https://build.palaso.org/viewLog.html?buildId=196661&buildTypeId=Keyboards_Upload&tab=buildLog&_focus=13357
Results of this are broken keyboard search, broken keyboard downloads, broken kmw toolbar etc. So nasty.
For greater stability, I think we should update our keyboard database build to use rotation: e.g. build to keyboards_1 and at the very end, update a global database parameter specifying the version of the keyboard database to use. If we do this carefully, we should avoid any downtime at all.
When I do
https://api.keyman.com/search?q=l:*
it's what shows for Portuguese.
This is what the Portuguese data looks like:
{
"id": "pt",
"name": "Portuguese",
"keyboards": [
"basic_kbdbr",
"basic_kbdbr",
"basic_kbdpo",
"brazilian_portuguese",
"brazilian_portuguese",
"european",
"european2",
"european2",
"kbdbr",
"kbdpo",
"malar_braille",
"portuguese",
"portuguese_abnt",
"portuguese_abnt",
"portuguese_abnt2",
"portuguese_abnt2",
"sil_euro_latin",
"transazerty"
]
},
There are several duplicates there. I could improve my import routine to handle it, but it seems like ideally it should be fixed in the Keyman data, unless I'm misunderstanding something.
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$ban-bali in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$kaw-bali in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$ar-dz in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$aii-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$bjf-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$bhn-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$cld-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$hrt-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$tmr-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$kqd-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$lhs-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$arc-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
[25-Nov-2019 02:50:50 America/Los_Angeles] PHP Notice: Undefined property: stdClass::$oar-syrc in D:\home\site\wwwroot\script\legacy\legacy40.php on line 409
...
Goes with keymanapp/keyboards#337
Pattern: https://keyman.com/go/package/download[/type]/package_id?[version=package_version][&platform=platform][&tier=tier][&bcp47=bcp47][&update=0|1]
This pattern will be returned from package download APIs on api.keyman.com:
http://api.keyman.com.local/package-version?keyboard=khmer_angkor,bar&keyboard=baz&model=zoo,nrc.en.mtnt&platform=windows
http://api.keyman.com.local/windows/14.0/update?package_khmer_angkor=&tier=alpha&version=14.0
Also, lexical model support for packages. See type parameter above, any lexical model APIs below:
http://api.keyman.com.local/model?q=nrc.en.mtnt
Extracted from https://docs.google.com/document/d/1rhgMeJlCdXCi6ohPb_CuyZd0PZMoSzMqGpv1A8cMFHY/edit?ts=5f17e421#heading=h.mio0p3wdzdye
Depends on keymanapp/keyman.com#155.
One thing I noticed: if I search for the keyboard on keyman, there are zero search results if I include more than one word.
Let me show you what I mean.
The search results for just the first word: https://keyman.com/keyboards?q=aleph
The search results for the first two words: https://keyman.com/keyboards?q=aleph%20with
Can this be fixed?
This turns out to be an oddity in SQL Server Full Text Search which we haven't struck with other keyboards. 'With' is a 'stop word', and so it gets ignored in a phrase search. This does feel a little broken. It's possible to disable stop words; we'll think about doing that.
Originally posted by @mcdurdin in keymanapp/keyboards#2148 (comment)
FUTURE:
TODO:
Include hints on how to use search (advanced search link?) -- extrapolated from Doug's feedback: (FIX: keymanapp/keyman.com#149)
The new site gives 27 keyboards in response to fulfulde.
The old gives only 1, and it is not a good one.
However, the new site returns no results for the code ffm
whereas the old site gives one language, under which there are 4 keyboards.
Update database layer to use schemas for switching instead of separate databases. (FIX: #96, #97)
Separate deprecated keyboards out of default search: https://community.software.sil.org/t/looking-for-testers-keyboard-search-refresh/3570/3?u=marc_keyman (FIX: #99, keymanapp/keyman.com#147)
strip out generic words such as “language” and “keyboard”. https://community.software.sil.org/t/looking-for-testers-keyboard-search-refresh/3570/5 (FIX: keymanapp/keyman.com#148)
skip the “page 1 of 0” when there are no results! https://community.software.sil.org/t/looking-for-testers-keyboard-search-refresh/3570/5 (FIX: keymanapp/keyman.com#146)
http://api.keyman.com.local/search?q=l:id:str-latn is matching nothing. Searching for str-latn
returns str
as expected, but losing the
str-latn
which is needed for the keyboard itself. So there is something mismatching here. (via slack https://keymanapp.slack.com/archives/C6Q9WS09G/p1593670613153900) (FIX: #98)
Use https://keyman.com/go/package/download/ pattern for package download APIs on api.keyman.com. (https://docs.google.com/document/d/1rhgMeJlCdXCi6ohPb_CuyZd0PZMoSzMqGpv1A8cMFHY/edit#heading=h.mio0p3wdzdye)
Embedded search is a bit wasteful on space. Let's restructure it (e.g. Home link, keyman.com link could be together on right, with Keyboard Search and search box on left, all on one line) (Fix keymanapp/keyman.com#153, keymanapp/keyman#3463)
Uncaught TypeError: Cannot read property 'q' of undefined
at init (search.js:282)
at load_search (search.js:311)
when using in embedded search; I searched for "khmer" and then clicked the first link. URL: http://keyman.com.local/keyboards/khmer_angkor?bcp47=km&embed=windows (Fix: #150)
Setup internal links for back-end API calls to use internal.api.keyman.com (non-proxied form of api.keyman.com); these avoid trips outside the datacenter for PHP calls to the backend. Requires also DNS setup. (Will be fixed in #107)
Deleting all text from search box doesn't reset search (FIX: keymanapp/keyman.com#149)
Verify that all appropriate indexes are in place (especially check short searches such as 'r', 'ra'). (Will be fixed in #107)
Non-Unicode keyboards are not listed as 'obsolete' yet (FIX: keymanapp/keyman.com#151)
===========================================================================================================================
DONE:
Additional Notes: Notes on api.keyman.com changes for langtag consumption
example keyboard: burushaski_girminas, khw-latn "Khowar (Latin)"
ខ្មែរ finds zero results but ខ្មែ finds 7...
Show a 'popular keyboards' list for the empty search -- this can also be the search engine jumping-off point.
"Show obsolete keyboards" needs an indication of the change of status ("Hide obsolete keyboards") and needs to be outdented. Also
needs some thought with paginated results.
Too many pages leads to overwhelming number of page links at bottom (e.g. s:latin)
http://api.keyman.com.local/search?q=l gives 500
el_dinka appears to have non-canonical bcp47 codes -- search finds it no trouble.
Show list of associated languages+scripts+countries in keyboard deatils (and related keyboards?)
For in-app download links, include information on searched language code (if available), for default language install (#1456)
Match fields in json should be integer or float where possible, not string! (and update schema accordingly)
schema for match type should be restrictive to actual types used
Search "spa" vs "spanish" -- the weighting could be better. Similar "ger" vs "german". (probably need length-based match weight override)
REFACTOR: region vs country
REFACTOR: code vs id vs tag
Pagination
Need to give more detail on failed links (and make it easier to find in logs, so tweak the broken link search a node wrapper)
Searches for keyboard ids should work
Phrases are not working yet (need to split into either a phrase search or separate words)
Searches for bcp47 tags, scripts, regions should work
FAIL: http://api.keyman.com.local/search/2.0?f=1&q=l:%, c:%, etc.
PHP Fatal error: Uncaught PDOException: SQLSTATE[IMSSP]: The active result for the query contains no fields. in C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php:236
Stack trace:
#0 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php(236): PDOStatement->fetchAll()
#1 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php(77): KeyboardSearch->GetSearchQueries()
#2 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php(73): KeyboardSearch->WriteSearchResults()
#3 C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.php(50): KeyboardSearch->GetSearchMatches()
#4 {main}
thrown in C:\Projects\keyman\sites\api.keyman.com\script\search\2.0\search.inc.php on line 236
Default search should return a FLAT LIST of KEYBOARDS ONLY with highlights. e.g. 'Thai' should return keyboards with 'Thai' in the name, in a language name, or in the country associated with the language.
Search results must be weighted (summed?)
a) match of primary language name 1.0
b) match of alternate language name 0.3
c) match of keyboard name or id 1.0
d) match of script name 1.0
e) match of country name 0.5
f) match on term in description 0.5
g) match quality (whole word match = 1.0, down to 0.1 for further distance? as a multiplicand)
select * from t_langtag_name inner join containstable(t_langtag_name, name, 'isabout (thai weight (1.0), "thai*" weight (0.5))') as KEY_TBL ON t_langtag_name._id = KEY_TBL.[KEY] order by [RaNK] desc
5 / 5 = 1.0
4 / 5 = 0.8
1 / 5 = 0.2
NOTE: final weighting is different but ... let's see how it goes
Can also specify a search:
?q=l:<term>
search for keyboards that support a language, by name (does not check id)
?q=l:id:<id>
search for keyboards that support a language, by bcp 47 id
?q=c:<term>
search for keyboards that support languages within a country
?q=c::id:<id>
search for keyboards that support languages within a country, by iso 3166 id
?q=s:<term>
search for keyboards that support a script
?q=s:id:<id>
search for keyboards that support a script by script id
?q=id:<id>
search for keyboards that match the id
?q=legacy:<id>
search for keyboards that match the legacy id, only one returned!
Should be able to specify alternate names? Searches should match on NFKD with diacritics stripped.
<br />
<b>Warning</b>: sizeof(): Parameter must be an array or an object that implements Countable in <b>C:\Projects\keyman\sites\api.keyman.com\script\search\search.php</b> on line <b>202</b><br />
<br />
<b>Warning</b>: sizeof(): Parameter must be an array or an object that implements Countable in <b>C:\Projects\keyman\sites\api.keyman.com\script\search\search.php</b> on line <b>209</b><br />
Process:
langtags.json is the replacement for alltags and should contain enough information to allow us not to import most of the other data sources we currently reference.
Note also this from @mhosken:
We now have a staging for langtags.json, using the same &staging=1 (or &staging=true).
I have added two new fields to the langtags.json specifically to help Keyman. The "windows" field gives a form of the tag that is appropriate for use on Windows where the only script field that can be removed is for languages with an appropriates suppress-script field in the IANA registry. I have also added that flag as a "suppress" field.
These changes are simple additions and therefore cause a minimal version bump.
Other changes in the staging data set is the addition of new iso639 codes recently approved.
I am waiting on a new set of Ethnologue data before we are really ready to release. So I'm not ready to flick the switch quite yet. But this is certainly ready for testing.
Note also: this yet to be released public document from MS:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/70feba9f-294e-491e-b6eb-56532684c37f?redirectedfrom=MSDN
which puts the whole addition of a windows field into question.
Originally posted by @mcdurdin in #87 (comment)
After #20 and #21, api.keyman.com/cloud/4.0 now returns lower-cased language IDs
( e.g. from str-Latn
to now str-latn
).
Should /model and legacy APIs also be updated to lower-case language IDs?
Otherwise, we get into a situation where the apps don't associate lexical models with a keyboard language.
Describe the bug
In Keyman for Android, I clicked on Settings/ Installed Languages and searched for "German". I installed "EuroLatin (SIL) (German language)". On the "Select languages for EuroLatin (SIL)" I didn't select any language, but clicked Install.
I would have expected that this adds EuroLatin for German, but instead it gets installed for Plautdietsch.
If I select German on the "Select languages for EuroLatin (SIL)" page before clicking Install, it installs it for German, but additionally for Plautdietsch.
Keyman for Android:
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice: Undefined index: devanagari_inscript in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning: Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice: Undefined index: kyrghyz in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning: Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice: Undefined index: urdu in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning: Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Notice: Undefined index: uzbek in D:\home\site\wwwroot\script\legacy\legacy40.php on line 317
[18-Dec-2019 12:25:17 America/Los_Angeles] PHP Warning: Invalid argument supplied for foreach() in D:\home\site\wwwroot\script\legacy\legacy40.php on line 319
Since #45 is an issue, it'd probably be best for us to also ensure that Bookmarklets use the full, available BCP-47 code in their calls so that any issues with "closest match" scenarios may be minimized.
For example, noting the bookmarklet calling https://api.keyman.com/cloud/4.0/keyboards?jsonp=keyman.register&languageidtype=bcp47&version=13.0&keyboardid=sil_uganda_tanzania@ikx&timerid=3, it really should be using @ikx-latn
, not @ikx
- the former's the one actually in the cloud API database.
Currently, the last update data for keyboards is always the last publish date for the whole database. This is not right and should actually match the last time the keyboard is changed in the repository; probably we can use the git logs for this:
git log -n 1 --format="format:%cI" -- release/gff/gff_amharic
This probably needs to be done in the .keyboard_info build process though.
Builds on #5. At present, searching for a keyboard in the database with the legacy searches does not exclude deprecated keyboards. It should.
it looks like that keyboard isn’t being returned; looks like a recent adjustment to our keyboard API may have been a bit overzealous. @Marc_Keyman, it looks like deprecated keyboards are now never returned by the API, even if a deprecated keyboard is specifically requested. While this is probably fine for general, language-code only requests, this breaks pages that rely on specific keyboards once they are deprecated.
Seems the appropriate resolution here is to not return deprecated keyboards if generic searches are made (query: do we have a way of detecting whether the search is coming from the Keyman app as opposed to websites? would need to check this).
Discussion: https://community.software.sil.org/t/keymanweb-keyboard-is-not-funcitoning-php/3088/7
Two things we could improve in keyboard search. Neither of them are high priority.
c:id:in
gives disordered weighting due to multi-language keyboards which get exaggerated weights.max(weight)
instead of sum(weight)
in f_keyboard_search_results
but that reduces most searches tof_keyboard_search
patterns sp_keyboard_Search_by_legacy_id
, by_id
, by_language_bcp47_tag
?When using the URL https://keyman.com/go/linux/17.0/download-keyboards and searching for "German", the first result ("EuroLatin (SIL)(German language)") has the bcp47 code pdt instead of de.
Things work with the regular search page (https://keyman.com/keyboards?q=german).
A language-code lookup of 'km' now default-returns Khmer (NiDA) ( basic_kbdkni) instead of Khmer Angkor ( khmer_angkor), and 'und-fonipa' returns the deprecated ipauni11 instead of sil_ipa keyboard. I doubt it's intentional, but wonder what severity we'd call that.
Arose from KeymanWeb testing in keymanapp/keyman#2528.
http://keyman.com.local/keyboards/languages/el?page=2&obsolete=0
vs
http://keyman.com.local/keyboards/languages/el?page=2&obsolete=1
The stats need to exclude obsolete keyboards on this search.
If a keyboard is not present, it's okay to return 400 from the endpoint. But the onlineupdate check should avoid writing a warning to the log!
[25-Nov-2019 02:50:36 America/Los_Angeles] PHP Warning: file_get_contents(https://api.keyman.com/keyboard/pyidaungsutwo): failed to open stream: HTTP request failed! HTTP/1.1 400 Fail: Keyboard not found
in D:\home\site\wwwroot\tools\onlineupdate.php on line 116
Currently, we mark deprecation with the deprecates
relation. This gives us one direction. However, this requires a secondary query to determine whether a keyboard is deprecated, which is not ideal for searches. We should build the corresponding relation for each keyboard as a deprecatedBy
relation.
Then we need to update the search and cloud APIs to flag deprecated keyboards.
This work comes out of keymanapp/keyboards#374.
[02/25/2021 20:58:27 > fc5cb9: INFO] Running D:\home\site\wwwroot/.data/iso639-3.sql
[02/25/2021 20:58:30 > fc5cb9: INFO] Array
[02/25/2021 20:58:30 > fc5cb9: INFO] (
[02/25/2021 20:58:30 > fc5cb9: INFO] [0] => 21S01
[02/25/2021 20:58:30 > fc5cb9: INFO] [1] => 110
[02/25/2021 20:58:30 > fc5cb9: INFO] [2] => [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]There are fewer columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement.
[02/25/2021 20:58:30 > fc5cb9: INFO] )
[02/25/2021 20:58:30 > fc5cb9: INFO] Failure: PDOException: SQLSTATE[21S01]: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]There are fewer columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement. in D:\home\site\wwwroot\tools\db\build\build.inc.php:148
[02/25/2021 20:58:30 > fc5cb9: INFO] Stack trace:
[02/25/2021 20:58:30 > fc5cb9: INFO] #0 D:\home\site\wwwroot\tools\db\build\build.inc.php(148): PDO->exec()
[02/25/2021 20:58:30 > fc5cb9: INFO] #1 D:\home\site\wwwroot\tools\db\build\build.inc.php(67): BuildDatabaseClass->sqlrun()
[02/25/2021 20:58:30 > fc5cb9: INFO] #2 D:\home\site\wwwroot\tools\db\build\build_cli.php(30): BuildDatabaseClass->BuildDatabase()
[02/25/2021 20:58:30 > fc5cb9: INFO] #3 {main}
[02/25/2021 20:58:30 > fc5cb9: INFO]
There are fewer columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement.
This suggests the format of the ISO639-3 data has changed.
This error is preventing the build from completing.
Finish keyboard install page (aka universal link infrastructure) for:
Originally posted by @mcdurdin in #87 (comment)
It is observed that some keyboards are not included in returned JSON data of API call - https://api.keyman.com/cloud/4.0/keyboards
Examples :-
sil ipa
returns no resultsipa sil
returns a single result - the expected sil_ipa keyboard.The search should be able to handle searches with words out of order like this.
The gff_musnad keyboard was created in keymanapp/keyboards#1089
The keyboard_info description has GFF Musnad (𐩣𐩯𐩬𐩵)
But https://api.keyman.com/keyboard/gff_musnad lists the keyboard name as
GFF Musnad (????)
and that's how it appears on https://keyman.com/keyboards/gff_musnad
This could go on api.keyman.com/_status and return something like this:
{
"host": "api.keyman.com",
"tier": "TIER_PRODUCTION",
"schema": "k0",
"database_build_date": "2020-12-01T14:03:15Z",
"database_build_state": "READY"
}
Where database_build_state can be "READY", "MUST_REBUILD", "BUILDING". If "BUILDING", maybe it will give a "database_build_start_time"
schema can be "k0", "k1", "production_k0", "production_k1".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.