dyalog/jarvis

APL-based web service framework supporting JSON or REST

jarvis's Issues

All but GET method seem to need a none empty body

I created simple functions for Get, Post, Put, ... like so:

r←Get req
r←req.Endpoint

Then I issued a request from Insomnia.

For Get everything works fine.
For all other methods, I need something in my request body to make it work.
Otherwise jarvis ignores it.

Especially for Delete this makes no sense to me.

Help needed,
Thx,
Karsten

│Jarvis│1.16.3│2024-01-12│

IDE:
  Version: 4.4.3687
  Platform: MacIntel
  Date: 2022-02-23 13:44:34 +0100
  Git commit: a3d8123cfb690ead493e261ccced59407db04c64
  Preferences:{
    "colourScheme":"Dracula",
    "confirmations":"{\"SaveFileOptionsExtension:.aplf\":100,\"SaveFileOptionsExtension:.apln\":100}",
    "kbdLocale":"en_US_Mac",
    "lbarOrder":"← +-×÷*⍟⌹○!? |⌈⌊⊥⊤⊣⊢ =≠≤<>≥≡≢ ∨∧⍲⍱ ↑↓⊂⊃⊆⌷⍋⍒ ⍳⍸∊⍷∪∩~ /\\⌿⍀ ,⍪⍴⌽⊖⍉ ¨⍨.⍣∘⍤⍥@ ⍞⎕⍠⌸⌺⌶⍎⍕ ⋄⍝→⍵⍺∇& ¯⍬ ",
    "menu":"# see below for syntax\n\nDyalog                          {mac}\n  About Dyalog             =ABT\n  -\n  Preferences...           =PRF\n  -                            \n  &Quit                    =QIT\n&File                           {!browser}\n  &Open...                 =OWS {local}\n  &New Session             =NEW\n  &Connect...              =CNC\n  -                             {!mac}\n  &Quit                    =QIT {!mac}\n&Edit\n  Undo                     =UND {!browser}\n  Redo                     =RDO {!browser}\n  -                             {!browser}\n  Cut                      =CT  {!browser}\n  Copy                     =CP  {!browser}\n  Paste                    =PT  {!browser}\n  Select All               =SA  {mac}\n  -                             {mac||!browser}\n  &Find...                 =SC\n  Find and &Replace...     =RP\n  -                             {!mac}\n  Preferences...           =PRF {!mac}\n&View\n  Show Language Bar        =LBR\n  Show Status Bar          =SBR\n  Show Workspace Explorer  =WSE\n  Show Debug               =DBG\n  Line Wrapping in Session =WRP\n  -                             {!browser}\n  Show Status Window       =SSW\n  Auto Status              =ASW\n  -                             {!browser}\n  Stops                    =TVB\n  Line Numbers             =LN\n  Outline                  =TVO\n  -                             {!browser}\n  Increase Font Size       =ZMI {!browser}\n  Decrease Font Size       =ZMO {!browser}\n  Reset Font Size          =ZMR {!browser}\n  -                             {!browser}\n  Toggle Full Screen            {!browser}\n&Window\n  Close All Windows        =CAW\n&Action\n  Edit                     =ED\n  Trace                    =TC\n  -\n  Clear all trace/stop/monitor =CAM\n  Weak Interrupt           =WI\n  Strong Interrupt         =SI\n&Threads                        {rp21}\n  Pause on Error           =POE\n  Pause all Threads        =PAT\n  Unpause all threads      =UAT\n  Continue all threads     =MA\n&My\n  Dyalog\n    Mastering APL          =https://mastering.dyalog.com/README.html\n  -\n  APL Cart                 =https://aplcart.info\n  APL Wiki                 =https://aplwiki.com\n&Help\n  Getting &Started         =https://dyalog.com/introduction.htm\n  -\n  Dyalog &Help             =DHI\n  &Language Elements       =LEL\n  &Documentation Centre    =DOX\n  -\n  Dyalog &Website          =https://dyalog.com/\n  &Email Dyalog            =EMD\n  -\n  Latest &Enhancements     =ENH\n  Read &Me                 =RME\n  &Third Party Licences    =TPL\n  &About                   =ABT {!mac}\n\n# Syntax:\n#   &x   access key, alt+x\n#   =CMD command code; some are special:\n#          LBR WRP WSE render as checkboxes\n#   =http://example.com/  open a URL\n#   {}   conditional display, a boolean expression\n#          operators: && || ! ( )\n#          variables: local browser mac win\n#   -    separator (when alone)\n#   #    comment\n\n# The =PRF (\"Preferences...\") menu item must be present.",
    "otherExe":"/Applications/Dyalog-18.2.app/Contents/Resources/app/../Dyalog/mapl",
    "wse":"1",
    "zoom":"3"
  }

Interpreter:
  Version: 18.2.45505
  Platform: Mac-64
  Edition: Unicode/64
  Date: Apr  7 2022 at 00:12:12

Consider making IncludeFns and ExcludeFns properties rather than fields

As suggested at Dyalog'22 SA2 workshop

Provide setting to kill request-handling thread on connection close

If the client connection is closed while an endpoint (requested by the connection) is executing, provide a switch to tell Jarvis to kill the request-handling thread.

Sort out CodeLocation and JarvisConfig in relation to Jarvis.dws Autostart

Currently, if Jarvis is loaded using the Jarvis.dws workspace, relative file paths are determined from the workspace location. This has some implications for the dyalog/jarvis container because it uses that workspace. I believe a more flexible (and user friendly) solution would be to use the location of the JarvisConfig file, if it exists, as the root folder from which relative folders are found. This makes Jarvis apps more portable.

Web Interface does not work?

Am I correct in understanding that the web interface does not currently work? When visiting the root of a server I get the following:

      LAUNCH
2020/02/12 @ 21:20:02 - Jarvis started in "JSON" mode on port 8080
2020/02/12 @ 21:20:02 - Serving code in #.server
2020/02/12 @ 21:20:02 - Click http://localhost:8080 to access web interface
0   
2:VALUE ERROR: Undefined name: n
ScriptFollows[2] r←{⍵/⍨'⍝'≠⊃¨⍵}{1↓¨⍵/⍨∧\'⍝'=⊃¨⍵}{⍵{((∨\⍵)∧⌽∨\⌽⍵)/⍺}' '≠⍵}¨(1+n⊃⎕LC)↓↓(180⌶)2⊃⎕XSI

KillSession is not Optionally Called

The KillSession test here will fail at some point because when Power Op is called with a function instead of a value, it will always run at least once. This means that KillSession will always run at least once. This will also result in potential errors depending on the datatypes used for sessions and a number of other undesirable behaviors.

Jarvis/Source/Jarvis.dyalog

Line 548 in d2c5e53

→resp⊣KillSession⍣~ns.Req.Fail 400×0∊⍴ns.Req.GetHeader SessionIdHeader

Web interface fails if exposed functions contain ∆

The function HtmlPage needs a UTF-8 conversion of the result, else the result has type 160 and cannot be returned via Conga.

Allow dynamic session creation

Allow a setting such that the client can be assigned a session without first calling StartSessionEndpoint.
Such a request could also include basic HTTP authentication credentials.

Allow .apla (APL array) files to be loaded

Currently, only code files (.dyalog, .apl[fonc]) are loaded. .apla files could also be loaded.

One complication is that the APL array notation has evolved over time and we need to take that into account.

Add MaxPayloadSize setting

Add a setting to limit the maximum payload size for a request. Default ¯1 meaning "no limit".

Not an issue

Definitely not an issue: more a curiosity. I am intrigued by this line:

Jarvis/Source/Jarvis.dyalog

Line 479 in 8c53cea

Connections←#.⎕NS''

Why the #.? I have very similar code in one of my projects (unsurprisingly since I copied it from JsonServer) but I don't think I went for the "#.". Is there a good reason to create that particular namespace as a child of root?
Thank you!

Don't close connection for in-process request

The problem:
Jarvis has a "ConnectionTimeout" setting intended to clean up connections that have gone dormant. The default value is 30 seconds. Currently, we may inadvertently close a connect for a request to endpoint that runs for longer than ConnectionTimeout.

The suggestion:
Add a "in-progress" flag to the connection namespace that the connection reaper will respect and not close connections for requests that are being serviced.

Adding default support for CORS in the REST paradigm

You’ll notice that I set up the interface for the HandleCORSRequest function so that it will signal whether there was a CORS request in its result. The hope here is that by doing this, the same function might be able to be used for the REST handler as well, just that the REST handler will be able to “pass along” instead of returning right away as the JSON handler does. The question remains then, how to appropriately set the Methods allowed header for the REST defaults.

As an initial default, so that there are no API changes required, it might be good to just use the RESTMethods defaults. The HandleCORSRequest handler could be modified to take an optional left argument that contains the methods that will be supported. This might allow the handler to be more flexibly deployed on demand at different locations once the support methods are known.

My current vote for "fast to initial working" solution is to add an optional left argument to HandleCORSRequest that will set the methods, and then pass RESTMethods as that left argument when we call HandleCORSRequest from the HandleRESTRequest handler. [That's a lot of hands.]

Break links to source files when building workspace

DYALOG_JARVIS_THREAD should default to "1" and not "DEBUG"

If the default is DEBUG, then the container will exit immediately unless -it or RIDE_INIT are set.

Unlocalized `r` in `HandleRequest`

With the merge of the KillSession adjustments, they use a r under the assumption that it was the return value.

Jarvis/Source/Jarvis.dyalog

Line 547 in 03ae42e

→resp⊣KillSession⍣(~ns.Req.Fail 400×0∊⍴r)⊢r←ns.Req.GetHeader SessionIdHeader

With HandleRequest being changed to not return a value, this r is no longer localized.

RFE: _htmlDefaultPage should be a public field, ideally also configurable via jarvis.json

And if empty, any requests that would return the defaultPage should return a 404. (minimal processing to avoid too much CPU load through "invalid activity")

Implement better handling in event where user fails the request

If the user does a request.Fail in the endpoint code, Jarvis should not do any further processing on the response.

Simpler `sint`

Jarvis/Source/Jarvis.dyalog

Line 1723 in 4c23d76

sint←{⎕IO←0 ⋄ 83=⎕DR ⍵:⍵ ⋄ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 ¯128 ¯127 ¯126 ¯125 ¯124 ¯123 ¯122 ¯121 ¯120 ¯119 ¯118 ¯117 ¯116 ¯115 ¯114 ¯113 ¯112 ¯111 ¯110 ¯109 ¯108 ¯107 ¯106 ¯105 ¯104 ¯103 ¯102 ¯101 ¯100 ¯99 ¯98 ¯97 ¯96 ¯95 ¯94 ¯93 ¯92 ¯91 ¯90 ¯89 ¯88 ¯87 ¯86 ¯85 ¯84 ¯83 ¯82 ¯81 ¯80 ¯79 ¯78 ¯77 ¯76 ¯75 ¯74 ¯73 ¯72 ¯71 ¯70 ¯69 ¯68 ¯67 ¯66 ¯65 ¯64 ¯63 ¯62 ¯61 ¯60 ¯59 ¯58 ¯57 ¯56 ¯55 ¯54 ¯53 ¯52 ¯51 ¯50 ¯49 ¯48 ¯47 ¯46 ¯45 ¯44 ¯43 ¯42 ¯41 ¯40 ¯39 ¯38 ¯37 ¯36 ¯35 ¯34 ¯33 ¯32 ¯31 ¯30 ¯29 ¯28 ¯27 ¯26 ¯25 ¯24 ¯23 ¯22 ¯21 ¯20 ¯19 ¯18 ¯17 ¯16 ¯15 ¯14 ¯13 ¯12 ¯11 ¯10 ¯9 ¯8 ¯7 ¯6 ¯5 ¯4 ¯3 ¯2 ¯1[utf8 ⍵]}

can be simplified to

sint←{⎕IO←0 ⋄ 83=⎕DR ⍵:⍵ ⋄ ((⍳,-+⍳)128)[utf8 ⍵]}

same goes for HttpCommand:
https://github.com/Dyalog/HttpCommand/blob/2659ea89357f85cbe10d38797500c629c1f2cb8b/source/HttpCommand.dyalog#L958

Seemingly random crashes related to "OSError" "Undefined name: CON000000XX"

I am very new to APL, but I have been working on a proof-of-concept web application to demonstrate the potential use of Dyalog / Jarvis to others. However, I keep getting these seemingly random crashes - sometimes it only takes a minute, sometimes it takes half an hour, but Jarvis always seems to exit with a cryptic "OS / Undefined Name" error. I could be wrong, but I don't think this is related to the code I am running. I've tried hitting my endpoints with good data and error cases, but I can't find a pattern.

I've attached the error I've been receiving along with the code I am running under Jarvis. I am using the Jarvis docker container, and have included the command invoking it. Is this an issue with Jarvis itself, or just a problem with the code I'm running?

apl-birdie-app.zip
output.txt

For what its worth, I've tried this both under WSL and Xubuntu and it seems to behave the same in both cases.

Let me know if you have any questions :)

Update Workspaces for Dyalog 17.0 Compatibility

As our policy is to support the 3 most recent releases of Dyalog, that would at present be 18.2, 18.0, and 17.0. As such we need to use 819⌶ for case conversion (at least until Dyalog 19.0 is released and we can use ⎕C).

The Jarvis and JarvisService workspaces contained an earlier version of Jarvis that inadvertently used ⎕C. This was corrected in the Jarvis source code, but the workspaces had not been updated.

No response from Server?

I am getting no response when I run the Jarvis Demo:

      srv.CodeLocation←'..\Samples\JSON\'
      srv.Start
2020/02/12 @ 21:53:35 - Jarvis started in "JSON" mode on port 8080
2020/02/12 @ 21:53:35 - Serving code in #.CodeLocation (populated with code from "C:/Users/arcfi/OneDrive/Documents/Jarvis/
      Samples/JSON/")
2020/02/12 @ 21:53:35 - Click http://localhost:8080 to access web interface
0   
      ]load HTTPCommand
#.HttpCommand
      cmd←⎕NEW HttpCommand
      cmd.(Command URL)←'POST' 'localhost:8080/GetSign'
      cmd.Headers⍪←'content-type' 'application/json'
      cmd.Params←'[10,31]'
      q←cmd.Run
      q.rc
0
      q.Data

      q
[rc: 0 | msg: "" | HTTP Status: 200 "OK" | ⍴Data: 0]

Any chance of supporting GZip, deflate please.

It would be really useful.

Thanks Michael

Should we handle Cross-origin Request handling automatically?

The current Jarvis doesn't appear to handle CORS situations when browsers access the data. Is there a clean and minimal way to do this in Jarvis itself that doesn't get too "over-engineered?"

EndPoints needs to be ⎕ML/⎕IO insensitive

Since Jarvis.EndPoints now looks at the ⎕AT of possible endpoints and this is executed within the CodeLocation hierarchy, and CodeLocation may have non-default settings for ⎕ML and ⎕IO, EndPoints needs to be ⎕ML and ⎕IO insensitive.

Update Jarvis container to run under v19

Please update the container to use v19.

"ConfigFile" settings conflicts with name of Dyalog Settings

Maybe prefix with JARVIS_ to avoid clashes?

Add HTTP compression

Add HTTP compression to Jarvis

Should URLDecode come after splitting the query parameters?

It seems in the following line, that the URL is decoded after splitting on '?' but before splitting on & and = in the query string. This would seem to be incorrect, as this would treat encoded & and = values as query separators, which they are not.

Jarvis/Source/Jarvis.dyalog

Line 874 in 048bf3b

QueryParams←2↑[2]↑'='(≠⊆⊢)¨'&'(≠⊆⊢)query

POST support error only triggers on a get method

The following line will only be checked if the method is a 'get' method.

Jarvis/Source/Jarvis.dyalog

Line 561 in d2c5e53

→0 If('Request method should be POST')ns.Req.Fail 405×~HtmlInterface

There are many more methods that might be passed besides 'get' and 'post' and the current handler assumes that if the method is not a 'get' method, then it is a 'post' method.

Are we checking for the right headers all the time?

It appears that the check below is intended to prevent the standard JSON Server header checking in cases when we are trying to access the HTML interface. However, it appears that checking is also disabled when the HTMLInterface is set to 0. If this is the case, then it seems that we will never check the headers for correctness, even though it seems that we should?

Jarvis/Source/Jarvis.dyalog

Line 528 in ccd05ff

:If HtmlInterface>(⊂ns.Req.Endpoint)∊(,'/')'/favicon.ico'

Add hook for custom HTTP failures

When a request fails (HTTP Status ≠ 2XX) enable the user to define a custom handler

Failing to respond on unexpected data in JSON Server mode

If you are attempting to access a route and something unexpected happens, such as a GET request instead of a POST request for the JSON Server, the server does not respond with any data. I believe this is due to a resource leak located around here:

Jarvis/Source/Jarvis.dyalog

Line 529 in ccd05ff

→0⍴⍨'(Request method should be POST)'ns.Req.Fail 405×'post'≢ns.Req.Method

This line and the two/three after it appear to return from the HandleRequest function having updated the response object with errors, but never actually sending the response. As a result, I believe that HandleRequest returns a 0 code at the end and does not expunge the object, but then does nothing with it. This essentially leaves the object in la la land for an indeterminant amount of time and the result is an unresponsive system on the front end.

When a new, valid request comes in, it appears to trigger something that pushes out some clean up and results in an error code 1105 for these responses:

2020/02/15 @ 20:59:20 - RunServer: DRC.Wait reported error 1105  ERR_RECV  Could not receive data  on SRV00000000.CON00000000 (IP Address [::1]:53760)
2020/02/15 @ 20:59:20 - RunServer: DRC.Wait reported error 1105  ERR_RECV  Could not receive data  on SRV00000000.CON00000001 (IP Address [::1]:53761)

AutoStart fails if AuthenticateFn has wrong valence, but does not show the info about the error.

I happened to define an ambivalent AuthenticateFn where a monadic one was expected.
The session output (after starting APL with jarvis.dws and a configfile-argument) was:

⍎Unable to start server - "#.CITA_Server. 0
      Server←AutoStart
      ∧

Bug in running secure server

CreateSecureParms has a logic error when not using ServerCertSKI

Is it necessary to expunge the connection here?

It appears to me that expunging the obj is handled in the call to HandleRequest when it returns a 1. Is there a reason that we use an r=0 and then manually expunge the object at the following line?

Jarvis/Source/Jarvis.dyalog

Line 539 in ccd05ff

Connections.⎕EX obj

Allocate Port <1024 under *nix

Add code to allow starting as root to allocate a port <1024 and then switch to non-root userid

Add support for Swagger UI

Swagger UI is a free library used for API documentation.
The use of it is so wide spread it can be considered industry standard.
In C# the library is available as a nuget package that can be installed and configured for each individual project.
https://learn.microsoft.com/en-us/aspnet/core/tutorials/web-api-help-pages-using-swagger?view=aspnetcore-7.0
It would be nice if this was available in Jarvis as well, preferably by default.

Return 431 status when headers are too large

When a request's headers exceed BufferSize we currently handle it as a DRC.Wait error and close the connection. I believe the more proper way to handle it is to issue a HTTP 431. This will give the client more information on how to correct the request.

431 Request Header Fields Too Large

The server is unwilling to process the request because its header fields are too large. The request may be resubmitted after reducing the size of the request header fields.