Change the amount of data returned by the api

[stoplightio]

Change the amount of data returned by the api

[manwithsteelnerves]

My suggestion would be
As pagination is done by using a common parameter for all api's, would request to allow an option in config to set the param name for quantity. This way we don't need to set it every time when requesting. And also, what ever rules we apply for limit will be considered inherently. Prism should see if the query parameter exists (as per the configuration) and serves accordingly.

ex: http://www.example.com/products?limit=30
This should return 30 products if we set in prism config that quantityKey=limit.

If you are not using a common query parameter for paging (which is more unlikely), there should be an extra extension which defines it. Ex: x-stoplight-prism-quantity-key : count (For API's where limit is replaced with count)

Hope it makes sense.

[Mvrlex]

I like the suggestion from @manwithsteelnerves
But as you want to implement data persistence somewhere in the future I would suggest directly implementing both parameters for pagination. limit and offset, with offset being optional.
What also has to be considered is the collection in which the data will be generated into (nesting) and sizes of non paginated collections.
Consider following examples:

Nested collection

{
  "entrySet": [ 
    { "id": 1, "name": "string" } 
  ],
  itemCount: 50
]

In this case we cannot directly use limit, because we don't know where to generate the data.
My suggestion for this would be to add a extension like: x-prism-pagination-collection, to mark the correct collection.

For example:

    "EntrySet": {
      "type": "object",
      "x-prism-pagination-collection": "entrySet",
      "properties": {
        "entrySet": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/Entry"
          }
        },
        "itemCount": {
          "type": "integer",
          "format": "int32"
        }
      }
    }

Or just use the first array you find :P

Non paginated collections
Not every collection will be paginated (for example some deeply nested collection), but we may still want to change the size of those collections for testing. I would suggest just adding another extension: x-prism-collection-min and x-prism-collection-max to the schema definitions. So these collections will be generated with a range of items by default.

For example:

    "EntrySet": {
      "type": "object",
      "properties": {
        "entrySet": {
          "type": "array",
          "x-prism-collection-min": 0,
          "x-prism-collection-max": 5,
          "items": {
            "$ref": "#/definitions/Entry"
          }
        },
        "itemCount": {
          "type": "integer",
          "format": "int32"
        }
      }
    }

Now we dont have to use limit to get a range of items. It just generates them automatically.

We could set 0-5 as a default for collections and maybe also add a parameter __collection-max=10 so you can dynamically change it.

[philsturgeon]

I wouldn’t say a limit is required no, APIs could chose to simply say 100 is how many you’ll get. It could also be a different parameter name than “limit” could be “num” or “number” but we could look for multiple param names.

[XVincentX]

Check out https://github.com/stoplightio/prism/pull/984/files — the PR has also the documentation changes. Write me in case you need more details

[digitalcrafted]

Hi @enyosolutions

Looking at the same issue as you.

Did you ever get to solve this ?

[CameronFraser]

hi was there ever a resolution to this?

[enyosolutions]

No never :( Le sam. 18 mai 2019 à 10:05 PM, Cameron F. <[email protected]> a écrit :

…

hi was there ever a resolution to this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20?email_source=notifications&email_token=ACALAYDLW7AX5FQQZVQRCD3PWBVTPA5CNFSM4DMND2Y2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVWV73Q#issuecomment-493707246>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACALAYA5P632S5NB3H6ECG3PWBVTPANCNFSM4DMND2YQ> .

[CameronFraser]

Is this open source version purposefully stunted because there is a paid product? The api generation works so well, yet there doesn't seem to be anyway to add additional configuration. There seems to be a config option on the stoplight.io mock reference for prism but this prism doesn't seem to recognize the --config command https://help.stoplight.io/prism/getting-started/custom-configuration

[philsturgeon]

Hey @CameronFraser, definitely not. Prism v2 development slowed down because Prism v3 was in the works, a total rewrite which has just hit alpha. We are working really hard on getting a whole bunch of awesome features into it, and some of these older issues were closed down to clear the deck a bit.

So talk to me about this feature request.

@enyosolutions when you say you want to return more than just one or two lines, what are we talking about here? Are we saying that for collections we want to control how many dynamically generated resources are placed in that collection?

I could get behind that.

[CameronFraser]

Hey @philsturgeon, glad to hear and thanks for your work on the project, what I've tried so far works really well. The request you described for controlling how many dynamically generated resources are placed in a collection is what I was hoping for. It is a nice feature for data being served to paginated views. However I think with those before and after hooks I saw in the config documentation I should be able to do this myself (once available)? Does the schema get passed to those functions so one could use jsf to generate more data?

[enyosolutions]

@philsturgeon yes exactly that. This woul very usefull for example when mocking a collection for a scroll related behavior for example..

[philsturgeon]

Excellent ideas. Working with a collection of 1 - or however many happen to be in any examples is not going to be enough in a lot of situations.

v3 has a --dynamic / ?__dynamic flag, meaning the default is "static" examples, or you can pick examples. It sounds like the functionality we want is part of the dynamic examples logic, and you're right @CameronFraser JSON Schema Faker would certainly help out here.

Would something like ?__dynamicCollection=50 be the sort of thing you had in mind?

It's possible down the line we could offer more specific pagination options which can autodetect popular pagination standards (JSON:API kinda, Links / Ranges), etc., but I think for now we need a more DIY solution, especially for v3.0.

[philsturgeon]

Cone on folks, help me help you! :D

How would you imagine this feature to look if you could have it any way you liked. Give me something to work with.

[enyosolutions]

Hi @philsturgeon,

I would personally keep it simple with a --quantity parameter that would accept a number or a value dynamic, or using
--minItems N --maxItems XXX.

A --dynamic item by itself would be great if the range would very variable. In order to tests scrolls / paginations scenarios.

Few examples of what i had in mind.

prism --quantity dynamic --minItems 10 --maxItems 50
prism --quantity 15
prism --minItems 10 --maxItems 50

[pytlesk4]

What about a prism config file, thoughts?

This is a personal opinion, but I don't like the idea of using CLI options/arguments to control finer grain settings.

• .prism or prism.json or prism.yaml, could specify the location, or it would prism would just try to load config from the CWD.

Example Structure

{
  "dynamic": {
    "minItems": 10,
    "maxItems": 50,
    "quantity": 15
  }
}

Don't know what quantity is, but included it in the example above.

If we do use a config file, then we should map all command line options/arguments to properties in the config.

Example Usage

prism mock
prism mock --config path/to/config

[philsturgeon]

As pagination is done by using a common parameter for all api's...

Unfortunately this is not true. A lot of APIs work with limit/offset, but some use cursors, and various forms of continuation token.

https://phauer.com/2017/web-api-pagination-continuation-token/

Pagination is about as consistent for APIs as "Security" and that has a whole construct in OpenAPI.

Do you think we could do something as cowboy as --records=100 which will just set the first, highest array in the response to have 100 items?

[philsturgeon]

We can certainly add support for "limit" and "num" to make this work in runtime for those who do happen to use those variable names.

[manwithsteelnerves]

True about using cursors(ex: Firestore uses cursors for pagination) but still limit is a required attribute to limit number of entries retrieved right?

[VenkatRamReddyK]

Can you please let me know if we could achieve this in any way ?

[XVincentX]

@VenkatRamReddyK Unfortunately there's no way at the moment.

[lornajane]

I am +1 on having more dynamic data generated and having that controlled by configuration and/or command line switches. I use the mock server when working on our SDKs and I'm alarmingly confident at how well they work with one record returned!

I'm not sure about trying to link to API pagination behaviour, that sounds more involved. Maybe it would be possible to start with the simplest implementation of multiple records and then get feedback on further linking into API behaviour?

[XVincentX]

@lornajane I am leaning towards using the Prefer header to instruct how much data you want back, what do you think?

[lornajane]

A header would also be welcome, feels similar to the query params like __example or __code that I'm already using and enjoying!

[XVincentX]

Nice! Keep in mind we want to remove the query parameters support in the next breaking change of Prism, so I suggest to start migrating to the Prefer header!

[lornajane]

Where can I read more about what I can send with Prefer? I only found ?__code mentioned in an issue when I was looking for something else. (Links to source code also accepted if docs are not available!)

[AnthonyPorthouse]

With the Prefer method approach perhaps we could allow both a random amount, say between 10 and 25, or an exact amount, i.e. 100.

For a range: Prefer: collection_min=10,collection_max=25
For a specific amount Prefer: collection_count=100

This would still need some extra thought on how you'd handle nested collections.

For example

{
    "items": [
        {
            "tags": ["ABC", "XYZ"]
        }
    ]
}

How would any implementation interact with tags?

I think we may also need a way of specifying which collections get affected by this behaviour.

[AnthonyPorthouse]

@philsturgeon

Do you think we could do something as cowboy as --records=100 which will just set the first, highest array in the response to have 100 items?

If you were using a HAL implementation then the _links key could very well include the first array you'd encounter at the same highest level as a collection under a key.

[philsturgeon]

Seeing as there are three things being discussed here, I'd like to split these down into distinct issues.

1. Making a specific number of records appear somehow #1139
2. Making a random number of records appear somehow
3. Pagination

I have the issue for 1 (#1139), and this seems like a good amount of scope for now.

If anyone here feels strongly about the other issues then please create an issue and link it back here so others can follow along.