dashbitco / nimble_options Goto Github PK
View Code? Open in Web Editor NEWA tiny library for validating and documenting high-level options. π½
License: Apache License 2.0
A tiny library for validating and documenting high-level options. π½
License: Apache License 2.0
Based on this schema, which, as far as I know, at the root is a keyword_list
:
iex(3)> schema = [
...(3)> max_connections: [type: :pos_integer, default: 100],
...(3)> timeout: [type: :pos_integer, default: :timer.seconds(5)]
...(3)> ]
[
max_connections: [type: :pos_integer, default: 100],
timeout: [type: :pos_integer, default: 5000]
]
iex(4)> NimbleOptions.validate([], schema)
{:ok, [timeout: 5000, max_connections: 100]}
Works as expected, meaning that if you don't provide any value for those options, it will take the default value. But when you have a nested keyword_list
like this:
iex(8)> schema = [
...(8)> http_adapter: [
...(8)> type: :keyword_list,
...(8)> keys: [
...(8)> max_connections: [type: :pos_integer, default: 100],
...(8)> timeout: [type: :pos_integer, default: 5000]
...(8)> ]
...(8)> ]
...(8)> ]
[
http_adapter: [
type: :keyword_list,
keys: [
max_connections: [type: :pos_integer, default: 100],
timeout: [type: :pos_integer, default: 5000]
]
]
]
iex(9)> NimbleOptions.validate([], schema)
{:ok, []}
You don't get the same behavior, I mean, the response from NimbleOptions.validate([], schema)
should be {:ok, [http_adapter: [timeout: 5000, max_connections: 100]]}
, right?
But, when you "execute" at least one key from the nested keyword_list
it works as expected:
iex(10)> NimbleOptions.validate([http_adapter: [max_connections: 50]], schema)
{:ok, [http_adapter: [timeout: 5000, max_connections: 50]]}
Shouldn't we traverse the nested keyword_list
regardless if it's given or not as key looking for default values inside?
As a workaround, or I don't know if this is the recommended schema configuration, you can define the following:
iex(11)> schema = [
...(11)> http_adapter: [
...(11)> type: :keyword_list,
...(11)> default: [max_connections: 100, timeout: :timer.seconds(5)],
...(11)> keys: [
...(11)> max_connections: [type: :pos_integer, default: 100],
...(11)> timeout: [type: :pos_integer, default: :timer.seconds(5)]
...(11)> ]
...(11)> ]
...(11)> ]
[
http_adapter: [
type: :keyword_list,
default: [max_connections: 100, timeout: 5000],
keys: [
max_connections: [type: :pos_integer, default: 100],
timeout: [type: :pos_integer, default: 5000]
]
]
]
iex(12)> NimbleOptions.validate([], schema)
{:ok, [http_adapter: [max_connections: 100, timeout: 5000]]}
iex(13)> NimbleOptions.validate([http_adapter: [timeout: 2000]], schema)
{:ok, [http_adapter: [max_connections: 100, timeout: 2000]]}
iex(14)> NimbleOptions.validate([http_adapter: []], schema)
{:ok, [http_adapter: [timeout: 5000, max_connections: 100]]}
But it feels kind of redundant having this block:
default: [max_connections: 100, timeout: 5000],
keys: [
max_connections: [type: :pos_integer, default: 100],
timeout: [type: :pos_integer, default: 5000]
]
Don't you think? Please let me know what do you think about this.
I think we are missing at least:
:string
(or :binary
?):boolean
I think those are pretty common and they showed up in the first time I tried out nimble_options on a work project. Thoughts?
Currently we call it "spec" but it is a very overloaded term and people will confuse it with libraries that actually use specs for option validation.
I proposed "definition" in an earlier PR but @whatyouhide proposed "schema". Do you have any preferences? /cc @wojtekmach @msaraiva
A list of choices is already supported via {:in, ...}
. Unfortunately, there's no easy way to document the choices.
Example:
@pets [:cat, dog]
@schema [
pet: [
type: {:in, @pets},
doc: "Type of pet."
]
]
@doc """
## Options
#{NimbleOptions.docs(@schema)}
"""
If you do this, there is no reference in the docs to the actual values the :pet
key can assume. If you want to document that, you need to do so manually:
@schema [
pet: [
type: {:in, @pets},
doc: ~s"""
Type of pet.
Must be one of:
* `:cat` - feline
* `:dog` - canine
"""
]
]
This isn't ideal. And it has a habit of growing out of sync with the actual values.
We add a struct of some sort that allows choices to self-document. For example:
@pets [
%NimbleOptions.Choice{
value: :cat,
doc: "feline"
},
%NimbleOptions.Choice{
value: :dog,
doc: "canine"
},
]
@schema [
pet: [
type: {:in, @pets},
doc: "Type of pet."
]
]
@doc """
## Options
#{NimbleOptions.docs(@schema)}
"""
Then if the type is {:in, ...}
and the element is a %NimbleOptions.Choice{}
(or whatever name makes sense), then the docs builder can add the "Must be one of..." part automatically.
It would be nice to be able to:
1.) Include the key's in the error messages and
2.) Support getting all of the errors at once
I have another layer of error reporting that I pass the result of NimbleParsec.validate/2
into, and previously I was using a library I wrote https://github.com/albert-io/optimal for validating options. However, I'd rather use this, even though it doesn't have some of the features, since it is backed by a pillar of the community :D
I was surprised to find that only :non_neg_integer
and :pos_integer
were supported, but this seems like a pretty common type that should be included by default. Thoughts?
If I'm not mistaken, the type_doc
option seems to be unused in the project? Maybe I'm mistaken, but I can't see any effect of using this option.
A big quality of life improvement for our usage of NimbleOptions
would be the following:
into
option for nested keyword listsAn example would be something like:
defmodule Discombobulator do
require NimbleOptions
@whirligig_schema [...]
NimbleOptions.define_struct(WhirligigOptions, @whirligig_schema)
@schema [
times: [
type "How many times to frobulate"
],
distance: [
type: "How far to reticulate the splines"
],
whirligigs: [
type: :keyword,
schema: @whirligig_schema,
into: WhirligigOptions
]
]
NimbleOptions.define_struct(Options, @schema)
def discombobulate(opts) do
case NimbleOptions.validate(opts, @schema, into: Options) do
{:ok, %Options{}} -> ...
{:error, error} -> ...
end
end
end
The benefit of having options specified as a struct is that you don't need to traverse the options list many times to lookup keys (small but tangible for options validated/accessed often), and you can get editor/language server hints on your options.
I think we could get a lot of the benefits without #1, where we require that the structs be hand written, but then you do end up needing to duplicate the keys and do up the typespecs yourself, that kind of thing.
Thoughts?
non_neg_integer | :infinity
thoughts?
As I was looking for a recursive type, for my use case to validate a recursive tree. I discovered that #3 added recursive type support using the lazy function but no longer works. I suspect a regression when a dead code cleanup took place in #26.
As the documentation generation has a different implementation, the @options_schema in the NimbleOptions moduledoc has worked since but the validation don't.
Using one of the unit test as example:
defmodule Foo do
def recursive_schema() do
[
*: [
type: :keyword_list,
keys: [
type: [
type: :atom,
required: true,
doc: "The type of the option item."
],
required: [
type: :boolean,
default: false,
doc: "Defines if the option item is required."
],
keys: [
type: :keyword_list,
doc: "Defines which set of keys are accepted.",
keys: &Foo.recursive_schema/0
],
default: [
doc: "The default."
]
]
]
]
end
end
> NimbleOptions.validate([], Foo.recursive_schema())
** (ArgumentError) invalid schema given to NimbleOptions.validate/2. Reason: expected :keys to be a keyword list, got: &Foo.recursive_schema/0 (in options [:*, :keys, :keys])
(nimble_options 0.3.5) lib/nimble_options.ex:233: NimbleOptions.validate/2
> NimbleOptions.docs(Foo.recursive_schema())
" * `:type` - Required. The type of the option item.\n\n * `:required` - Defines if the option item is required. The default value is `false`.\n\n * `:keys` - Defines which set of keys are accepted.\n\n * `:default` - The default.\n\n"
But as this is not officially supported in the documentation, I don't know if it is a feature or a bug.
I posted a question about Nimble Options' one_of
here. Is it possible to express nested options as arguments of one_of
? Is there an example I can reference? I'd be happy to post a PR for the docs if someone can explain what choices
can be in the usage Β {:one_of, choices}
Is there any interest in having a macro something along the lines of
@definition nimble_schema do
key :connections, :non_neg_integer, default: 5
key :url, :string, required: true
end
that generates
@definition [
connections: [
type: :non_neg_integer,
default: 5
],
url: [
type: :string,
required: true
]
]
If so, I'd be happy to work on this :)
I'm trying to write a schema to parse options for a HTTP library, and I'd love to do something like:
@options_schema NimbleOptions.new!([
url: [
type: {:or, [:atom, {:struct, URI}]}
]
])
Given this module:
defmodule Test do
@options_schema NimbleOptions.new!(
map: [type: :map]
)
@type option() :: unquote(NimbleOptions.option_typespec(@options_schema))
end
Loading this up in Visual Studio Code with ElixirLS produces a Dialyzer error
(CaseClauseError) no case clause matching: :map
Stacktrace:
β (nimble_options 0.5.0) lib/nimble_options/docs.ex:163: NimbleOptions.Docs.type_to_spec/1
β (nimble_options 0.5.0) lib/nimble_options/docs.ex:156: anonymous fn/1 in NimbleOptions.Docs.schema_to_spec/1
β (elixir 1.14.1) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
β (nimble_options 0.5.0) lib/nimble_options/docs.ex:155: NimbleOptions.Docs.schema_to_spec/1
β test.ex:5: (module)
It looks like NimbleOptions.Docs.type_to_spec/1
doesn't support :map
or {:map, key_type, value_type}
.
Broken off from #44
When trying to validate a list of keyword lists, I would assume the following would work:
definitions = [
my_items: [
type: {:list, :keyword_list},
keys: [key: [type: :string]]
]
]
NimbleOptions.validate!([
my_items: [
[key: "value"],
[key: "value"]
]
], definitions)
But instead I'm met with the following errors
** (ArgumentError) expected a keyword list, but an entry in the list is not a two-element tuple with an atom as its first element, got: [key: "value"]
(elixir 1.12.1) lib/keyword.ex:475: Keyword.keys/1
(nimble_options 0.4.0) lib/nimble_options.ex:388: NimbleOptions.validate_unknown_options/2
(nimble_options 0.4.0) lib/nimble_options.ex:376: NimbleOptions.validate_options_with_schema_and_path/3
(nimble_options 0.4.0) lib/nimble_options.ex:409: NimbleOptions.reduce_options/2
(elixir 1.12.1) lib/enum.ex:4251: Enumerable.List.reduce/3
(elixir 1.12.1) lib/enum.ex:2402: Enum.reduce_while/3
(nimble_options 0.4.0) lib/nimble_options.ex:402: NimbleOptions.validate_options/2
(nimble_options 0.4.0) lib/nimble_options.ex:377: NimbleOptions.validate_options_with_schema_and_path/3
As a developer I want to have the ability to treat some options as aliases of others, similar to command line arguments where -q
could be equivalent to --quiet
Specifically in scenic the goal is to allow developers to specify the translate (as one example) as either :t
or :translate
:
https://github.com/boydm/scenic/blob/b2e57b50737443d8fb7a6d7ef2e88ef704cf42c2/lib/scenic/primitive/transform/translate.ex#L21-L29
This is currently implemented using :rename_to
which is now deprecated in 0.4:
https://github.com/boydm/scenic/blob/b2e57b50737443d8fb7a6d7ef2e88ef704cf42c2/lib/scenic/primitive/transform/transform.ex#L61-L68
It would be nice to be able to do this without having to post-process the output of nimble_options (one specific downside to this is having to customize the output of NimbleOptions.docs/2).
This is effectively a request to include :other_names
mentioned in #68 (comment) although I think :aliases
would be a better name for it.
The docs for :rename_to
state:
Renames a option item allowing one to use a normalized name internally, e.g. rename a deprecated item to the currently accepted name.
Based on those docs I wouldn't expect that the position of :rename_to
in the schema would matter, but it does. Specifically I'd expect this test to pass:
test "the order of rename_to does not matter" do
schema1 = [
context: [rename_to: :new_context],
new_context: [type: {:custom, __MODULE__, :string_to_integer, []}]
]
schema2 = [
new_context: [type: {:custom, __MODULE__, :string_to_integer, []}],
context: [rename_to: :new_context]
]
assert NimbleOptions.validate([context: "1"], schema1) ==
assert(NimbleOptions.validate([context: "1"], schema2))
end
But instead it fails with this error:
1) test :rename_to the order of rename_to does not matter (NimbleOptionsTest)
test/nimble_options_test.exs:165
Assertion with == failed
code: assert NimbleOptions.validate([context: "1"], schema1) == assert(NimbleOptions.validate([context: "1"], schema2))
left: {:ok, [context: "1", new_context: 1]}
right: {:ok, [context: "1", new_context: "1"]}
stacktrace:
test/nimble_options_test.exs:176: (test)
So it appears that a value that goes through :rename_to
does not get the custom module's validation function run on it if :rename_to
is after the key that is being renamed.
Note: This caused a bug in Scenic
π hello!
I have a minimal example repo here: https://github.com/the-mikedavis/nimble_example, with the schema definition here and tests that exhibit the behaviour here
Given a schema like so (similar to a snippet in the documentation):
[
foo: [
type: :keyword_list,
keys: [*: [type: :atom]]
]
]
we can properly validate a set of input options:
iex(1)> NimbleOptions.validate([foo: [bar: :baz]], schema())
{:ok, [foo: [bar: :baz]]}
but trying to document this schema will fail:
iex(2)> NimbleOptions.docs(schema())
** (Protocol.UndefinedError) protocol Enumerable not implemented for nil of type Atom. This protocol is implemented for the following type(s): Map, Stream, Range, Date.Range, List, GenEvent.Stream, Function, File.Stream, HashDict, MapSet, IO.Stream, HashSet
(elixir 1.11.0) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir 1.11.0) lib/enum.ex:141: Enumerable.reduce/3
(elixir 1.11.0) lib/enum.ex:3461: Enum.reduce/3
(nimble_options 0.3.5) lib/nimble_options/docs.ex:61: NimbleOptions.Docs.option_doc/2
(elixir 1.11.0) lib/enum.ex:2181: Enum."-reduce/3-lists^foldl/2-0-"/3
(nimble_options 0.3.5) lib/nimble_options/docs.ex:6: NimbleOptions.Docs.generate/2
I'm using these versions
tool | version |
---|---|
elixir | 1.11.0 |
erlang | 23.1 |
nimble_options | 0.3.5 |
OS | ubuntu 16.04 |
thanks!
Broken off from #44
I've been using NimbleOptions for a while and there is one thing I can't figure out how to handle. Sometimes I want to have an option that doesn't have to be set. For example, I have a function that can query a database in different ways. I want to have two options: name_contains
and category
, both should be string
, but I want to be able to do a query with only one of these options (or both).
I can set require
to false
, but then I need a default. But if I set default
to nil
, the validation will fail, since nil
isn't a string.
I can of course set the type to type: {:or, [:string, :atom]}
, but then I allow every atom and it doesn't really communicate what I actually mean.
Since this has happened almost every time I've used NimbleOptions I'm wondering if I'm missing something? If not, would it be a good idea to add an option for allow_nil
? Or add :nil
as a type, so I can at least do {:or, [:string, :nil]}
to not allow all atoms and more clearly communicate that nil is ok?
It seems that the latest nimble_options
release was Nov 9, 2020, that's quite a while :)... and I could really benefit from the float
support added after #63
Thanks <3
https://hexdocs.pm/finch/Finch.html#start_link/1-pool-configuration-options
** (EXIT) an exception was raised:
** (NimbleOptions.ValidationError) unknown options [:size], valid options are: [:protocol, :size, :count, :max_idle_time, :conn_opts, :pool_max_idle_time, :conn_max_idle_time]
(finch 0.12.0) lib/finch.ex:145: anonymous fn/2 in Finch.pool_options!/1
unknown options [:size], valid options are: [:protocol, :size, ....]
looks not so obvious
Is there any appetite for a :in_domain
type to be supported directly by the library? I've been using custom type to do it but I figure that it's probably a common enough use case that others might benefit from it being in the library.
Something like:
def in_range(value, left_bracket, min, max, right_bracket) do
in_range? =
case {left_bracket, min, max, right_bracket} do
{_, nil, nil, _} ->
true
{_, nil, max, :closed} ->
value <= max
{_, nil, max, :open} ->
value < max
{:closed, min, nil, _} ->
value >= min
{:open, min, nil, _} ->
value > min
{:closed, min, max, :closed} ->
value >= min and value <= max
{:open, min, max, :closed} ->
value > min and value <= max
{:closed, min, max, :open} ->
value >= min and value < max
{:open, min, max,:open} ->
value > min and value < max
end
if in_range?, do: {:ok, value}, else: {:error, "Value #{value} is not in range"}
end
EDIT: Originally I used the term range
but I changed it to domain
to avoid confusion w/ the keyword
Occasionally, providing a default at compile-time just doesn't work. For example, if I want to provide a timestamp and have it default to DateTime.utc_now()
, I can't with nimble options. Instead I have to default to nil
and manually handle it (or something like that).
It would be cool if run-time-evaluated defaults were supported. Perhaps they can be provided as functions, like so?
schema = [
at: [
default: &DateTime.utc_now/0,
doc: "when foo took place"
]
]
I could really use the new rich error messages change in hex (so I can release my hex packages since you can't use GH dependencies in hex).
Is it within scope for NimbleOptions to return all errors and fallback to any specified defaults for fields that have an error? Alternatively if it was possible to get all the failing errors (instead of just the first) it would be reasonable to add the fallback logic in my application code.
Also right now it would be difficult to do that fallback logic because the key is not returned as an atom, only the error message is returned within the %NimbleOptionsVendored.ValidationError{}
.
A pretty common possible option value is a member of a list. An example that comes to mind is Broadway's :batch_mode
in Broadway.test_messages/3
, which is :bulk
or :flush
.
Do we want to support a "member" type?
schema = [
batch_mode: [type: {:member, [:bulk, :flush]}, default: :flush]
]
Do you think it's too soon, too specific, or a good addition?
I was reviewing this lib as I'd like to try it in a project but I missed checks for lists, so I couldn't properly write some schemas. Sure I could write a custom {:custom, mod, fun, args}
type but turns out that list is a common type and maybe could be supported out of the box. So, my proposal is to add the types :list
and :non_empty_list
, similar to the keyword list pair. Does that make sense? Let me know and I could give a shot sending a PR. Thanks for the lib :)
Sometimes some options are internal but you still want to validated them. What do you folks think if we add support for doc: false
to hide an option? I think that aligns with how we do docs in general.
Looks like (in options [:stages])
shouldn't be returned at
keys_path
.
I'm not sure what's the best way to fix this without breaking other cases. Care to give some instructions? :)
I took a look at whether it would be possible and I think it is. Macros don't get expanded inside @type
for some reason so I didn't get this working:
require NimbleOptions.Type
@type opts :: NimbleOptions.Type.generate(schema)
But I think this could work:
require NimbleOptions.Type
NimbleOptions.Type.generate(opts, schema)
Is there a plan to add ability to generate a @type
spec from the schema? Would that be a welcome addition in this project? I could publish a separate package instead
I have a spec that looks like this:
cluster_options_schema = [
connections: [type: :pos_integer, default: 2],
urls: [type: {:custom, __MODULE__, :validate_cluster_urls, []}, required: true]
]
@start_options_schema [
general_cluster: [type: :keyword_list, keys: cluster_options_schema],
messages_cluster: [type: :keyword_list, keys: cluster_options_schema],
ephemeral_cluster: [type: :keyword_list, keys: cluster_options_schema]
]
Now if I validate this and don't pass the required :url
option for one of the three clusters, like this:
NimbleOptions.validate([general_cluster: []], @start_options_schema)
then the returned error doesn't point me to which cluster's options are invalid:
{:error, "required option :urls not found, received options: [:channels_per_connection, :connections]"}
Should we have a :context
key that is supported in :keyword_list
s? Like this:
@start_options_schema [
general_cluster: [type: :keyword_list, keys: cluster_options_schema, context: :general_cluster],
]
The error could become:
{:error, "required option :urls not found, received options: [:channels_per_connection, :connections] (in :general_cluster)"}
Thoughts on how to achieve this, if this belongs in nimble_options, or if there's a better way to do it?
Hello,
What do you think about validating the schema once, so we can define an @attribute
and then directly call the validation with a valid schema?
Thank you
Seeing that: :non_empty_keyword_list, :pos_integer, :non_neg_integer
exist, I think it would make sense to have a :non_empty_string
as well?
What do you think?
I find myself copy-pasting the following implementation to be used to validate a dependency injection via module:
@doc false
def existing_module?(value) do
case Code.ensure_compiled(value) do
{:module, ^value} ->
{:ok, value}
{:error, error} ->
{:error, "Cannot find the requested module βΉ" <> inspect(value) <> "βΊ (#{error})"}
end
end
and use it in options schema declaration as
{:custom, MyValidators, :existing_module?, []}
I believe itβd be a great candidate for another default type. I am all-in to provide a PR if this would be desired.
What do you think about having an option to validate against and set number of options?
For example:
definition = [
status: [
type: :atom,
allowed_values: [:waiting, :started, :finished],
default: :waiting
],
]
The docs for :rename_to
state:
Renames a option item allowing one to use a normalized name internally, e.g. rename a deprecated item to the currently accepted name.
Based on a general understanding of what "rename" means, I would expect :rename_to
to remove the old key, but it does not. Specifically I'd expect this test to pass:
test "removes the old key" do
schema = [
context: [rename_to: :new_context],
new_context: [type: {:custom, __MODULE__, :string_to_integer, []}]
]
assert NimbleOptions.validate([context: "1"], schema) == {:ok, [{:new_context, 1}]}
end
Instead it fails with this error:
1) test :rename_to removes the old key (NimbleOptionsTest)
test/nimble_options_test.exs:156
Assertion with == failed
code: assert NimbleOptions.validate([context: "1"], schema) == {:ok, [new_context: 1]}
left: {:ok, [{:context, "1"}, {:new_context, 1}]}
right: {:ok, [new_context: 1]}
stacktrace:
test/nimble_options_test.exs:162: (test)
I'm not sure if this can be considered a bug, but this behavior did surprise me.
Today, I have this schema:
schema = [
pool_options: [
type: :keyword_list,
default: [],
keys: [
protocol: [type: :atom, default: :http1],
size: [type: :integer, default: 10]
]
]
]
However, if I validate []
I get this:
NimbleOptions.validate!([], schema)
#=> [pool_options: []]
when in reality I'd expect that the nested defaults are propagated up, having the expected behavior be:
NimbleOptions.validate!([], schema)
#=> [pool_options: [protocol: :http1, size: 10]]
@josevalim @msaraiva I can work on this if we agree it's the right call. Thoughts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.