Giter Site home page Giter Site logo

poison's Introduction

Poison

Build Status Coverage Status Hex.pm Version Hex.pm Download Total

Poison is a new JSON library for Elixir focusing on wicked-fast speed without sacrificing simplicity, completeness, or correctness.

Poison takes several approaches to be the fastest JSON library for Elixir.

Poison uses extensive sub binary matching, a hand-rolled parser using several techniques that are known to benefit BeamAsm for JIT compilation, IO list encoding and single-pass decoding.

Poison benchmarks sometimes puts Poison's performance close to jiffy and usually faster than other Erlang/Elixir libraries.

Poison fully conforms to RFC 7159, ECMA 404, and fully passes the JSONTestSuite.

Installation

First, add Poison to your mix.exs dependencies:

def deps do
  [{:poison, "~> 5.0"}]
end

Then, update your dependencies:

$ mix deps.get

Usage

Poison.encode!(%{"age" => 27, "name" => "Devin Torres"})
#=> "{\"name\":\"Devin Torres\",\"age\":27}"

Poison.decode!(~s({"name": "Devin Torres", "age": 27}))
#=> %{"age" => 27, "name" => "Devin Torres"}

defmodule Person do
  @derive [Poison.Encoder]
  defstruct [:name, :age]
end

Poison.encode!(%Person{name: "Devin Torres", age: 27})
#=> "{\"name\":\"Devin Torres\",\"age\":27}"

Poison.decode!(~s({"name": "Devin Torres", "age": 27}), as: %Person{})
#=> %Person{name: "Devin Torres", age: 27}

Poison.decode!(~s({"people": [{"name": "Devin Torres", "age": 27}]}),
  as: %{"people" => [%Person{}]})
#=> %{"people" => [%Person{age: 27, name: "Devin Torres"}]}

Every component of Poison (encoder, decoder, and parser) are all usable on their own without buying into other functionality. For example, if you were interested purely in the speed of parsing JSON without a decoding step, you could simply call Poison.Parser.parse.

Parser

iex> Poison.Parser.parse!(~s({"name": "Devin Torres", "age": 27}), %{})
%{"name" => "Devin Torres", "age" => 27}
iex> Poison.Parser.parse!(~s({"name": "Devin Torres", "age": 27}), %{keys: :atoms!})
%{name: "Devin Torres", age: 27}

Note that keys: :atoms! reuses existing atoms, i.e. if :name was not allocated before the call, you will encounter an argument error message.

You can use the keys: :atoms variant to make sure all atoms are created as needed. However, unless you absolutely know what you're doing, do not do it. Atoms are not garbage-collected, see Erlang Efficiency Guide for more info:

Atoms are not garbage-collected. Once an atom is created, it will never be removed. The emulator will terminate if the limit for the number of atoms (1048576 by default) is reached.

Encoder

iex> Poison.Encoder.encode([1, 2, 3], %{}) |> IO.iodata_to_binary
"[1,2,3]"

Anything implementing the Encoder protocol is expected to return an IO list to be embedded within any other Encoder's implementation and passable to any IO subsystem without conversion.

defimpl Poison.Encoder, for: Person do
  def encode(%{name: name, age: age}, options) do
    Poison.Encoder.BitString.encode("#{name} (#{age})", options)
  end
end

For maximum performance, make sure you @derive [Poison.Encoder] for any struct you plan on encoding.

Encoding only some attributes

When deriving structs for encoding, it is possible to select or exclude specific attributes. This is achieved by deriving Poison.Encoder with the :only or :except options set:

defmodule PersonOnlyName do
  @derive {Poison.Encoder, only: [:name]}
  defstruct [:name, :age]
end

defmodule PersonWithoutName do
  @derive {Poison.Encoder, except: [:name]}
  defstruct [:name, :age]
end

In case both :only and :except keys are defined, the :except option is ignored.

Key Validation

According to RFC 7159 keys in a JSON object should be unique. This is enforced and resolved in different ways in other libraries. In the Ruby JSON library for example, the output generated from encoding a hash with a duplicate key (say one is a string, the other an atom) will include both keys. When parsing JSON of this type, Chromium will override all previous values with the final one.

Poison will generate JSON with duplicate keys if you attempt to encode a map with atom and string keys whose encoded names would clash. If you'd like to ensure that your generated JSON doesn't have this issue, you can pass the strict_keys: true option when encoding. This will force the encoding to fail.

Note: Validating keys can cause a small performance hit.

iex> Poison.encode!(%{:foo => "foo1", "foo" => "foo2"}, strict_keys: true)
** (Poison.EncodeError) duplicate key found: :foo

Benchmarking

$ MIX_ENV=bench mix run bench/run.exs

Current Benchmarks

As of 2021-07-22:

License

Poison is released under the public-domain-equivalent 0BSD license.

poison's People

Contributors

aesthetikx avatar asaaki avatar bennyhallett avatar binaryseed avatar ch4s3 avatar cmpitg avatar devinus avatar filipmnowak avatar fishcakez avatar frost avatar ivan avatar jisaacstone avatar jorge-d avatar jtmoulia avatar kalys avatar mitchellhenke avatar msch avatar nettofarah avatar niku avatar ono avatar optikfluffel avatar rschmukler avatar stevedomin avatar sunaku avatar waldyrious avatar whatyouhide avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

poison's Issues

Poison.Parser.parse failure on weird surrogate pair

I have some json data that may contain some wonky surrogate pairs. jq reports this for it:
parse error: Invalid \uXXXX\uXXXX surrogate pair escape at line L, column C

Here's how poison handles the same:

iex(3)> Poison.Parser.parse("{\"field\":\"\\uD83C\"}")
** (ArgumentError) argument error
    lib/poison/parser.ex:229: Poison.Parser.string_escape/2
    lib/poison/parser.ex:78: Poison.Parser.object_pairs/3
    lib/poison/parser.ex:36: Poison.Parser.parse/2

Cryptic error aside, is there any way I could instruct poison to ignore these errors? Or tell it to not parse some parts? Do you have any suggestions on how to deal with this sort of data?

I'm using 2.2.0 btw.

Thanks a bunch and kudos for an awesome library. โค๏ธ

Edit: Hmm for some reason http://jsonlint.com/ can handle the same input...

possible to improve performance?

in the flamegraph of a simple api server, json parsing is apparently the bottleneck, besides reducing the payload, any idea of making this faster?
screen shot 2016-01-11 at 7 45 18 pm

Encoding a map changes the order of the keys

Encoding a map changes the order of the keys;

iex(6)> Poison.encode %{a: 1, b: 2, c: 3}
{:ok, "{\"c\":3,\"b\":2,\"a\":1}"} # reversed keys

Similarly decoding json changes the order of the keys;

iex(7)> Poison.decode "{\"c\":3,\"b\":2,\"a\":1}", keys: :atoms!
{:ok, %{a: 1, b: 2, c: 3}} # reversed keys

issue when encoding a string with escaped quotes

x = "a "b" c"

Poison.encode!(x) |> Poison.decode!

** (Poison.SyntaxError) Unexpected token: b
(poison) lib/poison/parser.ex:56: Poison.Parser.parse!/2
(poison) lib/poison.ex:30: Poison.decode!/2

IO.puts(Poison.encode!(x) |> IO.iodata_to_binary)

"a "b" c"

Performance issue with struct encoding

I have encountered some performance issues when generating JSON response in Phoenix framework, encoding Ecto.Model to JSON using poison. To nail down the issue, I added some benchmarks,

bench "structs (Poison)", [struct: gen_struct] do
  Poison.encode!(struct)
end

defmodule TheStruct do
  defstruct name: "John", age: 27
end

defp gen_struct do
  1..10 |> Enum.map(&(%TheStruct{name: "John#{&1}", age: &1 }))
end

Encoding structs appears to be extremely slow, e.g.

structs (Poison)                           100   19073.38 ยตs/op

while the same benchmark with Map instead of struct

defp gen_struct do
  1..10 |> Enum.map(&(%{name: "John#{&1}", age: &1 }))
end

gives orders of magnitude better perf,

structs (Poison)                         10000   109.68 ยตs/op

I am total noob as far as elixir is concerned, so I might be doing something totally flawed in the benchmark. The issue is probably not even directly related to the Poison as I have noticed the same performance drop in other Elixir JSON libraries, e.g.

structs (JSX)                              100   18999.76 ยตs/op
structs (Jazz)                             100   19457.10 ยตs/op

Benchmark for structs can be found in commit 6c6827f

Version info:
Elixir 1.0.5
Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Unable to encode keywords

iex(2)> Poison.encode! [{:a, "b"}]
** (Poison.EncodeError) unable to encode value: {:a, "b"}
(poison) lib/poison/encoder.ex:185: Poison.Encoder.Any.encode/2
(poison) lib/poison/encoder.ex:143: Poison.Encoder.List.encode/2
iex(2)> :jsx.encode [{:a, "b"}]
"{"a":"b"}"
iex(3)>

Handling camelCase to snake_case

The Forecast API returns some fields in camelCase and I would like to convert them to snake_case during decoding. Specifically the fields in currently

{
  "latitude": 0,
  "longitude": 0,
  "timezone": "Etc\/GMT",
  "offset": 0,
  "currently": {
    "time": 1420344625,
    "summary": "Light Rain",
    "icon": "rain",
    "precipIntensity": 0.0276,
    "precipProbability": 1,
    "precipType": "rain",
    "temperature": 74.93,
    "apparentTemperature": 74.93,
    "dewPoint": 68.85,
    "humidity": 0.81,
    "windSpeed": 10.29,
    "windBearing": 137,
    "cloudCover": 0.12,
    "pressure": 1013.09,
    "ozone": 249.33
  },
  "flags": {
    "sources": [
      "fnmoc",
      "cmc",
      "gfs"
    ],
    "units": "us"
  }
}

Adding README.md examples as Parser Tests

Hi!

I was looking for a schema-validating library for elixir, and yours seems to best shaped to do what I what.

However, trying to get more out of the as: option, I went straight to the tests and noticed that the examples in the README.md aren't in the test suite. I really think they should be (if you want me to, I can PR for that, but obviously it's your code).

I'm also wondering another thing: what's the limit of the "as" syntax? Say, for example, that I have a top-level array containing objects (that I want to be structs), how should I go around doing that?

This is my current code:

defmodule Parent
  defstruct [:name, :children]
end

defmodule Child
  defstruct [:name]
end

Poison.decode!(
  ~s([{โ€œnameโ€: โ€œaโ€, โ€œchildrenโ€: [{โ€œnameโ€: โ€œchild1โ€}]}, {โ€œnameโ€: โ€œbโ€, โ€œchildren": []}],
  as: ??)

Lack of MapSet encoder

both Set and HashSet in Elixir are being deprecated in favor of MapSet. I think that the lack of this encoder will become a real issue when they both will disappear. I can try to make it happen, but I don't know if I will be able to pull this off.

Nested struct decoding still doesn't seem to work

Steps to recreate:

  • Pulled the master branch (052ad9a)
  • Compiled the app with mix deps.get and mix compile
  • Booted up IEx like so: iex -S mix
  • Pasted the following struct (from the Usage example in the README) into the console:
      defmodule Person do
        @derive [Poison.Encoder]
        defstruct [:name, :age]
      end
  • Pasted the decode call into the console:
Poison.decode!(~s({"people": [{"name": "Devin Torres", "age": 27}]}),
  as: %{"people" => [Person]})

And got:

%{"people" => [%{"age" => 27, "name" => "Devin Torres"}]}
iex(3)>

Seems like this should have been fixed by #61, no?

Thanks ahead of time for looking at this and making this lib! โค๏ธ

Escaping UTF-8

I was expecting the following to return the original string

"รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ" |> Poison.encode! |> Poison.decode!

Instead it returns:

"รƒยกรƒยฉรƒยญรƒยณรƒยบรƒย รƒยจรƒยฌรƒยฒรƒยนรƒยขรƒยชรƒยฎรƒยดรƒยปรƒยฃรกยบยฝร„ยฉรƒยตร…ยฉ"

For comparison

Elixir with Poison:

iex(1)> "รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ" |> Poison.encode!
"\"\\u00C3\\u00A1\\u00C3\\u00A9\\u00C3\\u00AD\\u00C3\\u00B3\\u00C3\\u00BA\\u00C3\\u00A0\\u00C3\\u00A8\\u00C3\\u00AC\\u00C3\\u00B2\\u00C3\\u00B9\\u00C3\\u00A2\\u00C3\\u00AA\\u00C3\\u00AE\\u00C3\\u00B4\\u00C3\\u00BB\\u00C3\\u00A3\\u00E1\\u00BA\\u00BD\\u00C4\\u00A9\\u00C3\\u00B5\\u00C5\\u00A9\""

In Ruby:

irb(main):001:0> require 'json'
irb(main):002:0> "รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ".to_json
=> "\"รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ\""

In Python:

>>> import json
>>> json.dumps("รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ")
'"\\u00e1\\u00e9\\u00ed\\u00f3\\u00fa\\u00e0\\u00e8\\u00ec\\u00f2\\u00f9\\u00e2\\u00ea\\u00ee\\u00f4\\u00fb\\u00e3\\u1ebd\\u0129\\u00f5\\u0169"'

>>> json.loads( json.dumps("รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ") )
'รกรฉรญรณรบร รจรฌรฒรนรขรชรฎรดรปรฃแบฝฤฉรตลฉ'

Trouble encoding nil

Using "poison": {:hex, :poison, "1.5.0"},

    ** (Poison.EncodeError) unable to encode value: {nil, "shows"}
        (poison) lib/poison/encoder.ex:339: Poison.Encoder.Any.encode/2
        (poison) lib/poison/encoder.ex:213: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:213: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:232: anonymous fn/3 in Poison.Encoder.List.encode/3
        (poison) lib/poison/encoder.ex:233: Poison.Encoder.List."-encode/3-lists^foldr/2-1-"/3
        (poison) lib/poison/encoder.ex:233: Poison.Encoder.List.encode/3
        (poison) lib/poison.ex:41: Poison.encode!/2
        (phoenix) lib/phoenix/controller.ex:628: Phoenix.Controller.do_render/4
        (televised) web/controllers/api/v1/shows_controller.ex:1: Televised.Api.V1.ShowsController.phoenix_controller_pipeline/2
        (televised) lib/phoenix/router.ex:265: Televised.Router.dispatch/2
        (televised) web/router.ex:1: Televised.Router.do_call/2
        (televised) lib/televised/endpoint.ex:1: Televised.Endpoint.phoenix_pipeline/1
        (televised) lib/plug/debugger.ex:90: Televised.Endpoint."call (overridable 3)"/2
        (televised) lib/phoenix/endpoint/render_errors.ex:34: Televised.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

Big Ints

I was wondering if you have any suggestion how to handle BigInts or the best way to go about it since JavaScript will truncate the number.

In my internal code I did this.

defimpl Poison.Encoder, for: Integer do
  def encode(integer, options) when integer > 9007199254740992 do
    if options[:bigint_as_string] do
      ~w("#{integer}")
    else
      Integer.to_string(integer)
    end
  end

  def encode(integer, _options) do
    Integer.to_string(integer)
  end
end

Which the linter hates but it does work. I am assuming the Erlang compiler optimizes this to not hit the keyword lookup if the number is small.

It might be nice to have an option similar to Python's simplejson bigint_as_string

http://simplejson.readthedocs.org/en/latest/#basic-usage

Ability to not encode key/values with nil value

I have searched the code base and documentation and have not found an option for this behavior. Does this feature already exist and I am missing it. If not, is it something you can envision adding to Poison? I would be willing to work on the feature if you are open to a pull request.

Thanks for your work on this project!

Console.log doesn't work

Adding console.log to the javascript breaks the code.

function update(request) { console.log("testing"); return request; }

[error] GenServer #PID<0.407.0> terminating
** (Poison.SyntaxError) Unexpected token: t
(poison) lib/poison/parser.ex:56: Poison.Parser.parse!/2
(poison) lib/poison.ex:83: Poison.decode!/2
lib/execjs.ex:69: Execjs.extract_result/1
lib/execjs.ex:40: Execjs.exec/1

What am I missing here?

Correct way to coerce struct fields from strings to other types?

I have a struct like this:

defmodule StreamingBuffer do
  defstruct [:estimatedRows, :estimatedBytes, :oldestEntryTime]

  @type t :: %__MODULE__{
    estimatedRows: non_neg_integer,
    estimatedBytes: non_neg_integer,
    oldestEntryTime: non_neg_integer
  }
end

But the API that I'm calling returns each of those fields as strings, not as integers, and Poison decodes the struct with string fields:

x = "{\"estimatedRows\" => \"12\", \"estimatedBytes\" => \"1024\", \"oldestEntryTime\" => \"1\"}"
Poison.decode!(x, as: %StreamingBuffer{})

%StreamingBuffer{estimatedBytes: "1024", estimatedRows: "12",
 oldestEntryTime: "1"}

Is there a good way to tell Poison to decode those fields as integers? Or is the correct approach to write a Decoder implementation for each of my structs that have this issue?

As an aside, it seems a little weird to me that the value passed in to a custom decoder is an instance of the struct, as opposed to a plain map? I'm sure there are reasons why that I'm missing, but I'd love to have a little more insight into how & why the decodes work this way? I haven't done much with protocols in elixir yet.

Error when using with `keys: :atoms!` and the atom hasn't yet existed

With Poison master and release 1.3.1, this happens:

iex(3)> Poison.decode ~s({"what": "where"}), keys: :atoms!
** (ArgumentError) argument error
    :erlang.binary_to_existing_atom("what", :utf8)
    lib/poison/parser.ex:97: Poison.Parser.object_name/2
    lib/poison/parser.ex:82: Poison.Parser.object_pairs/3
    lib/poison/parser.ex:36: Poison.Parser.parse/2
    lib/poison.ex:69: Poison.decode/2
iex(3)> :what
:what
iex(4)> Poison.decode ~s({"what": "where"}), keys: :atoms!
{:ok, %{what: "where"}}

I'm using Elixir 1.0.2 and Erlang/OTP 17.

Is this library still considered "experimental"?

I've noticed that this library is past version 1.0 yet the description still says this project is "experimental'. I'm just curious if that is still true or if this library should be considered useable for production applications. If the latter we should probably update that description ๐Ÿ˜„.

screen shot 2015-03-31 at 4 27 31 pm

FunctionClauseError

I am a Rails dev just getting started with Elixir. Thanks for your patience.

In a Phoenix app I am fetching a list of objects from MongoDB and rendering as json, like so:

def index(conn, _params) do
  mongo = Mongo.connect!
  db = mongo |> Mongo.db("my_db")
  collection = db |> Mongo.Db.collection("my_collection")
  items = collection |> Mongo.Collection.find() |> Enum.to_list
  json conn, items
end

I get this error:

** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Poison.Encoder.BitString.chunk_size/3
        (poison) lib/poison/encoder.ex:127: Poison.Encoder.BitString.chunk_size(<<176, 87, 109, 84, 105, 109, 111, 60, 0, 0, 18>>, nil, 1)
        (poison) lib/poison/encoder.ex:122: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:84: Poison.Encoder.BitString.encode/2
        (poison) lib/poison/encoder.ex:213: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:213: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:232: anonymous fn/3 in Poison.Encoder.List.encode/3
        (poison) lib/poison/encoder.ex:233: Poison.Encoder.List."-encode/3-lists^foldr/2-1-"/3
        (poison) lib/poison/encoder.ex:233: Poison.Encoder.List.encode/3
        (poison) lib/poison.ex:41: Poison.encode!/2
        (phoenix) lib/phoenix/controller.ex:186: Phoenix.Controller.json/2
        (my_app) web/controllers/items_controller.ex:2: MyApp.ItemsController.action/2
        (my_app) web/controllers/items_controller.ex:2: MyApp.ItemsController.phoenix_controller_pipeline/2
        (my_app) lib/phoenix/router.ex:261: MyApp.Router.dispatch/2
        (my_app) web/router.ex:1: MyApp.Router.do_call/2
        (my_app) lib/my_app/endpoint.ex:1: MyApp.Endpoint.phoenix_pipeline/1
        (my_app) lib/plug/debugger.ex:93: MyApp.Endpoint."call (overridable 3)"/2

items is a list of maps. Each map represents a MongoDB document. Each document has a MongoDB assigned id like this _id: ObjectId(56b0576d54696d6f3c000011). Some of the values in each document are lists of string and others are lists of maps. Some of values are strings with escaped double quotes in them.

Any idea what might be wrong?

Is `attachments` parameter supported for `post_message`?

Hi,

Is it possible to send attachments via the Web API post_message function?

Something like:

image_url = "https://images-na.ssl-images-amazon.com/images/G/01/img15/pet-products/small-tiles/30423_pets-products_january-site-flip_3-cathealth_short-tile_592x304._CB286975940_.jpg"

opts = %{as_user: true, unfurl_links: true, attachments: [%{image_url: image_url}]}                                                                                 

Slack.Web.Chat.post_message("#general", "test", opts)                                                                                                                                            

Invalid output when mixing strings and atoms

This was my mistake, but I got highly unexpected behaviour out of Poison when it happened. I have a dictionary that is a mixture of atoms and strings for keys, and in the output there is a duplicated key, which is not valid JSON.

Minimal test case:

iex(12)> Poison.encode! %{ :foo => "bar", "foo" => "baz" }
"{\"foo\":\"baz\",\"foo\":\"bar\"}"

The result is that the other side of the HTTP endpoint would ambiguously parse the output. Very tricky to debug.

keys: :existing_atoms option

Hey there!
Usually when I'm dealing with json, it's heavily nested. In elixir, I tend to want to do something like keys: :atoms, but then I'm not sanitizing very well. instead of having to write nested loops to change everything to atoms, is there any way we could implement a keys: :existing_atoms option?

Implement Poison for Elixir v1.3 calendar types

Hi @devinus,

Could you please implement Poison.Encoder for Elixir v1.3 calendar types and make a new release?

Elixir protocol implementation does not actually check if the thing you are implementing for exists at the moment of implementation, this means you can implement the protocols and be backwards incompatible. Here is how we implemented those for Phoenix.HTML.Safe protocol:

phoenixframework/phoenix_html@d78a082

I think the only difference is that you will want to invoke to_iso8601 on every type instead of to_string invoked in the link above.

Thank you! โค๏ธ

Encoding structs with nil values

I have a struct which and I do not want keys to be included in the encoded payload when the associated value is nil. Is this possible? Maybe there should be an option for this?

Possibility of adding options for manipulating key names

Hi Devin,

I was wondering how you feel about adding the option to supply a function which would enable the users of Poison to manipulate the names of keys when both encoding and decoding. It would be a simple string ->string function that can be supplied as a option with the default value being the identify function. The use case for this feature is that we use snake_case in our elixir backend but camelCase in our javascript frontend. We would like to have the serializer take of doing this conversion.

Currently we have a solution where we in case of encoding redefine the Poison.Encoder protocol implementation for Map and for decoding do a recursive run though the map resulting from calling Poison.decode. It would be nice if it was something that could be build into the serializer using a simple hook as described above.

Best Regards

Simon, Denmark

Decoding nested json objects to structs

I have a json object which has nested relationships that maps to different structs.

eg.

{
  "firstname": "John",
  "lastname": "Smith"
  "address": {
    "street": "123",
    "zip": ...
  }
}

Can poison decode in format of %Person{"firstname"...., %Address{}} ?

Helpers for writing Poison.Decoder implementations

For decoding nested structs we can use custom Poison.Decoder implementations, but they're not really well documented and can get quite messy when you need to do a lot of conversions. For example:

defimpl Poison.Decoder, for: TaskList do
  def decode(task_list, options) do
    Map.update! task_list, :items, fn items ->
      Poison.Decode.decode(items, Keyword.merge(options, as: [TaskItem]))
    end
  end
end

I think it's currently a bit too hard to write them correctly, especially given the tiny but fatal difference between Poison.Decoder and Poison.Decode. Maybe one of those should be renamed so that they can be differentiated more easily. Anyway, it would be really nice if we could write the above example with a cleaner syntax, such as this:

defimpl Poison.Decoder, for: TaskList do
  use Poison.Decoder.Helpers

  decode_nested do
    decode :items, as: [TaskItem]
  end
end

Let me know what you think, If you're interested I'd love to give it a go.

Decode 'null' to struct

At the moment, this is the behaviour from Poison when creating a struct from the JSON "null"

defmodule MyStruct do
  defstruct one: "", two: ""
end

{:ok, nil} = Poison.decode("null", as: %MyStruct{})

I would have expected:

{:ok,  {:ok, %MyStruct{one: "", two: ""}} = Poison.decode("null", as: %MyStruct{})

or even better, Poison will generate an error like:

{:error, {:invalid, "expected an object, got null"} = Poison.decode("null", as: %MyStruct{})

Please let me know what the expected behaviour is. Then I will try to create a pull request.

Struct not being encoded

Hi,

Thank you for putting this library together.

I am a newbie to elixir I am following an article
https://robots.thoughtbot.com/testing-a-phoenix-elixir-json-api
when I get to the encoding part

contact_as_json = %Contact{name: "Rambo", phone: "801-555-5555"} |> Repo.insert |> List.wrap |> Poison.encode!

above throws following error

** (Poison.EncodeError) unable to encode value: {:ok, %Hawk.Contact{__meta__: #Ecto.Schema.Metadata<:loaded>, id: 4, inserted_at: #Ecto.DateTime<2015-10-09T00:57:37Z>, name: "Rambo", phone: "801-555-5555", updated_at: #Ecto.DateTime<2015-10-09T00:57:37Z>}}

regards

nested struct decoding

example code here: https://gist.github.com/jschoch/b9c4f9c08fa5be127c67

The example is a bit of a hack and i'm wondering why this isn't a feature, assuming there is some reason.

Goal is to be able to run the decoder on nested structs.

%Foo{tuple: {1,2}} encodes it's to [1,2] and decodes to {1,2}

encode/decode should work on the below and have the :tuple converted to a list and back

%Bar{name: :baz, foo: %Foo{ tuple: {1,2}}

Decoding arrays with numeric elements fails

Hi, there is some strange decoding behaviour when decoding a simple array with numeric elements that start with '1':

>Poison.decode("[1]")
{:ok, [1]}
>Poison.decode("[12]")
{:ok, ['\f']}
>Poison.decode("[123]")
{:ok, ['{']}
>Poison.decode("[1234]")
{:ok, [1234]}
>Poison.decode("[12, 12, 12]")
{:ok, ['\f', '\f', '\f']}
>Poison.decode("[22, 22, 22]")
{:ok, [22, 22, 22]}

Using Poison 2.2.0

[Question] How to pretty print?

Hi,

how can I use Poison.encode to pretty print a map?

I have been looking at the source code and I believe this should work, but it doesn't:

iex(1)> Poison.encode(%{"a"=>1, "b" => 2}, [pretty: true])    
{:ok, "{\"b\":2,\"a\":1}"}

Thank you!

keys: :atoms! sometime tries to convert values to keys

I've an example that when fed to Poison.decode! with keys: :atoms! option, errors out with ArgumentError where it tried converting a value to atom (which wouldn't be initialised of course)

It seems to be just a rare anomaly, because I wasn't able to reproduce the error with a simpler example. Am I missing something here?

This gist contains the example that triggers this anomaly.

Using poison v2.2.0

Poison.decode very slow for JSON documents with large strings

When trying to decode a 6MB JSON document (essentially an array of very long strings), Poison decodes quite slow. The JSON document has 6,045,531 characters in it totally. I ran exprof on this simple code:

{:ok, string} = File.read "test.json"
profile do
  Poison.decode!(string)
end

And the result was:

FUNCTION                                        CALLS        %     TIME  [uS / CALLS]
--------                                        -----  -------     ----  [----------]
erlang:send/2                                       1     0.00        0  [      0.00]
'Elixir.Poison.Decode':decode/2                     1     0.00        1  [      1.00]
'Elixir.Poison':'decode!'/2                         1     0.00        1  [      1.00]
'Elixir.Poison':'decode!'/1                         1     0.00        2  [      2.00]
'Elixir.Poison.Parser':'parse!'/2                   1     0.00        3  [      3.00]
'Elixir.Access':get/3                               2     0.00        3  [      1.50]
lists:keyfind/3                                     2     0.00        3  [      1.50]
'Elixir.Poison.Parser':parse/2                      1     0.00        4  [      4.00]
'Elixir.Access':fetch/2                             2     0.00        5  [      2.50]
'Elixir.Access':get/2                               2     0.00        5  [      2.50]
'Elixir.MyApp.Document':'-get/1-fun-0-'/1         1     0.00        6  [      6.00]
lists:reverse/2                                    43     0.00       38  [      0.88]
maps:from_list/1                                  102     0.00      114  [      1.12]
'Elixir.Poison.Parser':object_pairs/3             205     0.00      202  [      0.99]
lists:reverse/1                                   102     0.00      207  [      2.03]
'Elixir.Poison.Parser':number_frac/2              244     0.00      223  [      0.91]
'Elixir.Poison.Parser':number_exp/3               244     0.00      232  [      0.95]
erlang:binary_to_integer/1                        244     0.00      234  [      0.96]
'Elixir.Poison.Parser':number_start/1             244     0.00      235  [      0.96]
'Elixir.Poison.Parser':number_int/2               244     0.00      237  [      0.97]
'Elixir.Poison.Parser':array_values/3             341     0.01      320  [      0.94]
'Elixir.Poison.Parser':object_name/2              205     0.01      357  [      1.74]
'Elixir.Poison.Parser':number_complete/2          244     0.01      413  [      1.69]
'Elixir.Poison.Parser':number_digits/1            244     0.01      435  [      1.78]
'Elixir.Poison.Parser':value/2                    547     0.02     1010  [      1.85]
'Elixir.Poison.Parser':number_digits_count/2      960     0.02     1062  [      1.11]
'Elixir.Poison.Parser':skip_whitespace/1         1504     0.05     2664  [      1.77]
erlang:iolist_to_binary/1                         549     0.09     5206  [      9.48]
'Elixir.Poison.Parser':string_escape/2          21120     0.38    20854  [      0.99]
'Elixir.Poison.Parser':string_continue/2        32242     0.55    30085  [      0.93]
'Elixir.Poison.Parser':string_chunk_size/2    6011582    98.83  5440137  [      0.90]
--------------------------------------------  -------  -------  -------  [----------]
Total:                                        6071225  100.00%  5504298  [      0.91]

Not sure why there are so many calls to Elixir.Poison.Parser:string_chunk_size/2, but the number of calls is roughly equivalent to the number of characters in the document.

Parsing issue

I'm not sure if the following is valid json.

{
  "type": [
    {"a": "b"},
  ]
}

When it is parsed:

Poison.decode! '{"type": [{"a": "b"},]}'
# =>  %{"type" => []}

If the JSON is valid, it should return the nested object, or if not valid, raise an error.

Custom Decoder

I have a json string which I would like to decode into a struct. However, the json is nested with some arbitrary fields. How can I use Poison to map to only key fields to a struct? I am not finding a way for me to pass a mapping expression. Is it possible to define a custom decoder similar to the custom encoder in the README? Appreciate any suggestions on how to better use the as: or decode patterns.

Example JSON from Reddit, where I only want to decode each children's "title" and "url" but they are nested under multiple "data" that I don't want to clutter the struct with.

{"kind": "Listing", "data": {"modhash": "", "children": [{"kind": "t3", "data": {"domain": "stephanebegaudeau.tumblr.com", "banned_by": null, "media_embed": {}, "subreddit": "dartlang", "selftext_html": null, "selftext": "", "likes": null, "secure_media": null, "link_flair_text": null, "id": "2d9zao", "gilded": 0, "secure_media_embed": {}, "clicked": false, "stickied": false, "author": "sbegaudeau", "media": null, "score": 11, "approved_by": null, "over_18": false, "hidden": false, "thumbnail": "", "subreddit_id": "t5_2sut9", "edited": false, "link_flair_css_class": null, "author_flair_css_class": null, "downs": 0, "saved": false, "is_self": false, "permalink": "/r/dartlang/comments/2d9zao/introducing_dart_designer/", "name": "t3_2d9zao", "created": 1407824113.0, "url": "http://stephanebegaudeau.tumblr.com/post/94470472420/introducing-dart-designer", "author_flair_text": null, "title": "Introducing Dart Designer", "created_utc": 1407795313.0, "ups": 11, "num_comments": 0, "visited": false, "num_reports": null, "distinguished": null}}]}}

ArgumentError in Poison.Parser.string_escape/2, char range `\\ud800` to `\\udfff`

ArgumentError on decode:

Poison.decode("{\"s\": \"\\ud83d\"}")
** (ArgumentError) argument error
    (poison) lib/poison/parser.ex:229: Poison.Parser.string_escape/2
    (poison) lib/poison/parser.ex:78: Poison.Parser.object_pairs/3
    (poison) lib/poison/parser.ex:36: Poison.Parser.parse/2
    (poison) lib/poison.ex:69: Poison.decode/2

Poison 2.2.0

Poison.Decode.decode doesn't use defaults when decodable has atom keys

The title is a mouthful, but I think this code example will explain better.

defmodule X do
  defstruct y: 1,
            z: 2
end

Poison.Decode.decode(%{y: 3}, as: %X{}, keys: :atoms!) # => %X{y: 3, z: nil}
Poison.Decode.decode(%{"y" => 3}, as: %X{}) # => %X{y: 3, z: 2}

The expectation would be that the first decode would have the same result as the second.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.