Comments (17)
FWIW I just ran into this as well. I really think this should be an option. Sometimes APIs are just wrong but we still have to deal with it.
from cheshire.
Another use case: I want to test a REST API I am building. In the tests I want to verify the whole JSON response body by matching it as a String to the content of a JSON file with expected response body. In order to prevent whitespace-related issues when comparing the Strings I need to re-format both the expected and actual JSONs using the same formatter. In case of mismatch I also want to be able to replace the template file content with the actual response, obviously without re-ordering the fields of large maps/objects.
Doing this in plain Java using Jackson works OK (using TreeNode
as the intermediary data structure and .readTree
and .writeTree
methods).
Would be great to be able to use Cheshire for this on a Clojure project.
Also, I'd like to point out that although the JSON standard might not guarantee the order of map keys, all JSON processing libraries I know of do keep it, including Javascript's own JSON.parse
, and Java's Jackson which is used by Cheshire.
The different order of the keys is NOT because of JSON as the resolution might imply. The reason is the way Clojure manages map datastructures, as already pointed out by @ztellman above, and that has nothing to do with JSON itself.
I would therefore dare to propose not only to open this feature to extensibility (although even this would be great), but to consider using ordered-map
by default. This would make Cheshire consistent with all other JSON libraries out there.
I would prefer consistency over trying to minimize memory overhead, which is not quantified anyway, I know these are just my personal priorities, but extensibility would allow for more efficient datastructure (like Clojure's {}
), should the user so desire.
from cheshire.
@dyba --
As a relatively new Clojure user, I was feeling the need to learn a bit more this. Here is what I found:
It seems you are referring to ordered-map
from [flatland.ordered.map "1.5.2"]
Firstly, just to very explicit, array-coerce-fn
needs to return an actual collection. Unless ordered-map
in your code is referring to a var, it will need to used a function call, as below:
(fn [field-name]
(if (= field-name "contributionReceipt")
(ordered-map)
{}))
Secondly, it seems Cheshire only support coercing function for vectors. Here is a patch that makes it work for maps too.
diff --git a/src/cheshire/parse.clj b/src/cheshire/parse.clj
index 312b119..3e528ca 100644
--- a/src/cheshire/parse.clj
+++ b/src/cheshire/parse.clj
@@ -14,9 +14,11 @@
(definline parse-object [^JsonParser jp key-fn bd? array-coerce-fn]
(let [jp (tag jp)]
- `(do
+ `(let [object-field-name# (.getCurrentName ~jp)]
(.nextToken ~jp)
- (loop [mmap# (transient {})]
+ (loop [mmap# (transient (if ~array-coerce-fn
+ (~array-coerce-fn object-field-name#)
+ {}))]
(if-not (identical? (.getCurrentToken ~jp)
JsonToken/END_OBJECT)
(let [key-str# (.getText ~jp)
I am not submitting as a pull request because:
- It needs some cleaning up (should really be an additional parameter
map-coerce-fn
) - Unsure if this addition really makes sense for wider use. I'll let the primary contributors chime in.
from cheshire.
A (hacky?) alternative solution is to use clj-yaml which supports ordered maps:
bb -e '(-> (clj-yaml.core/parse-string "{\"a\": 1, \"b\": 2, \"c\": 2, \"d\": 2, \"e\": 2, \"f\": 2, \"g\": 2, \"i\": 2}")
(assoc :a 2) (cheshire.core/generate-string))'
"{\"a\":2,\"b\":2,\"c\":2,\"d\":2,\"e\":2,\"f\":2,\"g\":2,\"i\":2}"
if you replace clj-yaml.core/parse-string
with cheshire.core/parse-string
the order will change, but when reading it as yaml, the order is preserved. FWIW.
from cheshire.
I don't understand the problem.
When I
decode
the template in the body of the response, the resulting map reorders the keys in the original template.
clojure.repl=> (decode "{\"foo\": \"bar\", \"foo2\": \"bar\"}")
{"foo" "bar", "foo2" "bar"}
clojure.repl=> (decode "{\"foo2\": \"bar\", \"foo\": \"bar\"}")
{"foo2" "bar", "foo" "bar"}
Ordering seems to be maintained. Can you give an example of re-ordering?
from cheshire.
That's because smaller maps use PersistentArrayMap
, which is ordered. With over 8 entries, it'll spill over into PersistentHashMap
, which is not. A custom map implementation can be used, but it will involve memory overhead.
from cheshire.
Ahh, I see. Thanks for the explanation @ztellman.
clojure.repl=> (decode (str "{" (clojure.string/join ", " (map #(str "\"foo" % "\": \"bar\"") (range 8))) "}"))
{"foo0" "bar", "foo1" "bar", "foo2" "bar", "foo3" "bar", "foo4" "bar", "foo5" "bar", "foo6" "bar", "foo7" "bar"}
clojure.repl=> (decode (str "{" (clojure.string/join ", " (map #(str "\"foo" % "\": \"bar\"") (range 9))) "}"))
{"foo1" "bar", "foo4" "bar", "foo7" "bar", "foo8" "bar", "foo2" "bar", "foo6" "bar", "foo0" "bar", "foo3" "bar", "foo5" "bar"}
from cheshire.
@ztellman 👍 Nice explanation! I knew Clojure reordered the keys as part of an optimization but now I understand conceptually what happens.
from cheshire.
@jayp Hmm.. I'm on the fence about this. The JSON website/spec specifically states that "An object is an unordered set of name/value pairs" (emphasis mine) and that arrays should be used when order is important.
Is there any chance the server that is expecting an ordered map could be fixed? That's probably the most correct way to handle this, though I understand that servers are not always under your control.
from cheshire.
@dakrone I referenced the same line you quoted from the JSON spec to the third-party vendor who maintains the server. I've yet to hear back from them if they are able to change it.
from cheshire.
HI @dakrone - I appreciate software that allow for a more liberal interpretation of standards. For instance, web browsers attempt to follow the standard, but are forgiving. With that said, it's your call. No sleep lost if this issue were closed without any affect.
from cheshire.
@jayp @dakrone I just got a response from the third-party. They are indeed using an XML generator under the hood to generate their JSON, which explains why they require any JSON templates in a request sent to them to be ordered.
I think the workaround would be to use their API and request the data as XML rather than JSON. Thank you both for taking the time to read my issue and provide your feedback. 👍
from cheshire.
If anyone else needs this, here is a fork that includes the map-coerce-fn
as described by @jayp : https://github.com/px0/cheshire
from cheshire.
I need to load json docs and preserve order of keys aswell - seems to me that if you provide an API for overriding implementation of "array" (defaulting to vector) you should do the same for impl of "object" (defaulting to hash-map).
I might want to load directly into a sorted-map, to save sorting later...
I might want to load into custom record types....
I might want to preserve key order in spite of json spec...
I might want to...
Jules
from cheshire.
I was reviewing our project's having to have it's own copy of cheshire/parse.clj
just to hack it to allow overriding with a custom map type for all json-objects.
I thought I had given pretty good reasons above as to why this would be a useful and sensible addition to the API but I see that two releases of Cheshire have been done this year and neither appears to include this feature :-( Perhaps we needed to shout louder to get it in ?
- it nicely complements the similar
array-coerce-fn
- most of a json doc is made of json objects, it is not unlikely that I might have my own impl and reasons for its use
- as long as my impl supports map and transient/persistent interfaces (which it does) it is a trivial change
- there seem to be a number of people asking for it
- a library should allow the user to extend it to fit snugly into their usecase - this is real pain-point in my use of cheshire
I would really appreciate it if this feature could be considered for the next release.
many thanks
Jules
(definline parse-object [^JsonParser jp key-fn bd? array-coerce-fn]
(let [jp (tag jp)]
`(do
(.nextToken ~jp)
(loop [mmap# (transient (MY-CUSTOM-MAP))]
(if-not (identical? (.getCurrentToken ~jp)
JsonToken/END_OBJECT)
(let [key-str# (.getText ~jp)
_# (.nextToken ~jp)
key# (~key-fn key-str#)
mmap# (assoc! mmap# key#
(parse* ~jp ~key-fn ~bd? ~array-coerce-fn))]
(.nextToken ~jp)
(recur mmap#))
(persistent! mmap#))))))
from cheshire.
Another use-case for which I needed ordered keys is consistently serializing JSON to allow for hash-based comparisons on the generated string.
Since consumers must be able to do these comparisons as well they need to work on the serialized string and can't be based on the internal Clojure data structure.
If there is a JSON schema for the JSON then this can be achieved always serialize properties in the order defined in the associated JSON schema.
That only works if the JSON schema can first be read the provided order and then have the output serialized against that order.
from cheshire.
How are you handling the absence of this feature nowadays? Is there a solution?
I needed to edit some JSON for a code review, and the lack of order in keys has made the diff comparison very much unreadable, so that might be another use-case (albeit admittedly quite the trivial one).
from cheshire.
Related Issues (20)
- Always update to latest version of Jackson, when available
- Any way to not escape the quotes HOT 2
- Suggestion: prove equivalence with clojure.data.json HOT 6
- Add support for capturing source position/locations when parsing HOT 2
- Allow key-fn like hook for value HOT 2
- Converting nested fields from java objects to clojure types doesnot work
- Custom conversion when parsing?
- `parsed-smile-seq` actually parses JSON
- Run tests against JSONTestSuite
- Resulting Array from parse not processed lazily HOT 1
- :key-fn no longer works in generate-string? HOT 1
- Using the Ubuntu Cheshire package with Clojure HOT 1
- lein unable to find cheshire on classpath HOT 1
- Security vulnerability in Jackson dependency HOT 4
- Can't convert object of type org.postgresql.util.PGobject HOT 1
- support jackson's fast number parsing/writing features HOT 1
- Update test.generative HOT 2
- Possible to support parsing copy-paste JSON? HOT 1
- Test failures with Java 21
- Deterministic parse and generate HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cheshire.