No easy option for creating lazy ring responses

[gfredericks]

No easy option for creating lazy ring responses

[boxxxie]

just incase anyone is interested in a work around for this issue

(ring.util.io/piped-input-stream
   (fn [out] (->> out
                 (OutputStreamWriter.)
                 (BufferedWriter.)
                 (cheshire.core/generate-stream data))))

[dakrone]

Okay, let me make sure I have this correct in my understanding.

Would it be helpful to have something like:

(def mydata {:foo "bar" :etc "...etc..."})

;; of InputStream type
(def my-inputstream (ring/get-response-inputstream))

;; with "generate-to-stream" or whatever it's named to be added:
(json/generate-to-stream mydata my-inputstream)

? (this is heavily pseudo-coded since I don't know exactly what it looks like from the ring side)

[dakrone]

Also, I'm not quite sure what you mean by a "lazy seq of strings", since the usual output from generate-string is a single string; can you elaborate? (I'm not familiar with what enlive does in this case)

[gfredericks]

The simplest case would be

(json/generate-to-stream my-data) ;; returns an input stream

Then the input stream could be returned as the response body to the ring adapter which would read from it and shoot the bytes down the wire as needed.

The lazy seq of strings tactic would be something like

(json/generate-strings {:foo [1 2 3]})
;; => ("{" "\"foo\"" ":" "[" "1" "," "2" "," "3" "]" "}")

how exactly the string is broken up isn't as important as the fact that it's lazy, and so the whole thing doesn't have to be assembled into a single spot in memory -- it can be consumed by the ring adapter lazily, in a manner analogous to using an InputStream.

[dakrone]

Okay, that clarifies it; thanks!

[dakrone]

@fredericksgary You may also want to check out the cheshire.experimental namespace, as we've been experimenting with stream parsing in there.

[aviflax]

+1 on this, and I don’t understand how to employ @boxxxie’s suggested workaround.

[paul-lshift]

@aviflax Use this function to turn data into an output stream:

(defn generate-stream
  ([data] (generate-stream data nil))
  ([data options]
    (ring.util.io/piped-input-stream
      (fn [out] (cheshire.core/generate-stream data 
                                               (-> out
                                                 (OutputStreamWriter.)
                                                 (BufferedWriter.))
                                               options)))))

[jconti]

Endpoints from systems like ElasticSearch package results as multiple top level objects, which facilitates this.