I would like to write huge trees but don't retain the entire tree in memory

[skinkie]

I would like to write huge trees but don't retain the entire tree in memory

[tefra]

We need to fully support the Iterable type annotation for infinite generators in the data models, and the serializers.

The pr is a good first attempt @skinkie but it needs some more work

[skinkie]

Doing a 3.4GB file using generators, takes ~12GB of memory to write using LxmlEventwriter. XmlEventWriter does absolutely not take any memory while writing to disk, and it does it in a streaming fashion. I think this must be investigated, especially if LxmlEventWriter is the default. I rewrote my whole project to split up stuff because I was under the impression I couldn't get it stored in memory.

[tefra]

It's mentioned in a few places in the docs

https://xsdata.readthedocs.io/en/latest/data_binding/xml_serializing/#alternative-writers
https://xsdata.readthedocs.io/en/latest/api/formats/dataclass/serializers/writers/lxml/#xsdata.formats.dataclass.serializers.writers.lxml.LxmlEventWriter
https://xsdata.readthedocs.io/en/latest/api/formats/dataclass/serializers/writers/native/#xsdata.formats.dataclass.serializers.writers.native.XmlEventWriter

For normal use cases, the lxml writer is always faster, 3.4GB xml is not very common 😄

[skinkie]

@tefra it is mentioned that there are alternatives, but not the characteristics of the two.