Community data payload is too large

[tgxn]

Community data payload is too large

[pcamilo89]

Hello, I noticed that for me the community JSON always fails.

I know I'm a minority, but I have a 1mbit connection at home and a 4mb file at that speed would be downloaded in 32 seconds (a 5mb file would be 40sec, just to point out an upper limit to be safe).

So, can the timeout be longer tan 15 seconds, even if it's just 16–20 seconds it would cover download speeds of 2mbit or higher.

[pcamilo89]

@pcamilo89 give it a go now... they are smaller but there's a bunch.

That does work, I can see the list, and it slowly loads images and other data after, if some of the data fails to be retrieved it retries and gets it eventually.

The next image is just a reference for the retries (both for the lists of communities and images):

Tested a few times without cache active, and sometimes it can ask you to reload the page (I assume it has a bit of issues with the amount of requests), with cache active, there is no problem after the initial download, as it uses the cached data.

Thank you, now it is usable for low speed connections like mine.

I guess there is room to make improvements later, so in the case you decide to work on it, I will leave notifications for this issue on and can do a test later if needed.

[tgxn]

Yeah there is room to improve. This whole project I've not had to create any server backend, since it's all based on client-side JSON. I don't really plan on making a featured backend for this, cost and complexity ++.

I'm glad the fix made it possible to use, I added 3 re-tries on each payload in case they fail :) Maybe a future improve the loading, as the data grows I can see maybe doign more advanced splitting or minifying the data further to reduce size.

[tgxn]

Yeah that means it failed to load the ~4MB JSON file that has the community data in it. I'll extend the timeout to 15 seconds instead of the current 3.5 seconds.

[tgxn]

Yeah it's getting bigger and bigger :S

Compressed it is still around ~3MB

[tgxn]

I think the solution is to have either multiple files (small index file and separate files for each ~1000 communities) or try to remove some data from the communities json and seperate the metrics/metadata out.
Probably a combination of both.

Ideally the main load would be <1MB.

[tgxn]

my thinking on implementation, which I've written the code for tonight.

• split list into files with ITEMS_PER_FILE
• implement loader that will fetch these on-demand.

i guess the goal is to "make it work on slower connections" not "make it totally fast"

[tgxn]

@pcamilo89 give it a go now... they are smaller but there's a bunch.

#78