breaking under heavy http insert load

[ponzu-cms]

breaking under heavy http insert load

[nilslice]

I don't think it's the root of the problem, but could you update to 1.8 stable and see if you're still hitting this?

I have a feeling it's just that with each write to Bolt, Ponzu runs a sort in a goroutine on the bucket you're impacting and there could be some race issues at the transaction level if too many writes are happening at once.

[nilslice]

If you do a search for SortContent in the codebase you'll see what I'm referring to. This sorting method isn't an ideal process and should be rethought. But, for the systems I built Ponzu to support the write performance is a very low priority.

I would be happy to find a way to improve this though.

[nilslice]

There is some code pushed to the throttle-sort (6c34044) branch. If you can test it out, pull it into your master and re-install ponzu by running go install ./... from inside $GOPATH/src/github.com/ponzu-cms/ponzu

On the very first 'load' command, I am seeing a bug that is preventing the SortContent call from being evoked at the right time. Maybe the HTTP calls to github API can finish sooner than the 4-second waitDuration before a secondary / throttled call to SortContent can happen? will have to test a shorter waitDuration value and see.

[nilslice]

I reduced the waitDuration value and it seems to have resolved this issue. I think the loading feature of the gobservatory project is fairly niche so i don't want to over optimize for it, but it would be good to be sure that those HTTP calls are the cause of the bug I caught.

[nilslice]

I'm going to close this up for now, since a fix was merged to master. If you get a chance to try it out and successfully replicate the panic you encountered, let me know and I'll dig in.

Thanks for reporting this!

[kkeuning]

I was able to test and confirm that with this fix I am no longer getting the panic under high load.

There is a remaining issue with SortContent being not called after the loading completes, submitted #123 to fix.