error with big project on JSONWriter - broken pipe

[stefanrinderle]

error with big project on JSONWriter - broken pipe

/cc @nixel2007

[yvo-niedrich]

Debugging

That's one impressively huge project. Mainly the amount of files should negativly impact the plugins behavoir, as metrics like lines of code are just passed as simple integer values.

I'm duplicating a lot of code in one of my projects in order to recreate this behavior, but so far the only problem is the available memory on my virtual machine. Once the code analysis is scanned by SonarQube, the Softvis-Plugin renders the city just fine.

My project-Size:

• Lines Of Code: 1.507.486
• Files: 8.742
• Directories: 1.186

I tried to create a bigger project, but after about 2 hours of scanning a SonarQube-Component fails. I will try again and update this post, once i have more information on that.

Identifing the error

Aside from the project size there may be other factors, causing the component to fail. My first guess is the composition of the available data.

Question: Do you use cyrillic characters in class- or filenames?

Fixing the error

Usually @stefanrinderle would take care of that as the backend is his speciality, but he's hardly available till the end of month.

Maybe I will have a crack at it (once I can recreate the bug).

[nixel2007]

Question: Do you use cyrillic characters in class- or filenames?

Cyrillic characters are presented in file paths and files itself.
Filenames are in latin in most cases if not always.

[stefanrinderle]

Hi,

thanks for reporting this issue and @Ungolianth for looking into this. You got a pretty big project :-)

I would expect that this is caused by a timeout of the sonarqube webservice offered by the plugin. The time between the initialization and the exception is about 100 seconds. I tried to set a wait timer in the webservice to check that - but it worked even after 130 seconds processing time with SonarQube 6.0. Which SonarQube version do you have?

Or could it be the network connection? Is it a local SonarQube instance, within the same network or do you connect through internet?

So maybe it's really about the characters: Do you have the same issue on similar character based smaller projects or is it only at this big one?

Stefan

[nixel2007]

We have projects start with 500k lines minimum :)

We have a dedicated sonarqube 6.1 server in private cloud, connected via Internet.

The "small" one project (near 433k of LOC) renders properly without any errors (both contain Cyrillic in paths and files)

[stefanrinderle]

Hey,

unfortunately we couldn't yet reproduce the issue. Apart from the timout of the sonarqube webservice, i have no other idea.

Good news is that we are planning to rework the data processing in the next (not the current) milestone. Maybe it works after the change.

We will definetly contact you after we made the first changes here.

If you have an idea what could be the issue - we are happy to look into it. But currently we have no idea ;-(

[nixel2007]

@allustin project on the screenshot is our erp system, isn't it? Can we give guys a private access to that repo for limited time?

[stefanrinderle]

it would be great to have access to the project in order to test it against a local SonarQube instance.

missed the notification from github, sorry

@nixel2007 yes it erp. let discuss private acccess tomorow in real-life. Ok ?

[nixel2007]

@allustin sure.

[stefanrinderle]

Hey @allustin and @nixel2007,

the current snapshot version of the plugin includes a change of the backend calls. Instead of one big request, we now have smaller junks. This should fix your issue you had with the plugin.

The new version works with SQ >= 6.3. The last time we talked you where at 6.1. Did you already upgrade?

I attached a current snapshot of the plugin. Could you give it a try?

Thanks.

sonar-softvis3d-plugin-0.9.1-SNAPSHOT.jar.zip

[nixel2007]

@stefanrinderle hello!

yeah, we've upgraded to 6.3.1 already.
The 0.9.0 version shows error on loading. 0.9.1-SNAPSHOT - no error, but also no view =(

Current size -

[stefanrinderle]

@nixel2007 - Thanks for your quick response. Would you mind taking a look into the javascript console if there is any error present?

[nixel2007]

@stefanrinderle the only messages are (verbose log)

THREE.WebGLRenderer 85 three.min.js:160 <--- this message is on INFO level, others - on VERBOSE
bundle.js:7 [Violation] Added non-passive event listener to a scroll-blocking 'wheel' event. Consider marking event handler as 'passive' to make the page more responsive.
bundle.js:7 [Violation] Added non-passive event listener to a scroll-blocking 'touchstart' event. Consider marking event handler as 'passive' to make the page more responsive.
bundle.js:7 [Violation] Added non-passive event listener to a scroll-blocking 'touchmove' event. Consider marking event handler as 'passive' to make the page more responsive.
[Violation] 'requestAnimationFrame' handler took 488ms
[Violation] 'requestAnimationFrame' handler took 61ms
[Violation] 'requestAnimationFrame' handler took 55ms
[Violation] 'requestAnimationFrame' handler took 61ms
[Violation] 'requestAnimationFrame' handler took 73ms
[Violation] 'requestAnimationFrame' handler took 67ms
[Violation] 'requestAnimationFrame' handler took 56ms
[Violation] 'requestAnimationFrame' handler took 55ms
[Violation] 'requestAnimationFrame' handler took 53ms
[Violation] 'requestAnimationFrame' handler took 55ms
[Violation] 'requestAnimationFrame' handler took 51ms
[Violation] 'requestAnimationFrame' handler took 53ms
[Violation] 'requestAnimationFrame' handler took 50ms
[Violation] 'requestAnimationFrame' handler took 58ms
[Violation] 'requestAnimationFrame' handler took 111ms
[Violation] 'requestAnimationFrame' handler took 213ms
[Violation] 'requestAnimationFrame' handler took 210ms
[Violation] 'requestAnimationFrame' handler took 211ms
[Violation] 'requestAnimationFrame' handler took 180ms
[Violation] 'requestAnimationFrame' handler took 57ms
[Violation] 'requestAnimationFrame' handler took 52ms
[Violation] 'requestAnimationFrame' handler took 51ms
[Violation] 'requestAnimationFrame' handler took 50ms
[Violation] 'requestAnimationFrame' handler took 74ms
[Violation] 'requestAnimationFrame' handler took 67ms
[Violation] 'requestAnimationFrame' handler took 62ms
[Violation] 'requestAnimationFrame' handler took 52ms

[nixel2007]

Sonar logs are empty :(

[stefanrinderle]

Hey,
sorry for not getting back to this within the last days. Was pretty busy.

Thanks for the browser logs. I can reproduce the "[Violation] 'requestAnimationFrame' handler took ...ms" with larger projects on my machine, but i could still see the visualization.

So maybe it's an browser issue? What system and browser are you using?
Or, it could be a camera position issue. Could you try to type "r" while focussing the visualization, or try to move around with the mouse.

Thanks for trying

[nixel2007]

What system and browser are you using?

Windows 10 Pro x64 with Chrome 58.0.3029.110 (64-bit) (latest)
SonarQube runs at Ubuntu Server x64
r button nor moving with mouse doesn't help.

I've noticed that Violation messages continue spam to console after rendering page.
I'll try to wait some time (10 mins? 1 hour? :D ).

[nixel2007]

I could enable debug logs at our sonarqube server if it may helps

[stefanrinderle]

@nixel2007 Thanks again for the infos.

The violation messages show that the rendering is taking a bit too long, but this shouldn't be an issue. I have the same messages on bigger project, but visualization itself is not affected.

The messages appear after leaving the plugin because we didn't yet close out application correctly. I create the issue #122 for that, which is already fixed. Thanks for recognizing this.

But unfortunately, i still have no idea why it does not work for your project. I therefore created a version with a bit more logging which maybe helps us to identify what's really broken. You should see the follwing info level logging in your browser with the attached version:

load complete SONAR_LOAD_METRICS bundle.js:1 Options: bundle.js:1 e {layout: e, footprint: e, height: e, scale: e…} bundle.js:1 load complete SONAR_LOAD_MEASURES bundle.js:1 prepare model bundle.js:1 build city three.min.js:160 THREE.WebGLRenderer 85 bundle.js:1 shapes: bundle.js:1 t {$mobx: e} bundle.js:1 load complete BUILD_CITY

Could you try this out? Thanks.
sonar-softvis3d-plugin-0.9.1-SNAPSHOT.jar.zip

[nixel2007]

t-object contains 31833 entries...

[nixel2007]

Maybe we can plan some pair teamviewer time? I'll open this project, chrome dev tools and you could add some breakpoints...

[stefanrinderle]

@nixel2007 Hey again ;-)

Thanks for using the provided version of the plugin. As far as i can see, this is possibly not an issue withe the plugin itself. There is no error shown and the shapes should be loaded correctly to the scene. That the scene shoudl contain aroung 30k objects is a good marking point. I found the following website:

http://www.smartjava.org/tjscb/02-geometries-meshes/02.05-handle-large-number-of-object.html

How many cubes can you add before your browser gives up? For me, it works with 22000 objects, but fps is down to 4. But unfortunately our cubes are not equally optimized as in the example.

I also realized that there is nothing to see during the loading. How long did you wait for something to appear? I added two versions of the plugin.

1. Log output during the shapes are added to the plugin
2. Same log output, but max 15000 elements will be added to the scene

Also thanks for suggesting a Teamviewer session, but i hope that this time it will be more clear what happens.

sonar-softvis3d-plugin-0.9.1-SNAPSHOT_2.jar.zip
sonar-softvis3d-plugin-0.9.1-SNAPSHOT_1.jar.zip

[nixel2007]

hello, @stefanrinderle!

How many cubes can you add before your browser gives up?

I have an empty screen near 33k at my home machine. I've checked sonar plugin at my place also :)

I also realized that there is nothing to see during the loading. How long did you wait for something to appear?

I have a loading screen with flying dots ~2-3 minutes with

Request measures from SonarQube
0%
0 of 2

Than the canvas with keyboard shortcuts (left side of the screen) renders - all white/empty.
In that state I tried to wait near 12 hours - nothing happens.

I'll try to check this versions in a short time.
Thanks for your work!

[nixel2007]

Oh my god, guys, i've noticed something... strange, but understandable.

So... If I select the evostreet mode I get very... no... VERY-VERY-VERY LONG street. It is so huge, that it can not be visible at default scale. But if you scroll up...

District-mode is working well.

I've loaded the first of two new versions. I'll try to load this page tomorrow at my work pc.

P.S. I've tried to move aroud with mouse at old versions, and i didn't notice any... long lines as at the gif above. But at home i have a 29" screen, and only 17" at work, so (probably?) I just couldn't see them. Did you change something noticable in the code that could (finally!) helps me with rendering?

[stefanrinderle]

Hey @nixel2007 ,

great to see it finally working ;-)

The only thing i really changed in the version i attached here, was that the objects have been kind of "shrinked" in size. I thought that this could probably help. So the last thing you could do for us is to check the attached versions. One has the shrink activated (sonar-softvis3d-plugin-0.9.1-SNAPSHOT_shrinked.jar.zip) the other one is a current default snapshot version.

Do both version work or only the shrinked one?

Thanks

[stefanrinderle]

I would assume it works without shrinking the view. Thanks to everyone who was involved in finding the issue.