Comments (13)
@kltm have we seen anything like this for the owltools loader?
from golr-loader.
Yes, things like it.
At first blush, I think it looks like a timeout before finish--either slow over wire or too large. Are you still doing the single blob?
from golr-loader.
from golr-loader.
You may be right at the limit then. Maybe try cutting out proxies or loading from local? It may be hard to figure it out without getting really good numbers on usage and time at each step.
If you get more fails on a smaller or more "remote" machine, that may also be an indication.
from golr-loader.
This sounds like a priority ticket - is it still happening?
from golr-loader.
Yes, it still randomly happens. It failed, the generated json can still be indexed afterwards, it's a manual step.
@kshefchek is on it, we talked about it on Friday and this morning.
from golr-loader.
The PR here: #33 is merely duct tape to fix this. It looks like we've outgrown the current approach, the run requires 155g of memory, with 64g in cached mem, we end up swapping which likely causes these intermittent time out issues. Some thoughts on a way forward:
- Fix at the systems level:
- Adjust swappiness
- Throw more resources at it - ie higher memory machine
Pros: no additional development needed
Cons: Short sited
- Optimize current code
- Reduce the need to build huge mapDB objects, while these are stored on disk there appears to still be some memory requirement (either that or there's a memory leak). The approach now is to cache subject,object pairs for a single query, then iterate over all paths per s,o to generate evidence, source, is_defined_by fields. If we simply ordered the results by subject we could generate solr docs, clear out the mapDB object, and move on.
- Write a golr updater
- Instead of clearing the index and reloading, only update certain fields for each build. This might be complicated given all the moving parts, graph isomorphism, etc. But this is more traditional than the scorched earth approach. Solr also supports concurrent updates.
from golr-loader.
On ordering the results - apparently you cannot apply an ORDER BY on a UNION of queries, I remember this was discussed when we initially developed the stack, and likely the reason for the huge mapDB approach:
I can see if there have been any updates since that could serve as a workaround.
from golr-loader.
@kshefchek Honestly, I think that 1b and 2 are the only real ways forward.
- I think 1a is a non-starter; once issues of swap come up, the battle is already lost, no matter the "stickiness"
- I think that updating the indexes starts getting a little weird--if anything falls out of phase, you can have dangling links and nonsense creep in very quickly
from golr-loader.
@kshefchek Practically speaking though, it seems like there is probably a fundamental non-scaling issue in the code. While techincally 1b might work, it is likely only deferring the issue a little while. At the very least, I'd vote that 2 should be done until the exact nature of the scaling issue is known.
from golr-loader.
Neo4J now has procedures that can be used as a workaround for post union processing
https://github.com/neo4j-contrib/neo4j-apoc-procedures
https://neo4j.com/developer/kb/post-union-processing/
On a test query it seems to work just fine, I can test this out if it sounds like a good way forward.
from golr-loader.
Taking a closer look, these timeouts occur when neo4j has a long GC pause, for example:
Golr loader:
Jul 11, 2017 8:22:52 AM org.monarch.golr.GolrLoader addAndCommitToSolr
WARNING: Caught: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr/golr
Neo4J:
2017-07-11 08:21:30.606+0000 WARN [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 2048ms.
2017-07-11 08:22:52.318+0000 WARN [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 60676ms.
2017-07-11 08:23:13.748+0000 WARN [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 789ms.
I have a test branch that uses post union processing to order the results, which results in less memory usage but has longer GC pauses, so the timeouts are more frequent. Knowing now that neo4j is the issue perhaps we can play around with some tuning of memory and GC settings, but we may need to switch to a server with more memory as our graph grows.
from golr-loader.
This is resolved now, the resolution was replacing UNION with UNION ALL and no longer having neo4j uniquify query results, where this is done in the indexing code anyway. The loader now has a much smaller memory footprint.
from golr-loader.
Related Issues (20)
- Investigate on slow queries HOT 1
- Queries should accept CURIEs in START statements
- chromose queries throw NPE
- Consider aligning getNeighbors and getClosure methods
- add flag to delete json as soon as they're indexed
- Use de-reification in evidence graphs HOT 3
- Extend golr pattern to include subj/obj equivalents and xrefs HOT 2
- NotInTransactionException in golr loader HOT 2
- Rewrite golr loader to convert results to SolrInputDocument instead of JSON HOT 2
- Indexing behavior when a sub-object pair is linked by multiple relations HOT 6
- Version mismatch? HOT 3
- Consider indexing non clique leaders
- Too many values for UnInvertedField faceting on field object_closure HOT 10
- Remove IRI field from search core HOT 2
- Index all associations instead of merging subject-object pairs per query
- Index synonyms from non-clique leader IDs
- ArrayIndexOutOfBoundsException when splitting curies
- English spelling/synonyms HOT 3
- Filter out obsoleted terms from search HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from golr-loader.