bfergerson/gitdetective

Find the use and users of open-source code

gitdetective's Issues

Latest build log should take into account deleted jobs

Deleting an index job leaves the project page in a weird place. Doesn't know which logs to show

Don't need functions_open-source.txt

Anything in there should be in defs/refs

Remove kythe:// from qualified_name

Project import limits

Configurable limit for project import : 20k max each file

Empty zpop set

Proven to be rather elusive:

java.lang.IllegalStateException: Empty zpop set
        at io.vertx.blueprint.kue.queue.KueWorker.lambda$zpop$5(KueWorker.java:157)
        at io.vertx.redis.impl.Command.handle(Command.java:155)
        at io.vertx.redis.impl.RedisConnection.handleReply(RedisConnection.java:397)
        at io.vertx.redis.impl.ReplyParser.handle(ReplyParser.java:182)
        at io.vertx.redis.impl.ReplyParser.handle(ReplyParser.java:21)
        at io.vertx.core.net.impl.NetSocketImpl.handleDataReceived(NetSocketImpl.java:321)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMsgReceived(NetClientImpl.java:197)
        at io.vertx.core.net.impl.VertxNetHandler.lambda$channelRead$0(VertxNetHandler.java:64)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:335)
        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:193)
        at io.vertx.core.net.impl.VertxNetHandler.channelRead(VertxNetHandler.java:64)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:122)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

Merge duplicate functions

Will need issue brought up in https://discuss.grakn.ai/t/what-does-key-actually-do/786 to be fixed for a permanent solution moving forward. In the mean time though there are tons of references not being attributed to projects because of duplicate functions. Need to create a script to merge them

Upgrade to Kythe 0.28

Unit tests

Lots and lots of unit tests + setup travis-ci

Indexer module incorrect Kue usage

Remove PROCESS_NEXT_JOB from indexer

Implement offsets in reference calculation

Gathering a project's references increases the calculation time for each reference added. Need to implement offsets to scale

Leaderboard doesn't always show highest score

Leaderboard can miss references and never increase to the correct amount. Periodically the leaderboard should be updated with the true count. Should happen when project page is visited. Maybe once an hour max.

Add enum class to myproject/otherproject

Make release distribution

Projects with no references

Projects with no external references still have a lot of data associated with them. Could show which projects they uses the most, external/internal methods they use the most, etc.

Why don't apache/commons-* work?

https://gitdetective.io/apache/commons-lang
https://gitdetective.io/apache/commons-io
https://gitdetective.io/apache/commons-math

Kythe isn't producing output on these projects. Need to build latest Kythe and see if this issue goes away.

Make indexers slaves of web's redis

Filter data out on indexer by hitting local redis

Remove Javascript source files

Would be cool if GitHub knew this was a Groovy project

Use KytheURI instead of string parsing

Gradle support

Reference calculation timeout

Bazel support

Store reference/definition data to disk not to memory

Data is getting too big. Needs to be cached to disk

Reference code location to line on Github

Need to translate Kythe offsets to Github line numbers

Fix broken pipe exception

Can't figure this one out. Pretty sure it's because Vert.x is being left open on the client side during refreshes. Should somehow detect when the user is leaving the page and close the connection gracefully?

Jun 15, 2018 1:09:02 PM io.vertx.ext.web.impl.RoutingContextImplBase
SEVERE: Unexpected exception in route
java.io.IOException: Broken pipe
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at io.netty.channel.DefaultFileRegion.transferTo(DefaultFileRegion.java:145)
	at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:379)
	at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:215)
	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:406)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:974)
	at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:244)
	at io.vertx.core.net.impl.ConnectionBase.endReadAndFlush(ConnectionBase.java:88)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:335)
	at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:193)
	at io.vertx.core.net.impl.VertxHandler.channelReadComplete(VertxHandler.java:106)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:399)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:381)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:374)
	at io.netty.handler.codec.ByteToMessageDecoder.channelReadComplete(ByteToMessageDecoder.java:307)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:399)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:381)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1339)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:399)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:381)
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:932)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)

Look into odd file references

https://gitdetective.io/fasterxml/jackson-core
JsonParser.nextToken() has a lot of file references that don't look correct

GithubArchiveSync blocks event loop

Clean code

• Remove code/queries not being used
• Comment stuff
• Vertx consumers should be on static final string not literals

Test references between groovy projects

gradle + groovy test

Use single JSch session in GraknImporter

Should get rid of:

com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: Connection reset
        at com.jcraft.jsch.Session.connect(Session.java:565)
        at com.jcraft.jsch.Session.connect(Session.java:183)
        at com.jcraft.jsch.Session$connect$19.call(Unknown Source)
        at io.gitdetective.web.work.importer.GraknImporter.downloadAndExtractImportFiles(GraknImporter.groovy:539)
        at io.gitdetective.web.work.importer.GraknImporter.this$3$downloadAndExtractImportFiles(GraknImporter.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
        at io.gitdetective.web.work.importer.GraknImporter.processImportJob(GraknImporter.groovy:140)
        at io.gitdetective.web.work.importer.GraknImporter.this$3$processImportJob(GraknImporter.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
        at io.gitdetective.web.work.importer.GraknImporter$_start_closure2_closure10.doCall(GraknImporter.groovy:118)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at groovy.lang.Closure.call(Closure.java:414)
        at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
        at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
        at com.sun.proxy.$Proxy5.handle(Unknown Source)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:271)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

KytheUsageExtractor support for languages other than Java

Way to safely start/stop/reset/update web/indexer

java.lang.*, javax.*, etc off limits to define

https://gitdetective.io/find-sec-bugs/find-sec-bugs includes javax.* class files locally and is getting reference counts off them. Need to block the ability to define main Java classes.

Forked project support

Web module blocks event loop periodically

Todo: get stack trace

Index out of bounds

Throwing index out of bounds on qualified name:

kythe://github?lang=java?com.google.common.collect.CollectCollectors.<T,K extends java.lang.Comparable<? super K>,V>toImmutableRangeMap(java.util.function.Function<? super T,com.google.common.collect.Range>,java.util.function.Function<? super T,? extends V>)

Throwing index out of bounds on qualified name:

kythe://github?lang=java?com.google.common.collect.ImmutableSortedMap.<K extends java.lang.Comparable<?>,V>naturalOrder()

Indexed SLOC statistics

For the aesthetics

Indexer should replace kythe urls with ids when possible

Better maven support

This is maven: https://gitdetective.io//h2database/h2database

Project showing < 10 top functions

https://gitdetective.io/google/gson has more than 10 externally referenced functions. Only listing 8

Indexers should send create job request to web

Indexers run off slaved Redis instances. Need to have them send job requests to web to save everything correctly.

No auto-build if project is building/importing/calculating

Add reference leaderboard rank to project page

Track project to project references

Don't think Grakn even needs to be queried. Should be enough in cache

Default constructors aren't saved correctly

Internal references still getting through

https://gitdetective.io/pietermartin/sqlg is still getting internal references through. About 1/4th of the 40k references in this project are internal like:

kythe://github?lang=java?path=sqlg-core/target/classes/org/umlg/sqlg/sql/dialect/SqlDialect.java#fb15e78081bd54383501511be31e4df994bc3118e513c2e91d54a922366b2c9a

Sitemap

For SEO

Add license and release badge to readme

Track reference round instead of reference instance

Currently system is setup so each function instance gets incremented by an offset. This is used to ensure only new references are queried. Need to change this from incremented each function instance to incrementing only after each calculation. Will help with scaling.

Disk backed storage

Redis alone won't cut it for too long

Add navigation to leaderboard pages

List top 100