alexwl/haskell-code-explorer

Web application for exploring and understanding Haskell codebases

haskell-code-explorer's Issues

Comparison with HLS

Hello,

I just stumbled upon this repo and the first question I had was "How does this compare to haskell-language-server" given that it has progressed quite fast in the past years. Searching through the issues brought up :

My goals with haskell-code-explorer are to create a code-browsing tool that deeply understands Haskell and to experiment with code intelligence features that are Haskell-specific and are not considered 'standard' (not supported by LSP yet).

Wondering if a section in the Readme would be beneficial ? I'm also curious about possible integration with tools like sourcegraph.

Cheers,

Jun

Enhancement: keyboard shortcuts with completing search

It would be great to be able to navigate the UI with keyboard shortcuts combined with fuzzy/completing search. A few specific ones I would find useful:

• Switching files
• Switching packages
• Finding a definition in this file
• Finding a definition in this project
• Finding a definition in any project

Hackage links are broken

Links to documentation on Hackage are broken since they're missing
src/ in the URL.

e.g. clicking on "Go to definition" for Enum leads to
https://hackage.haskell.org/package/base-4.9.1.0/docs/GHC-Enum.html#t:Enum
instead of
https://hackage.haskell.org/package/base-4.9.1.0/docs/src/GHC-Enum.html#t:Enum.

ghc-8.10.1 support?

Hi, thanks for the awesome project :) I wonder if there's plan to add support for building with ghc-8.10.1?

submodule

The stack project contains multiple submodules, each with its own .stack-work/dist directory. The dist directory in the root directory of the project does not have a .stack-work directory.
How do you index the package? Can I automatically index all submodule packages? For example: the libs directory and the services directory of the wire project, all submodules in these two directories have their own .stack-work directory.

https://github.com/wireapp/wire-server

Find references across packages

Hi again!

A feature request, or a confused user request :)

The project I'm looking at has a structure like:

MyProjectFoo/MyProjectFoo.cabal
MyProjectFoo/bar/bar.cabal

I've indexed these separately, and loaded them all into one server instance. If I'm looking at MyProjectFoo, then it finds references to symbols defined in bar.

However, if I'm looking at a symbol in bar and find all references to that symbol, it only seems to find those in bar (and not references coming from the top level project MyProjectFoo).

My goal is to be able to look at a symbol defined in bar and discover the references in other projects. Is this possible?

Thanks!

Part of the `ghc-8.2.2/typecheck/FamInst.hs` file is missing

For example, this function should be indexed:
https://haskell-code-explorer.mfix.io/package/ghc-8.2.2/show/typecheck/FamInst.hs#L1224

Error while indexing: cannot satisfy -package-id

I built this project with the following Nix file:

{ pkgsOld   ? import <nixpkgs> { }
, compiler  ? "ghc844"
}:

let
  config = {
    packageOverrides = pkgs: rec {
      haskellPackages =
        # builtins.trace (builtins.concatStringsSep "\n" (builtins.attrNames pkgs.haskell.packages))
        pkgs.haskell.packages."${compiler}".override {
        overrides = haskellPackagesNew: haskellPackagesOld: {
          cabal-helper = pkgs.haskell.lib.doJailbreak haskellPackagesOld.cabal-helper;
        };
      };
    };
  };
  pkgs = import <nixpkgs> { inherit config; };
in with pkgs; haskellPackages.callPackage ./haskell-code-explorer.nix { }

where haskell-code-explorer was generated by cabal2nix.

I then dropped into a nix-shell for the project I want to index, and called

./result/bin/haskell-code-indexer -p ../llvm-pretty-bc-parser --dist ../llvm-pretty-bc-parser/dist

and got:

2019-01-16 10:25:18.166980006 PST : [info] haskell-code-indexer version 0.1.0.0, GHC version 8.4.4.0                                                         
2019-01-16 10:25:18.694597024 PST : [info] Indexing llvm-pretty-bc-parser-0.4.0.0                                                                            
2019-01-16 10:25:18.786759929 PST : [error] Error while indexing component lib : <command line>: cannot satisfy -package-id array-0.5.3.0                    
    (use -v for more information)

But ghc-pkg lists it:

ghc-pkg list | grep array
    array-0.5.3.0

Any idea why it can't be found? cabal build works just fine.

(One thing that might be helpful in debugging this is if the debug logging actually printed the full GHC command line rather than just the flags.

Enhancement: Kind of type parameter on hover

Could the code viewer perhaps show the inferred kind of a type parameter on hover, perhaps with a similar UI to type-on-hover? e.g. it might show Type or * when hovering over a in the following declaration (on either the LHS or RHS):

newtype Foo a = MkFoo a

Make more mobile browser friendly

I just tried it for the first time, but happened to be on my phone - the layout (scrolling) is not so mobile browser friendly.

I am sure desktop browsing is the main target, maybe some smaller layout tweaks could improve the mobile experience. :)

Consider including GHC version in executable name

My understanding is that the GHC version used to build haskell-code-explorer needs to match the GHC version of the project you're indexing (eg see #16 (comment)).

This wasn't initially clear to me when initially trying to get haskell-code-indexer working, and using a haskell-code-indexer built with the wrong version of GHC lead to confusing error messages, eg

2019-03-20 17:18:19.736738839 EDT : [error] Error while indexing component li
KSLx7jSwdIM:
    MonadRandom-0.5.1.1-9vmCiRoafqTKSLx7jSwdIM is unusable due to missing dep
      primitive-0.6.4.0-1mvPxVOk6Q6KOkWCZxqESf transformers-0.5.5.0 transform
    (use -v for more information)

Because of this, would it make sense to adopt Haskell IDE Engine's convention for executable naming? They append each executable with the GHC version used to build it, eg:

$ ls hie*
hie      hie-8.2.1  hie-8.4    hie-8.4.3  hie-8.6    hie-8.6.2
hie-8.2  hie-8.2.2  hie-8.4.2  hie-8.4.4  hie-8.6.1  hie-wrapper

As you can see they also build one version of hie without the GHC version appended, but I'm not sure we even need that.

This would also make it really easy to have lots of different versions of haskell-code-indexer on your machine at once.

Failing to index

Hello,
First things first, thank you for this awesome project.
I'm trying to index a simple project (just a blank project I created with stack new ...) but I'm failing so far :-(
I'm using the stack-8.8.3.yml for haskell-code-explorer.

Whether I have ghc-8.8.3 installed or not, whether I use the installed haskell-code-explorer binary or run stack exec haskell-code-indexer -- -p myProject it fails with cabal-helper-wrapper: Installing Cabal version 3.0.1.0 failed.

I have tried to install cabal 3.0.0.0 (with ghcup if that's relevant) but same problem.

I'm not too sure what I'm doing wrong and I'd love to have your input on this.

This is the log when I run with --verbose. Note that it says that GHC version 8.6.5.0 is used instead of 8.8.3 (also not sure if this is normal).

Add support for GHC 8.4.3

Strange lines source code of conduit

Namely https://haskell-code-explorer.mfix.io/package/conduit-1.3.0.2/show/src/Data/Conduit/Combinators.hs seems to contain some output from CPP preprocessor or some other tool

Immediately fail on GHC mismatch

I ran haskell-code-explorer on a medium size project and got huge wall of error output. When I started up haskell-code-server only a few modules had been indexed.

It turns out the culprit was a GHC version mismatch, but I hadn't noticed the problem because the notification about it was scrolled of the screen.

Might a better default be to fail immediately when a GHC mismatch is detected?

Add instructions on how to index the GHC code base

Hi, I am using the online version of your haskell-code-explorer to quickly jump around the GHC code base. That works great and thank you for creating this. But I was not able to setup the local version with a cloned repo of GHC, could you provide some documentation on how to do that?

Add support for Cabal-2.2.x

cabal-helper-0.8.0.2 that is used by haskell-code-indexer supports only Cabal >=1.14 && <1.26 || ==2.0.*

Attempt to index a package that was built using Cabal 2.2.0.0 results in an error : Could not find module ‘Distribution.PackageDescription.Parse’ (#4)

Testing team

Would it be a good idea to assemble a team of testers for various platforms?

I'd be happy to be the Ubuntu tester.

GHC version match check seems to be checking the version that Cabal was compiled with

I'm not 100% but it seems that getPackageGhcVersion is actually parsing out the GHC version that cabal-install/Cabal was built with, not the GHC version for the package.

Here's a snippet of the setup-config I have:

Saved package config for shake-0.17.8 written by Cabal-2.4.1.0 using ghc-8.4
<BLAH> exe:shake
<BLAH BLAH> /Users/adam/.stack/programs/x86_64-osx/ghc-8.6.4/bin/ghc

Problem indexing xeno package

It fails with the following message:

$ stack exec -- haskell-code-indexer -p ../xeno
2018-10-05 00:33:44.493431958 MSK : [info] haskell-code-indexer version 0.1.0.0, GHC version 8.2.2.0
cabal-helper-wrapper: /home/qrilka/ws/h/xeno/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.0.1.0/setup-config: openFile: does not exist (No such file or directory)
2018-10-05 00:33:44.688234312 MSK : [error] readCreateProcess: /home/qrilka/ws/h/haskell-code-explorer/.stack-work/install/x86_64-linux-tinfo6/lts-11.3/8.2.2/libexec/x86_64-linux-ghc-8.2.2/cabal-helper-0.8.0.2/cabal-helper-wrapper "--with-ghc=ghc" "--with-ghc-pkg=ghc-pkg" "--with-cabal=cabal" "/home/qrilka/ws/h/haskell-code-explorer/../xeno" ".stack-work/dist/x86_64-linux-tinfo6/Cabal-2.0.1.0" "package-db-stack" "flags" "compiler-version" "ghc-merged-pkg-options" "config-flags" "non-default-config-flags" "ghc-src-options" "ghc-pkg-options" "ghc-lang-options" "ghc-options" "source-dirs" "entrypoints" "needs-build-output" (exit 1): failed

The output of that command could be found at https://gist.github.com/qrilka/422fefa3208bb94e7d1093bf450f6116

Are there any new features?

1. Whether the left directory tree can be sorted by directory and alphabetically.

2. Whether the code area can be edited. For example, when viewing the source code, you want to add some comments and save them for future viewing.

3. After viewing the file A.hs/B.hs/C.hs sequentially, can you provide a shortcut key to quickly jump to the previous browsed file. For example, when viewing C.hs, use the shortcut key to return to the B.hs file.

4. Integrate with development tools such as vs-code or IntelliJ IDEA？

"Needed for compilation but not listed in your .cabal file's other-modules"

Hi!

VERSIONS: I'm using Haskell 8.2.2 and Cabal 2.0.1.0. I installed with stack --stack-yaml=stack-8.2.2.yaml install. I built master of haskell-code-explorer at d4827af.

I'm an experienced programmer but new to Haskell, so feeling a bit stupid here. Hoping to use this lovely tool to help me find my way around a large legacy codebase. I've done my best to follow the README and docs but apologies if I missed something obvious.

When running stack exec --no-ghc-package-path haskell-code-indexer -- -p ., I get:

[error] Error while indexing component exe-FooBarMyProject : These modules are needed for compilation but not listed in your .cabal file's other-modules: 
...
<Huge list of about 20 modules>
...

There are quite a lot of components. When I look at the cabal file for these failing components, I see:

other-modules:   Paths_FooBarMyProject

I found this defined in the Cabal docs. I'm guessing that in fact this is a red herring.

One thing the strikes me about the cabal file is that there are many executables/benchmarks defined which all depend on FooBarMyProject through build-depends:, and that FooBarMyProject.cabal looks like this:

name:                FooBarMyProject
...
<author etc>
...

library
  exposed-modules:
    <huge list of modules>

So my best guess is that those modules which haskell-code-indexer complains about are meant to be depended on through that library, somehow.

Thanks in advance if you can spare a moment to help with this.

segfault related to TemplateHaskell

Building the following code (e.g. with stack build) and then running haskell-code-explorer results in a segfault (yes, a segfault in Haskell, yikes!):

module Lib where

import Prelude

import qualified Data.Yaml as Yaml
import Language.Haskell.TH (Exp, Q, runIO)
import Language.Haskell.TH.Syntax (addDependentFile)

migrations :: ()
migrations = $(do
  _ <- runIO (Yaml.decodeFileEither "file.txt" :: IO (Either Yaml.ParseException Yaml.Value))
  [|()|])

The above snippet is of course silly, but I encountered the bug in a real project and the above is the result of deleting as much as possible from that project in order to create a minimal repro case. I have put the code in a git repository for your convenience: https://github.com/gelisam/haskell-code-explorer-bug

Note that since this is TemplateHaskell code, the Yaml.decodeFileEither "file.txt" is executed at compile time. Building the project succeeds, so it doesn't seem to be a bug in the yaml package. However, replacing Yaml.decodeFileEither "file.txt" with the simpler readFile "file.txt" does not trigger the bug, so there must be some weird interaction between the yaml library and haskell-code-explorer. Since the symptom is a segfault, I suspect it has to do with the libyaml C library it uses under the hood.

Use public Hoogle API to get documentation for not indexed packages

Hoogle supports exact queries that return Hackage URL, type signature, and HTML documentation, e.g.,
https://hoogle.haskell.org/?hoogle=map%20is%3Aexact%20package%3Abase%20module%3APrelude&mode=json

Fails to index

$ haskell-code-indexer --package /home/maerwald/git/hpath --dist dist-newstyle
2018-10-05 11:06:27.113302971 +08 : [info] haskell-code-indexer version 0.1.0.0, GHC version 8.2.2.0
cabal-helper-wrapper: /home/maerwald/git/hpath/dist-newstyle/setup-config: openFile: does not exist (No such file or directory)
2018-10-05 11:06:27.124410555 +08 : [error] readCreateProcess: /home/maerwald/git/haskell-code-explorer/.stack-work/install/x86_64-linux-tinfo6/lts-11.3/8.2.2/libexec/x86_64-linux-ghc-8.2.2/cabal-helper-0.8.0.2/cabal-helper-wrapper "--with-ghc=ghc" "--with-ghc-pkg=ghc-pkg" "--with-cabal=cabal" "/home/maerwald/git/hpath" "/home/maerwald/git/hpath/dist-newstyle" "package-db-stack" "flags" "compiler-version" "ghc-merged-pkg-options" "config-flags" "non-default-config-flags" "ghc-src-options" "ghc-pkg-options" "ghc-lang-options" "ghc-options" "source-dirs" "entrypoints" "needs-build-output" (exit 1): failed

I have no idea why it's trying to get into .stack-work directory of my package. I don't use stack for that package and told it to use dist-newstyle.

Docker image for easy and reproducible deployment

Hi,
I encountered a number of a issues in trying to build and install the binaries. Still there are a few issues in getting it to work.

This is a fantastic tool but having difficult build setup makes it difficult to access. It will be great if you can create a Dockerfile that allows this tool to be built reproducibly.

Fails to build

The Glorious Glasgow Haskell Compilation System, version 8.4.4

onfiguring library for haskell-code-explorer-0.1.0.0..
Warning: Packages using 'cabal-version: >= 1.10' must specify the
'default-language' field for each component (e.g. Haskell98 or Haskell2010).
If a component uses different languages in different modules then list the
other ones in the 'other-languages' field.
Preprocessing library for haskell-code-explorer-0.1.0.0..
Building library for haskell-code-explorer-0.1.0.0..
[1 of 7] Compiling HaskellCodeExplorer.Types ( src/HaskellCodeExplorer/Types.hs, /home/maerwald/git/haskell-code-explorer/dist-newstyle/build/x86_64-linux/ghc-8.4.4/haskell-code-explorer-0.1.0.0/build/HaskellCodeExplorer/Types.o )
[2 of 7] Compiling HaskellCodeExplorer.Preprocessor ( src/HaskellCodeExplorer/Preprocessor.hs, /home/maerwald/git/haskell-code-explorer/dist-newstyle/build/x86_64-linux/ghc-8.4.4/haskell-code-explorer-0.1.0.0/build/HaskellCodeExplorer/Preprocessor.o )
[3 of 7] Compiling HaskellCodeExplorer.GhcUtils ( src/HaskellCodeExplorer/GhcUtils.hs, /home/maerwald/git/haskell-code-explorer/dist-newstyle/build/x86_64-linux/ghc-8.4.4/haskell-code-explorer-0.1.0.0/build/HaskellCodeExplorer/GhcUtils.o )

src/HaskellCodeExplorer/GhcUtils.hs:1113:10: error:
    • Couldn't match type ‘String
                           -> Documentation.Haddock.Types.MetaDoc
                                mod0 Documentation.Haddock.Parser.Identifier’
                     with ‘Documentation.Haddock.Types.MetaDoc
                             (ModuleName, OccName) Documentation.Haddock.Parser.Identifier’
      Expected type: [HsDocString]
                     -> Documentation.Haddock.Types.MetaDoc
                          (ModuleName, OccName) Documentation.Haddock.Parser.Identifier
        Actual type: [HsDocString]
                     -> String
                     -> Documentation.Haddock.Types.MetaDoc
                          mod0 Documentation.Haddock.Parser.Identifier
    • Probable cause: ‘(.)’ is applied to too few arguments
      In the second argument of ‘(.)’, namely
        ‘parseParas . concatMap (unpackFS . (\ (HsDocString s) -> s))’
      In the second argument of ‘(.)’, namely
        ‘_doc
           . parseParas . concatMap (unpackFS . (\ (HsDocString s) -> s))’
      In the second argument of ‘(.)’, namely
        ‘overIdentifier (parseIdent flags)
           . _doc
               . parseParas . concatMap (unpackFS . (\ (HsDocString s) -> s))’
     |
1113 |   _doc . parseParas . concatMap (unpackFS . (\(HsDocString s) -> s))
     |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

src/HaskellCodeExplorer/GhcUtils.hs:1113:23: error:
    • Couldn't match type ‘[Char]’
                     with ‘Maybe Documentation.Haddock.Types.Package’
      Expected type: [HsDocString]
                     -> Maybe Documentation.Haddock.Types.Package
        Actual type: [HsDocString] -> [Char]
    • In the second argument of ‘(.)’, namely
        ‘concatMap (unpackFS . (\ (HsDocString s) -> s))’
      In the second argument of ‘(.)’, namely
        ‘parseParas . concatMap (unpackFS . (\ (HsDocString s) -> s))’
      In the second argument of ‘(.)’, namely
        ‘_doc
           . parseParas . concatMap (unpackFS . (\ (HsDocString s) -> s))’
     |
1113 |   _doc . parseParas . concatMap (unpackFS . (\(HsDocString s) -> s))
     |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cabal: Failed to build haskell-code-explorer-0.1.0.0 (which is required by
exe:haskell-code-indexer from haskell-code-explorer-0.1.0.0).

No "open in new tab" option after right clicking an identifier

Right click followed by "open in new tab" works for me for the files in the sidebar, but not for identifiers. Instead I see:

Including more packages in https://haskell-code-explorer.mfix.io/

First off, many thanks for creating this project! :)

I often go to https://haskell-code-explorer.mfix.io/ to see how some library export is being used in other packages. Recently I started getting involved with prettyprinter which unfortunately isn't included in the current package set.

What would it take to get it included? Would it be feasible to extend the package set to the entirety of e.g. LTS-14.23?

No cabal files

I get this error when trying to index one of the packages in a project.

cabal-helper-wrapper: user error (Pattern match failure in do expression at src/CabalHelper/Compiletime/Wrapper.hs:164:7-13)

After adding a trace to the line:

[cfile] <- traceShowId . filter isCabalFile <$> getDirectoryContents projdir

I get [].

Port to GHC 9.x.x and emacs package

Hello,

I ported haskell-code-explorer to GHC 9.x.x, more specifically 9.2.2 but probably at most only minor changes are needed for other versions with major version 9. Recently I have been happily using an Emacs package to talk to the server exclusively, so I removed the js client. You may find the last commit with a web client at https://g.ypei.me/hcel.git/tree/?h=last-web-client. Though my fork is currently just an indexer / server now, I might add back a simpler web client that does not require handling npm vulnerabilities once I have improved the server more (e.g. removing dependency on cabal-helper if possible).

I am happy to send pull requests here, if folks are ok with the license (AGPLv3+) covering my code.

Best,
Yuchen

Client

Cool project!

Is it possible to have a client which reads the indexed data?

This project could maybe be alternative solution for intero and so making the IntelliJ Haskell plug-in also work for Cabal projects.

Find identifiers across packages

Haskell-code-explorer is great for exploring a particular definition, but finding that definition is a bit more difficult than it should be, because I need to know in which package it is defined before I can use the identifier search to find that definition by name. Would it be possible to select more than one packages in which to search? Or perhaps, if I search for an identifier and find 0 results, offer the option to search in all other indexed packages?

Indexing stack sub-project fails to find dist directory

I have a number of cabal packages which live within a stack project.

MyProjectFoo/stack.yaml
MyProjectFoo/MyProjectFoo.cabal
MyProjectFoo/bar/bar.cabal
MyProjectFoo/baz/baz.cabal

I'm new to Haskell, stack and cabal, so things are a little confusing. Indexing works fine on MyProjectFoo, but fails on the subprojects bar and baz because it fails to find the dist directory and setup-config.

In this issue I found that using --dist $(stack path --dist-dir) from the bar or baz directory works.

Would it be reasonable for the indexer to use this if there exists a stack.yaml at any level in the filesystem above the current working directory?

Moved from #4 (comment).

Enhancement: Link to pragma documentation

It would be quite convenient if GHC pragmas like {-# OVERLAPS #-} or {-# LANGUAGE OverloadedStrings #-} linked to their documentation!

How to index all locally installed packages?

I'd like to index not just the package I'm working on, but also the rest of the packages in the stack.yaml's snapshot, as well as the packages in extra-deps.

Is there a way to do this, perhaps what haskell-code-explorer.mfix.io does?

Comparison with haskell-indexer

Hey, this looks really slick, great job!

There is a similar tool out there currently, haskell-indexer. Are you aware of it? If yes, could you shortly compare it to your approach? If no, could you summarise how your haskell-code-indexer works? Is it also a wrapper around ghc, or a plugin, does it use a custom parser, or something else?

We are currently building a tool that takes as input call graph information and lets you do queries on it (e.g. "give me all code paths from function f to function g): https://github.com/qrilka/haskell-navigation

Right now we are based on haskell-indexer; perhaps it would make sense for us to be able to work on the output of your tool as well?

Enhancement: Static site generation

I'd love to index a package set and host the results on e.g. Github's static site feature. Is that at all realistic? I see that the app/ folder contains quite a bit of server code, so this might be out of reach.

Default `--package` to `.`

This might make haskell-code-indexer and haskell-code-explorer a little more convenient to use.

If you think this is a good idea I can make a PR for it.

Static js files location

I've been giving this a spin, great work! I was wondering if there is some hardcoded path for the js files? Or is it always expecting those specified?

I've been using it with stack build --copy-compiler-tools for all the yamls and then stack exec which makes use much easier with stack, since I don't need to keep track of which version it was complied with for different projects.

But I'm a bit unsure about where to put the js files...
The best alternative from a use senario I guess would be to embed them at compile time?