Client

[alexwl]

Client

[alexwl]

Thanks!

It is certainly possible to have a client that reads the indexed data.

Currently, the only client is a web application, so the output format of the indexer (haskell-code-indexer) is optimized for serving over HTTP: one gzipped JSON file for each indexed Haskell module; the code that saves the data to disk is here

haskell-code-explorer/app/Indexer.hs

Lines 118 to 135 in 74f9297

	BS.writeFile
	(outputDir </> HCE.packageInfoBinaryFileName)
	(S.encode $ HCE.toCompactPackageInfo packageInfo)
	mapM_
	(\(HCE.HaskellModulePath path, modInfo) ->
	let (compressFunction, compressExtension) =
	case configCompression config of
	Gzip -> (compress, ".gz")
	NoCompression -> (id, "")
	filePath =
	outputDir </>
	(encode (T.unpack path) ++ ".json" ++ compressExtension)
	in BSL.writeFile filePath . compressFunction . A.encode $ modInfo) .
	HM.toList $
	HCE.moduleMap (packageInfo :: HCE.PackageInfo HCE.ModuleInfo)
	BSL.writeFile
	(outputDir </> HCE.packageInfoJsonFileName)
	(A.encode packageInfo)

It is possible to add an option to the indexer to save the output in a form more suitable for reading by external applications (e.g., one binary file).

I think it is worth noting that the indexer does not support incremental reindexing, so I'm not sure if it is feasible to use the data from the indexer in IntelliJ Haskell plugin.

[rikvdkleij]

It is possible to add an option to the indexer to save the output in a form more suitable for reading by external applications (e.g., one binary file).

Also, the interface of client should have options to request which kind of info and which location (e.g. filepath/module name, rownr and columnnr)
(I took a quick look on the uncompressed indexed data but could not find e.g. the type of an identifier.)

I think it is worth noting that the indexer does not support incremental reindexing, so I'm not sure if it is feasible to use the data from the indexer in IntelliJ Haskell plugin.

I do not know if that would be deal-breaker. Have to find out...
Is is doable to make support incremental reindexing?

Something else, when I ran the indexer the first time, it was automatically building cabal-install while the cabal binary of the right version was on the path.

[alexwl]

Here is an example of JSON with the data:
https://haskell-code-explorer.mfix.io/files/amazonka-cloudsearch-domains-1.6.0/.haskell-code-explorer/test%252FTest%252FAWS%252FCloudSearchDomains.hs.json
(the Haskell module: https://haskell-code-explorer.mfix.io/package/amazonka-cloudsearch-domains-1.6.0/show/test/Test/AWS/CloudSearchDomains.hs)

The root JSON object has the following attributes:

• identifiers

  A map from the internal id of an identifier (a unique key of an identifier in the current module) to the identifier info

• occurrences

  A map from the location in the Haskell module to the internal id of an identifier and location-specific information (binder or not, instantiated type and instance resolution tree)

• sourceCodeHtml

  The source code of the module in HTML format

• declarations

  All declarations in the module

• name

  Module name

• id

  Module path

For example, information about "tests" function:

{
  "isExported": true,
  "idType": {
    "components": [
      {
        "tag": "Text",
        "contents": "["
      },
      {
        "tag": "TyCon",
        "name": "TestTree",
        "internalId": "4"
      },
      {
        "tag": "Text",
        "contents": "]"
      }
    ]
  },
  "locationInfo": {
    "packageId": {
      "name": "amazonka-cloudsearch-domains",
      "version": "1.6.0"
    },
    "tag": "ExactLocation",
    "moduleName": "Test.AWS.CloudSearchDomains",
    "startLine": 23,
    "endLine": 23,
    "modulePath": "test/Test/AWS/CloudSearchDomains.hs",
    "startColumn": 1,
    "endColumn": 6
  },
  "nameSpace": "VarName",
  "internalId": "0",
  "details": "VanillaId",
  "sort": "External",
  "occName": "tests",
  "externalId": "amazonka-cloudsearch-domains-1.6.0|Test.AWS.CloudSearchDomains|Val|tests",
  "demangledOccName": "tests"
}

Type of an identifier is the value of idType attribute.

Is is doable to make support incremental reindexing?

It is hard to say at the moment (I need to investigate this). I guess it would require significant changes in the indexing process.

Something else, when I ran the indexer the first time, it was automatically building cabal-install while the cabal binary of the right version was on the path.

I think cabal-helper didn't find the appropriate version of Cabal library.

From cabal-helper documentation (http://hackage.haskell.org/package/cabal-helper):

Since we do not want to bind the user of a development tool which utilises this library to a specific version of Cabal we compile the code which interfaces with the Cabal library's API on the user's machine, at runtime, against whichever version of Cabal was used to write the on disk information for a given project.

If this version of Cabal is not available on the users machine anymore, which is fairly likely since cabal-install is usually linked statically, we have support for compiling the Cabal library also. In this case the library is installed into a private, isolated, package database in $XDG_CACHE_HOME/cabal-helper so as to not interfere with the user's package database."

[rikvdkleij]

Closing this issue because I think it's get too complicated for integration with IntelliJ. It has to work out-of-the-box in various environments.

Thanks for answering my questions!