Add support for query-based views

[dmcassel]

Add support for query-based views

[grtjn]

I was mainly sharing my runScript task as a workaround :) (and for other uses ;)

[rjrudin]

@dmcassel @grtjn How about the following design for a user:

1. ml-schemas/qbv is now a "special" folder, similar to ml-schemas/tde.
2. A file in the "qbv" directory should be either an sjs or xqy script that ends with generateView("mySchema", "myView") (user can pass in any args they want to that method).

When the user deploys their app or loads schemas, ml-gradle will then send each script to /v1/eval, which will return an XML plan:query-based-view. ml-gradle then sends that to /v1/documents to insert the schema - or the eval call does that as well.

I think collections.properties and permissions.properties would then be applied to the XML view that's sent to /v1/documents. Though the special QBV collection will automatically be added, just like what's done for a TDE template.

That seem like a good design?

[rjrudin]

Good idea @dmcassel - just curious, do you have a custom task or anything that does this already?

This would be a nice addition for all the Optic-based testing we do on our connectors too; we typically only use TDE's. Having this would make it easier to include QBV's for testing, to at least sanity check that those work fine too.

[dmcassel]

@rjrudin I don't have a custom task yet. This idea came from exploring the QBV feature and thinking about how to deploy it. I can picture a custom task that would include the query and do the insert, but I haven't written one.

[grtjn]

We have deployed QBVs by pre-generating them in QC or such, although we have a Task that can evaluate adhoc queries too (I'll share below). Once generated, you can put them in ml-schemas/qbv/ (folder is arbitrary I think). Put a collections.properties next to it which adds 'http://marklogic.com/xdmp/qbv'. permissions.properties works in the same way as with TDEs, although it is suggested to use/add 'query-view-admin'. mlLS works well after that.

Here my runScript task, which expects either -Pxquery={path} or -Pjavascript={path}, and also supports a -Poutput={path}. It is not well tested, but seems to work fine so far:

def decodeMultipart(responseEntity) {
  def boundary = responseEntity.getHeaders().getFirst('Content-Type').replaceAll(/^.*boundary=(.*)$/, '$1')
  def parts = responseEntity.getBody().replace('\r\n', '\n').replace('\r', '\n').split('--' + boundary)
  def body = ''
  parts.each {
    def splits = it.split('\n\n')
    if (splits.length > 1) {
      body = body + splits[1];
    }
  }
  return body
}

def runScript(xquery, jscript) {
  def url = mlHost + ":" + mlAppServicesPort + '/v1/eval'
  logger.lifecycle('Running script ' + (xquery ?: jscript))

  try {

    def manageConfig = getProject().property("mlManageConfig")
    if (mlAppServicesSimpleSsl == 'true') {
      manageConfig.setScheme("https")
      manageConfig.setConfigureSimpleSsl(true)
    } else {
      manageConfig.setScheme("http")
      manageConfig.setConfigureSimpleSsl(false)
    }

    def manageClient = new com.marklogic.mgmt.ManageClient(manageConfig)
    def code = resolveTokens(new File(xquery ?: jscript).getText('UTF-8'))
    def body = decodeMultipart(manageClient.postForm("/v1/eval", xquery ? "xquery" : "javascript", code))
    logger.debug("Success: ${body}")

    return body

  } catch (java.net.ConnectException ex) {
    logger.warn("Host not reachable: "+ mlHost + ":" + mlAppServicesPort)
  } catch (Exception ex) {
    logger.error("Running script failed:" + ex.getMessage())
  }
}

task runScript(group: project.name) {
  doLast {
    def xquery = project.findProperty("xquery")
    def jscript = project.findProperty("jscript")
    if (xquery == '' && jscript == '') {
      logger.error('-' * 30)
      logger.error('ERR: No script provided. Use -Pxquery=.. or -Pjscript=.. to provide a script.')
    }

    def body = runScript(xquery, jscript)

    def file = project.findProperty("output")
    if (file) {
      logger.lifecycle("Results written to " + file)
      new File(file).write(body)
    } else {
      print body
    }
  }
}

[dmcassel]

We have deployed QBVs by pre-generating them in QC or such, although we have a Task that can evaluate adhoc queries too (I'll share below). Once generated, you can put them in ml-schemas/qbv/ (folder is arbitrary I think). Put a collections.properties next to it which adds 'http://marklogic.com/xdmp/qbv'. permissions.properties works in the same way as with TDEs, although it is suggested to use/add 'query-view-admin'. mlLS works well after that.

I considered the idea of using QC to generate, then writing the results and using mlLoadSchemas to deploy. While that would work, the what we'd have in git is an artifact, rather than code we could reasonably work with. If we wanted to make a modification to the QBV, the file in git would be the artifact rather than the original query.

What you provided is interesting for ad hoc queries; thanks!

[grtjn]

Yeah, you want to have the code in the repo as well. We wanted to have both. That is the reason why I came up with the runScript task. We put the code in src/main/runScripts/. You can still decide to copy-paste it into QC, and run it from there.

But I can see why generating, and including it in deployment automatically would be convenient.

[BillFarber]

@dmcassel @grtjn - I've created a PR for this feature in the ml-javaclient-util project. Check it out and let me know if you have any thoughts: marklogic/ml-javaclient-util#182

[dmcassel]

@BillFarber I haven't tested it, but from the code I think it looks great -- thanks for taking this on!

[rjrudin]

This is all implemented in ml-javaclient-util.