Debugging module loading

Under the hood, ml-gradle uses the MarkLogic Java Client to connect to a MarkLogic REST API server and load modules. So when you run into errors with loading modules, it's often helpful to run a quick test that uses the Java Client to confirm that you can connect to your REST API server outside the scope of ml-gradle.

Debugging module loading without involving ml-gradle

Here's a task that you can customize and run for loading modules via port 8000 (this includes all "asset" modules - i.e. not REST API services, transforms, or options, which must be loaded via your application-specific REST API server):

task testLoadModule {
    doLast {
	// See https://docs.marklogic.com/javadoc/client/com/marklogic/client/DatabaseClientFactory.html
	def host = "localhost"
	def port = 8000
	def database = "example-modules"
	def username = "admin"
	def password = "admin"

	// See https://docs.marklogic.com/javadoc/client/com/marklogic/client/DatabaseClientFactory.SecurityContext.html
	def context = new com.marklogic.client.DatabaseClientFactory.DigestAuthContext(username, password)
	def client = com.marklogic.client.DatabaseClientFactory.newClient(host, port, database, context)
	try {
		client.newDocumentManager().write("/test/module.xqy", new com.marklogic.client.io.StringHandle("<hello>world</hello>"))
	} finally {
		client.release()
	}
    }
}

You can use this as a starting point for any sort of debugging test, such as for loading a service or transform. Just check out the Java Client javadocs to see what calls need to be made.

Verifying that modules are in the right directories

See How modules are loaded to ensure that you have your modules in the directories that ml-gradle expects.

Module loading taking a long time?

Some projects have reported that when there are hundreds of non-REST modules to load or more (i.e. those under src/main/ml-config/root or src/main/ml-config/ext), the single call (defaults to port 8000) that loads the modules can take significantly different amounts of time on different machines - i.e. a few seconds on one machine, a few minutes on another machine.

As of ml-gradle 3.4.0, you can set the following property (e.g. in gradle.properties) to control how many modules are loaded in a single call to MarkLogic:

mlModulesLoaderBatchSize=100

This property has no value by default, which means all non-REST modules are loaded in a single call.

Prior to ml-gradle 3.4.0 (but not before ml-gradle 3.0.0), you can do some build.gradle surgery to configure the batch size that ml-gradle uses when loading non-REST module. You can put the following into build.gradle to specify a batch size to control how many modules are loaded at a time:

ext {
  def loadModulesCommand = mlAppDeployer.getCommand("LoadModulesCommand")
  loadModulesCommand.initializeDefaultModulesLoader(mlCommandContext)
  loadModulesCommand.modulesLoader.assetFileLoader.batchSize = 10
}

Understanding errors from the Java Client

The Java Client can throw an error based on a variety of conditions. Below are some common error messages to see and what might be causing them.

Invalid username/password - if the Java Client fails to connect to MarkLogic, you'll see an error like this:

Error occurred while loading REST modules: Local message: /config/query write failed: Unauthorized. Server Message: Unauthorized

To debug this, see Configuring security to understand what username/password properties are used for loading modules.

Resource not found - sometimes, the Java Client may connect to a MarkLogic app server that isn't a valid REST app server (i.e. it's not using the REST API rewriter). And you'll get an error like this:

Caused by: com.marklogic.client.ResourceNotFoundException: Local message: /config/query not found for write. Server Message: Request failed. Error body not received from server

To debug this, check the value of the "mlRestPort" property that's logged when using Gradle info-level logging (-i or --info on the command line). This tells you what application-specific REST server ml-gradle is trying to connect to. You can try using the script near the top of this page to debug the connection and loading of the REST module as well.

Configuring SSL

The Java Client SecurityContext javadocs show the methods for configuring an SSL context and hostname verifier on the SecurityContext. You can call these methods to configure an SSL connection to your REST server:

def context = new com.marklogic.client.DatabaseClientFactory.DigestAuthContext(username, password)
// This is one possible implementation of an SSLContext
context.withSslContext(com.marklogic.client.ext.modulesloader.ssl.SimpleX509TrustManager.newSSLContext())
context.withHostnameVerifier(com.marklogic.client.DatabaseClientFactory.SSLHostnameVerifier.ANY)

Authenticating with a certificate

If port 8000 and/or your REST API server (for loading REST extensions like services, transforms, and options) requires authenticating with a certificate, you can use the below task as a starting point for debugging authenticating with that app server. It doesn't load a module; it just evaluates a simple XQuery expression, which should suffice for verifying that you can authenticate correctly.

task testAuthenticateWithCertificate {
  doLast {
    def host = "localhost"
    def port = 8123 // the port of your REST API server
    def certFile = "path/to/cert.p12"
    def certPassword = "not-required"

    // See https://docs.marklogic.com/javadoc/client/com/marklogic/client/DatabaseClientFactory.CertificateAuthContext.html
    def context = new com.marklogic.client.DatabaseClientFactory.CertificateAuthContext(certFile, certPassword)
    def client = com.marklogic.client.DatabaseClientFactory.newClient(host, port, context)
    try {
      println client.newServerEval().xquery("fn:current-dateTime()").evalAs(String.class)
    } finally {
      client.release()
    }
  }
}

Debugging module loading