Load JDBC SQL driver at runtime in DOTS and amgr

Recently there was a discussion in the German Notes forum about an error when loading a JDBC driver. https://atnotes.de/index.php/topic,63166.0.html To be clear, the problem has so far only occurred with the driver for MySQL.
The driver used is the same as the one used for HCL Traveler when Traveler uses a MySQL database as backend datastore in HA mode.
I could not reproduce the described error in Traveler, but in a DOTS plugin and in a Java agent.
My tests were performed with Domino 12. But I assume that the behavior is also reproducible in V11.0.1.x.
Here is a description of my tests and instructions on how to solve the problem.
Once again. The problem is only with the JDBC MySQL driver and is not with Domino. Rather, it is a problem in the driver’s code, but it is affecting Domino.

Here is the code I used in a Java agent. I also use the same code in my DOTS plugin.

import java.io.File;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.sql.Driver;
import java.sql.DriverManager;

import lotus.domino.AgentBase;

public class JavaAgent extends AgentBase {
	static final String MYSQL_DRIVER = "com.mysql.cj.jdbc.Driver";
	static final String MYSQL_DRIVER_FILEPATH = "Traveler\\lib\\mysql-connector-java-8.0.22.jar";

	public void NotesMain() {

		try {
			File dbFile = new File(MYSQL_DRIVER_FILEPATH);
			updateClasspath(dbFile.toURI().toURL());
			Class.forName(MYSQL_DRIVER);

			Driver driver = DriverManager.getDriver("jdbc:mysql:");

			System.out.println(String.format("%s loaded and registered. Version: %d.%d", dbFile.getName(),
					driver.getMajorVersion(), driver.getMinorVersion()));
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	private void updateClasspath(URL path) {
		try {
			URLClassLoader cl = (URLClassLoader) ClassLoader.getSystemClassLoader();
			Method m = URLClassLoader.class.getDeclaredMethod("addURL", new Class[] { URL.class });
			m.setAccessible(true);
			m.invoke(cl, new Object[] { path });
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

In the simplest variant, the drivers to be used are simply copied to DominoPrgmDir/ndext and can then be accessed via

Class.forName(MYSQL_DRIVER);

The directory ndext is already in the classpath. But I use the drivers from the HCL Traveler installation. The directory is not in the classpath because Traveler brings its own JVM. So we need to extend the classpath before using the driver. This is done by the updateClasspath() method.

When you run the agent, you will get the following error.

AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent  error: Exception in thread "AgentThread: JavaAgent" 
Agent Manager: Agent  error: java.lang.ExceptionInInitializerError
Agent Manager: Agent  error:     at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
Agent Manager: Agent  error:     at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.initialize(J9VMInternals.java:87)
Agent Manager: Agent  error:     at java.lang.Class.forName(Class.java:347)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.NonRegisteringDriver.(NonRegisteringDriver.java:98)
Agent Manager: Agent  error:     at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.initialize(J9VMInternals.java:87)
Agent Manager: Agent  error:     at java.lang.Class.forName(Class.java:347)
Agent Manager: Agent  error:     at JavaAgent.NotesMain(Unknown Source)
Agent Manager: Agent  error:     at lotus.domino.AgentBase.runNotes(Unknown Source)
Agent Manager: Agent  error:     at lotus.domino.NotesThread.run(Unknown Source)
Agent Manager: Agent  error: Caused by: 
Agent Manager: Agent  error: java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "setContextClassLoader")
Agent Manager: Agent  error:     at java.security.AccessController.throwACE(AccessController.java:176)
Agent Manager: Agent  error:     at java.security.AccessController.checkPermissionHelper(AccessController.java:238)
Agent Manager: Agent  error:     at java.security.AccessController.checkPermission(AccessController.java:385)
Agent Manager: Agent  error:     at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
Agent Manager: Agent  error:     at lotus.notes.AgentSecurityManager.checkPermission(Unknown Source)
 Agent Manager: Agent  error:     at java.lang.Thread.setContextClassLoader(Thread.java:840)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.lambda$static$0(AbandonedConnectionCleanupThread.java:77)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread$$Lambda$8/0x0000000000000000.newThread(Unknown Source)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
Agent Manager: Agent  error:     at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.(AbandonedConnectionCleanupThread.java:80)
Agent Manager: Agent  error:     … 10 more
AMgr: Agent 'driver' in 'driver.nsf' completed execution

The relevant line in the stack trace is

Agent Manager: Agent  error: java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "setContextClassLoader")

A little causal research, and the reason was found. It is an error in the code of the driver.Here is the source https://bugs.mysql.com/bug.php?id=88172.
To solve the problem I have added the following lines to the java.policy in DominoPrgmDir/jvm/lib/security

grant codeBase "file:${notes.binary}/Traveler/lib/-" {
	permission java.lang.RuntimePermission "setContextClassLoader";
};

This fixed the problem in the DOTS plugin, but the error in the Java agent persisted. So i moved the “permission” to the

// default permissions granted to all domains

grant {

section at the top of the file.

This now solved the issue for DOTS and the Java agent. To be more precise. This solved it for the Java agent only, if you have the driver copied to the ndext directory. If you use the updateClasspath() method, you must add additional permissions to make your code work.

grant {
	permission java.lang.RuntimePermission "getClassLoader";
	permission java.lang.RuntimePermission "setContextClassLoader";
	permission java.lang.RuntimePermission "accessDeclaredMembers";
	permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

Modifying the java.policy file directly is generally not a good idea. The customizations may be lost during an upgrade.
HCL Domino uses the parameter javaOptionsFile= to make adjustments to the JVM.
Using this file we can now include our own custom.policy file. The content of the file is added to the DEFAULT java.policy at runtime. My custom.policy file has the following content

// =============================================================
//  custom.policy file added at runtime
//  -Djava.security.manager -Djava.security.policy=custom.policy
// =============================================================

grant {
	permission java.lang.RuntimePermission "getClassLoader";
	permission java.lang.RuntimePermission "setContextClassLoader";
	permission java.lang.RuntimePermission "accessDeclaredMembers";
	permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};

Save the file to the DominoDataDir for example.

Create a new file in the DominoDataDir javaOptionsFile.opt and add the following line to the file.

-Djava.security.manager -Djava.security.policy=./data/custom.policy

Save the file and add the following line to the server notes.ini

JavaOptionsFile=<DominoDataDir>/javaoptions.opt

Replace <DominoDataDir> according to your environment. After restarting Domino, you should now be able to load the driver without issues.

But wait. Here is one more thing. When you run the agent to load the driver, it will be loaded but you will see the following on the console.

te amgr run "driver.nsf" 'driver'
JVM: Java Virtual Machine initialized.
AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent printing: mysql-connector-java-8.0.22.jar
Agent Manager: Agent printing: Driver loaded.
Agent Manager: Agent  error: Error cleaning up agent threads
Agent Manager: Agent  error: Exception in thread "mysql-cj-abandoned-connection-cleanup" 
Agent Manager: Agent  error: java.lang.IllegalMonitorStateException
Agent Manager: Agent  error: 	at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
Agent Manager: Agent  error: 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
Agent Manager: Agent  error: 	at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
Agent Manager: Agent  error: 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Agent Manager: Agent  error: 	at java.lang.Thread.run(Thread.java:823)
AMgr: Agent 'driver' in 'driver.nsf' completed execution

The errors only occur when the JVM is not yet initialized. Any subsequent agent run does not show the output. Also, running the code in a DOTS plugin does not show any errors.

lo dots
JVM: Java Virtual Machine initialized.
Domino OSGi Tasklet Container started ( profile DOTS )
te dots run de.eknori.driver.main com.mysql.cj.jdbc.Driver C:\\Domino\\Traveler\\lib\\mysql-connector-java-8.0.22.jar
[DOTS] (de.eknori.driver.main) [driver]: Main started.
[DOTS] (de.eknori.driver.main) mysql-connector-java-8.0.22.jar
[DOTS] (de.eknori.driver.main) [driver]: Main disposed.

te amgr run "driver.nsf" 'driver'
AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent printing: mysql-connector-java-8.0.22.jar
Agent Manager: Agent printing: Driver loaded.
AMgr: Agent 'driver' in 'driver.nsf' completed execution

So far I can’t see why the error doesn’t occur when loading the JDBC MySQL driver in HCL Traveler. HCL uses the exact same code in Notes.jar.

I can only assume that the required adjustments to the java.policy are already in place when initializing the HCL Traveler JVM. I have not yet run my tests in a V11 environment. That is still pending.

Maybe someone is fancy to do this in a V11 environment and give feedback.