Load JDBC SQL driver at runtime in DOTS and amgr

Recently there was a discussion in the German Notes forum about an error when loading a JDBC driver. https://atnotes.de/index.php/topic,63166.0.html To be clear, the problem has so far only occurred with the driver for MySQL.
The driver used is the same as the one used for HCL Traveler when Traveler uses a MySQL database as backend datastore in HA mode.
I could not reproduce the described error in Traveler, but in a DOTS plugin and in a Java agent.
My tests were performed with Domino 12. But I assume that the behavior is also reproducible in V11.0.1.x.
Here is a description of my tests and instructions on how to solve the problem.
Once again. The problem is only with the JDBC MySQL driver and is not with Domino. Rather, it is a problem in the driver’s code, but it is affecting Domino.

Here is the code I used in a Java agent. I also use the same code in my DOTS plugin.

import java.io.File;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.sql.Driver;
import java.sql.DriverManager;

import lotus.domino.AgentBase;

public class JavaAgent extends AgentBase {
	static final String MYSQL_DRIVER = "com.mysql.cj.jdbc.Driver";
	static final String MYSQL_DRIVER_FILEPATH = "Traveler\\lib\\mysql-connector-java-8.0.22.jar";

	public void NotesMain() {

		try {
			File dbFile = new File(MYSQL_DRIVER_FILEPATH);
			updateClasspath(dbFile.toURI().toURL());
			Class.forName(MYSQL_DRIVER);

			Driver driver = DriverManager.getDriver("jdbc:mysql:");

			System.out.println(String.format("%s loaded and registered. Version: %d.%d", dbFile.getName(),
					driver.getMajorVersion(), driver.getMinorVersion()));
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	private void updateClasspath(URL path) {
		try {
			URLClassLoader cl = (URLClassLoader) ClassLoader.getSystemClassLoader();
			Method m = URLClassLoader.class.getDeclaredMethod("addURL", new Class[] { URL.class });
			m.setAccessible(true);
			m.invoke(cl, new Object[] { path });
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

In the simplest variant, the drivers to be used are simply copied to DominoPrgmDir/ndext and can then be accessed via

Class.forName(MYSQL_DRIVER);

The directory ndext is already in the classpath. But I use the drivers from the HCL Traveler installation. The directory is not in the classpath because Traveler brings its own JVM. So we need to extend the classpath before using the driver. This is done by the updateClasspath() method.

When you run the agent, you will get the following error.

AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent  error: Exception in thread "AgentThread: JavaAgent" 
Agent Manager: Agent  error: java.lang.ExceptionInInitializerError
Agent Manager: Agent  error:     at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
Agent Manager: Agent  error:     at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.initialize(J9VMInternals.java:87)
Agent Manager: Agent  error:     at java.lang.Class.forName(Class.java:347)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.NonRegisteringDriver.(NonRegisteringDriver.java:98)
Agent Manager: Agent  error:     at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.initialize(J9VMInternals.java:87)
Agent Manager: Agent  error:     at java.lang.Class.forName(Class.java:347)
Agent Manager: Agent  error:     at JavaAgent.NotesMain(Unknown Source)
Agent Manager: Agent  error:     at lotus.domino.AgentBase.runNotes(Unknown Source)
Agent Manager: Agent  error:     at lotus.domino.NotesThread.run(Unknown Source)
Agent Manager: Agent  error: Caused by: 
Agent Manager: Agent  error: java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "setContextClassLoader")
Agent Manager: Agent  error:     at java.security.AccessController.throwACE(AccessController.java:176)
Agent Manager: Agent  error:     at java.security.AccessController.checkPermissionHelper(AccessController.java:238)
Agent Manager: Agent  error:     at java.security.AccessController.checkPermission(AccessController.java:385)
Agent Manager: Agent  error:     at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
Agent Manager: Agent  error:     at lotus.notes.AgentSecurityManager.checkPermission(Unknown Source)
 Agent Manager: Agent  error:     at java.lang.Thread.setContextClassLoader(Thread.java:840)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.lambda$static$0(AbandonedConnectionCleanupThread.java:77)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread$$Lambda$8/0x0000000000000000.newThread(Unknown Source)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
Agent Manager: Agent  error:     at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.(AbandonedConnectionCleanupThread.java:80)
Agent Manager: Agent  error:     … 10 more
AMgr: Agent 'driver' in 'driver.nsf' completed execution

The relevant line in the stack trace is

Agent Manager: Agent  error: java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "setContextClassLoader")

A little causal research, and the reason was found. It is an error in the code of the driver.Here is the source https://bugs.mysql.com/bug.php?id=88172.
To solve the problem I have added the following lines to the java.policy in DominoPrgmDir/jvm/lib/security

grant codeBase "file:${notes.binary}/Traveler/lib/-" {
	permission java.lang.RuntimePermission "setContextClassLoader";
};

This fixed the problem in the DOTS plugin, but the error in the Java agent persisted. So i moved the “permission” to the

// default permissions granted to all domains

grant {

section at the top of the file.

This now solved the issue for DOTS and the Java agent. To be more precise. This solved it for the Java agent only, if you have the driver copied to the ndext directory. If you use the updateClasspath() method, you must add additional permissions to make your code work.

grant {
	permission java.lang.RuntimePermission "getClassLoader";
	permission java.lang.RuntimePermission "setContextClassLoader";
	permission java.lang.RuntimePermission "accessDeclaredMembers";
	permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

Modifying the java.policy file directly is generally not a good idea. The customizations may be lost during an upgrade.
HCL Domino uses the parameter javaOptionsFile= to make adjustments to the JVM.
Using this file we can now include our own custom.policy file. The content of the file is added to the DEFAULT java.policy at runtime. My custom.policy file has the following content

// =============================================================
//  custom.policy file added at runtime
//  -Djava.security.manager -Djava.security.policy=custom.policy
// =============================================================

grant {
	permission java.lang.RuntimePermission "getClassLoader";
	permission java.lang.RuntimePermission "setContextClassLoader";
	permission java.lang.RuntimePermission "accessDeclaredMembers";
	permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};

Save the file to the DominoDataDir for example.

Create a new file in the DominoDataDir javaOptionsFile.opt and add the following line to the file.

-Djava.security.manager -Djava.security.policy=./data/custom.policy

Save the file and add the following line to the server notes.ini

JavaOptionsFile=<DominoDataDir>/javaoptions.opt

Replace <DominoDataDir> according to your environment. After restarting Domino, you should now be able to load the driver without issues.

But wait. Here is one more thing. When you run the agent to load the driver, it will be loaded but you will see the following on the console.

te amgr run "driver.nsf" 'driver'
JVM: Java Virtual Machine initialized.
AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent printing: mysql-connector-java-8.0.22.jar
Agent Manager: Agent printing: Driver loaded.
Agent Manager: Agent  error: Error cleaning up agent threads
Agent Manager: Agent  error: Exception in thread "mysql-cj-abandoned-connection-cleanup" 
Agent Manager: Agent  error: java.lang.IllegalMonitorStateException
Agent Manager: Agent  error: 	at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
Agent Manager: Agent  error: 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
Agent Manager: Agent  error: 	at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
Agent Manager: Agent  error: 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Agent Manager: Agent  error: 	at java.lang.Thread.run(Thread.java:823)
AMgr: Agent 'driver' in 'driver.nsf' completed execution

The errors only occur when the JVM is not yet initialized. Any subsequent agent run does not show the output. Also, running the code in a DOTS plugin does not show any errors.

lo dots
JVM: Java Virtual Machine initialized.
Domino OSGi Tasklet Container started ( profile DOTS )
te dots run de.eknori.driver.main com.mysql.cj.jdbc.Driver C:\\Domino\\Traveler\\lib\\mysql-connector-java-8.0.22.jar
[DOTS] (de.eknori.driver.main) [driver]: Main started.
[DOTS] (de.eknori.driver.main) mysql-connector-java-8.0.22.jar
[DOTS] (de.eknori.driver.main) [driver]: Main disposed.

te amgr run "driver.nsf" 'driver'
AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent printing: mysql-connector-java-8.0.22.jar
Agent Manager: Agent printing: Driver loaded.
AMgr: Agent 'driver' in 'driver.nsf' completed execution

So far I can’t see why the error doesn’t occur when loading the JDBC MySQL driver in HCL Traveler. HCL uses the exact same code in Notes.jar.

I can only assume that the required adjustments to the java.policy are already in place when initializing the HCL Traveler JVM. I have not yet run my tests in a V11 environment. That is still pending.

Maybe someone is fancy to do this in a V11 environment and give feedback.


Java agents with imported .jar files

Many of us Notes developers know the problem. You have developed a Java agent that contains methods from imported .jar files in addition to the actual code.
When the agent is executed on a HCL Domino server, it works for a while ( sometimes longer, sometimes shorter ), but then causes problems due to memory leaks.
The problem lies in the architecture of Domino. The imported .jar files are reloaded every time the agent is started and over time they use more and more RAM.

The workaround was to dump the .jar files to the server’s file system. This works, but always causes problems where you as a developer do not always have access to the server.
Or there is already a .jar in the file directory; but not in the required version. Copying your own .jar into the file system can then lead to undesirable side effects.

I had therefore created a case at HCL to get a fix for the problem. The problem was tracked under SPR # BHUY8PRMKK. Yesterday I received an update from support:

The Product Development Team had developed a possible fix that mitigates the issue but the fix was of medium risk and it was decided not to submit the fix. As an alternative they have come up with a workaround. Set the following notes.ini, this will cache the agentloader objects and reuse them.

EnableJavaAgentCache=2

I have set the parameter on my test system. At the moment it looks like it solves the problem. I will continue testing.

Many thanks to HCL Development & HCL Support ( especially to Abhaysingh Shirke ).


Create random files with random content with Java

I was playing with DAOS in Domino 12 recently and needed a way to create thousands of test files with a given file size and random content.

I did not want to use existing files with real data for my tests. There are several programs available for Linux and Windows. Google for it. But as a developer, I should be able to create my one tool.

Here is some sample Java code that uses java.util.Random to create filnames and content in an easy way.

package de.eknori;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Random;

public class DummyFileCreator {
	static int filesize = 129 * 1024;
	static int count = 9500;

	static File dir = new File("c:\\temp\\dummy\\");
	static String ext = ".txt";

	public static void main(String[] args) {
		byte[] bytes = new byte[filesize];
		BufferedOutputStream bos = null;
		FileOutputStream fos = null;

		try {
			for (int i = 0; i < count; i++) {
				Random rand = new Random();

				String name = String.format("%s%s", System.currentTimeMillis(), rand.nextInt(100000) + ext);
				File file = new File(dir, name);

				fos = new FileOutputStream(file);
				bos = new BufferedOutputStream(fos);

				rand.nextBytes(bytes);
				bos.write(bytes);

				bos.flush();
				bos.close();
				fos.flush();
				fos.close();
			}

		} catch (FileNotFoundException fnfe) {
			System.out.println("File not found" + fnfe);
		} catch (IOException ioe) {
			System.out.println("Error while writing to file" + ioe);
		} finally {
			try {
				if (bos != null) {
					bos.flush();
					bos.close();
				}
				if (fos != null) {
					fos.flush();
					fos.close();
				}
			} catch (Exception e) {
				System.out.println("Error while closing streams" + e);
			}
		}
	}

}

The code is self-explanatory. Adjust the variables to your own needs, and you are ready to go


Gradle – Execution failed for task ‘clean’. Unable to delete file

I am using Gradle to build my Java projects. This works well on a Mac, but the build process fails on a Windows machine, when the task clean is going to be executed.

Task :clean FAILED
FAILURE: Build failed with an exception.
What went wrong:
Execution failed for task ':clean'.
Unable to delete file: C:\0.GIT\travelerrules-dots\de.midpoints.travelerrules.dots\build\unpuzzle_temp\maven-ant-tasks-2.1.4-SNAPSHOT

The jar file is build when I omit the clean task, but I always wanted the build process to to all the build steps on Windows and on the Mac. I never found a good solution, and also upgrading Gradle did not solve the problem.

Today, I found at least a workaround. Run the following command from the Windows command prompt.

TASKKILL /F /IM java.exe

now you can use

gradle cleanEclipse eclipse clean build

without any issues.


DAOS – JAR files in Java agents.

Martin Vogel from sirius-net GmbH informed me in an email about a possible problem with DAOS and Java agents.

If DAOS is activated on a database, in addition to the attachments from data documents, the jar files contained in a Java agent are also transferred to the DAOS repository if they exceed the set size.

I have been able to reproduce the problem with Domino V11.
I started with a freshly installed server, activated DAOS on the server

and enabled the database for DAOS.

load compact -c -DAOS ON barcode.nsf
[0B1C:0004-2C3C] 23.01.2020 18:26:48   Informational, DAOS has been enabled for database barcode.nsf.
[0B1C:0004-2C3C] 23.01.2020 18:26:48   Compacting barcode.nsf (barcode),  -c -DAOS ON barcode.nsf
[0B1C:0004-2C3C] 23.01.2020 18:26:48   Recovery Manager: Assigning new DBIID for C:\Domino\Data\barcode.nsf (need new backup for media recovery).
[0B1C:0004-2C3C] Clearing DBIID 5B9C3856 for DB C:\Domino\Data\barcode.ORIG
[0B1C:0004-2C3C] 23.01.2020 18:26:49   Compacted  barcode.nsf, 5K bytes recovered (<1%),  -c -DAOS ON barcode.nsf
[0B1C:0002-2F14] 23.01.2020 18:26:50   Database compactor process shutdown 

Result: Both archive files contained in the agent were transferred to the DAOS repository as NLO.

The agent continues to run without any issue.


AutoPopulateGroup – Scheduled Agent

In yesterdays post about how to automatically populate a group document, I published code to do the job in the foreground only. One of my blog readers complained about this. Maybe I was to naive to think that even an unexperienced java developer like me could modify the given code to run on a scheduled basis on the server.

Well, here is the code for an scheduled agent.

import lotus.domino.*;
import javax.naming.*;
import javax.naming.directory.*;
import java.util.Hashtable;
import java.util.Vector; 

public class LDAPSearchWithFilter extends AgentBase { 

	private static String fldMembers = "Members";

    	public void NotesMain() { 

    	try {
        	Database _db;
        	Document _doc;
        	Session session = getSession();
        	AgentContext agentContext = session.getAgentContext();
        	_db = agentContext.getCurrentDatabase(); 

        	Agent ag1 = agentContext.getCurrentAgent(); 

        	String ldapCF = "com.sun.jndi.ldap.LdapCtxFactory";
        	String ldapURL = "ldap://localhost:389/";
        	String ldapBaseDN = "";
        	String ldapUserID = "";
        	String ldapPassword = ""; 

        	Hashtable env = new Hashtable(4);
        	env.put(Context.INITIAL_CONTEXT_FACTORY, ldapCF);
        	env.put(Context.PROVIDER_URL, ldapURL + ldapBaseDN);
        	env.put(Context.SECURITY_PRINCIPAL, ldapUserID);
        	env.put(Context.SECURITY_CREDENTIALS, ldapPassword); 

      	DocumentCollection _dc = _db.getAllDocuments();
      	Document doc = _dc.getFirstDocument();

      	while (doc != null) {
        	String searchCriteria = doc.getItemValueString("SelectionCriteria");
        	DirContext ctx = new InitialDirContext(env);
        	SearchControls ctls = new SearchControls();
		NamingEnumeration answer = ctx.search("", searchCriteria, ctls);
		PopulateGroup (answer, doc);
     		ctx.close();
        	doc = _dc.getNextDocument();
		} // end of while
	  } // end of try
		catch (Exception e) {
    		e.printStackTrace(); }
	} // end of Main 

	public static void PopulateGroup(NamingEnumeration col, Document doc) { 

    	try {
    	Item item = doc.getFirstItem(fldMembers);
    	Vector v = new Vector();
    	String result;
	if (col.hasMore()) {
        	while (col.hasMore()) {
            	  SearchResult sr = (SearchResult)col.next();
            	  result = (String)sr.getName();
                  v.addElement(result.replace(',','/'));
           	} // end of while
		  doc.replaceItemValue(fldMembers, v);
    		  doc.save(true);
		} // end of if
    	} // end of try
	catch (NamingException e) {
    		e.printStackTrace(); }
	catch (Exception e) {
    		e.printStackTrace(); }
	} // end of PopulateGroup
} // end of class

AutoPopulateGroup (If You Do Not Run Domino 8.5)

A few days ago, I wrote about a new feature of Domino 8.5 to automatically populate groups via a LDAP selectioncriteria. This is a great feature and I have successfully tested it on my sandbox server.
Since we run Domino 8.0.1 on our productive servers, we cannot use this very useful feature. …

But, with a few lines of JAVA and Lotusscript code, you can build your own solution to auto populate groups. Here is what I came out with.

I’ve created a subform with two fields and a button.

  • HiddenMembers, Text, hidden
  • SelectionCriteria, Text, editable

The “Populate Group” button contains the following code

'/* Declaration
Const fldMEMBERS = "Members"
Const fldHIDDEN = "HiddenMembers"
Const agntDOLDAP = "AutoPopulateGroup"

Sub Click(Source As Button)
	Dim s As New NotesSession
	Dim ws As New NotesUIWorkspace
	Dim db As NotesDatabase
	Dim agent As NotesAgent
	Dim doc As NotesDocument
	Dim uidoc As NotesUIDocument
	Dim searchResultItem As NotesItem
	Dim paramid As String 

	Set db = s.CurrentDatabase
	Set uidoc = ws.CurrentDocument
	Set doc = uidoc.Document
	Set agent = db.GetAgent(agntDOLDAP)
	Call doc.save(True, False)
	paramid = doc.NoteID
	Call agent.RunOnServer(paramid)
	Delete doc
	Set doc = db.GetDocumentByID(paramid) 

	Set searchResultItem = doc.getFirstItem(fldHIDDEN)
	Call uidoc.FieldSettext( fldMEMBERS,  "")
	Forall values In searchResultItem.Values
		Call uidoc.FieldAppendText(fldMEMBERS,  values)
		Call uidoc.FieldAppendText(fldMEMBERS, Chr(10))
	End Forall
	Call uidoc.Refresh
	doc.Remove(True)
End Sub

Like in Domino 8.5 you’ll have to run LDAP on your server. To access LDAP and do a search according to the SelectionCriteria, you need an agent with the following piece of Java code.

import lotus.domino.*;
import javax.naming.*;
import javax.naming.directory.*;
import java.util.Hashtable;
import java.util.Vector; 

public class LDAPSearchWithFilter extends AgentBase { 

	private static String fldTmpMembers = "HiddenMembers";

    	public void NotesMain() { 

    	try {
        Database _db;
        Document _doc;
        Session session = getSession();
        AgentContext agentContext = session.getAgentContext();
        _db = agentContext.getCurrentDatabase(); 

        Agent ag1 = agentContext.getCurrentAgent();
        String paramid = ag1.getParameterDocID();
        Document doc = _db.getDocumentByID(paramid); 

        String searchCriteria = doc.getItemValueString("SelectionCriteria"); 

        String ldapCF = "com.sun.jndi.ldap.LdapCtxFactory";
        String ldapURL = "ldap://localhost:389/";
        String ldapBaseDN = "";
        String ldapUserID = "";
        String ldapPassword = ""; 

        Hashtable env = new Hashtable(4);
        env.put(Context.INITIAL_CONTEXT_FACTORY, ldapCF);
        env.put(Context.PROVIDER_URL, ldapURL + ldapBaseDN);
        env.put(Context.SECURITY_PRINCIPAL, ldapUserID);
        env.put(Context.SECURITY_CREDENTIALS, ldapPassword); 

    try {
       	DirContext ctx = new InitialDirContext(env);
        	SearchControls ctls = new SearchControls();
		NamingEnumeration answer = ctx.search("", searchCriteria, ctls);
		PopulateGroup (answer, doc);
        	ctx.close(); 

	    		} catch(NamingException e) {
     	   		e.printStackTrace();
    		} 

	    } catch (Exception e) {
    		e.printStackTrace();
    }
} // end of Main 

public static void PopulateGroup(NamingEnumeration col, Document doc) { 

    try {
    Item item = doc.getFirstItem(fldTmpMembers);
    Vector v = new Vector();
    String result;    

	if (col.hasMore()) { 

        		while (col.hasMore()) {
            		SearchResult sr = (SearchResult)col.next();
            		result = (String)sr.getName();
                	v.addElement(result.replace(',','/'));
           	} // end of while

			doc.replaceItemValue(fldTmpMembers, v);
    			doc.save(true);
		}
    }
	catch (NamingException e) {
    		e.printStackTrace();
    		} catch (Exception e) {
    			e.printStackTrace();
    			}
	} // end of PopulateGroup
} // end of class

Set the agent’s runtime security to “2”, to allow restricted operations. When you have all code in place, you can test the function by typing a selection criteria and clicking the button.

The members field in your form should now show all persons that have “serv01” as their mailserver.

Download sample database