Add configuration option to prevent Timers being expunged

[stuartpullinger]

Glassfish behaves differently to JBoss and Geronimo in expunging timers after an application exception rather than just logging the exception. Expunging the timers in Glassfish can only be fixed by redeploying the application.

We should add the 'reschedule-failed-timer' option to the setup-glassfish.py script to prevent the Timers from being expunged.

Some references:
https://stackoverflow.com/a/18303655
javaee/ejb-spec#111 - requesting clarity in the spec
https://github.com/javaee/glassfish/issues/20749 - reporting issue to Glassfish. Doesn't look like it will be addressed.
https://bitbucket.org/obfischer/glassfish-timerproblem/src - example application to produce the error

[stuartpullinger]

Instructions to edit the domain.xml file (found at payara41/glassfish/domains/domain1/config/domain.xml) are here.

I think this would be equivalent to the asadmin command:

asadmin set server.ejb-container.ejb-timer-service.property.reschedule-failed-timer="true"

but I haven't tested it.

[stuartpullinger]

The Expunged timers seem to occur when there is a problem with the connection to the database. Database queries hang which leads to scheduled actions failing which leads to the timers being expunged.

One guard against this happening could be to set a timeout for database queries. It can be done in the persistence.xml file and in the code. We currently have Payara version 4.1.2-173 in production which ships with EclipseLink 2.6.4. The reference documentation for setting the timeout in Eclipselink 2.6 is here.

More info (using a more recent version of the spec) is here. The JPA spec states that this new query hint is optional but the documentation for our implementation (EclipseLink) suggest it is supported in version 2.7. The spec states that the value should be milliseconds but the EclipseLink docs say they default to seconds so, if we upgrade Payara and start using the newer javax.persistence.query.timeout hint, care should be taken to also define the eclipselink.jdbc.timeout.unit hint to make clear the units.