BuildException DatabaseManagerNonTransientException

[mhardin18]

com.ibm.dbb.build.BuildException: com.ibm.db2.jcc.am.DatabaseManagerNonTransientException: [jcc][t4][2055][14191][4.31.18] The database manager is not able to accept new requests, has terminated all requests in progress,
or has terminated this particular request due to unexpected error conditions detected at the target system. ERRORCODE=-1224, SQLSTATE=55032

This is most likely a DBB api issue, but it is causing me pain. The code in question comes from the ImpactUtilities script.

deletedFiles.each { file -> // files in a collection are stored as relative paths from a source directory if (props.verbose) println "*** Deleting logical file for $file" logicalFile = buildUtils.relativizePath(file) metadataStore.getCollection(props.applicationCollectionName).deleteLogicalFile(logicalFile) metadataStore.getCollection(props.applicationOutputsCollectionName).deleteLogicalFile(logicalFile) }

In here, when we it calling the Collection.deleteLogicalFile method. What appears to be happening is that this is causing queued up transactions in db2, that when you have hundreds of deleted files, an under-powered db2 system can't keep up. That is guess based on what I know of how jcc operates under the covers.

There should be a more efficient method in the DBB api that allows for deletion of a list of logical files, just as it has one for adding a list of logical files.

So I know this is an edge case, as this case probably doesn't come up much in reality. However, during one of migrations, the plan was to migrate all source to git, and then once there, we would archive all obsolete members to a separate repository. As such, I have to rethink the plan to avoid this.

One question someone maybe able to answer, as I have not tried, but could do so without a lot of effort; if I did migrate all the code and then moved, which means deleting it from the original repo, can I just delete the collections data and scan anew and being that there is no record of the original logical file, it wouldn't see the need to delete it? I have tried this with a separate branch, but since it wants to clone the main branch, I haven't found the workaround for that.

Additional info:
Related issue: SQL1224N while applications try to connect to database.

Full stack trace:
Caught: com.ibm.dbb.build.BuildException: com.ibm.db2.jcc.am.DatabaseManagerNonTransientException: [jcc][t4][2055][14191][4.31.18] The database manager is not able to accept new requests, has terminated all requests in progress, or has terminated this particular request due to unexpected error conditions detected at the target system. ERRORCODE=-1224, SQLSTATE=55032 com.ibm.dbb.build.BuildException: com.ibm.db2.jcc.am.DatabaseManagerNonTransientException: [jcc][t4][2055][14191][4.31.18] The database manager is not able to accept new requests, has terminated all requests in progress, or has terminated this particular request due to unexpected error conditions detected at the target system. ERRORCODE=-1224, SQLSTATE=55032 at com.ibm.dbb.metadata.jdbc.CollectionImpl.getLogicalFilesFromDB(CollectionImpl.java:535) at com.ibm.dbb.metadata.jdbc.CollectionImpl.getLogicalFiles(CollectionImpl.java:189) at com.ibm.dbb.metadata.jdbc.CollectionImpl.getLogicalFile(CollectionImpl.java:334) at com.ibm.dbb.metadata.jdbc.CollectionImpl.deleteLogicalFile(CollectionImpl.java:321) at ImpactUtilities$_updateCollection_closure8.doCall(ImpactUtilities.groovy:537) at ImpactUtilities.updateCollection(ImpactUtilities.groovy:533) at ImpactUtilities.createMergeBuildList(ImpactUtilities.groovy:192) at build.createBuildList(build.groovy:601) at build.run(build.groovy:48) Caused by: com.ibm.db2.jcc.am.DatabaseManagerNonTransientException: [jcc][t4][2055][14191][4.31.18] The database manager is not able to accept new requests, has terminated all requests in progress, or has terminated this particular request due to unexpected error conditions detected at the target system. ERRORCODE=-1224, SQLSTATE=55032 at com.ibm.db2.jcc.am.b7.c(b7.java:413) at com.ibm.db2.jcc.t4.z.Z(z.java:1532) at com.ibm.db2.jcc.t4.z.y(z.java:760) at com.ibm.db2.jcc.t4.ac.h(ac.java:319) at com.ibm.db2.jcc.t4.ac.a(ac.java:256) at com.ibm.db2.jcc.t4.ac.c(ac.java:31) at com.ibm.db2.jcc.t4.q.a(q.java:32) at com.ibm.db2.jcc.t4.h.readFetch_(h.java:266) at com.ibm.db2.jcc.am.ResultSet.flowFetch(ResultSet.java:4065) at com.ibm.db2.jcc.t4.c.f(c.java:2447) at com.ibm.db2.jcc.am.bi.a(bi.java:257) at com.ibm.db2.jcc.t4.c.a(c.java:129) at com.ibm.db2.jcc.am.ResultSet.nextX(ResultSet.java:464) at com.ibm.db2.jcc.am.ResultSet.next(ResultSet.java:383) at com.ibm.dbb.metadata.jdbc.CollectionImpl.getLogicalFilesFromDB(CollectionImpl.java:523) ... 8 more

[mhardin18]

I did try adding a sleep(1000) call between each member deletion, and intial test still failed, though it went on for longer than it had been. So in theory I could step that up large enough value to manage the load on the db. However, the amount of time to delete some 4k files, would mean that is going to run for quite some time. And, unfortunately, I have no control over the physical resources for the db.

I'm only getting through about 500 members, with the sleep, as it is.

[mhardin18]

Well it turns out that the Collection.addLogicalFiles is doing it one by one as well. As I just got the same issue doing a scan of a branch. So this issue is not just an edge case. Will be opening a PMR regarding it.

As a workaround, looking at the source, I'm going to try and alter

// save logical files in batches of 500 to avoid running out of heap space if (logicalFiles.size() == 500) { if (props.verbose) println "** Storing ${logicalFiles.size()} logical files in MetadataStore collection '$props.applicationCollectionName'" metadataStore.getCollection(props.applicationCollectionName).addLogicalFiles(logicalFiles) logicalFiles.clear() }

Setting the size limit to a configurable property.

[M-DLB]

Hi Matthew - Indeed, the way the logical files are added is in bunch of 500 files, it was more to avoid building large objects in memory, but the benefit is that it can also give some relief to the metadatastore.
This might be also a solution when deleting lots of files.
If you happen to implement the same logic for deleting files, would you agree to contribute back to zAppBuild by opening a Pull Request with your proposed changes?

[dennis-behm]

@mhardin18
Moved from an issue into discussions. I agree, this needs investigation from the dev team.

[mhardin18]

I implemented it, and the subsequent scans have not ended up with the exception. But it wasn't something I could produce on demand; so I'm not entirely sure that it solves my issue. After discussions with the client, we are going to pursue some avenues with IBM to increase the performance of db2.

I added the pull request #571

[mhardin18]

Unfortunately, this did not put a band aide on my issue. Still randomly failing.