Keyspace: Buggy program behavior if rows have been removed between discovery and replication phase

[johannes-gehrs]

Describe the bug

Looking at this logic

        def processNonCounterRow(row: Row, whereClause: String): Unit = {
          if (ttlColumn.equals("None")) {
            val rs = getSourceRow(selectStmtWithTs, whereClause, cassandraConnPerPar, customFormat)
            if (rs.nonEmpty) {
              processRowWithTimestamp(row, whereClause, rs)
            } else {
              val rs = getSourceRow(selectStmtWithTTL, whereClause, cassandraConnPerPar, customFormat)
              if (rs.nonEmpty) {
                processRowWithTTL(row, whereClause, rs)
              }
            }
          }
        }

I think the normal case is val rs = getSourceRow(selectStmtWithTs... returns a result.

But if the row was deleted after the row-sets to be worked on have been created in dataReplicationProcess, then rs is empty.

It then falls back to getSourceRow(selectStmtWithTTL... i.e. the version with TTL!

However, in our case we do not pass a TTL column.

But if you look at how you construct selectStmtWithTTL

      case s if s.equals("None") => ""...

because TTL column is not set, it returns an empty string.
We then do getSourceRow with cls being... an empty string.

The error handling Column list (cls) cannot be null or empty will then trigger and stop the whole replication process.

To Reproduce

Not trivial, would need to delete rows at the right time.

Expected behavior

Not very sure, tbh. As far as I can tell this row could be skipped because it would show up in the next newDeletesDF.

Screenshots
n/a

Additional context
n/a

[nwheeler81]

Hi @johannes-gehrs this is a known issue with multiple use cases. You’ve already addressed two scenarios: deletes and TTLs. However, there’s an additional concern: inconsistencies arising from prolonged repair processes, node failures, or misconfigurations in a Cassandra cluster, which could lead to inconsistencies post-migration. CQLReplicator may skip mutations like deletes or TTLs, but it cannot reliably differentiate between actual mutations and inconsistencies, particularly when the root cause is indistinguishable from either scenario.
So, I’m thinking of adding another metric in the stats section to indicate the number of skipped rows due to TTLs and DELETEs, or to inform the user about what happened and which might require some attention and temporarily track skipped primary keys, but maybe you have a better idea :)

[johannes-gehrs]

Skipping the row is one thing, but what happens is that we get a runtime error Column list (cls) cannot be null or empty and the whole migration process stops as soon as one row is not found.

And executing the statement selectStmtWithTTL when the same is an empty string probably doesn't make sense, does it?

The code should verify if TTL_COLUMN is None, (the default), and if that is the case then afaict the select statement selectStmtWithTTL should never be executed.