Validation R2 drastically different in Keras 3 (eager mode) compared to Keras 2 (graph mode)

[DaniCosta92]

Hi Keras community,

I’m seeing a large discrepancy in validation metrics between Keras 2.13 with eager disabled and Keras 3 with eager enabled. I’m training a model with the Adam optimizer and a custom R2 metric:

def r2_keras(y_true, y_pred):
    ss_res = K.sum(K.square(y_true - y_pred))
    ss_tot = K.sum(K.square(y_true - K.mean(y_true)))
    return 1 - ss_res / (ss_tot + K.epsilon())

model.compile(loss='mean_squared_error', optimizer=adam, metrics=['mse', 'mae', r2_keras])

Using the same training and validation data (Xtrain, labels_train; Xtest, labels_test) and the call

history = model.fit(
    Xtrain,
    labels_train,
    validation_data=(Xtest, labels_test),
    epochs=n_epochs,
    batch_size=batch_size,
    verbose=0
)

In Keras 2.13 (graph mode) the validation R2 per epoch is around 0.6, similar to training, but in Keras 3 (eager mode) the validation R2 drops drastically to around -17,000 while training R2 remains the same.

Interestingly, if I set **validation_batch_size=len(Xtest)** in Keras 3, the validation R2 matches the old Keras 2 result.

This suggests that Keras 2 (graph mode) and Keras 3 (eager) handle validation metrics differently: Keras 2 may have effectively computed R2 over the entire validation set per epoch, while Keras 3 computes metrics per batch and averages them, which drastically affects non-linear metrics like R2.

I cannot find any web evidence that explains the difference I am seeing between Keras 2 (graph mode) and Keras 3 (eager mode). Any suggestions on why this behavior occurs would be very much appreciated.

[Sameekshagithub]

This behavior is expected because R2 is not an additive metric and cannot be calculated correctly by averaging batch-wise results.

In Keras 3, metrics are generally updated batch by batch during validation. The custom r2_keras function calculates R2 using only the current batch:

ss_res = sum((y_true - y_pred)^2)
ss_tot = sum((y_true - mean(y_true))^2)

Since R2 depends on the mean and variance of the complete dataset, calculating it separately for each validation batch and then averaging can produce very different values, especially when batch sizes are small.

Setting validation_batch_size=len(Xtest) works because the metric is then calculated over the complete validation set in one batch.

A better solution in Keras 3 is to implement R2 as a custom Metric class that accumulates y_true and y_pred statistics across batches, then calculates the final R2 at epoch end.

For example, maintain:

• sum of y_true
• sum of y_true squared
• sum of residual squared error
• number of samples

Then compute R2 in result().

This will make the metric consistent between eager and graph execution.