Repository navigation
Handle HTTP 413 as a batch-size error on the insertAll path - #256
Open
james-johnston-thumbtack wants to merge 1 commit into
Open
james-johnston-thumbtack wants to merge 1 commit into
james-johnston-thumbtack wants to merge 1 commit into
Conversation
TableWriter halves the batch and retries when BigQuery rejects a streaming
insert for being too large, but it only recognized HTTP 400 responses with a
JSON body. For very large payloads Google's HTTP frontend rejects the request
before it reaches BigQuery, answering with an HTTP 413 and an HTML body. That
arrives as a BigQueryException with code 413 and a null reason and error, so it
matched no predicate, the batch was never split, and the task failed.
Which response comes back is not deterministic: a slow first call draws the 400,
while an immediate repeat of the same 50MB payload is answered quickly with the
413.
% curl -s -i \
-X POST \
-H "Content-Type: application/octet-stream" \
--data-binary @/tmp/payload.bin \
'https://bigquery.googleapis.com/bigquery/v2/projects/myproj/datasets/testds/tables/testtable/insertAll?prettyPrint=false'
HTTP/2 413
content-type: text/html; charset=UTF-8
content-length: 2393
<!DOCTYPE html>
<html lang=en>
<title>Error 413 (Request Entity Too Large)!!1</title>
<p><b>413.</b> <ins>That's an error.</ins>
<p>Your client issued a request that was too large.
<ins>That's all we know.</ins>
isContentTooLargeError matches on the status code alone, since the HTML body
leaves nothing else to go on, and is ORed into isBatchSizeError.
Without this change, the above scenario will cause the task to fail with:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:659)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:360)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:262)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:226)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:243)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:298)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:254)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: A write thread has failed with an unrecoverable error
Caused by: Failed to write to table
at com.wepay.kafka.connect.bigquery.write.batch.KcbqThreadPoolExecutor.lambda$maybeThrowEncounteredError$0(KcbqThreadPoolExecutor.java:111)
at java.base/java.util.Optional.ifPresent(Unknown Source)
at com.wepay.kafka.connect.bigquery.write.batch.KcbqThreadPoolExecutor.maybeThrowEncounteredError(KcbqThreadPoolExecutor.java:110)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.maybeThrowErrors(BigQuerySinkTask.java:557)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.writeSinkRecords(BigQuerySinkTask.java:215)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.put(BigQuerySinkTask.java:285)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:629)
... 11 more
Caused by: com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Failed to write to table
Caused by: 413 Request Entity Too Large
POST https://bigquery.googleapis.com/bigquery/v2/projects/tt-dp-staging/datasets/cdc_pg_customerinfo/tables/multimodal_queries_v1_tmp_0_75540e19_bca6_41b8_8034_665e34643f75_1787095725666/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 413 (Request Entity Too Large)!!1</title>
<snip>
at com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run(TableWriter.java:149)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask$MdcContextThreadFactory.lambda$newThread$0(BigQuerySinkTask.java:671)
... 1 more
Caused by: com.google.cloud.bigquery.BigQueryException: 413 Request Entity Too Large
POST https://bigquery.googleapis.com/bigquery/v2/projects/tt-dp-staging/datasets/cdc_pg_customerinfo/tables/multimodal_queries_v1_tmp_0_75540e19_bca6_41b8_8034_665e34643f75_1787095725666/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 413 (Request Entity Too Large)!!1</title>
<snip>
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:548)
at com.google.cloud.bigquery.BigQueryImpl$28.call(BigQueryImpl.java:1119)
at com.google.cloud.bigquery.BigQueryImpl$28.call(BigQueryImpl.java:1116)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1115)
at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest(AdaptiveBigQueryWriter.java:131)
at com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows(BigQueryWriter.java:150)
at com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run(TableWriter.java:136)
... 4 more
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 413 Request Entity Too Large
POST https://bigquery.googleapis.com/bigquery/v2/projects/tt-dp-staging/datasets/cdc_pg_customerinfo/tables/multimodal_queries_v1_tmp_0_75540e19_bca6_41b8_8034_665e34643f75_1787095725666/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 413 (Request Entity Too Large)!!1</title>
<snip>
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:546)
... 13 more
After making this change, the task halves the batch sizes as expected,
and then runs to completion. With this change, the task logs warnings
as one might expect. We tested this in our staging environment where
version 2.12.0 was consistently crashing with this error. Now, the
staging environment logs a new message at warning level. The connector
does not fail and the backfilling eventually finishes successfully since
batch sizes are being halved. From our logs:
Message: Could not write batch of size 500 to BigQuery.
Error code: 413, underlying error (if present): null
Stack trace:
com.google.cloud.bigquery.BigQueryException: 413 Request Entity Too Large
POST https://bigquery.googleapis.com/bigquery/v2/projects/tt-dp-staging/datasets/cdc_pg_customerinfo/tables/multimodal_queries_v1_tmp_0_01M0X9WZ654SJEHRA4PS71PFZD_1787689860314/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 413 (Request Entity Too Large)!!1</title>
<snip>
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:548)
at com.google.cloud.bigquery.BigQueryImpl$28.call(BigQueryImpl.java:1119)
at com.google.cloud.bigquery.BigQueryImpl$28.call(BigQueryImpl.java:1116)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1115)
at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest(AdaptiveBigQueryWriter.java:134)
at com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows(BigQueryWriter.java:185)
at com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run(TableWriter.java:149)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask$MdcContextThreadFactory.lambda$newThread$0(BigQuerySinkTask.java:699)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 413 Request Entity Too Large
POST https://bigquery.googleapis.com/bigquery/v2/projects/tt-dp-staging/datasets/cdc_pg_customerinfo/tables/multimodal_queries_v1_tmp_0_01M0X9WZ654SJEHRA4PS71PFZD_1787689860314/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 413 (Request Entity Too Large)!!1</title>
<snip>
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:546)
... 13 more
james-johnston-thumbtack
force-pushed
the
handle-http-413-batch-size-error
branch
from
October 7, 2026 23:00
f5031c2 to
9378fc7
Compare
Author
|
Today I had more difficulty reproducing the HTTP 413 errors with the above curl command. It was definitely intermittent before for a couple weeks. Maybe fixed on Google's end for now, or maybe not. I still think the changes in this PR are the right ones to make, as I don't know how there would be any other reasonable interpretation of HTTP 413 than to cut batch sizes. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TableWriter halves the batch and retries when BigQuery rejects a streaming insert for being too large, but it only recognized HTTP 400 responses with a JSON body. For very large payloads Google's HTTP frontend rejects the request before it reaches BigQuery, answering with an HTTP 413 and an HTML body. That arrives as a
BigQueryExceptionwith code 413 and a null reason and error, so it matched no predicate, the batch was never split, and the task failed.Which response comes back is not deterministic: a slow first call draws the 400, while an immediate repeat of the same 50MB payload is answered quickly with the 413.
It is very easy to reproduce this using curl. First, create a
payload.binof 50 megabytes. It must be incompressible (e.g. copy from/dev/random). Next, try posting it.In this PR, the new error classification function
isContentTooLargeErrormatches on the status code alone, since the HTML body leaves nothing else to go on, and is ORed into the existingisBatchSizeErrorimplementation.Without this change, the above scenario will cause the task to fail with:
After making this change, the task halves the batch sizes as expected, and then runs to completion. With this change, the task logs warnings as one might expect. We tested this in our staging environment where version 2.12.0 was consistently crashing with this error. Now, the staging environment logs a new message at warning level. The connector does not fail and the backfilling eventually finishes successfully since batch sizes are being halved. From our logs:
This manual testing in our deployment environment proved to our satisfaction that this PR helps with HTTP 413 errors that were not previously addressed.
Developer Certificate of Origin:
The contribution was created in whole or in part by me and I have the right to submit it under the open source license
indicated in the file