restcatalog::namespace_exists always fails

[gfee-home]

Apache Iceberg Rust version

0.4.0 (latest version)

Describe the bug

Calling this method always generates an error with content:

Unexpected, context: { code: 200 OK, method: HEAD, url: http://localhost:8181/v1/namespaces/default, json: } => Failed to parse response from rest catalog server!, source: EOF while parsing a value at line 1 column 0

To Reproduce

Repro:

1. Bring up minio and rest catalog using your test harness configuration
2. Run this:

const REST_CATALOG_IP: &str = "localhost";
const REST_CATALOG_PORT: i16 = 8181;
const S3_ENDPOINT: &str = "localhost:9000";
const MINIO_IP: &str = "localhost";
const S3_ACCESS_KEY_ID: &str = "admin";
const S3_SECRET_ACCESS_KEY: &str = "password";
const S3_REGION: &str = "us-east-1";

async fn repro() -> () {
let config = RestCatalogConfig::builder()
.uri(format!("http://{}:{}", REST_CATALOG_IP, REST_CATALOG_PORT))
.props(HashMap::from([
(
S3_ENDPOINT.to_string(),
format!("http://{}:{}", MINIO_IP, 9000),
),
(S3_ACCESS_KEY_ID.to_string(), "admin".to_string()),
(S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()),
(S3_REGION.to_string(), "us-east-1".to_string()),
]))
.build();

let catalog = RestCatalog::new(config);
let namespace_ident = NamespaceIdent::new("does_not_matter".to_string());

catalog.namespace_exists(&namespace_ident).await.unwrap();

}

Expected: runs without error
Actual: panics

Expected behavior

This method should return an Ok(false) for the repro above. It should return an Ok(true) if you add a line to create the namespace first.

Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community

[Fokko]

@gfee-home Thanks for raising this.

This is actually a bug in the catalog, as it should return 204 instead of 200:

https://github.com/apache/iceberg/blob/321e66ffc97349f7221096e15129a7eddb5ab81d/open-api/rest-catalog-open-api.yaml#L380-L381

However, I think we can adapt the Rust code to also accept 200. Are you interested in providing a PR?

[dentiny]

I seems to suffer the same issue, so temporarily I workaround with list + contains:

let namespaces = catalog.list_namespaces(None).await?;
let namespace_exists = namespaces.contains(&namespace_ident);

[kevinjqliu]

Similar to apache/iceberg-python#1363

[Xuanwo]

Let me get this fixed. The specs requires 204, but the real world disagree with that. Let's just be more open to the input.

[Xuanwo]

After deeply investigating this issue, I've found that the real problem isn't related to how we handle the status code.

We already allow both 204 and 200:

iceberg-rust/crates/catalog/rest/src/catalog.rs

Line 449 in 5677d44

StatusCode::NO_CONTENT | StatusCode::OK => Ok(true),

The real problem here is that HEAD request won't return a http body, but we still trying to decode it into json:

iceberg-rust/crates/catalog/rest/src/client.rs

Lines 252 to 267 in 5677d44

	/// Deserializes a unexpected catalog response into an error.
	///
	/// TODO: Eventually, this function should return an error response that is custom to the error
	/// codes that all endpoints share (400, 404, etc.).
	pub(crate) async fn deserialize_unexpected_catalog_error(response: Response) -> Error {
	let (status, headers) = (response.status(), response.headers().clone());
	let bytes = match response.bytes().await {
	Ok(bytes) => bytes,
	Err(err) => return err.into(),
	};
	Error::new(ErrorKind::Unexpected, "Received unexpected response")
	.with_context("status", status.to_string())
	.with_context("headers", format!("{:?}", headers))
	.with_context("json", String::from_utf8_lossy(&bytes))
	}

So it's in fact a bug that need to fix instead of just a workaround.

[Xuanwo]

Should be fixed by #1266

[gfee-home]

thank you!