Reading fsst arrays

[duarten]

Hi,

I'm struggling a bit to read a FSST-encoded column. Once I get the ArrayRef from scan(), I can't seem to get the corresponding FSSTArray?

My goal would be to get the FSSTArray to then use the compressor to encode a string, and then match the result against the compressed codes to locate matches of the string in the array.

[gatesn]

So we actually already implement this short-circuit comparison logic here:

vortex/encodings/fsst/src/compute/compare.rs

Lines 47 to 121 in 9a76489

	fn compare_fsst_constant(
	left: &FSSTArray,
	right: &Scalar,
	operator: Operator,
	) -> VortexResult<Option<ArrayRef>> {
	let is_rhs_empty = match right.dtype() {
	DType::Binary(_) => right
	.as_binary()
	.is_empty()
	.vortex_expect("RHS should not be null"),
	DType::Utf8(_) => right
	.as_utf8()
	.is_empty()
	.vortex_expect("RHS should not be null"),
	_ => vortex_bail!("VarBinArray can only have type of Binary or Utf8"),
	};
	if is_rhs_empty {
	let buffer = match operator {
	// Every possible value is gte ""
	Operator::Gte => BitBuffer::new_set(left.len()),
	// No value is lt ""
	Operator::Lt => BitBuffer::new_unset(left.len()),
	_ => {
	let uncompressed_lengths = left.uncompressed_lengths().to_primitive();
	match_each_integer_ptype!(uncompressed_lengths.ptype(), |P| {
	compare_lengths_to_empty(
	uncompressed_lengths.as_slice::<P>().iter().copied(),
	operator,
	)
	})
	}
	};
	return Ok(Some(
	BoolArray::from_bit_buffer(
	buffer,
	Validity::copy_from_array(left.as_ref())
	.union_nullability(right.dtype().nullability()),
	)
	.into_array(),
	));
	}
	// The following section only supports Eq/NotEq
	if !matches!(operator, Operator::Eq | Operator::NotEq) {
	return Ok(None);
	}
	let compressor = left.compressor();
	let encoded_buffer = match left.dtype() {
	DType::Utf8(_) => {
	let value = right
	.as_utf8()
	.value()
	.vortex_expect("Expected non-null scalar");
	ByteBuffer::from(compressor.compress(value.as_bytes()))
	}
	DType::Binary(_) => {
	let value = right
	.as_binary()
	.value()
	.vortex_expect("Expected non-null scalar");
	ByteBuffer::from(compressor.compress(value.as_slice()))
	}
	_ => unreachable!("FSSTArray can only have string or binary data type"),
	};
	let encoded_scalar = Scalar::new(
	DType::Binary(left.dtype().nullability() | right.dtype().nullability()),
	encoded_buffer.into(),
	);
	let rhs = ConstantArray::new(encoded_scalar, left.len());
	compare(left.codes().as_ref(), rhs.as_ref(), operator).map(Some)
	}

Therefore you just need to use the compare compute function over the resulting array.

[duarten]

No no, I meant getting the compressor associated with an FSSTArray. Just like in compare(), where you do let compressor = left.compressor();.

[duarten]

Here: https://github.com/vortex-data/vortex/blob/develop/encodings/fsst/src/array.rs#L326

[connortsui20]

So the reason it isn't public is the same as the decompressor method above, I'll get back to you by tomorrow and let you know if we can relax that constraint.

[connortsui20]

Fixed in #5752!

[duarten]

Awesome, thank you! :D

[connortsui20]

See #5709 (reply in thread) and #5709 (reply in thread) for answers