2024-02-22 kernel meeting notes

[zachschuermann]

summary

action items

PRs in flight:

• Switch to new data passing API #109
• start implementing FFI #91

attendees

@ryan-johnson-databricks @roeap @zachschuermann @nicklan @hntd187 @vkorukanti @tdas

notes stream

• big rebase in progress for Nick's PR (Switch to new data passing API #109)
• removing a layer of indirection
• pass in schema with primitive leaves -> engine gives back primitives
• trait that the engine has to implement: "you are a list that gives out an indexed item and length" and also gives out a vec[string] etc. (does allocation etc. - does this from the kernel so we know we are allocating it) example: read feature list defined in protocol
• arrow has int32 array, we implement trait that has getters to get primitives from it
• the 'visit' through the data has a tightly-scoped lifetime. if you want the data, gotta make a copy (the ref goes out of scope after the getter)
• nick has docs updates, some evangelization to do!
• with current implementation you can't have more than one type of data that goes between kernel/engine
• expressions evaluate to arrays rather than full schema-ed record batches
• with array + datatype we don't have schema because we need a name. does this matter?
• there are places where we define a schema with a dummy name (e.g. in data skipping)
• data skipping has interesting pattern of going in and out of engine a lot (lots of kernel/engine back and forth)
  • this currently passes arrays and it's fast
  • if we do some 'wrapping' this will add some overhead
• for expressions: we don't need full record batch etc. so robert was just doing simple array
• this is slightly at odds with the new pattern of 'everything is record batch'
• top-level struct fields don't have names but everything else does (structs have names)
• schema vs struct array: struct that makes up the schema has to be non-nullable, need names. datatype -> schema OK if you have names. just have to name the top-level columns
  • could just have some convention for this
  • do we think there will be enough places where we want schema-less arrays where we create some one-off to allow this?
  • JVM kernel solves this by always having kernel provide input/output schemas
  • note we (rust kernel) already have the type, just need the names
• robert: JVM world doesn't have record batch? always deals with struct arrays. seems like this high level concept is omitted?
  • one direction: fold everything into the trait
  • or: expression evaluator mechanics change to extractors with this row-based nature
• ryan: favor the idea of moving to arrays instead of record batches everywhere. concern: one place might get complicated to do struct explode to get columns to visit (because today it is name based). engine data doesnt expose schema. we need it when we pass in schema and say "get me this set of columns". funny we can do 'things' with schema but can't ask for schema. idea: move to arrays.
• nick: issue - when we ask engine to read data and give schema we have to read in the same order. need some sort of convention?
• then we could just do everything index-based?
• other potential: require the top-level to always be a struct. then the struct is corresponding to the schema, and just wraps everything. ArrowSchema is just this - one more struct on top to give everything names. if we do this might solve all problems and we can get rid of record batch.
• record batch likely exists due to historical reasons
• today expressions can be any type (primitive, array, struct)
• we give ArrayRef back now. we turn into array when doing evaluation
• expression handling could always return struct array back? then top-level array really just schema
• ACTION ITEM: return struct array, then rebase nick's PR, then consider remove record batch and only use arrays?
• nick: can't the expression change the data type from the incoming data? we can't use the top-level data that came in, we have to compute the output type. but if we require expressions are struct then yay happy.
• do AI above and this will unblock and we can iterate.
• venki: JVM side has col batch not adding much value. batch has get schema/datatype. struct array would be similar to this.
• arrow has record batch and struct array. almost the same thing but a few subtle differences like nullability.
• ryan - 3 pieces:
  • (1) still doing FFI - trying to show off next week. traits across FFI boundary. minimizing unsafe.
  • (2) errors from kernel to engine
  • (3) how can engine present a trait to rust? vtable stuff
• EngineInterface trait. at the FFI boundary we need more trait methods than the regular kernel trait. In particular: how to convert kernel error into engine error. there will be some new trait which is a superset/wraps normal EngineInterface
• EngineInterface naming concern. once FFI is working possibly revisit names. zach likes Engine
• Steve likes the name ExternalEngineAdapterTraitEngineProviderInterface
• robert's next PR: testing refactoring/factory could help.