Do not check parquet directory for consistency

Maybe there is a better way to actually do some form of check?
The problem is that if we have multiple nodes/utility workers
then one of them may lag behind when the director already contains
files.
However, blocking on this also seems not better.  Is there a better
way that would allow checking then just an option to disable this
check?

Signed-off-by: Sebastian Berg <[email protected]>

Author: seberg

cpp/include/legate_dataframe/parquet.hpp

@@ -98,6 +98,10 @@ class ParquetReadArray : public Task<ParquetReadArray, OpCode::ParquetReadArray>
  *          ├── part-2.parquet
  *          └── ...
  *
+ * This function may create the directory but does not ensure it is empty.
+ * If a previous write wrote more partitions the old files will remain
+ * leaving the directory in an inconsistent state.
+ *
  * @param tbl The table to write.
  * @param path Destination directory for data.
  */

cpp/src/parquet.cpp

@@ -492,9 +492,7 @@ ParquetReadInfo get_parquet_info(const std::vector<std::string>& file_paths,
 void parquet_write(LogicalTable& tbl, const std::string& dirpath)
 {
   std::filesystem::create_directories(dirpath);
-  if (!std::filesystem::is_empty(dirpath)) {
-    throw std::invalid_argument("if path exist, it must be an empty directory");
-  }
+
   auto runtime = legate::Runtime::get_runtime();
   legate::AutoTask task =
     runtime->create_task(get_library(), task::ParquetWrite::TASK_CONFIG.task_id());

python/legate_dataframe/lib/parquet.pyx

@@ -68,6 +68,11 @@ def parquet_write(LogicalTable tbl, path: pathlib.Path | str) -> None:
             ├── part.2.parquet
             └── ...
 
+    .. note::
+        This function will create the directory but does not ensure it is empty.
+        If a previous write had more partitions the old files will remain
+        leaving the directory in an inconsistent state.
+
     See Also
     --------
     parquet_read: Read parquet data