FEAT-865: LAZ writer (#104)

* Fix precommit bug with black.

* Fix precommit unauthenticated git protocol error.

* Rename elements to prepare for LAZ writer.

* Add scaffolding for LAZ writer.

* Update LAZ writer init function.

* Implement LazConfig and LazConfigWriter.

* Fixing LazConfig tests.

* Start to implement BaseWriter class.

* Finish implementation of LAZ base writer.

* Prototype LAZ writer.

* Finish and test LAZ writer.

* Hide base writer header file.

* Implement LasHeader UpdateBounds.

* Typo.

* Rename LasHeader::UpdateBounds to CheckAndUpdateBounds and small clean-ups.

* Revert "Rename LasHeader::UpdateBounds to CheckAndUpdateBounds and small clean-ups."

This reverts commit a97719c.

* Fix bug with laz writer VLRs.

* Refactor OffsetToPointData.

* Fix OffsetToPointData.

* Implement pointer casting between laz and copc config classes to avoid code duplication.

* Small changes from PR review.

* Clean-up.

* Refactor file writing into a base class.

* Refactor WriteChunk into BaseWriter to remove code duplication.

* Clean-up.

* Clean-up.

* Clean-up.

* Miscellaneous improvements and clean-up.

* Miscellaneous improvements and clean-up.

* Fix seekg types.

* Rename CheckAndUpdateBounds and add SetGpsTimeBit.

* Refactor header space fill to BaseWriter and rename LasHeader::SIZE_BYTES to LasHeader::HEADER_SIZE_BYTES.

* Fix github test runs.

* Update LasHeader GpsTime type bit tests.

* Revert "Update LasHeader GpsTime type bit tests."

This reverts commit 187dbb1.

* Revert "Refactor header space fill to BaseWriter and rename LasHeader::SIZE_BYTES to LasHeader::HEADER_SIZE_BYTES."

This reverts commit e843c4d.

* Revert "Rename CheckAndUpdateBounds and add SetGpsTimeBit."

This reverts commit 18ec87b.

* Revert "Revert "Refactor header space fill to BaseWriter and rename LasHeader::SIZE_BYTES to LasHeader::HEADER_SIZE_BYTES.""

This reverts commit d404323.

* Fix refactoring of std::fill_n.

* Revert "Revert "Rename CheckAndUpdateBounds and add SetGpsTimeBit.""

This reverts commit 7f8309c.

* Fix tests after CheckAndUpdateBounds name change.

* Revert "Revert "Update LasHeader GpsTime type bit tests.""

This reverts commit 94fc920.

* Fix return type of LasHeader::ApplyInverseScale.

* Fix return type of LasHeader::ApplyInverseScale.

* Fix bug in filepath in writer_test.py.

* Remove remaining static_cast<long>.

* Revert fill_n changes.

* Change type of LasHeader::RemoveScale from int64 to in32.

* Fix IsCopc typos.

Co-authored-by: Christopher Lee <[email protected]>

Author: leo-stan

CHANGELOG.md

@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- **\[C++\]** Add `LazConfig` and `LazConfigWriter` classes as parent class to `CopcConfig` to handle LAZ files.
+- **\[C++\]** Add `LazWriter` and `LazFileWriter` classes to write LAZ files.
+- **\[C++\]** Add `example-laz-writer.cpp` to show how to use the LAZ writer.
+- **\[Python/C++\]** Add `CheckAndUpdateBounds` to `LasHeader` to check values in a `Point` and update the bounds if necessary.
+- **\[Python/C++\]** Add `SetGpsTimeBit` to `LasHeader` to easily set the bit value to 1.
+
+### Changed
+- **\[Python/C++\]** Rename `LasHeader::ApplyInverseScale` to `LasHeader::RemoveScale` to be consistent with `Point` class.
+
+### Fixed
+- **\[Python/C++\]** Change return type of `LasHeader::ApplyInverseScale` from `double` to `int32_t`.
+
 ## [2.2.3] - 2022-02-02
 
 ### Fixed

README.md

@@ -82,7 +82,7 @@ add_executable(funlib fun-main.cpp)
 target_link_libraries(funlib COPCLIB::copc-lib LAZPERF::lazperf)
 ```
 
-The primary public interface will be your [FileReader](./cpp/include/copc-lib/io/reader.hpp) and [FileWriter](./cpp/include/copc-lib/io/writer.hpp) objects. Check the headers and [example files](./example) for documentation.
+The primary public interface will be your [FileReader](./cpp/include/copc-lib/io/copc_reader.hpp) and [FileWriter](./cpp/include/copc-lib/io/copc_writer.hpp) objects. Check the headers and [example files](./example) for documentation.
 
 ```cpp
 #include <iostream>

cpp/CMakeLists.txt

@@ -4,21 +4,24 @@ set(LIBRARY_TARGET_NAME copc-lib)
 set(${LIBRARY_TARGET_NAME}_HDR
         include/${LIBRARY_TARGET_NAME}/copc/info.hpp
         include/${LIBRARY_TARGET_NAME}/copc/extents.hpp
-        include/${LIBRARY_TARGET_NAME}/copc/config.hpp
+        include/${LIBRARY_TARGET_NAME}/copc/copc_config.hpp
         include/${LIBRARY_TARGET_NAME}/geometry/box.hpp
         include/${LIBRARY_TARGET_NAME}/geometry/vector3.hpp
         include/${LIBRARY_TARGET_NAME}/hierarchy/entry.hpp
         include/${LIBRARY_TARGET_NAME}/hierarchy/key.hpp
         include/${LIBRARY_TARGET_NAME}/hierarchy/node.hpp
         include/${LIBRARY_TARGET_NAME}/hierarchy/page.hpp
-        include/${LIBRARY_TARGET_NAME}/io/base_io.hpp
-        include/${LIBRARY_TARGET_NAME}/io/reader.hpp
-        include/${LIBRARY_TARGET_NAME}/io/writer.hpp
+        include/${LIBRARY_TARGET_NAME}/io/copc_base_io.hpp
+        include/${LIBRARY_TARGET_NAME}/io/copc_reader.hpp
+        include/${LIBRARY_TARGET_NAME}/io/copc_writer.hpp
+        include/${LIBRARY_TARGET_NAME}/io/laz_writer.hpp
         include/${LIBRARY_TARGET_NAME}/las/header.hpp
+        include/${LIBRARY_TARGET_NAME}/io/laz_base_writer.hpp
         include/${LIBRARY_TARGET_NAME}/las/point.hpp
         include/${LIBRARY_TARGET_NAME}/las/points.hpp
         include/${LIBRARY_TARGET_NAME}/las/utils.hpp
         include/${LIBRARY_TARGET_NAME}/las/vlr.hpp
+        include/${LIBRARY_TARGET_NAME}/las/laz_config.hpp
         include/${LIBRARY_TARGET_NAME}/laz/compressor.hpp
         include/${LIBRARY_TARGET_NAME}/laz/decompressor.hpp
 )
@@ -27,22 +30,25 @@ set(${LIBRARY_TARGET_NAME}_HDR
 set(${LIBRARY_TARGET_NAME}_SRC
         include/${LIBRARY_TARGET_NAME}/hierarchy/internal/page.hpp
         include/${LIBRARY_TARGET_NAME}/hierarchy/internal/hierarchy.hpp
-        include/${LIBRARY_TARGET_NAME}/io/internal/writer_internal.hpp
+        include/${LIBRARY_TARGET_NAME}/io/internal/copc_writer_internal.hpp
         src/copc/info.cpp
         src/copc/extents.cpp
-        src/copc/config.cpp
+        src/copc/copc_config.cpp
         src/geometry/box.cpp
         src/hierarchy/key.cpp
         src/hierarchy/page.cpp
-        src/io/base_io.cpp
-        src/io/reader.cpp
-        src/io/writer_internal.cpp
-        src/io/writer_public.cpp
+        src/io/copc_base_io.cpp
+        src/io/copc_reader.cpp
+        src/io/copc_writer_internal.cpp
+        src/io/copc_writer_public.cpp
+        src/io/laz_base_writer.cpp
+        src/io/laz_writer.cpp
         src/las/header.cpp
         src/las/point.cpp
         src/las/points.cpp
         src/las/utils.cpp
         src/las/vlr.cpp
+        src/las/laz_config.cpp
 )
 
 # Compile static library for pip wheels

cpp/include/copc-lib/copc/config.hpp renamed to cpp/include/copc-lib/copc/copc_config.hpp

@@ -7,42 +7,32 @@
 #include "copc-lib/copc/extents.hpp"
 #include "copc-lib/copc/info.hpp"
 #include "copc-lib/las/header.hpp"
+#include "copc-lib/las/laz_config.hpp"
 #include "copc-lib/las/utils.hpp"
 #include "copc-lib/las/vlr.hpp"
 
 namespace copc
 {
-const int COPC_OFFSET = 429;
 
-class CopcConfig
+class CopcConfig : public las::LazConfig
 {
   public:
     CopcConfig() = default;
     CopcConfig(const las::LasHeader &header, const CopcInfo &copc_info, const CopcExtents &copc_extents,
                const std::string &wkt, const las::EbVlr &extra_bytes_vlr)
-        : header_(std::make_shared<las::LasHeader>(header)), copc_info_(std::make_shared<copc::CopcInfo>(copc_info)),
-          copc_extents_(std::make_shared<copc::CopcExtents>(copc_extents)), wkt_(wkt),
-          eb_vlr_(std::make_shared<las::EbVlr>(extra_bytes_vlr)){};
-
-    virtual las::LasHeader LasHeader() const { return *header_; }
+        : LazConfig(header, wkt, extra_bytes_vlr), copc_info_(std::make_shared<copc::CopcInfo>(copc_info)),
+          copc_extents_(std::make_shared<copc::CopcExtents>(copc_extents)){};
 
     virtual copc::CopcInfo CopcInfo() const { return *copc_info_; }
 
     virtual copc::CopcExtents CopcExtents() const { return *copc_extents_; }
 
-    std::string Wkt() const { return wkt_; }
-
-    las::EbVlr ExtraBytesVlr() const { return *eb_vlr_; }
-
   protected:
     CopcConfig(const int8_t &point_format_id, const Vector3 &scale, const Vector3 &offset, const std::string &wkt,
                const las::EbVlr &extra_bytes_vlr, bool has_extended_stats);
 
-    std::shared_ptr<las::LasHeader> header_;
     std::shared_ptr<copc::CopcInfo> copc_info_;
     std::shared_ptr<copc::CopcExtents> copc_extents_;
-    std::string wkt_;
-    std::shared_ptr<las::EbVlr> eb_vlr_;
 };
 
 class CopcConfigWriter : public CopcConfig

cpp/include/copc-lib/copc/extents.hpp

@@ -2,6 +2,7 @@
 #define COPCLIB_COPC_EXTENTS_H_
 
 #include <string>
+#include <utility>
 #include <vector>
 
 #include <lazperf/vlr.hpp>
@@ -129,43 +130,49 @@ class CopcExtents
     }
 
     // Setters for python bindings
-    void Intensity(std::shared_ptr<CopcExtent> intensity) { extents_[0] = intensity; }
-    void ReturnNumber(std::shared_ptr<CopcExtent> return_number) { extents_[1] = return_number; }
-    void NumberOfReturns(std::shared_ptr<CopcExtent> number_of_returns) { extents_[2] = number_of_returns; }
-    void ScannerChannel(std::shared_ptr<CopcExtent> scanner_channel) { extents_[3] = scanner_channel; }
-    void ScanDirectionFlag(std::shared_ptr<CopcExtent> scan_direction_flag) { extents_[4] = scan_direction_flag; }
-    void EdgeOfFlightLine(std::shared_ptr<CopcExtent> edge_of_flight_line) { extents_[5] = edge_of_flight_line; }
-    void Classification(std::shared_ptr<CopcExtent> classification) { extents_[6] = classification; }
-    void UserData(std::shared_ptr<CopcExtent> user_data) { extents_[7] = user_data; }
-    void ScanAngle(std::shared_ptr<CopcExtent> scan_angle) { extents_[8] = scan_angle; }
-    void PointSourceId(std::shared_ptr<CopcExtent> point_source_id) { extents_[9] = point_source_id; }
-    void GpsTime(std::shared_ptr<CopcExtent> gps_time) { extents_[10] = gps_time; }
+    void Intensity(std::shared_ptr<CopcExtent> intensity) { extents_[0] = std::move(intensity); }
+    void ReturnNumber(std::shared_ptr<CopcExtent> return_number) { extents_[1] = std::move(return_number); }
+    void NumberOfReturns(std::shared_ptr<CopcExtent> number_of_returns) { extents_[2] = std::move(number_of_returns); }
+    void ScannerChannel(std::shared_ptr<CopcExtent> scanner_channel) { extents_[3] = std::move(scanner_channel); }
+    void ScanDirectionFlag(std::shared_ptr<CopcExtent> scan_direction_flag)
+    {
+        extents_[4] = std::move(scan_direction_flag);
+    }
+    void EdgeOfFlightLine(std::shared_ptr<CopcExtent> edge_of_flight_line)
+    {
+        extents_[5] = std::move(edge_of_flight_line);
+    }
+    void Classification(std::shared_ptr<CopcExtent> classification) { extents_[6] = std::move(classification); }
+    void UserData(std::shared_ptr<CopcExtent> user_data) { extents_[7] = std::move(user_data); }
+    void ScanAngle(std::shared_ptr<CopcExtent> scan_angle) { extents_[8] = std::move(scan_angle); }
+    void PointSourceId(std::shared_ptr<CopcExtent> point_source_id) { extents_[9] = std::move(point_source_id); }
+    void GpsTime(std::shared_ptr<CopcExtent> gps_time) { extents_[10] = std::move(gps_time); }
     void Red(std::shared_ptr<CopcExtent> red)
     {
         if (point_format_id_ > 6)
-            extents_[1] = red;
+            extents_[1] = std::move(red);
         else
             throw std::runtime_error("CopcExtents::Red: This point format does not have Red");
     }
     void Green(std::shared_ptr<CopcExtent> green)
     {
         if (point_format_id_ > 6)
-            extents_[12] = green;
+            extents_[12] = std::move(green);
         else
             throw std::runtime_error("CopcExtents::Green: This point format does not have Green");
     }
     void Blue(std::shared_ptr<CopcExtent> blue)
     {
         if (point_format_id_ > 6)
-            extents_[13] = blue;
+            extents_[13] = std::move(blue);
         else
             throw std::runtime_error("CopcExtents::Blue: This point format does not have Blue");
     }
 
     void Nir(std::shared_ptr<CopcExtent> nir)
     {
         if (point_format_id_ == 8)
-            extents_[14] = nir;
+            extents_[14] = std::move(nir);
         else
             throw std::runtime_error("CopcExtents::NIR: This point format does not have NIR");
     }

cpp/include/copc-lib/copc/info.hpp

@@ -11,6 +11,9 @@ namespace copc
 class CopcInfo
 {
   public:
+    static const int VLR_OFFSET = 375;     // COPC Info VLR must be the first VLR after the LAS header block
+    static const int VLR_SIZE_BYTES = 160; // COPC Info VLR payload is 160 bytes, https://copc.io/
+
     CopcInfo() = default;
 
     CopcInfo(const lazperf::copc_info_vlr &copc_info_vlr);

cpp/include/copc-lib/hierarchy/entry.hpp

@@ -17,7 +17,7 @@ class Entry
     static const int ENTRY_SIZE = 32;
 
     Entry() : offset(-1), byte_size(-1), key(VoxelKey::InvalidKey()), point_count(-1){};
-    Entry(VoxelKey key, int64_t offset, int32_t size, int32_t point_count)
+    Entry(VoxelKey key, uint64_t offset, int32_t size, int32_t point_count)
         : offset(offset), byte_size(size), key(key), point_count(point_count){};
 
     virtual bool IsValid() const { return offset >= 0 && byte_size >= 0 && key.IsValid(); }

cpp/include/copc-lib/hierarchy/internal/page.hpp

@@ -3,6 +3,7 @@
 
 #include <set>
 #include <unordered_map>
+#include <utility>
 
 #include "copc-lib/hierarchy/page.hpp"
 
@@ -13,7 +14,7 @@ class PageInternal;
 class PageInternal : public Page
 {
   public:
-    PageInternal(Entry e) : Page(e){};
+    PageInternal(Entry e) : Page(std::move(e)){};
     PageInternal(VoxelKey key, int64_t offset, int32_t byte_size) : Page(key, offset, byte_size){};
     PageInternal(VoxelKey key) : Page(key, -1, -1){};
 

cpp/include/copc-lib/hierarchy/page.hpp

@@ -3,6 +3,7 @@
 
 #include <istream>
 #include <memory>
+#include <utility>
 
 #include "copc-lib/hierarchy/entry.hpp"
 #include "copc-lib/hierarchy/node.hpp"
@@ -13,7 +14,7 @@ class Page;
 class Page : public Entry
 {
   public:
-    Page(Entry e) : Entry(e){};
+    Page(Entry e) : Entry(std::move(e)){};
     Page(VoxelKey key, int64_t offset, int32_t byte_size) : Entry(key, offset, byte_size, -1){};
 
     // If a page is "loaded" it doesn't matter the offset/size

cpp/include/copc-lib/io/base_io.hpp renamed to cpp/include/copc-lib/io/copc_base_io.hpp

@@ -1,7 +1,7 @@
-#ifndef COPCLIB_IO_BASE_H_
-#define COPCLIB_IO_BASE_H_
+#ifndef COPCLIB_IO_COPC_BASE_IO_H_
+#define COPCLIB_IO_COPC_BASE_IO_H_
 
-#include "copc-lib/copc/config.hpp"
+#include "copc-lib/copc/copc_config.hpp"
 #include "copc-lib/copc/extents.hpp"
 #include "copc-lib/copc/info.hpp"
 #include "copc-lib/hierarchy/node.hpp"
@@ -34,4 +34,4 @@ class BaseIO
 };
 
 } // namespace copc
-#endif // COPCLIB_IO_BASE_H_
+#endif // COPCLIB_IO_COPC_BASE_IO_H_

cpp/include/copc-lib/io/reader.hpp renamed to cpp/include/copc-lib/io/copc_reader.hpp

@@ -1,14 +1,14 @@
-#ifndef COPCLIB_IO_READER_H_
-#define COPCLIB_IO_READER_H_
+#ifndef COPCLIB_IO_COPC_READER_H_
+#define COPCLIB_IO_COPC_READER_H_
 
 #include <istream>
 #include <limits>
 #include <map>
 #include <string>
 
-#include "copc-lib/copc/config.hpp"
+#include "copc-lib/copc/copc_config.hpp"
 #include "copc-lib/hierarchy/key.hpp"
-#include "copc-lib/io/base_io.hpp"
+#include "copc-lib/io/copc_base_io.hpp"
 #include "copc-lib/las/points.hpp"
 #include "copc-lib/las/vlr.hpp"
 
@@ -84,7 +84,7 @@ class Reader : public BaseIO
     static uint64_t FetchVlr(const std::map<uint64_t, las::VlrHeader> &vlrs, const std::string &user_id,
                              uint16_t record_id);
     // Finds and loads the COPC vlr
-    CopcInfo ReadCopcInfoVlr();
+    CopcInfo ReadCopcInfoVlr(std::map<uint64_t, las::VlrHeader> &vlrs);
     // Finds and loads the COPC vlr
     CopcExtents ReadCopcExtentsVlr(std::map<uint64_t, las::VlrHeader> &vlrs, const las::EbVlr &eb_vlr) const;
     // Finds and loads the WKT vlr
@@ -126,4 +126,4 @@ class FileReader : public Reader
 };
 
 } // namespace copc
-#endif // COPCLIB_IO_READER_H_
+#endif // COPCLIB_IO_COPC_READER_H_