Merlin: HugeCTR 23.05.01

What's New in Version 23.05

In this release, we have fixed issues and enhanced the code.

• 3G Embedding Updates:

  • Refactored the DataDistributor related code
  • New SOK load() and dump() APIs are usable in TensorFlow 2. To use the API, specify sok_vars in addition to path.
  • sok_vars is a list of sok.variable and/or sok.dynamic_variable.
  • If you want to store optimizer states such as m and v of Adam, the optimizer must be specified as well.
  • The optimizer must be a tf.keras.optimizers.Optimizer or sok.OptimizerWrapper while their underlying type must be SGD, Adamax, Adadelta, Adagrad, or Ftrl.

  import sparse_operation_kit as sok
  
  sok.load(path, sok_vars, optimizer=None)
  
  sok.dump(path, sok_vars, optimizer=None)

  These APIs are independent from the number of GPUs in use and the sharding strategy. For instance, a distributed embedding table trained and dumped with 8 GPUs can be loaded to train on a 4-GPU machine.

• Issues Fixed:

  • Fixed the segmentation fault and wrong initialization when the embedding table fusion is enabled in using the HPS UVM implementation
  • cudaDeviceSynchronize() is removed when building the HugeCTR in the debug mode, so you can enable the CUDA Graph even in the debug mode.
  • Modified some Notebooks to use the most recent version of NGC container
  • Fixed the EmbeddingTableCollection utest to run correctly with multiple GPUs

• Known Issues:

  • HugeCTR can lead to a runtime error if client code calls RMM’s rmm::mr::set_current_device_resource() or rmm::mr::set_current_device_resource() because HugeCTR’s Parquet Data Reader also calls rmm::mr::set_current_device_resource(), and it becomes visible to other libraries in the same process. Refer to [this issue] (#356) . As a workaround, set an environment variable HCTR_RMM_SETTABLE to 0 to disable HugeCTR to set a custom RMM device resource, if they know rmm::mr::set_current_device_resource() is called outside HugeCTR. But be cautious, as it could affect the performance of parquet reading.

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also this NCCL known issue and this GitHub issue.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka,make sure that a sufficient number of Kafka brokers are running, operating properly, and reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers. Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR 23.04

What's New in Version 23.04

• Hierarchical Parameter Server Enhancements:

  • HPS Table Fusion: From this release, you can fuse tables of the same embedding vector size in HPS. We support this feature in the HPS plugin for TensorFlow and the Triton backend for HPS.. To turn on table fusion, set fuse_embedding_table to true in the HPS JSON file. This feature requires that the key values in different tables do not overlap and the embedding lookup layers are not dependent on each other in the model graph. For more information, refer to HPS configuration and HPS table fusion demo notebook. This feature can reduce the embedding lookup latency significantly when there are multiple tables and GPU embedding cache is employed. About 3x speedup is achieved on V100 for the fused case demonstrated in the notebook compared to the unfused one.

  • UVM Support: We have upgraded the static embedding solution. For embedding tables whose size exceeds the device memory, we will save high-frequency embeddings in the HBM as an embedding cache and offload the remaining embeddings to the UVM. Compared with the dynamic cache solution that offloads the remaining embeddings to the Volatile DB, the UVM solution has higher CPU lookup throughput. We will support online updating of the UVM solution in a future release. Users can switch between different embedding cache solutions through the embedding_cache_type configuration parameter.

  • Triton Perf Analayzer’s Request Generator: We have added an inference request generator to generate the JSON request format required by Triton Perf Analyzer. By using this request generator together with the model generator, you can use the Triton Perf Analyzer to profile the HPS performance and do stress testing. For API documentation and demo usage, please refer to README

• General Updates:

  • DenseLayerComputeConfig: MLP and CrossLayer support asynchronous weight gradient computations with data gradient backpropagation when training. We have added a new member hugectr DenseLayerComputeConfig to hugectr.DenseLayer for configuring the computing behavior. The knob for enabling asynchronous weight gradient computations has been moved from hugectr.CreateSolver to hugectr.DenseLayerComputeConfig.async_wgrad. The knob for controlling the fusion mode of weight gradients and bias gradients has been moved from hugectr.DenseLayerSwitchs to hugectr.DenseLayerComputeConfig.fuse_wb.
  • Hopper Architecture Support: Users can build HugeCTR from scratch with the compute capability 9.0 (DSM=90), so that it can run on Hopper architectures. Note that our NGC container does not support the compute capability yet. Users who are unfamiliar with how to build HugeCTR can refer to the HugeCTR Contribution Guide.
  • RoCE Support for Hybrid Embedding: With the parameter CommunicationType.IB_NVLink_Hier in HybridEmbeddingParams, the RoCE is supported. We have also added 2 environment variables HUGECTR_ROCE_GID and HUGECTR_ROCE_TC so that a user can control the RoCE NIC's GID and traffic class.
    https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#hybridembeddingparam-class

• Documentation Updates:

  • Data Reader: We have enhanced our Raw data reader to read multi-hot input data, connecting with an embedding collection seamlessly. The raw dataset format is strengthened as well. Refer to our online documentation for more details. We have refined the description for Norm datasest as well.
  • Embedding Collection: We have added the knob is_exclusive_keys to enable potencial acceleration if a user has already preprocessed the input of embedding collection to make the resulting tables exclusive with one another. We have also added the nob comm_strategy in embedding collection for user to configure optimized communication strategy in multi-node training
  • HPS Plugin: We have fixed the unit of measurement for DLRM inference benchmark results that leverage the HPS plugin. We have updated the user guide for the HPS plugin for TensorFlow and the HPS plugin for TensorRT
  • Embedding Cache: We have updated the usage of three types of embedding cache. We have updated the descriptions of the three types of embedding cache as well.

• Issues Fixed:

  • We added a slots emptiness check to prevent SparseParam from being misused.
  • We revised MPI lifetime service to become MPI init service with slightly greater scope and clearer interface. In this effort, we also fixed a rare bug that could lead access violations during the MPI shutdown procedure.
  • We fixed a segment fault that occurs when a GPU has no embedding wgrad to update.
  • SOK build & runtime error related to TF version: We made the SOK Experiment](https://github.com/NVIDIA-Merlin/HugeCTR/tree/main/sparse_operation_kit/experiment) compatible with the Tensorflow >= v2.11.0. The legacy SOK doesn’t support that and newer versions of Tensorflow.
  • HPS requires CPU memory to be at least 2.5x larger than the model size during its initialization. From this release, we parse the model embedding files through chunks and reduce the required memory to 1.3x model size.

• Known Issues:

  • HugeCTR can lead to a runtime error if client code calls RMM’s rmm::mr::set_current_device_resource() or rmm::mr::set_current_device_resource() because HugeCTR’s Parquet Data Reader also calls rmm::mr::set_current_device_resource(), and it becomes visible to other libraries in the same process. Refer to [this issue] (#356) . As a workaround, a user can set an environment variable HCTR_RMM_SETTABLE to 0 to disable HugeCTR to set a custom RMM device resource, if they know rmm::mr::set_current_device_resource() is called outside HugeCTR. But be cautious, as it could affect the performance of parquet reading.

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also this NCCL known issue and this GitHub issue](#243).

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR 23.02

What's New in Version 23.02

• HPS Enhancements:

  • Enabled the HPS Tensorflow plugin.
  • Enabled the max_norm clipping for the HPS Tensorflow plugin.
  • Optimized the performance of HPS HashMap fetch.
  • Enabled the HPS Profiler.

• Google Cloud Storage (GCS) Support:

  • Added the support of Google Cloud Storage(GCS) for both training and inference. For more details, check out the GCS section in the training with remote filesystem notebook.

• Issues Fixed:

  • Fixed a bug in HPS static table, which leads to a wrong results when the batch size is larger than 256.
  • Fixed a preprocessing issue in the wdl_prediction notebook.
  • Corrected how devices are set and managed in HPS and InferenceModel.
  • Fixed the debug build error.
  • Fixed the build error related with the CUDA 12.0.
  • Fixed reported issues with respect to Multi-Process HashMap in notebook and a couple of minor issues on the side.

• Known Issues:

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR V4.3.1 (Merlin 22.12.1)

What's New in Version 4.3

In January 2023, the HugeCTR team plans to deprecate semantic versioning, such as `v4.3`.
Afterward, the library will use calendar versioning only, such as `v23.01`.

• Support for BERT and Variants:
  This release includes support for BERT in HugeCTR.
  The documentation includes updates to the MultiHeadAttention layer and adds documentation for the SequenceMask layer.
  For more information, refer to the samples/bst directory of the repository in GitHub.

• HPS Plugin for TensorFlow integration with TensorFlow-TensorRT (TF-TRT):
  This release includes plugin support for integration with TensorFlow-TensorRT.
  For sample code, refer to the Deploy SavedModel using HPS with Triton TensorFlow Backend notebook.

• Deep & Cross Network Layer version 2 Support:
  This release includes support for Deep & Cross Network version 2.
  For conceptual information, refer to https://arxiv.org/abs/2008.13535.
  The documentation for the MultiCross Layer is updated.

• Enhancements to Hierarchical Parameter Server:

  • RedisClusterBackend now supports TLS/SSL communication.
    For sample code, refer to the Hierarchical Parameter Server Demo notebook.
    The notebook is updated with step-by-step instructions to show you how to setup HPS to use Redis with (and without) encryption.
    The Volatile Database Parameters documentation for HPS is updated with the enable_tls, tls_ca_certificate, tls_client_certificate, tls_client_key, and tls_server_name_identification parameters.
  • MultiProcessHashMapBackend includes a bug fix that prevented configuring the shared memory size when using JSON file-based configuration.
  • On-device input keys are supported now so that an extra host-to-device copy is removed to improve performance.
  • A dependency on the XX-Hash library is removed.
    The library is no longer used by HugeCTR.
  • Added the static table support to the embedding cache.
    The static table is suitable when the embedding table can be placed entirely in GPU memory.
    In this case, the static table is more than three times faster than the embedding cache lookup.
    The static table does not support embedding updates.

• Support for New Optimizers:

  • Added support for SGD, Momentum SGD, Nesterov Momentum, AdaGrad, RMS-Prop, Adam and FTRL optimizers for dynamic embedding table (DET).
    For sample code, refer to the test_embedding_table_optimizer.cpp file in the test/utest/embedding_collection/ directory of the repository on GitHub.
  • Added support for the FTRL optimizer for dense networks.

• Data Reading from S3 for Offline Inference:
  In addition to reading during training, HugeCTR now supports reading data from remote file systems such as HDFS and S3 during offline inference by using the DataSourceParams API.
  The HugeCTR Training and Inference with Remote File System Example is updated to demonstrate the new functionality.

• Documentation Enhancements:

  • The set up instructions for running the example notebooks are revised for clarity.
  • The example notebooks are also updated to show using a data preprocessing script that simplifies the user experience.
  • Documentation for the MLP Layer is new.
  • Several 2022 talks and blogs are added to the HugeCTR Talks and Blogs page.

• Issues Fixed:

  • The original CUDA device with NUMA bind before a call to some HugeCTR APIs is recovered correctly now.
    This issue sometimes lead to a problem when you mixed calls to HugeCTR and other CUDA enabled libraries.
  • Fixed the occasional CUDA kernel launch failure of embedding when installed HugeCTR with macro DEBUG.
  • Fixed an SOK build error that was related to TensorFlow v2.1.0 and higher.
    The issue was that the C++ API and C++ standard were updated to use C++17.
  • Fixed a CUDA 12 related compilation error.

• Known Issues:

  • HugeCTR can lead to a runtime error if client code calls the RMM rmm::mr::set_current_device_resource() method or rmm::mr::set_current_device_resource() method.
    The error is due to the Parquet data reader in HugeCTR also calling rmm::mr::set_current_device_resource().
    As a result, the device becomes visible to other libraries in the same process.
    Refer to GitHub issue #356 for more information.
    As a workaround, you can set environment variable HCTR_RMM_SETTABLE to 0 to prevent HugeCTR from setting a custom RMM device resource, if you know that rmm::mr::set_current_device_resource() is called by client code other than HugeCTR.
    But be cautious because the setting can reduce the performance of Parquet reading.

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue #243.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR V4.3 (Merlin 22.12)

What's New in Version 4.3

In January 2023, the HugeCTR team plans to deprecate semantic versioning, such as `v4.3`.
Afterward, the library will use calendar versioning only, such as `v23.01`.

• Support for BERT and Variants:
  This release includes support for BERT in HugeCTR.
  The documentation includes updates to the MultiHeadAttention layer and adds documentation for the SequenceMask layer.
  For more information, refer to the samples/bst directory of the repository in GitHub.

• HPS Plugin for TensorFlow integration with TensorFlow-TensorRT (TF-TRT):
  This release includes plugin support for integration with TensorFlow-TensorRT.
  For sample code, refer to the Deploy SavedModel using HPS with Triton TensorFlow Backend notebook.

• Deep & Cross Network Layer version 2 Support:
  This release includes support for Deep & Cross Network version 2.
  For conceptual information, refer to https://arxiv.org/abs/2008.13535.
  The documentation for the MultiCross Layer is updated.

• Enhancements to Hierarchical Parameter Server:

  • RedisClusterBackend now supports TLS/SSL communication.
    For sample code, refer to the Hierarchical Parameter Server Demo notebook.
    The notebook is updated with step-by-step instructions to show you how to setup HPS to use Redis with (and without) encryption.
    The Volatile Database Parameters documentation for HPS is updated with the enable_tls, tls_ca_certificate, tls_client_certificate, tls_client_key, and tls_server_name_identification parameters.
  • MultiProcessHashMapBackend includes a bug fix that prevented configuring the shared memory size when using JSON file-based configuration.
  • On-device input keys are supported now so that an extra host-to-device copy is removed to improve performance.
  • A dependency on the XX-Hash library is removed.
    The library is no longer used by HugeCTR.
  • Added the static table support to the embedding cache.
    The static table is suitable when the embedding table can be placed entirely in GPU memory.
    In this case, the static table is more than three times faster than the embedding cache lookup.
    The static table does not support embedding updates.

• Support for New Optimizers:

  • Added support for SGD, Momentum SGD, Nesterov Momentum, AdaGrad, RMS-Prop, Adam and FTRL optimizers for dynamic embedding table (DET).
    For sample code, refer to the test_embedding_table_optimizer.cpp file in the test/utest/embedding_collection/ directory of the repository on GitHub.
  • Added support for the FTRL optimizer for dense networks.

• Data Reading from S3 for Offline Inference:
  In addition to reading during training, HugeCTR now supports reading data from remote file systems such as HDFS and S3 during offline inference by using the DataSourceParams API.
  The HugeCTR Training and Inference with Remote File System Example is updated to demonstrate the new functionality.

• Documentation Enhancements:

  • The set up instructions for running the example notebooks are revised for clarity.
  • The example notebooks are also updated to show using a data preprocessing script that simplifies the user experience.
  • Documentation for the MLP Layer is new.
  • Several 2022 talks and blogs are added to the HugeCTR Talks and Blogs page.

• Issues Fixed:

  • The original CUDA device with NUMA bind before a call to some HugeCTR APIs is recovered correctly now.
    This issue sometimes lead to a problem when you mixed calls to HugeCTR and other CUDA enabled libraries.
  • Fixed the occasional CUDA kernel launch failure of embedding when installed HugeCTR with macro DEBUG.
  • Fixed an SOK build error that was related to TensorFlow v2.1.0 and higher.
    The issue was that the C++ API and C++ standard were updated to use C++17.
  • Fixed a CUDA 12 related compilation error.

• Known Issues:

  • HugeCTR can lead to a runtime error if client code calls the RMM rmm::mr::set_current_device_resource() method or rmm::mr::set_current_device_resource() method.
    The error is due to the Parquet data reader in HugeCTR also calling rmm::mr::set_current_device_resource().
    As a result, the device becomes visible to other libraries in the same process.
    Refer to GitHub issue #356 for more information.
    As a workaround, you can set environment variable HCTR_RMM_SETTABLE to 0 to prevent HugeCTR from setting a custom RMM device resource, if you know that rmm::mr::set_current_device_resource() is called by client code other than HugeCTR.
    But be cautious because the setting can reduce the performance of Parquet reading.

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue #243.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR V4.2 (Merlin 22.11)

What's New in Version 4.2

In January 2023, the HugeCTR team plans to deprecate semantic versioning, such as `v4.2`.
Afterward, the library will use calendar versioning only, such as `v23.01`.

• Change to HPS with Redis or Kafka:
  This release includes a change to Hierarchical Parameter Server and affects deployments that use RedisClusterBackend or model parameter streaming with Kafka.
  A third-party library that was used for HPS partition selection algorithm is replaced to improve performance.
  The new algorithm can produce different partition assignments for volatile databases.
  As a result, volatile database backends that retain data between application startup, such as the RedisClusterBackend, must be reinitialized.
  Model streaming with Kafka is equally affected.
  To avoid issues with updates, reset all respective queue offsets to the end_offset before you reinitialize the RedisClusterBackend.

• Enhancements to the Sparse Operation Kit in DeepRec:
  This release includes updates to the Sparse Operation Kit to improve the performance of the embedding variable lookup operation in DeepRec.
  The API for the lookup_sparse() function is changed to remove the hotness argument.
  The lookup_sparse() function is enhanced to calculate the number of non-zero elements dynamically.
  For more information, refer to the sparse_operation_kit directory of the DeepRec repository in GitHub.

• Enhancements to 3G Embedding:
  This release includes the following enhancements to 3G embedding:

  • The API is changed.
    The EmbeddingPlanner class is replaced with the EmbeddingCollectionConfig class.
    For examples of the API, see the tests in the test/embedding_collection_test directory of the repository in GitHub.
  • The API is enhanced to support dumping and loading weights during the training process.
    The methods are Model.embedding_dump(path: str, table_names: list[str]) and Model.embedding_load(path: str, list[str]).
    The path argument is a directory in file system that you can dump weights to or load weights from.
    The table_names argument is a list of embedding table names as strings.

• New Volatile Database Type for HPS:
  This release adds a db_type value of multi_process_hash_map to the Hierarchical Parameter Server.
  This database type supports sharing embeddings across process boundaries by using shared memory and the /dev/shm device file.
  Multiple processes running HPS can read and write to the same hash map.
  For an example, refer to the Hierarchcal Parameter Server Demo notebook.

• Enhancements to the HPS Redis Backend:
  In this release, the Hierarchical Parameter Server can open multiple connections in parallel to each Redis node.
  This enhancement enables HPS to take advantage of overlapped processing optimizations in the I/O module of Redis servers.
  In addition, HPS can now take advantage of Redis hash tags to co-locate embedding values and metadata.
  This enhancement can reduce the number of accesses to Redis nodes and the number of per-node round trip communications that are needed to complete transactions.
  As a result, the enhancement increases the insertion performance.

• MLPLayer is New:
  This release adds an MLP layer with the hugectr.Layer_t.MLP class.
  This layer is very flexible and makes it easier to use a group of fused fully-connected layers and enable the related optimizations.
  For each fused fully-connected layer in MLPLayer, the output dimension, bias, and activation function are all adjustable.
  MLPLayer supports FP32, FP16 and TF32 data types.
  For an example, refer to the dgx_a100_mlp.py in the samples/dlrm directory of the GitHub repository to learn how to use the layer.

• Sparse Operation Kit installable from PyPi:
  Version 1.1.4 of the Sparse Operation Kit is installable from PyPi in the merlin-sok package.

• Multi-task Model Support added to the ONNX Model Converter:
  This release adds support for multi-task models to the ONNX converter.
  This release also includes an enhancement to the preprocess_census.py script in samples/mmoe directory of the GitHub repository.

• Issues Fixed:

  • Using the HPS Plugin for TensorFlow with MirroredStrategy and running the Hierarchical Parameter Server Demo notebook triggered an issue with ReplicaContext and caused a crash.
    The issue is fixed and resolves GitHub issue #362.
  • The 4_nvt_process.py sample in the samples/din/utils directory of the GitHub repository is updated to use the latest NVTabular API.
    This update resolves GitHub issue #364.
  • An illegal memory access related to 3G embedding and the dgx_a100_ib_nvlink.py sample in the samples/dlrm directory of the GitHub repository is fixed.
  • An error in HPS with the lookup_fromdlpack() method is fixed.
    The error was related to calculating the number of keys and vectors from the corresponding DLPack tensors.
  • An error in the HugeCTR backend for Triton Inference Server is fixed.
    A crash was triggered when the initial size of the embedding cache is smaller than the allowed minimum size.
  • An error related to using a ReLU layer with an odd input size in mixed precision mode could trigger a crash.
    The issue is fixed.
  • An error related to using an asynchronous reader with the AsyncParam class and specifying an io_alignment value that is smaller than the block device sector size is fixed.
    Now, if the specified io_alignment value is smaller than the block device sector size, io_alignment is automatically set to the block device sector size.
  • Unreported memory leaks in the GRU layer and collectives are fixed.
  • Several broken documentation links related to HPS are fixed.

• Known Issues:

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR V4.1.1 (Merlin 22.10)

What's New in Version 4.1.1

• Simplified Interface for 3G Embedding Table Placement Strategy:
  3G embedding now provides an easier way for you to configure an embedding table placement strategy.
  Instead of using JSON, you can configure the embedding table placement strategy by using function arguments.
  You only need to provide the shard_matrix, table_group_strategy, and table_placement_strategy arguments.
  With these arguments, 3G embedding can group different tables together and place them according to the shard_matrix argument.
  For an example, refer to dlrm_train.py file in the test/embedding_collection_test directory of the repository on GitHub.
  For comparison, refer to the same file from the v4.0 branch of the repository.

• New MMoE and Shared-Bottom Samples:
  This release includes a new shared-bottom model, an example program, preprocessing scripts, and updates to documentation.
  For more information, refer to the README.md, mmoe_parquet.py, and other files in the samples/mmoe directory of the repository on GitHub.
  This release also includes a fix to the calculation and reporting of AUC for multi-task models, such as MMoE.

• Support for AWS S3 File System:
  The Parquet DataReader can now read datasets from the Amazon Web Services S3 file system.
  You can also load and dump models from and to S3 during training.
  The documentation for the DataSourceParams class is updated.
  To view sample code, refer to the HugeCTR Training with Remote File System Example class is updated.

• Simplication for File System Usage:
  You no longer ’t need to pass DataSourceParams for model loading and dumping.
  The FileSystem class automatically infers the correct file system type, local, HDFS, or S3, based on the path URI that you specified when you built the model.
  For example, the path hdfs://localhost:9000/ is inferred as an HDFS file system and the path https://mybucket.s3.us-east-1.amazonaws.com/ is inferred as an S3 file system.

• Support for Loading Models from Remote File Systems to HPS:
  This release enables you to load models from HDFS and S3 remote file systems to HPS during inference.
  To use the new feature, specify an HDFS for S3 path URI in InferenceParams.

• Support for Exporting Intermediate Tensor Values into a Numpy Array:
  This release adds function check_out_tensor to Model and InferenceModel.
  You can use this function to check out the intermediate tensor values using the Python interface.
  This function is especially helpful for debugging.
  For more information, refer to Model.check_out_tensor and InferenceModel.check_out_tensor.

• On-Device Input Keys for HPS Lookup:
  The HPS lookup supports input embedding keys that are on GPU memory during inference.
  This enhancement removes a host-to-device copy by using the DLPack lookup_fromdlpack() interface.
  By using the interface, the input DLPack capsule of embedding key can be a GPU tensor.

• Documentation Enhancements:

  • The graphic for the Hierarchical Parameter Server library that shows relationship to other software packages is enhanced.
  • The sample notebook for Deploy SavedModel using HPS with Triton TensorFlow Backend is added to the documentation.
  • Style updates to the Hierarchical Parameter Server API documentation.

• Issues Fixed:

  • The InteractionLayer class is fixed so that it works correctly with num_feas > 30.
  • The cuBLASLt configuration is corrected by increasing the workspace size and adding the epilogue mask.
  • The NVTabular based preprocessing script for our samples that demonstrate feature crossing is fixed.
  • The async data reader is fixed. Previously, it would hang and cause a corruption issue due to an improper I/O block size and I/O alignment problem.
    The AsyncParam class is changed to implement the fix.
    The io_block_size argument is replaced by the max_nr_request argument and the actual I/O block size that the async reader uses is computed accordingly.
    For more information, refer to the AsyncParam class documentation.

• Known Issues:

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR V4.1 (Merlin 22.10)

What's New in Version 4.1

• Simplified Interface for 3G Embedding Table Placement Strategy:
  3G embedding now provides an easier way for you to configure an embedding table placement strategy.
  Instead of using JSON, you can configure the embedding table placement strategy by using function arguments.
  You only need to provide the shard_matrix, table_group_strategy, and table_placement_strategy arguments.
  With these arguments, 3G embedding can group different tables together and place them according to the shard_matrix argument.
  For an example, refer to dlrm_train.py file in the test/embedding_collection_test directory of the repository on GitHub.
  For comparison, refer to the same file from the v4.0 branch of the repository.

• New MMoE and Shared-Bottom Samples:
  This release includes a new shared-bottom model, an example program, preprocessing scripts, and updates to documentation.
  For more information, refer to the README.md, mmoe_parquet.py, and other files in the samples/mmoe directory of the repository on GitHub.
  This release also includes a fix to the calculation and reporting of AUC for multi-task models, such as MMoE.

• Support for AWS S3 File System:
  The Parquet DataReader can now read datasets from the Amazon Web Services S3 file system.
  You can also load and dump models from and to S3 during training.
  The documentation for the DataSourceParams class is updated.
  To view sample code, refer to the HugeCTR Training with Remote File System Example class is updated.

• Simplication for File System Usage:
  You no longer ’t need to pass DataSourceParams for model loading and dumping.
  The FileSystem class automatically infers the correct file system type, local, HDFS, or S3, based on the path URI that you specified when you built the model.
  For example, the path hdfs://localhost:9000/ is inferred as an HDFS file system and the path https://mybucket.s3.us-east-1.amazonaws.com/ is inferred as an S3 file system.

• Support for Loading Models from Remote File Systems to HPS:
  This release enables you to load models from HDFS and S3 remote file systems to HPS during inference.
  To use the new feature, specify an HDFS for S3 path URI in InferenceParams.

• Support for Exporting Intermediate Tensor Values into a Numpy Array:
  This release adds function check_out_tensor to Model and InferenceModel.
  You can use this function to check out the intermediate tensor values using the Python interface.
  This function is especially helpful for debugging.
  For more information, refer to Model.check_out_tensor and InferenceModel.check_out_tensor.

• On-Device Input Keys for HPS Lookup:
  The HPS lookup supports input embedding keys that are on GPU memory during inference.
  This enhancement removes a host-to-device copy by using the DLPack lookup_fromdlpack() interface.
  By using the interface, the input DLPack capsule of embedding key can be a GPU tensor.

• Documentation Enhancements:

  • The graphic for the Hierarchical Parameter Server library that shows relationship to other software packages is enhanced.
  • The sample notebook for Deploy SavedModel using HPS with Triton TensorFlow Backend is added to the documentation.
  • Style updates to the Hierarchical Parameter Server API documentation.

• Issues Fixed:

  • The InteractionLayer class is fixed so that it works correctly with num_feas > 30.
  • The cuBLASLt configuration is corrected by increasing the workspace size and adding the epilogue mask.
  • The NVTabular based preprocessing script for our samples that demonstrate feature crossing is fixed.
  • The async data reader is fixed. Previously, it would hang and cause a corruption issue due to an improper I/O block size and I/O alignment problem.
    The AsyncParam class is changed to implement the fix.
    The io_block_size argument is replaced by the max_nr_request argument and the actual I/O block size that the async reader uses is computed accordingly.
    For more information, refer to the AsyncParam class documentation.

• Known Issues:

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

  • Dumping Adam optimizer states to AWS S3 is not supported.

Merlin: HugeCTR V4.0 (Merlin 22.09)

What's New in Version 4.0

• 3G Embedding Stablization:
  Since the introduction of the next generation of HugeCTR embedding in v3.7, several updates and enhancements were made, including code refactoring to improve usability.
  The enhancements for this release are as follows:

  • Optimized the performance for sparse lookup in terms of inter-warp load imbalance.
    Sparse Operation Kit (SOK) takes advantage of the enhancement to improve performance.
  • This release includes a fix for determining the maximum embedding vector size in the GlobalEmbeddingData and LocalEmbeddingData classes.
  • Version 1.1.4 of Sparse Operation Kit can be installed with Pip and includes the enhancements mentioned in the preceding bullets.

• Embedding Cache Initialization with Configurable Ratio:
  In previous releases, the default value for the cache_refresh_percentage_per_iteration parameter of the InferenceParams was 0.1.

  In this release, default value is 0.0 and the parameter provides an additional purpose.
  If you set the parameter to a value greater than 0.0 and also set use_gpu_embedding_cache to True for a model, when Hierarchical Parameter Server (HPS) starts, HPS initializes the embedding cache for the model on the GPU by loading a subset of the embedding vectors from the sparse files for the model.
  When embedding cache initialization is used, HPS creates log records when it starts at the INFO level.
  The logging records are similar to EC initialization for model: "<model-name>", num_tables: <int> and EC initialization on device: <int>.
  This enhancement reduces the duration of the warm up phase.

• Lazy Initialization of HPS Plugin for TensorFlow:
  In this release, when you deploy a SavedModel of TensorFlow with Triton Inference Server, HPS is implicitly initialized when the loaded model is executed for the first time.
  In previous releases, you needed to run hps.Init(ps_config_file, global_batch_size) explicitly.
  For more information, see the API documentation for hierarchical_parameter_server.Init.

• Enhancements to the HDFS Backend:

  • The HDFS Backend is now called IO::HadoopFileSystem.
  • This release includes fixes for memory leaks.
  • This release includes refactoring to generalize the interface for HDFS and S3 as remote filesystems.
  • For more information, see hadoop_filesystem.hpp in the include/io directory of the repository on GitHub.

• Dependency Clarification for Protobuf and Hadoop:
  Hadoop and Protobuf are true third_party modules now.
  Developers can now avoid unnecessary and frequent cloning and deletion.

• Finer granularity control for overlap behavior:
  We deperacated the old overlapped_pipeline knob and introduces four new knobs train_intra_iteration_overlap/train_inter_iteration_overlap/eval_intra_iteration_overlap/eval_inter_iteration_overlap to help user better control the overlap behavior. For more information, see the API documentation for Solver.CreateSolver

• Documentation Improvements:

  • Removed two deprecated tutorials triton_tf_deploy and dump_to_tf.
  • Previously, the graphics in the Performance page did not appear.
    This issue is fixed in this release.
  • Previously, the API documentation for the HPS Plugin for TensorFlow did not show the class information. This issue is fixed in this release.

• Issues Fixed:

  • Fixed a build error that was triggered in debug mode.
    The error was caused by the newly introduced 3G embedding unit tests.
  • When using the Parquet DataReader, if a parquet dataset file specified in metadata.json does not exist, HugeCTR no longer crashes.
    The new behavior is to skip the missing file and display a warning message.
    This change relates to GitHub issue 321.

• Known Issues:

  • HugeCTR uses NCCL to share data between ranks and NCCL can require shared system memory for IPC and pinned (page-locked) system memory resources.
    If you use NCCL inside a container, increase these resources by specifying the following arguments when you start the container:

      -shm-size=1g -ulimit memlock=-1

    See also the NCCL known issue and the GitHub issue.

  • KafkaProducers startup succeeds even if the target Kafka broker is unresponsive.
    To avoid data loss in conjunction with streaming-model updates from Kafka, you have to make sure that a sufficient number of Kafka brokers are running, operating properly, and are reachable from the node where you run HugeCTR.

  • The number of data files in the file list should be greater than or equal to the number of data reader workers.
    Otherwise, different workers are mapped to the same file and data loading does not progress as expected.

  • Joint loss training with a regularizer is not supported.

Merlin: HugeCTR V3.9.1 (Merlin 22.08)

• fix compatibility issue of cudf 22.06
• some document refactors