Alias ILM Malcolm

[devilman85]

Dear Malcolm Development Team,

I would like to suggest an improvement for the ILM (Index Lifecycle Management) functionality in Malcolm. Specifically, it would be beneficial if, in future updates, the system could automatically:

• Add newly created indices to the Hot phase of the ILM policy upon creation.

• Configure a default rollover process for these indices, reducing the need for manual intervention.

This enhancement would simplify index management, especially for environments with high data volumes, and ensure seamless integration with ILM policies. It would also improve overall usability by reducing the time spent on manual configuration after new indices are generated.

Thank you for considering this improvement, and I would be happy to discuss further if needed.

Best regards

[mmguero]

Arkime already has support for ILM/ISM settings, which Malcolm can use. See the documentation:

Using ILM/ISM with Arkime

Arkime allows setting index management policies with its sessions and history indices. The Malcolm environment variables for configuring this behavior are set in arkime.env. These variables can be used for both OpenSearch and Elasticsearch instances (OpenSearch Index State Management (ISM) and Elasticsearch Index Lifecycle Management (ILM), respectively).

During Malcolm configuration, you will be prompted as such:

• Enable index management policies (ILM/ISM) in Arkime?
  • Choose Y to proceed to the following related questions about using ILM/ISM with Arkime.
  • Should Arkime use a hot/warm design in which non-session data is stored in a warm index?
    • This quesion allows users to specify if Arkime should store non-session indices (arkime-history) indices in a warm index. This requires additional configuration as demonstrated in the Arkime documentation.
  • How long should Arkime keep an index in the hot node? (e.g. 25h, 5d, etc.)
    • This question allows users to specify how long an Arkime index should remain in the hot state before moving into a warm state.
  • How long should Arkime retain SPI data before deleting it? (e.g. 25h, 90d, etc.)
    • This question is used to set the maximum age at which Arkime session indices are deleted.
  • How many segments should Arkime use to optimize?
    • This question asks for the number of segments to use for optimization.
  • How many replicas should Arkime maintain for older session indices?
    • This defines how many additional copies of older session indices Arkime should store.
  • How many weeks of history should Arkime keep?",
    • This defines the retention period (in weeks) for arkime-history indices.

Closing as duplicate of idaholab#300.