IBM · williammorrison2 · Sep 20, 2021 · Sep 29, 2021 · Sep 29, 2021 · Nov 3, 2021
diff --git a/docs/source/cast-big-data/beats.rst b/docs/source/cast-big-data/beats.rst
@@ -24,6 +24,8 @@ for a more generalized installation guide please consult the official `Filebeats
 
 1. Install the filebeats rpm on the node:
 
+This node is usually the CSM master where transaction logs are located.
+
 .. code-block:: bash
 
    rpm -ivh filebeat-*.rpm 

diff --git a/docs/source/cast-big-data/common-problems.rst b/docs/source/cast-big-data/common-problems.rst
@@ -20,7 +20,7 @@ There was a typo in a previous version of CAST. The field "close_removed" is a b
 Logstash Not Starting
 ---------------------
 
-In ELK 6.8.1, Logstash may not start and run on Power, due to an arch issue. 
+In ELK 7.5.1, Logstash may not start and run on Power, due to an arch issue. 
 
 .. code-block:: none
 
@@ -35,7 +35,7 @@ GitHub Issue: https://github.com/elastic/logstash/issues/10755
 
 IBM and the CAST team have made a script to fix this packaging issue. 
 
-The patch can be found in the CAST repo at: https://github.com/IBM/CAST/blob/master/csm_big_data/logstash/patches/csm_logstash_6-8-1_patch.sh and in the install dir at: ``/opt/ibm/csm/bigdata/logstash/patches/csm_logstash_6-8-1_patch.sh``.
+The patch can be found in the CAST repo at: https://github.com/IBM/CAST/blob/master/csm_big_data/logstash/csm_logstash_patch_jruby_9.2.8.sh and in the install dir at: ``/opt/ibm/csm/bigdata/logstash/csm_logstash_patch_jruby_9.2.8.sh``.
 
 Run this patch before starting Logstash. 
 

diff --git a/docs/source/cast-big-data/elasticsearch.rst b/docs/source/cast-big-data/elasticsearch.rst
@@ -14,8 +14,10 @@ storage or large numbers of drives are prefered.
 Configuration
 -------------
 
-.. note:: This guide has been tested using Elasticsearch 6.8.1, the latest RPM may be downloaded from
-    `the Elastic Site <https://www.elastic.co/downloads/elasticsearch>`_.
+.. note:: This guide has been tested using Elasticsearch 7.5.1, which is built for IBM System P.
+    At this writing,
+    `the Elastic Site <https://www.elastic.co/downloads/elasticsearch>`_ no longer provided rpms
+    for the ppc64le architecture.
 
 The following is a brief introduction to the installation and configuration of the elasticsearch service.
 It is generally assumed that elasticsearch is to be installed on multiple Big Data Nodes to take
@@ -26,11 +28,45 @@ CAST provides a set of sample configuration files in the repository at `csm_big_
 If the |csm-bds| rpm as been installed the sample configurations may be found
 in `/opt/ibm/csm/bigdata/elasticsearch/`.
 
-1. Install the elasticsearch rpm and java 1.8.1+ (command run from directory with elasticsearch rpm):
+Preparing for installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Create user ID for `Elasticsearch`, `Kibana`, and `Logstash` on all the BDS nodes. Each user ID should have unique ID number. For example:
+
+Verify the groups and users were submitted into the `/etc/passwd` and `/etc/group` files.
+
+.. code-block:: bash
+
+    grep 'logstash\|elasticsearch\|kibana' /etc/passwd /etc/group
+    /etc/passwd:elasticsearch:x:1900:1900::/home/elasticsearch:/bin/bash
+    /etc/passwd:kibana:x:1901:1901::/home/kibana:/bin/bash
+    /etc/passwd:logstash:x:1902:1902::/home/logstash:/bin/bash
+    /etc/group:elasticsearch:x:1900:
+    /etc/group:kibana:x:1901:
+    /etc/group:logstash:x:1902:
+
+1. Install the elasticsearch rpm and java-11 (command run from directory with elasticsearch rpm):
+
+.. code-block:: bash
+
+    yum install -y elasticsearch-*.rpm java*
+
+.. note:: After installing `Java`, set up the `JAVA` environment variables in root's `.bashrc` file. Assuming `Java` is installed in `/usr/lib/jvm/java-11`.
+
+.. code-block:: bash
+
+    export JAVA_HOME=/usr/lib/jvm/java-11
+    export PATH=/usr/lib/jvm/java-11/bin:$PATH
+
+The above dir. could also be a symbolic link. There could be multiple versions of `Java` installed by other software. Find out where java is:
 
 .. code-block:: bash
 
-    yum install -y elasticsearch-*.rpm java-1.8.*-openjdk
+    # which java
+    /usr/lib/jvm/java-11/bin/java
+
+    # ls -ld /usr/lib/jvm/java-11
+    lrwxrwxrwx 1 root root 29 Jul 12 15:59 /usr/lib/jvm/java-11 -> /etc/alternatives/java_sdk_11
 
 2. Copy the Elasticsearch configuration files to the `/etc/elasticsearch` directory. 
 
@@ -62,8 +98,8 @@ in `/opt/ibm/csm/bigdata/elasticsearch/`.
 .. note:: This is technically optional, however, data will have limited use. This script 
     configures Elasticsearch to properly parse timestamps.
 
-Elasticsearch should now be operational. If Logstash was properly configured there should already
-be data being written to your index.
+Elasticsearch should now be operational. Next logical step is to 
+    `install and configure Logstash <https://cast.readthedocs.io/en/cast_1.8.x/cast-big-data/logstash.html#installation-and-configuration>`_.
 
 Tuning Elasticsearch
 --------------------

diff --git a/docs/source/cast-big-data/kibana.rst b/docs/source/cast-big-data/kibana.rst
@@ -10,18 +10,77 @@ CAST provides a utility plugin for multistep searches of CSM jobs in Kibana dash
 Configuration
 -------------
 
-.. note:: This guide has been tested using Kibana 6.8.1, the latest RPM may be downloaded from
-    `the Elastic Site <https://www.elastic.co/downloads/kibana>`_.
+.. note:: This guide has been tested using Kibana 7.5.1, which is built for IBM System P.
+    At this writing,
+    `the Elastic Site <https://www.elastic.co/downloads/elasticsearch>`_ no longer provided rpms
+    for the ppc64le architecture.
 
 The following is a brief introduction to the installation and configuration of the Kibana service.
 
 At the current time CAST does not provide a configuration file in its RPM.
 
+.. note:: `Kibana` installation has dependency on `node.js`. Make sure the RH BASEOS media or repository is available.
+
 1. Install the Kibana rpm:
 
 .. code-block:: bash
 
-   yum install -y kibana-*.rpm
+    yum install -y kibana-*.rpm
+
+.. code-block:: bash
+
+    # yum install kibana-7.5.1-1_ol001.el8.ppc64le.rpm
+    Updating Subscription Management repositories.
+    Unable to read consumer identity
+
+    This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
+
+    Last metadata expiration check: 0:24:48 ago on Mon 12 Jul 2021 03:49:51 PM EDT.
+    Dependencies resolved.
+    =============================================================================================================================================================
+     Package  Arch      Version                                               Repository                                                                    Size
+    =============================================================================================================================================================
+    Installing:
+     kibana   ppc64le   7.5.1-1_ol001.el8                                     @commandline                                                                 202 M
+    Installing dependencies:
+     nodejs   ppc64le   1:10.24.0-1.module+el8.3.0+10166+b07ac28e             local-rhels8.4-ppc64le--install-REPO-os-rhels-8.4-rhels8.4-ga-repo-ppc64le   9.2 M
+     npm      ppc64le   1:6.14.11-1.10.24.0.1.module+el8.3.0+10166+b07ac28e   local-rhels8.4-ppc64le--install-REPO-os-rhels-8.4-rhels8.4-ga-repo-ppc64le   3.7 M
+    Enabling module streams:
+     nodejs             10
+
+    Transaction Summary
+    =============================================================================================================================================================
+    Install  3 Packages
+
+    Total size: 214 M
+    Total download size: 13 M
+    Installed size: 661 M
+    Is this ok [y/N]: y
+
+.. note:: `Kibana` requires the dir `/var/run/kibana` to store its pid. Create the directory if not there.
+
+.. code-block:: bash
+
+    mkdir /var/run/kibana; chown -R kibana:kibana /var/run/kibana
+    # ls -l /var/run/ | grep kibana
+    drwxr-xr-x  2 kibana        kibana          60 Sep 15 16:18 kibana
+
+After successful `Kibana` installation, check following directory.
+
+.. code-block:: bash
+
+    # ls -ld /usr/share/kibana
+    drwxr-xr-x 11 root root 4096 Jul 12 16:15 /usr/share/kibana
+    # ls -ld /var/run/kibana
+    drwxr-xr-x 2 kibana kibana 60 Sep 15 16:18 /var/run/kibana
+
+Each installation also put configuration files and templates into /etc. For example:
+
+.. code-block:: bash
+
+    # ls -l /etc/kibana
+    -rw-r--r-- 1 kibana kibana  344 Jul 12 16:26 kibana.yml
+    -rw-r--r-- 1 kibana kibana 5149 Jul 12 16:26 kibana.yml.inst
 
 2. Configure the Kibana YAML file (`/etc/kibana/kibana.yml`)
 
@@ -48,6 +107,8 @@ CAST recommends the following four values be set before starting Kibana:
 
     rpm -ivh ibm-csm-bds-kibana-*.noarch.rpm
 
+See `Cast Search <https://cast.readthedocs.io/en/cast_1.8.x/cast-big-data/kibana.html#cast-search>`_ for more information.
+
 4. Start Kibana:
 
 .. code-block:: bash
@@ -63,7 +124,7 @@ CAST Search
 
 CAST Search is a React plugin designed for interfacing with elastic search an building filters for 
 Kibana Dashboards. To maxmize the value of the plugin the `cast-allocation` index pattern should be 
-specified.
+specified. See detailed instructions for `Cast Search installation and configuration <https://cast.readthedocs.io/en/cast_1.8.x/cast-big-data/cast-search.html>`_.
 
 .. TODO describe funciton and feature in greater depth.
 

diff --git a/docs/source/cast-big-data/logstash.rst b/docs/source/cast-big-data/logstash.rst
@@ -23,8 +23,10 @@ Installation and Configuration
 Installation
 ^^^^^^^^^^^^
 
-.. note:: This guide has been tested using Logstash 6.8.1, the latest RPM may be downloaded from
-   `the Elastic Site <https://www.elastic.co/downloads/logstash>`_.
+.. note:: This guide has been tested using Logstash 7.5.1, which is built for IBM System P.
+    At this writing,
+    `the Elastic Site <https://www.elastic.co/downloads/elasticsearch>`_ no longer provided rpms
+    for the ppc64le architecture.
 
 For the official install guide of Logstash in the ELK stack go to: `Installing Logstash`_
 
@@ -33,11 +35,66 @@ CAST provides a set of sample configuration files in the repository at `csm_big_
 If the |csm-bds| rpm has been installed the sample configurations may be found 
 in `/opt/ibm/csm/bigdata/logstash/`.
 
-1. Install the logstash rpm and java 1.8.1+ (command run from directory with logstash rpm):
+Preparing for installation
+
+The Elastic software stack components are installed and run on multiple nodes with associated
+user name such as `elasticsearch`, `kibana`, and `logstash`. They are required to have
+the same user ID and group ID across nodes. Following are examples of creating and verifying
+these user IDs.
+
+.. code-block:: bash
+
+    # groupadd -g 1900 elasticsearch; groupadd -g 1901 kibana; groupadd -g 1902 logstash
+    # useradd -g 1900 -u 1900 elasticsearch; useradd -g 1901 -u 1901 kibana; useradd -g 1902 -u 1902 logstash
+
+.. code-block:: bash
+
+    grep 'logstash\|elasticsearch\|kibana' /etc/passwd /etc/group
+    /etc/passwd:elasticsearch:x:1900:1900::/home/elasticsearch:/bin/bash
+    /etc/passwd:kibana:x:1901:1901::/home/kibana:/bin/bash
+    /etc/passwd:logstash:x:1902:1902::/home/logstash:/bin/bash
+    /etc/group:elasticsearch:x:1900:
+    /etc/group:kibana:x:1901:
+    /etc/group:logstash:x:1902:
+
+1. Install the logstash rpm and java-11 (command run from directory with logstash rpm):
+
+.. code-block:: bash
+
+     yum install -y logstash-*.rpm java-11*
+
+.. note:: Java-11 is a prerequite and can be installed before logstash. In that case, you just
+     install logstash.
+
+.. code-block:: bash
+
+     yum install -y logstash-*.rpm
+
+After the JAVA-11 installation
+
+.. note:: After installing `Java`, set up the `JAVA` environment variables in root's `.bashrc` file. Assuming `Java` is installed in `/usr/lib/jvm/java-11`.
+
+Additionally, bundler and jruby could be installed outside of logstash, so these could also be include in the `/root/.bashrc` file as well.
+There could be incompatibility issues because logstash installs its own bundler and jruby. They have to be added in-front of the PATH so they are found first.
+The admin can try to run the two different versions by specifying absolute paths.
+
+.. code-block:: bash
+
+    export JAVA_HOME=/usr/lib/jvm/java-11
+    export PATH=/usr/lib/jvm/java-11/bin:$PATH
+    export BUNDLE_DIR=/usr/share/logstash/vendor/bundle/jruby/2.5.0/bin
+    export JRUBY_DIR=/usr/share/logstash/vendor/jruby/bin
+    export PATH=$BUNDLE_DIR:$JRUBY_DIR:$PATH
+
+The above dir. could also be a symbolic link. There could be multiple versions of `Java` installed by other software. Find out where java is:
 
 .. code-block:: bash
 
-     yum install -y logstash-*.rpm java-1.8.*-openjdk
+    # which java
+    /usr/lib/jvm/java-11/bin/java
+
+    # ls -ld /usr/lib/jvm/java-11
+    lrwxrwxrwx 1 root root 29 Jul 12 15:59 /usr/lib/jvm/java-11 -> /etc/alternatives/java_sdk_11
 
 2. Copy the Logstash pipeline configuration files to the appropriate directories. 
 
@@ -94,6 +151,9 @@ Configuration
    pipeline.batch.size: 2000 # This is the MAXIMUM, to prevent exceedingly long waits a delay is supplied.  
    pipeline.batch.delay: 50  # Maximum time to wait to execute an underfilled queue in milliseconds.
    queue.type: persisted
+   log.level: info
+   path.queue: /var/log/logstash/logstash-queue/
+   queue.max_bytes: 300gb
    ...
 
 Tuning logstash is highly dependant on your use case and environment. What follows is a set of
@@ -207,7 +267,7 @@ Now that every thing has been installed and configured. You can start Logstash.
 Logstash should now be operational. At this point data aggregators should be configured to point
 to your Logstash node as appropriate.
 
-.. note:: In ELK 6.8.1, Logstash may not start and run on Power, due to an arch issue. Please see: :ref:`Logstash_Not_Starting`
+.. note:: In ELK 7.5.1, Logstash may not start and run on Power, due to an arch issue. Please see: :ref:`Logstash_Not_Starting`
 
 CSM Event Correlator
 ---------------------

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -51,17 +51,17 @@
 
 # General information about the project.
 project = u'CAST'
-copyright = u'2020, IBM Corporation'
+copyright = u'2021, IBM Corporation'
 author = u'IBM Corporation'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = u'1.8.2'
+version = u'1.8.3'
 # The full version, including alpha/beta/rc tags.
-release = u'1.8.2'
+release = u'1.8.3'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.