Skip to content

Latest commit

 

History

History
496 lines (411 loc) · 21.6 KB

linux-installer.md

File metadata and controls

496 lines (411 loc) · 21.6 KB

The official Splunk documentation for this page is Install the Collector for Linux with the installer script. For instructions on how to contribute to the docs, see CONTRIBUTING.md.

Linux Installer Script

For non-containerized Linux environments, an installer script is available. The script deploys and configures:

IMPORTANT: systemd is required to use this script.

Supported Platforms

Currently, the following Linux distributions and versions are supported:

  • Amazon Linux: 2, 2023 (Note: Log collection with Fluentd not currently supported for Amazon Linux 2023.)
  • CentOS / Red Hat / Oracle: 7, 8, 9
  • Debian: 9, 10, 11, 12 (Note: Log collection with Fluentd is not supported for Debian 9 aarch64 or Debian 12.)
  • SUSE: 12, 15 (Note: Only for Collector versions v0.34.0 or higher. Log collection with Fluentd not currently supported.)
  • Ubuntu: 16.04, 18.04, 20.04, 22.04 (Note: Log collection with Fluentd is not supported for Ubuntu 16.04 aarch64.)

Getting Started

Download the latest release of the installer script and view all available options by running the script with the -h flag.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sh /tmp/splunk-otel-collector.sh -h

Run the following command on your host to begin installation with the default options. Replace these variables:

  • SPLUNK_REALM: Which realm to send the data to (for example: us0)
  • SPLUNK_ACCESS_TOKEN: Access token to authenticate requests
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN

After successful installation, run the following command to check the splunk-otel-collector service status:

sudo systemctl status splunk-otel-collector

The splunk-otel-collector service logs and errors can be viewed in the systemd journal:

sudo journalctl -u splunk-otel-collector

You can view the source for more details and available options.

Advanced Configuration

Additional Script Options

Additional configuration options supported by the script can be found by running the script with the -h flag.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sh /tmp/splunk-otel-collector.sh -h

One additional parameter that may need to be changed is --memory in order to configure the memory allocation.

By default, this variable is set to 512. If you have allocated more memory to the Collector then you must increase this setting.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --realm SPLUNK_REALM --memory SPLUNK_MEMORY_TOTAL_MIB \
    -- SPLUNK_ACCESS_TOKEN

By default, apt/yum/zypper repo definition files will be created to download the Collector and Fluentd (if enabled) deb/rpm packages from https://splunk.jfrog.io/splunk and https://packages.treasuredata.com, respectively. To skip these steps and use pre-configured repos on the target system that provide the splunk-otel-collector and td-agent deb/rpm packages, specify the --skip-collector-repo and/or --skip-fluentd-repo options. For example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --realm SPLUNK_REALM --with-fluentd --skip-collector-repo --skip-fluentd-repo \
    -- SPLUNK_ACCESS_TOKEN

Collector Configuration

The Collector comes with a default configuration which can be found at /etc/otel/collector/agent_config.yaml. This configuration can be modified as needed. Possible configuration options can be found in the receivers, processors, exporters, and extensions folders of either:

To use an existing Collector configuration file instead of the default, run the installer script with the --collector-config PATH_TO_CONFIG option, replacing PATH_TO_CONFIG with the absolute path to the desired configuration file on the host. For example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --realm SPLUNK_REALM --collector-config /etc/my-config.yaml \
    -- SPLUNK_ACCESS_TOKEN

If the Collector configuration file includes references to custom environment variables, these variables and values will need to be manually added to the /etc/otel/collector/splunk-otel-collector.conf systemd environment file after installation in order for the splunk-otel-collector systemd service to expand these variables. For example, if the Collector configuration file references ${MY_CUSTOM_VAR1} and ${MY_CUSTOM_VAR2}, add the following to /etc/otel/collector/splunk-otel-collector.conf:

MY_CUSTOM_VAR1=my_custom_value1
MY_CUSTOM_VAR2=my_custom_value2

See EnvironmentFile for more details about the systemd environment file.

If the Collector configuration file or /etc/otel/collector/splunk-otel-collector.conf is modified after installation, the Collector service needs to be restarted for the changes to take effect:

sudo systemctl restart splunk-otel-collector

The splunk-otel-collector service logs and errors can be viewed in the systemd journal:

sudo journalctl -u splunk-otel-collector

Collector Upgrade

To upgrade the Collector, run the following commands on your system (requires root privileges):

  • Debian: 
    sudo apt-get update
sudo apt-get install --only-upgrade splunk-otel-collector
    Note: If the default configuration files in /etc/otel/collector have been modified after initial installation, you may be prompted to keep the existing files or overwrite the files from the new Collector package.
  • RPM:
    • yum 
      sudo yum upgrade splunk-otel-collector
    • dnf 
      sudo dnf upgrade splunk-otel-collector
    • zypper 
      sudo zypper refresh
sudo zypper update splunk-otel-collector
    Note: If the default configuration files in /etc/otel/collector have been modified after initial installation, the existing files will be preserved and the files from the new Collector package may be installed with a .rpmnew extension.

Fluentd Configuration

If log collection with Fluentd is required and your platform is supported, run the installer script with the --with-fluentd option to install/configure Fluentd and the plugins/dependencies listed below. For example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --realm SPLUNK_REALM --with-fluentd -- SPLUNK_ACCESS_TOKEN

The Fluentd deb/rpm package (td-agent) will be installed, and the td-agent service will be configured to forward log events with the @SPLUNK label to the Collector (see below for how to add custom Fluentd log sources). The Collector will then send these events to the HEC ingest endpoint determined by the --realm SPLUNK_REALM option, i.e. https://ingest.SPLUNK_REALM.signalfx.com/v1/log.

The following Fluentd plugins will also be installed:

Additionally, the following dependencies will be installed as prerequisites for the Fluentd plugins:

  • Debian-based systems:

    • build-essential
    • libcap-ng0
    • libcap-ng-dev
    • pkg-config

  • RPM-based systems:

    • Development Tools
    • libcap-ng
    • libcap-ng-devel
    • pkgconfig

To configure the Collector to send log events to a custom HEC endpoint URL, you can specify the following parameters for the installer script:

  • --hec-url URL
  • --hec-token TOKEN

The main Fluentd configuration file will be installed to /etc/otel/collector/fluentd/fluent.conf. Custom Fluentd source config files can be added to the /etc/otel/collector/fluentd/conf.d directory after installation. Please note:

  • By default, Fluentd will be configured to collect log events from many popular services, like the systemd journal. Check the .conf files in this directory for the default configuration of the included sources. Note: The paths defined within these sources may need to be updated for the system or service.
  • Any new source added to this directory should have a .conf extension and have the @SPLUNK label to automatically forward log events to the Collector.
  • All files with a .conf extension in this directory will automatically be included when the Fluentd service starts/restarts.
  • The Fluentd service runs as the td-agent user/group. If adding/modifying any configuration file, ensure that the td-agent user/group has permissions to access the configuration file and the path(s) defined within.
  • After any configuration modification, the Fluentd service needs to be restarted: 
    sudo systemctl restart td-agent
  • The Fluentd service logs and errors can be viewed in /var/log/td-agent/td-agent.log.
  • See https://docs.fluentd.org/configuration for general Fluentd configuration details.

Note: If the td-agent package is upgraded after initial installation, Linux capabilities may need to be set for the new version by performing the following steps (only applicable for td-agent versions 4.1 or newer):

  1. Check for the enabled capabilities:
sudo /opt/td-agent/bin/fluent-cap-ctl --get -f /opt/td-agent/bin/ruby

The output should be:

Capabilities in '/opt/td-agent/bin/ruby',
Effective:   dac_override, dac_read_search
Inheritable: dac_override, dac_read_search
Permitted:   dac_override, dac_read_search
  1. If the output from the previous command does not include dac_override and dac_read_search as shown above, run the following commands:
sudo td-agent-gem install capng_c
sudo /opt/td-agent/bin/fluent-cap-ctl --add "dac_override,dac_read_search" -f /opt/td-agent/bin/ruby
sudo systemctl daemon-reload
sudo systemctl restart td-agent

Auto Instrumentation

Splunk OpenTelemetry Auto Instrumentation installs and supports configuration of the following Auto Instrumentation agent(s) to capture traces and forward them to the locally installed Collector:

To see all supported options and defaults before installation, run:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sh /tmp/splunk-otel-collector.sh -h

Prerequisites and Requirements

Installation

To install the Collector and the Splunk OpenTelemetry Auto Instrumentation packages, run the installer script with either of the following options:

  • --with-instrumentation: Install and activate the provided Auto Instrumentation agent(s) for all supported processes by automatically adding the libsplunk.so shared object library to /etc/ld.so.preload:

    curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN

  • --with-systemd-instrumentation: Install and activate the provided Auto Instrumentation agent(s) for all supported systemd services by installing a systemd drop-in file:

    curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN

By default, both the Java and Node.js Auto Instrumentation agents will be installed and activated. Run the installer script with either the --without-instrumentation-sdk java or --without-instrumentation-sdk node option to skip installation and activation of the respective agent. For example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --without-instrumentation-sdk node --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN

Additional options include:

  • --deployment-environment <value>: Set the deployment.environment resource attribute to <value>. If not specified, the "Environment" in the Splunk APM UI will appear as "unknown" for the instrumented application(s).
  • --service-name <name>: Override the auto-generated service names for all instrumented applications on the host with <name>.
  • --enable-profiler: Enable AlwaysOn Profiling (disabled by default).
  • --enable-profiler-memory: Enable AlwaysOn Memory Profiling (disabled by default).
  • --enable-metrics: Enable instrumentation metrics collection (disabled by default).

Note: After successful installation, reboot the host or manually start/restart the Java and/or Node.js applications/services on the host for Auto Instrumentation to take effect.

Node.js Requirements

If running the installer script with Node.js Auto Instrumentation, the following are required:

  • npm is required to install the Node.js Auto Instrumentation package. If npm is not installed or not found with the command -v npm shell command, the installer script will automatically skip installation and configuration of the Node.js agent. Use the --npm-path <path> option to specify a custom path to npm. For example: 
    curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --npm-path /path/to/npm --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN
  • For arm64/aarch64 architectures, the following package groups will automatically be installed in order to build/compile the Node.js Auto Instrumentation package:
    • Debian/Ubuntu: build-essential
    • CentOS/Oracle/Red Hat/Amazon: Development Tools
    • Suse: devel_basis and devel_C_C++
  • To explicitly skip Node.js Auto Instrumentation, run the installer script with the --without-instrumentation-sdk node option. For example: 
    curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --without-instrumentation-sdk node --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN

Post-Install Configuration

The activation and configuration of Auto Instrumentation with the steps below will be applied globally on the host, either for all supported processes with the --with-instrumentation option, or for all supported systemd services with the --with-systemd-instrumentation option. Configuration of individual processes/services to override the global configuration will need to be managed separately and manually.

  • If the --with-instrumentation option was used:

    • The /etc/ld.so.preload file will be automatically created/updated with the default path to the installed instrumentation library (/usr/lib/splunk-instrumentation/libsplunk.so). If necessary, custom library paths can be manually added to this file.
    • The /etc/splunk/zeroconfig/java.conf and/or /etc/splunk/zeroconfig/node.conf configuration files will be automatically created to configure the activated Auto Instrumentation agent(s), and the environment variables within these files can be manually configured after installation.
    • See System-wide Configuration for more details.

  • If the --with-systemd-instrumentation option was used:

    • The /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf systemd drop-in file will be automatically created to include environment variables that configure the activated Auto Instrumentation agent(s), and can be manually configured for additional environment variables.
    • See Systemd Configuration for more details.

Note: After any configuration changes, reboot the host or manually start/restart the Java and/or Node.js applications/services on the host for the changes to take effect.

Upgrade

To upgrade the Auto Instrumentation package, run the following commands on your system (requires root privileges):

  • Debian: 
    sudo apt-get update
sudo apt-get install --only-upgrade splunk-otel-auto-instrumentation
    Note: You may be prompted to keep or overwrite the configuration files in the /etc/splunk/zeroconfig/ directory. Choosing to overwrite will revert these files to the default file provided by the new package.
  • RPM:
    • yum 
      sudo yum upgrade splunk-otel-auto-instrumentation
    • dnf 
      sudo dnf upgrade splunk-otel-auto-instrumentation
    • zypper 
      sudo zypper refresh
sudo zypper update splunk-otel-auto-instrumentation

The latest splunk-otel-auto-instrumentation deb/rpm package may provide an updated version of the Node.js Auto Instrumentation agent (check the GitHub Release notes). To update the Node.js agent to the latest provided version, run the following command (requires npm):

cd /usr/lib/splunk-instrumentation/splunk-otel-js && \
sudo npm install /usr/lib/splunk-instrumentation/splunk-otel-js.tgz

# WARNING: The default Auto Instrumentation configuration expects the Node.js
# agent to be installed the /usr/lib/splunk-instrumentation/splunk-otel-js
# directory, as specified the command above. If the installation is changed to
# a different path, manually update the path for the NODE_OPTIONS
# environment variable in either /etc/splunk/zeroconfig/node.conf or
# /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
# accordingly.

Important: After the splunk-otel-auto-instrumentation is upgraded, the /usr/lib/splunk-instrumentation/libsplunk.so shared object library path will automatically be removed from /etc/ld.so.preload, if it exists. If system-wide Auto Instrumentation is required, manually add the /usr/lib/splunk-instrumentation/libsplunk.so path to /etc/ld.so.preload. See System-wide Activation for more details.

Note: After successful upgrade, reboot the host or manually start/restart the Java and/or Node.js applications/services on the host for the changes to take effect.

Discovery mode

If you wish to start the collector with discovery mode you can add the --discovery installation option. For more information see the discovery config provider documentation.

Uninstall

If you wish to uninstall the Collector, Fluentd, and Auto Instrumentation packages, you can run:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --uninstall

Note that configuration files may be left on the filesystem. On RPM-based systems, modified configuration files will be renamed with the .rpmsave extension and can be manually deleted if they are no longer needed. On Debian-based systems, modified configuration files will persist and should be manually deleted before re-running the installer script if you do not intend on re-using these configuration files.