diff --git a/ydb/docs/en/core/devops/deployment-options/ansible/initial-deployment.md b/ydb/docs/en/core/devops/deployment-options/ansible/initial-deployment.md index de61af476c21..6411925ad44e 100644 --- a/ydb/docs/en/core/devops/deployment-options/ansible/initial-deployment.md +++ b/ydb/docs/en/core/devops/deployment-options/ansible/initial-deployment.md @@ -2,7 +2,13 @@ -This guide outlines the process of deploying a {{ ydb-short-name }} cluster on a group of servers using [Ansible](https://www.ansible.com/). The recommended setup to get started is 3 servers with 3 disk drives for user data each. For reliability purposes each server should have as independent infrastructure as possible: they'd better be each in a separate datacenter or availability zone, or at least in different server racks. +This guide outlines the process of deploying a {{ ydb-short-name }} cluster on a group of servers using [Ansible](https://www.ansible.com/). This is the recommended approach for bare-metal or virtual machine environments. + +## Prerequisites + +### Server Setup + +The recommended setup to get started is 3 servers with 3 disk drives for user data each. For reliability purposes, each server should have as independent infrastructure as possible: they'd better be each in a separate datacenter or availability zone, or at least in different server racks. For large-scale setups, it is recommended to use at least 9 servers for highly available clusters (`mirror-3-dc`) or 8 servers for single-datacenter clusters (`block-4-2`). In these cases, servers can have only one disk drive for user data each, but they'd better have an additional small drive for the operating system. You can learn about redundancy models available in {{ ydb-short-name }} from the [{#T}](../../../concepts/topology.md) article. During operation, the cluster can be [expanded](../../../maintenance/manual/cluster_expansion.md) without suspending user access to the databases. @@ -18,89 +24,347 @@ Recommended server requirements: * OS: Ubuntu 18+, Debian 9+. * Internet access is needed to update repositories and download necessary packages. +See [{#T}](../../concepts/system-requirements.md) for more details. + {% endnote %} -Download the GitHub repository with examples for installing {{ ydb-short-name }} cluster – `git clone https://github.com/ydb-platform/ydb-ansible-examples.git`. This repository contains a few installation templates for deploying {{ ydb-short-name }} clusters in subfolders, as well as scripts for generating TLS certificates and requirement files for installing necessary Python packages. In this article, we'll use the `3-nodes-mirror-3-dc` subfolder for the most simple setup. Alternatively, you can similarly use `8-nodes-block-4-2` or `9-nodes-mirror-3-dc` if you have the necessary number of suitable servers. +If you plan to use virtual machines in a public cloud provider, consider following [{#T}](preparing-vms-with-terraform.md). -{% cut "Repository Structure" %} +### Software Setup -{% include [repo-tree](./_includes/repo-tree.md) %} +To work with the project on a local (intermediate or installation) machine, you will need: -{% endcut %} +- Python 3 version 3.10+ +- Ansible core version 2.15.2 or higher +- A working directory on a server with SSH access to all cluster servers -To work with the project on a local (intermediate or installation) machine, you will need: Python 3 version 3.10+ and Ansible core version 2.15.2 or higher. Ansible can be installed and run globally (installed in the system) or in a virtual environment. If Ansible is already installed – you can move on to the step ["Configuring the Ansible project"](#ansible-project-setup); if Ansible is not yet installed, install it using one of the following methods: +{% note tip %} -{% list tabs %} +It is recommended to keep the working directory and all files created during this guide in a [VCS](https://en.wikipedia.org/wiki/Version_control) repository such as [Git](https://git-scm.com/). If multiple DevOps engineers will be working with the cluster to be deployed, they should collaborate in the same repository. -- Installing Ansible globally (Ubuntu 22.04 LTS) +{% endnote %} - * Update the apt package list with `sudo apt-get update`. - * Upgrade packages with `sudo apt-get upgrade`. - * Install the `software-properties-common` package to manage your distribution's software sources – `sudo apt install software-properties-common`. - * Add a new PPA to apt – `sudo add-apt-repository --yes --update ppa:ansible/ansible`. - * Install Ansible – `sudo apt-get install ansible-core` (note that installing just `ansible` will lead to an unsuitable outdated version). - * Check the Ansible core version – `ansible --version` +If Ansible is already installed, you can move on to the step ["Configuring the Ansible project"](#ansible-project-setup). If Ansible is not yet installed, install it using one of the following methods: -- Installing Ansible in a Python virtual environment +{% cut "Installing Ansible globally" %} - * Update the apt package list – `sudo apt-get update`. - * Install the `venv` package for Python3 – `sudo apt-get install python3-venv` - * Create a directory where the virtual environment will be created and where the playbooks will be downloaded. For example, `mkdir venv-ansible`. - * Create a Python virtual environment – `python3 -m venv venv-ansible`. - * Activate the virtual environment – `source venv-ansible/bin/activate`. All further actions with Ansible are performed inside the virtual environment. You can exit it with the command `deactivate`. - * Install the recommended version of Ansible using the command `pip3 install -r requirements.txt`, while in the root directory of the downloaded repository. - * Check the Ansible core version – `ansible --version` +Using Ubuntu 22.04 LTS as an example: -{% endlist %} +* Update the apt package list with `sudo apt-get update`. +* Upgrade packages with `sudo apt-get upgrade`. +* Install the `software-properties-common` package to manage your distribution's software sources – `sudo apt install software-properties-common`. +* Add a new PPA to apt – `sudo add-apt-repository --yes --update ppa:ansible/ansible`. +* Install Ansible – `sudo apt-get install ansible-core` (note that installing just `ansible` will lead to an unsuitable outdated version). +* Check the Ansible core version – `ansible --version` + +Refer to [Ansible Installation Guide](https://docs.ansible.com/ansible/latest/installation_guide/index.html) for more details and other installation options. + +{% endcut %} + +{% cut "Installing Ansible in a Python virtual environment" %} -Navigate to the root directory of the downloaded repository and execute the command `ansible-galaxy install -r requirements.yaml` – this will download the Ansible collections `ydb_platform.ydb` and `community.general`, which contain roles and plugins for installing {{ ydb-short-name }}. +* Update the apt package list – `sudo apt-get update`. +* Install the `venv` package for Python3 – `sudo apt-get install python3-venv` +* Create a directory where the virtual environment will be created and where the playbooks will be downloaded. For example, `mkdir venv-ansible`. +* Create a Python virtual environment – `python3 -m venv venv-ansible`. +* Activate the virtual environment – `source venv-ansible/bin/activate`. All further actions with Ansible are performed inside the virtual environment. You can exit it with the command `deactivate`. +* Install the recommended version of Ansible using the command `pip3 install -r requirements.txt`, while in the root directory of the downloaded repository. +* Check the Ansible core version – `ansible --version` + +{% endcut %} ## Configure the Ansible Project {#ansible-project-setup} -### Edit the Inventory Files {#inventory-edit} +### Install {{ ydb-short-name }} Ansible Playbooks -Regardless of the chosen cluster topology (`3-nodes-mirror-3-dc`, `9-nodes-mirror-3-dc`, or `8-nodes-block-4-2`), the main parameters for installing and configuring {{ ydb-short-name }} are contained in the inventory file `50-inventory.yaml`, which is located in the `inventory/` directory. +{% list tabs %} -In the inventory file `50-inventory.yaml`, you need to specify the current list of FQDNs of the servers where {{ ydb-short-name }} will be installed. By default, the list appears as follows: +- Via requirements.yaml + + ```bash + $ cat < requirements.yaml + roles: [] + collections: + - name: git+https://github.com/ydb-platform/ydb-ansible + type: git + version: main + EOF + $ ansible-galaxy install -r requirements.yaml + ``` -```yaml -all: - children: - ydb: - static-node-1.ydb-cluster.com: - static-node-2.ydb-cluster.com: - static-node-3.ydb-cluster.com: +- One-time + + ```bash + $ ansible-galaxy collection install git+https://github.com/ydb-platform/ydb-ansible.git + ``` + +{% endlist %} + +### Configure Ansible + +Create `ansible.cfg` with Ansible configuration that makes sense for your environment. Refer to [Ansible Configuration Reference](https://docs.ansible.com/ansible/latest/reference_appendices/config.html) for more details on this. The further guide assumes that the `./inventory` subdirectory of the working directory is configured to be used for inventory files. + +{% cut "Example starter ansible.cfg" %} + +```ini +[defaults] +conditional_bare_variables = False +force_handlers = True +gathering = explicit +interpreter_python = /usr/bin/python3 +inventory = ./inventory +pipelining = True +private_role_vars = True +timeout = 5 +vault_password_file = ./ansible_vault_password_file +verbosity = 1 +log_path = ./ydb.log + +[ssh_connection] +retries = 5 +timeout = 60 ``` -Next, you need to make the following changes in the `vars` section of the inventory file: +{% endcut %} + +### Create the Primary Inventory File {#inventory-create} + +Create a `inventory/50-inventory.yaml` file using one of the templates below depending on the chosen [{{ ydb-short-name }} cluster topology](../../../concepts/topology.md): + +{% list tabs %} + +- 3 nodes + + ```yaml + all: + children: + ydb: + # Servers + hosts: + static-node-1.ydb-cluster.com: + location: + data_center: 'zone-a' + static-node-2.ydb-cluster.com: + location: + data_center: 'zone-b' + static-node-3.ydb-cluster.com: + location: + data_center: 'zone-c' + vars: + # Ansible + ansible_user: ubuntu + ansible_ssh_private_key_file: "~/ydb" + + # System + system_timezone: UTC + system_ntp_servers: pool.ntp.org + + # Nodes + ydb_version: "25.1.1" + ydb_storage_node_cores: 8 + ydb_database_node_cores: 8 + + # Storage + ydb_database_storage_groups: 8 + ydb_disks: + - name: /dev/vdb + label: ydb_disk_1 + - name: /dev/vdc + label: ydb_disk_2 + - name: /dev/vdd + label: ydb_disk_3 + ydb_allow_format_drives: true # replace with false after the initial setup + + # Database + ydb_user: root + ydb_domain: Root + ydb_database_name: database + ydb_config: + erasure: mirror-3-dc + fail_domain_type: disk + default_disk_type: SSD + security_config: + enforce_user_token_requirement: true + ``` + +- Cross-datacenter -* `ansible_user` – specify the user for Ansible to connect via SSH. -* `ansible_ssh_common_args: "-o ProxyJump=@"` – option for connecting Ansible to a server by IP, from which {{ ydb-short-name }} will be installed (including ProxyJump server). It is used when installing {{ ydb-short-name }} from a local machine not included in the private DNS zone. -* `ansible_ssh_private_key_file` – change the default private SSH-key path to the actual one: `"../"`. -* Choose one of the available options for deploying {{ ydb-short-name }} executables: + ```yaml + all: + children: + ydb: + # Servers + hosts: + static-node-1.ydb-cluster.com: + location: + data_center: 'zone-a' + rack: 'rack-1' + static-node-2.ydb-cluster.com: + location: + data_center: 'zone-a' + rack: 'rack-2' + static-node-3.ydb-cluster.com: + location: + data_center: 'zone-a' + rack: 'rack-3' + static-node-4.ydb-cluster.com: + location: + data_center: 'zone-b' + rack: 'rack-4' + static-node-5.ydb-cluster.com: + location: + data_center: 'zone-b' + rack: 'rack-5' + static-node-6.ydb-cluster.com: + location: + data_center: 'zone-b' + rack: 'rack-6' + static-node-7.ydb-cluster.com: + location: + data_center: 'zone-c' + rack: 'rack-7' + static-node-8.ydb-cluster.com: + location: + data_center: 'zone-c' + rack: 'rack-8' + static-node-9.ydb-cluster.com: + location: + data_center: 'zone-c' + rack: 'rack-9' + vars: + # Ansible + ansible_user: ubuntu + ansible_ssh_private_key_file: "~/ydb" + + # System + system_timezone: UTC + system_ntp_servers: pool.ntp.org + + # Nodes + ydb_version: "25.1.1" + ydb_storage_node_cores: 8 + ydb_database_node_cores: 8 + + # Storage + ydb_database_storage_groups: 8 + ydb_disks: + - name: /dev/vdb + label: ydb_disk_1 + ydb_allow_format_drives: true # replace with false after the initial setup + + # Database + ydb_user: root + ydb_domain: Root + ydb_database_name: database + ydb_config: + erasure: mirror-3-dc + default_disk_type: SSD + security_config: + enforce_user_token_requirement: true + ``` - * `ydb_version`: automatically download one of the [{{ ydb-short-name }} official releases](../../../downloads/index.md#ydb-server) by version number. For example, `23.4.11`. - * `ydb_git_version`: automatically download a {{ ydb-short-name }} build from the [CI/CD system](https://github.com/ydb-platform/ydb/actions/workflows/build-and-test-ondemand.yml) by Git commit hash. For example, `c1de4531c5316c46d92b602b83b0c5b8b5e5e8c8`. - * `ydb_archive`: a local filesystem path for a {{ ydb-short-name }} distribution archive [downloaded](../../../downloads/index.md#ydb-server) or otherwise prepared in advance. - * `ydbd_binary` and `ydb_cli_binary`: local filesystem paths for {{ ydb-short-name }} server and client executables, [downloaded](../../../downloads/index.md#ydb-server) or otherwise prepared in advance. +- Single datacenter -#### Installing fq-connector-go + ```yaml + all: + children: + ydb: + # Servers + hosts: + static-node-1.ydb-cluster.com: + location: + rack: 'rack-1' + static-node-2.ydb-cluster.com: + location: + rack: 'rack-2' + static-node-3.ydb-cluster.com: + location: + rack: 'rack-3' + static-node-4.ydb-cluster.com: + location: + rack: 'rack-4' + static-node-5.ydb-cluster.com: + location: + rack: 'rack-5' + static-node-6.ydb-cluster.com: + location: + rack: 'rack-6' + static-node-7.ydb-cluster.com: + location: + rack: 'rack-7' + static-node-8.ydb-cluster.com: + location: + rack: 'rack-8' + vars: + # Ansible + ansible_user: ubuntu + ansible_ssh_private_key_file: "~/ydb" + + # System + system_timezone: UTC + system_ntp_servers: pool.ntp.org + + # Nodes + ydb_version: "25.1.1" + ydb_storage_node_cores: 8 + ydb_database_node_cores: 8 + + # Storage + ydb_database_storage_groups: 8 + ydb_disks: + - name: /dev/vdb + label: ydb_disk_1 + ydb_allow_format_drives: true # replace with false after the initial setup + + # Database + ydb_user: root + ydb_domain: Root + ydb_database_name: database + ydb_config: + erasure: block-4-2 + default_disk_type: SSD + security_config: + enforce_user_token_requirement: true + ``` + +{% endlist %} + +Mandatory settings to adjust for your environment in the chosen template: + +1. **Server hostnames.** Replace `static-node-*.ydb-cluster.com` in `all.children.ydb.hosts` with the real [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name). +2. **Server locations.** The names in `data_center` and `rack` in `all.children.ydb.hosts.location` are arbitrary, but they should match between servers only if they are indeed located in the same datacenter (or availability zone) and rack, respectively. +3. **Filesystem paths to block devices** in `all.children.ydb.vars.ydb_disks`. The template assumes `/dev/vda` is for the operating system and the following disks like `/dev/vdb` are for {{ ydb-short-name }} storage layer. Disk labels are created by the playbooks automatically and their names can be arbitrary. +4. **Ansible-related settings** with `all.children.ydb.ansible_` prefix like username and private key to use for `ssh`. Add more of them like `ansible_ssh_common_args` as necessary. + +Recommended settings to adjust: + +* `ydb_domain`. It will be the first path component for all [scheme objects](../../../concepts/glossary.md#scheme-object) in the cluster. For example, you could put your company name there, cluster region, etc. +* `ydb_database_name`. It will be the second path component for all [scheme objects](../../../concepts/glossary.md#scheme-object) in the database. For example, you could put the use case or project name there. +* `default_disk_type`. If you are using [NVMe](https://en.wikipedia.org/wiki/NVM_Express) or rotating [HDD drives](https://en.wikipedia.org/wiki/Hard_disk_drive), change this setting to `NVME` or `ROT`, respectively. +* `ydb_config`: + * Any {{ ydb-short-name }} settings can be adjusted via this field, see [{#T}](../../../reference/configuration/index.md) for more details. + * {{ ydb-short-name }} playbooks automatically set some {{ ydb-short-name }} settings based on Ansible inventory (like `hosts` or [TLS-related](../../../reference/configuration/index.md) settings), if you configure them explicitly in `ydb_config` it will have a priority. + * If you prefer to keep {{ ydb-short-name }}-specific settings separate from the Ansible inventory, replace this whole setting with a string containing file path to a separate [YAML](https://en.wikipedia.org/wiki/YAML) file with {{ ydb-short-name }} configuration. +* `ydb_storage_node_cores` and `ydb_database_node_cores`. If your server has more than 16 CPU cores, increase these so they sum up to the actual available number. If you have over 64 cores per server, consider running multiple database nodes per server using `ydb_database_nodes_per_server`. Aim for $ydb\_storage\_node\_cores + ydb\_database\_nodes\_per\_server \times ydb\_database\_node\_cores = available\_cores$. + +{% cut "Optional settings" %} + +There are multiple options to specify which exactly {{ ydb-short-name }} executables you want to use for the cluster: + +* `ydb_version`: automatically download one of the [{{ ydb-short-name }} official releases](../../../downloads/index.md#ydb-server) by version number. For example, `23.4.11`. +* `ydb_git_version`: automatically compile the {{ ydb-short-name }} executables from the source code, downloaded from [the official GitHub repository](https://github.com/ydb-platform/ydb). The setting's value is a branch, tag, or commit name. For example, `main`. +* `ydb_archive`: a local filesystem path for a {{ ydb-short-name }} distribution archive [downloaded](../../../downloads/index.md#ydb-server) or otherwise prepared in advance. +* `ydbd_binary` and `ydb_cli_binary`: local filesystem paths for {{ ydb-short-name }} server and client executables, [downloaded](../../../downloads/index.md#ydb-server) or otherwise prepared in advance. Installing a [connector](../../../concepts/federated_query/architecture.md#connectors) may be necessary for using [federated queries](../../../concepts/federated_query/index.md). The playbook can deploy the [fq-connector-go](../manual/federated-queries/connector-deployment.md#fq-connector-go) to the hosts with dynamic nodes. Use the following settings: -* `ydb_install_fq_connector` - set `true` for installing the the connector. +* `ydb_install_fq_connector` - set to `true` for installing the connector. * Choose one of the available options for deploying fq-connector-go executables: * `ydb_fq_connector_version`: automatically download one of the [fq-connector-go official releases](https://github.com/ydb-platform/fq-connector-go/releases) by version number. For example, `v0.7.1`. * `ydb_fq_connector_git_version`: automatically compile the fq-connector-go executable from the source code, downloaded from [the official GitHub repository](https://github.com/ydb-platform/fq-connector-go). The setting's value is a branch, tag, or commit name. For example, `main`. * `ydb_fq_connector_archive`: a local filesystem path for a fq-connector-go distribution archive [downloaded](https://github.com/ydb-platform/fq-connector-go/releases) or otherwise prepared in advance. * `ydb_fq_connector_binary`: local filesystem paths for fq-connector-go executable, [downloaded](https://github.com/ydb-platform/fq-connector-go/releases) or otherwise prepared in advance. -#### Optional Changes in the Inventory Files - -Feel free to change these settings if needed, but it is not necessary in straightforward cases: - -* `ydb_cores_static` – set the number of CPU cores allocated to static nodes. -* `ydb_cores_dynamic` – set the number of CPU cores allocated to dynamic nodes. * `ydb_tls_dir` – specify a local path to a folder with TLS certificates prepared in advance. It must contain the `ca.crt` file and subdirectories with names matching node hostnames, containing certificates for a given node. If omitted, self-signed TLS certificates will be generated automatically for the whole {{ ydb-short-name }} cluster. * `ydb_brokers` – list the FQDNs of the broker nodes. For example: @@ -111,14 +375,14 @@ Feel free to change these settings if needed, but it is not necessary in straigh - static-node-3.ydb-cluster.com ``` -The optimal value of the `ydb_database_groups` setting in the `vars` section depends on available disk drives. Assuming only one database in the cluster, use the following logic: +The optimal value of the `ydb_database_storage_groups` setting in the `vars` section depends on available disk drives. Assuming only one database in the cluster, use the following logic: * For production-grade deployments, use disks with a capacity of over 800 GB and high IOPS, then choose the value for this setting based on the cluster topology: - * For `block-4-2`, set `ydb_database_groups` to 95% of your total disk drive count, rounded down. - * For `mirror-3-dc`, set `ydb_database_groups` to 84% of your total disk drive count, rounded down. -* For testing {{ ydb-short-name }} on small disks, set `ydb_database_groups` to 1 regardless of cluster topology. + * For `block-4-2`, set `ydb_database_storage_groups` to 95% of your total disk drive count, rounded down. + * For `mirror-3-dc`, set `ydb_database_storage_groups` to 84% of your total disk drive count, rounded down. +* For testing {{ ydb-short-name }} on small disks, set `ydb_database_storage_groups` to 1 regardless of cluster topology. -The values of the `system_timezone` and `system_ntp_servers` variables depend on the infrastructure properties where the {{ ydb-short-name }} cluster is being deployed. By default, `system_ntp_servers` includes a set of NTP servers without considering the geographical location of the infrastructure on which the {{ ydb-short-name }} cluster will be deployed. It is strongly recommended to use a local NTP server for on-premise infrastructure and the following NTP servers for cloud providers: +The values of the `system_timezone` and `system_ntp_servers` variables depend on the infrastructure properties where the {{ ydb-short-name }} cluster is being deployed. By default, `system_ntp_servers` includes a set of NTP servers without considering the geographical location of the infrastructure on which the {{ ydb-short-name }} cluster will be deployed. We strongly recommend using a local NTP server for on-premise infrastructure and the following NTP servers for cloud providers: {% list tabs %} @@ -142,161 +406,127 @@ The values of the `system_timezone` and `system_ntp_servers` variables depend on {% endlist %} -No changes to other sections of the `50-inventory.yaml` configuration file are required. +{% endcut %} -#### Changing the root User Password { #change-password } +### Changing the root User Password {#change-password} -Next, you can change the standard {{ ydb-short-name }} root user password contained in the encrypted inventory file `99-inventory-vault.yaml` and in the file `ansible_vault_password_file.txt`. To change the password – specify the new password in the `ansible_vault_password_file.txt` file and duplicate it in the `99-inventory-vault.yaml` file in the format: +Next, you can set the password for the initial user specified in the `ydb_user` setting (`root` by default). This user will initially have the full access rights in the cluster, but this can be later adjusted if necessary. Create `inventory/99-inventory-vault.yaml` with the following contents (replace `` with the actual password): ```yaml all: children: ydb: vars: - ydb_password: + ydb_password: ``` -To encrypt `99-inventory-vault.yaml`, execute the command `ansible-vault encrypt inventory/99-inventory-vault.yaml`. - -After modifying the inventory files, you can proceed to prepare the {{ ydb-short-name }} configuration file. - -### Prepare the {{ ydb-short-name }} Configuration File {#ydb-config-prepare} - -The {{ ydb-short-name }} configuration file contains the settings for {{ ydb-short-name }} nodes and is located in the subdirectory `/files/config.yaml`. A detailed description of the configuration file settings for {{ ydb-short-name }} can be found in the article [{#T}](../../../reference/configuration/index.md). - -The default {{ ydb-short-name }} configuration file already includes almost all the necessary settings for deploying the cluster. You need to replace the standard FQDNs of hosts with the current FQDNs in the `hosts` and `blob_storage_config` sections: +Encrypt this file using the command `ansible-vault encrypt inventory/99-inventory-vault.yaml`. This would require you to either manually enter the encryption password (which can be different) or have `vault_password_file` Ansible setting configured. See [Ansible Vault documentation](https://docs.ansible.com/ansible/latest/vault_guide/index.html) for more details on how this works. -* `hosts` section: +### Prepare the YDB Configuration File {#ydb-config-prepare} - ```yaml - ... - hosts: - - host: static-node-1.ydb-cluster.com - host_config_id: 1 - walle_location: - body: 1 - data_center: 'zone-a' - rack: '1' - ... - ``` - -* `blob_storage_config` section: +## Deploying the {{ ydb-short-name }} Cluster - ```yaml - ... - - fail_domains: - - vdisk_locations: - - node_id: static-node-1.ydb-cluster.com - pdisk_category: SSD - path: /dev/disk/by-partlabel/ydb_disk_1 - ... - ``` +After all the preparations explained above are complete, the actual initial cluster deployment is as simple as running the following command from the working directory: -The rest of the sections and settings in the configuration file can remain unchanged. - -### fq-connector-go Configuration File - -Configuration file for fq-connector-go located in the `/files/fq-connector-go/config.yaml`. In straightforward cases, it can remain unchanged. - -## Deploying the {{ ydb-short-name }} Cluster {#erasure-setup} - -{% note info %} - -The minimum number of servers in a {{ ydb-short-name }} cluster is eight servers for the `block-4-2` redundancy model and nine servers for the `mirror-3-dc` redundancy model. - -In `mirror-3-dc` servers should be distributed across three availability zones or datacenters as evenly as possible. - -{% endnote %} - -The [repository](https://github.com/ydb-platform/ydb-ansible-examples) contains two ready sets of templates for deploying a {{ ydb-short-name }} cluster of eight (redundancy model `block-4-2`) and nine servers (`mirror-3-dc`). Both options can be scaled to any required number of servers, considering a number of technical requirements. - -To prepare your template, you can follow the instructions below: +```bash +ansible-playbook ydb_platform.ydb.initial_setup +``` -1. Create a copy of the directory with the ready example (`3-nodes-mirror-3-dc`, `9-nodes-mirror-3-dc`, or `8-nodes-block-4-2`). -2. Specify the FQDNs of the servers in the file `TLS/ydb-ca-nodes.txt` and execute the script `ydb-ca-update.sh` to generate sets of TLS certificates. -3. Change the template's inventory files according to the [instructions](#inventory-edit). -4. Make changes to the {{ ydb-short-name }} configuration file according to the [instructions](#ydb-config-prepare). -5. In the directory of the cloned template, execute the command `ansible-playbook ydb_platform.ydb.initial_setup`. +Shortly after start, you'll be asked to confirm full wipe of the configured disk drives. Then, it can take tens of minutes to finish deployment depending on the environment and settings. Under the hood, this playbook follows roughly the same steps as explained in instructions for [manual {{ ydb-short-name }} cluster deployment](../manual/initial-deployment.md). -## Installation Script Execution Plan for {{ ydb-short-name }} {#ydb-playbook-run} +### Checking the Cluster State {#cluster-state} -The sequence of role executions and their brief descriptions: +On the last step, the playbook will run a few test queries using real temporary tables to double-check if everything is indeed working as intended. On success, you'll see the `ok` status, `failed=0` for each server, and results of these test queries (3 and 6) if the playbook output is set to be verbose enough. -1. The `packages` role configures repositories, manages APT preferences and configurations, fixes unconfigured packages, and installs necessary software packages depending on the distribution version. -1. The `system` role sets up system settings, including clock and timezone configuration, time synchronization via NTP with `systemd-timesyncd`, configuring `systemd-journald` for log management, kernel module loading configuration, kernel parameter optimization through `sysctl`, and CPU performance tuning using `cpufrequtils`. -1. The `ydb` role performs tasks related to checking necessary variables, installing base components and dependencies, setting up system users and groups, deploying and configuring {{ ydb-short-name }}, including managing TLS certificates and updating configuration files. -1. The `ydb_fq_connector` role (optional) performs tasks related to deploying and configuring fq-connector-go, including checking necessary variables, installing binaries, configuration files, creating and launching `systemd unit`. -1. The `ydb-static` role prepares and launches static nodes of {{ ydb-short-name }}, including checking necessary variables and secrets, formatting and preparing disks, creating and launching `systemd unit` for the storage node, as well as initializing the storage and managing database access. -1. The `ydb-dynamic` role configures and manages dynamic nodes of {{ ydb-short-name }}, including checking necessary variables, creating configuration and `systemd unit` files for each dynamic node, launching these nodes, obtaining a token for {{ ydb-short-name }} access, and creating a database in {{ ydb-short-name }}. +{% cut "Example output" %} -{% cut "Detailed step-by-step installation process description" %} +```txt +... -{% include [ansible-install-steps](./_includes/ansible-install-steps.md) %} +TASK [ydb_platform.ydb.ydbd_dynamic : run test queries] ******************************************************************************************************************************************************* +ok: [static-node-1.ydb-cluster.com] => (item={'instance': 'a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"instance": "a"}, "msg": "all test queries were successful, details: {\"count\":3,\"sum\":6}\n"} +ok: [static-node-1.ydb-cluster.com] => (item={'instance': 'b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"instance": "b"}, "msg": "all test queries were successful, details: {\"count\":3,\"sum\":6}\n"} +ok: [static-node-2.ydb-cluster.com] => (item={'instance': 'a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"instance": "a"}, "msg": "all test queries were successful, details: {\"count\":3,\"sum\":6}\n"} +ok: [static-node-2.ydb-cluster.com] => (item={'instance': 'b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"instance": "b"}, "msg": "all test queries were successful, details: {\"count\":3,\"sum\":6}\n"} +ok: [static-node-3.ydb-cluster.com] => (item={'instance': 'a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"instance": "a"}, "msg": "all test queries were successful, details: {\"count\":3,\"sum\":6}\n"} +ok: [static-node-3.ydb-cluster.com] => (item={'instance': 'b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"instance": "b"}, "msg": "all test queries were successful, details: {\"count\":3,\"sum\":6}\n"} +PLAY RECAP **************************************************************************************************************************************************************************************************** +static-node-1.ydb-cluster.com : ok=167 changed=80 unreachable=0 failed=0 skipped=167 rescued=0 ignored=0 +static-node-2.ydb-cluster.com : ok=136 changed=69 unreachable=0 failed=0 skipped=113 rescued=0 ignored=0 +static-node-3.ydb-cluster.com : ok=136 changed=69 unreachable=0 failed=0 skipped=113 rescued=0 ignored=0 +``` {% endcut %} -As a result of executing the playbook, a {{ ydb-short-name }} cluster will be created, with a test database named `database`, a `root` user with maximum access rights created, and [Embedded UI](../../../reference/embedded-ui/index.md) running on port 8765. To connect to the Embedded UI, you can set up SSH tunneling. For this, execute the command `ssh -L 8765:localhost:8765 -i @` on your local machine. After successfully establishing the connection, you can navigate to the URL [localhost:8765](http://localhost:8765): +As a result of executing the `ydb_platform.ydb.initial_setup` playbook, a {{ ydb-short-name }} cluster will be created. It will contain a [domain](../../../concepts/glossary.md#domain) named with the `ydb_domain` setting (`Root` by default), a [database](../../../concepts/glossary.md#database) named with the `ydb_database_name` setting (`database` by default), and an initial [user](../../../concepts/glossary.md#access-user) named with the `ydb_user` setting (`root` by default). + +## Optional Next Steps -![ydb-web-ui](../../_assets/ydb-web-console.png) +The easiest way to explore the newly deployed cluster is by using [Embedded UI](../../../reference/embedded-ui/index.md) running on port 8765 of each server. In the likely case that you don't have direct access to this port from your browser, you can set up SSH tunneling. For this, execute the command `ssh -L 8765:localhost:8765 -i @` on your local machine (add more options if necessary). After successfully establishing the connection, you can navigate to the [localhost:8765](http://localhost:8765) URL via browser. The browser might ask you to accept a security exception. An example of how it might look like: -## Monitoring the Cluster State {#troubleshooting} +![ydb-web-ui](../../../_assets/ydb-web-console.png) -After successfully creating the {{ ydb-short-name }} cluster, you can check its state using the Embedded UI – [http://localhost:8765/monitoring/cluster/tenants](http://localhost:8765/monitoring/cluster/tenants): +After successfully creating the {{ ydb-short-name }} cluster, you can check its state using the following Embedded UI page: [http://localhost:8765/monitoring/cluster/tenants](http://localhost:8765/monitoring/cluster/tenants). It might look like this: -![ydb-cluster-check](../../_assets/ydb-cluster-check.png) +![ydb-cluster-check](../../../_assets/ydb-cluster-check.png) This section displays the following parameters of the {{ ydb-short-name }} cluster, reflecting its state: -* `Tablets` – a list of running [tablets](../../../concepts/glossary.md#tablets). All tablet state indicators should be green; -* `Nodes` – the number and state of static and dynamic nodes launched in the cluster. The node state indicator should be green, and the ratio of created to launched nodes should be equal. For example, 27/27 for a nine-node cluster. +* `Tablets` – a list of running [tablets](../../../concepts/glossary.md#tablet). All tablet state indicators should be green. +* `Nodes` – the number and state of storage and database nodes launched in the cluster. The node state indicator should be green, and the number of created to launched nodes should be equal. For example, 18/18 for a nine-node cluster with a single database node per server. The `Load` indicators (amount of RAM used) and `Storage` (amount of disk space used) should also be green. You can check the state of the storage group in the `storage` section – [http://localhost:8765/monitoring/cluster/storage](http://localhost:8765/monitoring/cluster/storage): -![ydb-storage-gr-check](../../_assets/ydb-storage-gr-check.png) +![ydb-storage-gr-check](../../../_assets/ydb-storage-gr-check.png) The `VDisks` indicators should be green, and the `state` status (found in the tooltip when hovering over the Vdisk indicator) should be `Ok`. More about the cluster state indicators and monitoring can be read in the article [{#T}](../../../reference/embedded-ui/ydb-monitoring.md). -## Cluster Testing { #testing } +### Cluster Testing {#testing} -You can test the cluster using the built-in load tests in {{ ydb-short-name }} CLI. To do this, download {{ ydb-short-name }} CLI version [2.5.0](https://storage.yandexcloud.net/yandexcloud-ydb/release/2.5.0/linux/amd64/ydb) to the machine where Ansible is installed. For example, using wget: `wget https://storage.yandexcloud.net/yandexcloud-ydb/release/2.5.0/linux/amd64/ydb`. - -Make the downloaded binary file executable – `chmod +x ydb` and execute the connection check command: +You can test the cluster using the built-in load tests in {{ ydb-short-name }} CLI. To do this, [install {{ ydb-short-name }} CLI](../../../reference/ydb-cli/install.md) and create a profile with connection parameters, replacing the placeholders: ```shell -./ydb \ -config profile create \ --d /Root/database \ --e grpcs://< FQDN node >:2135 \ ---ca-file /CA/certs/ca.crt \ ---user root \ ---password-file /ansible_vault_password_file +{{ ydb-cli }} \ + config profile create \ + -d // \ + -e grpcs://:2135 \ + --ca-file $(pwd)/files/TLS/certs/ca.crt \ + --user root \ + --password-file ``` Command parameters and their values: * `config profile create` – This command is used to create a connection profile. You specify the profile name. More detailed information on how to create and modify profiles can be found in the article [{#T}](../../../reference/ydb-cli/profile/create.md). * `-e` – Endpoint, a string in the format `protocol://host:port`. You can specify the FQDN of any cluster node and omit the port. By default, port 2135 is used. -* `--ca-file` – Path to the root certificate for connections to the database using `grpcs`. The certificate is created by the `ydb-ca-update.sh` script in the `TLS` directory and is located at the path `TLS/CA/certs/` relative to the root of the `ydb-ansible-examples` repository. -* `--user` – The user for connecting to the database. By default, the user `root` is created when executing the `ydb_platform.ydb.initial_setup` playbook. -* `--password-file` – Path to the password file. In each folder with a YDB cluster deployment template, there is an `ansible_vault_password_file` that contains the password for the user `root`. - -You can check if the profile has been created using the command `./ydb config profile list`, which will display a list of profiles. After creating a profile, you need to activate it with the command `./ydb config profile activate `. To verify that the profile has been activated, you can rerun the command `./ydb config profile list` – the active profile will have an (active) mark. +* `--ca-file` – Path to the root certificate for connections to the database using `grpcs`. +* `--user` – The user for connecting to the database. +* `--password-file` – Path to the password file. Omit this to enter the password manually. -To execute a YQL query, you can use the command `./ydb sql -s 'select 1;'`, which will return the result of the `select 1` command in table form to the terminal. After checking the connection, you can create a test table with the command: -`./ydb workload kv init --init-upserts 1000 --cols 4`. This will create a test table `kv_test` consisting of 4 columns and 1000 rows. You can verify that the `kv_test` table was created and filled with test data by using the command `./ydb sql -s 'select * from kv_test limit 10;'`. +You can check if the profile has been created using the command `{{ ydb-cli }} config profile list`, which will display a list of profiles. After creating a profile, you need to activate it with the command `{{ ydb-cli }} config profile activate `. To verify that the profile has been activated, you can rerun the command `ydb config profile list` – the active profile will have an `(active)` mark. -The terminal will display a table of 10 rows. Now, you can perform cluster performance testing. The article [{#T}](../../../reference/ydb-cli/workload-kv.md) describes 5 types of workloads (`upsert`, `insert`, `select`, `read-rows`, `mixed`) and the parameters for their execution. An example of executing the `upsert` test workload with the parameter to print the execution time `--print-timestamp` and standard execution parameters is: `./ydb workload kv run upsert --print-timestamp`. +To execute a [YQL](../../../yql/reference/index.md) query, you can use the command `{{ ydb-cli }} sql -s 'SELECT 1;'`, which will return the result of the `SELECT 1` query in table form to the terminal. After checking the connection, you can create a test table with the command: +`{{ ydb-cli }} workload kv init --init-upserts 1000 --cols 4`. This will create a test table `kv_test` consisting of 4 columns and 1000 rows. You can verify that the `kv_test` table was created and filled with test data by using the command `{{ ydb-cli }} sql -s 'select * from kv_test limit 10;'`. -A report of the following type will be displayed in the terminal: +The terminal will display a table of 10 rows. Now, you can perform cluster performance testing. The article [{#T}](../../../reference/ydb-cli/workload-kv.md) describes multiple types of workloads (`upsert`, `insert`, `select`, `read-rows`, `mixed`) and the parameters for their execution. An example of executing the `upsert` test workload with the parameter to print the execution time `--print-timestamp` and standard execution parameters is: `{{ ydb-cli }} workload kv run upsert --print-timestamp`: ```text Window Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) Timestamp 1 727 0 0 11 27 71 116 2024-02-14T12:56:39Z 2 882 0 0 10 21 29 38 2024-02-14T12:56:40Z 3 848 0 0 10 22 30 105 2024-02-14T12:56:41Z +4 901 0 0 9 20 27 42 2024-02-14T12:56:42Z +5 879 0 0 10 22 31 59 2024-02-14T12:56:43Z ... ``` -After completing the tests, the `kv_test` table can be deleted with the command: `./ydb workload kv clean`. More details on the options for creating a test table and tests can be read in the article [{#T}](../../../reference/ydb-cli/workload-kv.md). +After completing the tests, the `kv_test` table can be deleted with the command: `{{ ydb-cli }} workload kv clean`. More details on the options for creating a test table and tests can be read in the article [{#T}](../../../reference/ydb-cli/workload-kv.md). + +## See Also + +* [Extra Ansible configuration examples](https://github.com/ydb-platform/ydb-ansible-examples) +* [{#T}](restart.md) +* [{#T}](update-config.md) +* [{#T}](update-executable.md) \ No newline at end of file