Write performance, Kubernetes or not, utilizing existing severs?

[sykp241095]

Original author: gmkt
Original time: 2023-11-02 15:11:46.740387
Original ID/Slug: 756 / write-performance-kubernetes-or-not-utilizing-existing-severs

We’re looking to migrate from MariaDB to TiDB. We currently have 16 MariaDB 10.5.x servers storing data that’s sharded in our application.

Much of our data is transient, only stored for 2 weeks. While we only have about 13 TB of total stored data, we have about 175 TB of writes per month (fairly evenly spread throughout the hours of the month).

Question 1) One of the concerns we have is write speed. I’ve seen it reported that TiDB can be 3 - 6 times slower than MariaDB for writes. Can someone provide current status and context on that? We have customer processes that run for say 4 hours. To have that start taking 12 - 24 hours would be a blocker.

Question 2) What are the advantages and disadvantages of using Kubernetes to manage the cluster? Is there a performance hit to using it?

Question 3) Most of our MariaDB servers have 4 TB SSDs that are less than 25% utilized, stay under 1 loadavg, and have plenty of RAM. Would it be possible / safe to use some of those servers as TiDB nodes while the MariaDB process stays up and we slowly migrate the data from MariaDB to TiDB as our confidence in TiDB grows? To our software the TiDB server would be one of the shards that we could migrate specific groups of data to as slow or as fast as we’d feel comfortable. This would allow our TiDB cluster to start with many more servers from the start (maybe 10 vs 3), though it would be sharing resources with MariaDB on a given server.

Question 4) Any other recommendations based on our situation? Or pitfalls to avoid?

Thanks in advance.

Application environment:

Theoretical production

TiDB version:

Latest

Reproduction method:

N/A

Problem:

Knowing how to get started in production

Resource allocation:

Our current DB servers are 16 core Intel(R) Xeon(R) E-2288G CPU @ 3.70GHz, have 128GB of RAM, 4TB enterprise SSDs, and a 10 Gbps NIC.

Other servers could be added with more cores, etc.

Attachment:

N/A

[sykp241095]

Original author: TiDB-AI-Assistant
Original time: 2023-11-03 02:50:55.615108

Question 1) The write performance of TiDB compared to MariaDB can vary depending on various factors such as hardware configuration, workload characteristics, and optimization techniques used. While it is true that TiDB may have slightly lower write performance compared to MariaDB in some scenarios, it is important to note that TiDB is designed to provide strong consistency and scalability, which may result in a trade-off with write performance. However, TiDB has been continuously optimized and improved over time, and the performance gap between TiDB and MariaDB has been significantly reduced.

To ensure optimal write performance in TiDB, you can consider the following best practices:

• Properly configure the TiDB cluster according to your workload and hardware specifications. This includes adjusting the number of TiKV nodes, PD nodes, and TiDB nodes, as well as tuning various configuration parameters.
• Utilize TiDB’s distributed architecture to scale out your cluster horizontally by adding more TiKV nodes. This allows you to distribute the write load across multiple nodes and improve overall write performance.
• Optimize your schema design and queries to minimize unnecessary writes and improve efficiency. This includes using appropriate data types, indexing strategies, and avoiding unnecessary data duplication.

For more detailed information on optimizing TiDB performance, you can refer to the TiDB Performance Tuning Guide .

Question 2) Using Kubernetes to manage your TiDB cluster has several advantages. Kubernetes provides a scalable and flexible platform for deploying and managing containerized applications, including TiDB. It allows you to easily scale your TiDB cluster by adding or removing nodes, perform rolling upgrades, and handle node failures automatically. Kubernetes also provides features like load balancing and service discovery, which can improve the availability and reliability of your TiDB cluster.

However, there might be a slight performance overhead when running TiDB on Kubernetes compared to running it directly on bare metal or virtual machines. This overhead is mainly due to the additional layer of abstraction introduced by Kubernetes. Nevertheless, the performance impact is generally minimal and can be mitigated by properly configuring and tuning your Kubernetes cluster.

For more information on running TiDB on Kubernetes, you can refer to the TiDB on Kubernetes Best Practices .

Question 3) It is possible to use your existing servers that currently run MariaDB as TiDB nodes while keeping the MariaDB process running. This approach allows you to gradually migrate your data from MariaDB to TiDB as your confidence in TiDB grows. However, there are a few considerations to keep in mind:

• Ensure that the hardware resources (CPU, RAM, disk I/O) on the servers are sufficient to handle the combined workload of both MariaDB and TiDB. You mentioned that the servers have 4TB SSDs and are underutilized, which is a good starting point.
• Plan the migration process carefully to avoid any disruptions to your application. You can start by migrating specific groups of data to the TiDB cluster and gradually increase the data migration rate as you gain confidence in TiDB’s performance and stability.
• Monitor the resource usage and performance of both MariaDB and TiDB to ensure that they coexist without impacting each other’s performance.

For more information on migrating from MariaDB to TiDB, you can refer to the TiDB Migration Guide .

Question 4) Based on your situation, here are some additional recommendations and pitfalls to avoid:

• Perform thorough testing and benchmarking before migrating your production workload to TiDB. This will help you understand the performance characteristics of TiDB and identify any potential bottlenecks or issues.
• Consider implementing a proper backup and disaster recovery strategy for your TiDB cluster to ensure data safety and availability.
• Stay up to date with the latest TiDB releases and documentation to take advantage of performance improvements and new features.
• Engage with the TiDB community and seek assistance from the TiDB team or community members if you encounter any challenges or have specific requirements.

Overall, migrating from MariaDB to TiDB can bring significant benefits in terms of scalability, high availability, and ease of management. With proper planning, configuration, and optimization, you can achieve good write performance in TiDB while leveraging your existing servers.

: TiDB Documentation: TiDB Introduction | PingCAP Docs

[sykp241095]

Original author: qiang.cao
Original time: 2023-11-04 00:03:47.255032

1.For TiDB data ingestion, some best practice here:
a) How to achieve High performance - small batch insert with high-concurrency:
Use Batch API
b) Try to avoid the hotspot when ingest data into TiDB:
Hotspot case
c) In our customer’s use case, TiDB cluster can support about 30TB per day, 150K rows per second (row size is 1.5KB).

3.For MariaDB migration, you can use DM tool to migrate from MariaDB, it not only support existing data but also support incremental data, which means you can use DM tool to sync data from MariaDB to TiDB in real-time, and you can choose running your business in MariaDB or in TiDB cluster.
a) About DM tool: dm-overview
b) A detailed use case in DM sync data in real-time: https://www.pingcap.com/case-study/reduced-batch-processing-time-by-58-percent-with-a-scale-out-mysql-alternative/

Meanwhile, we (PingCAP team) have extensive experience in data migration, if you want to discuss about the solution detail or need any help, feel free to reach out me.

[BonreeOne]

We are Bonree, a company specializing in Application Performance Management (APM) and Observability solutions.

1. Write Speed Concerns (TiDB vs MariaDB)

Regarding the write speed concerns with TiDB, it's true that TiDB might experience slower write performance compared to MariaDB, especially under heavy write loads. Bonree ONE can help you monitor and analyze the performance differences between TiDB and MariaDB during the migration process:

Real-time Write Latency Monitoring: Bonree ONE can track write latency in real-time, helping you evaluate the performance impact of switching to TiDB compared to MariaDB.
Resource Usage Analysis: By monitoring CPU, memory, and disk I/O usage, Bonree ONE helps identify any potential bottlenecks in TiDB's write process.
Bottleneck Diagnosis: If write performance degrades, Bonree ONE provides deep diagnostics to help pinpoint the root causes, whether it's disk I/O, memory consumption, or CPU load.
2. Kubernetes and Performance Impact

Using Kubernetes to manage TiDB clusters may introduce some overhead, especially in resource scheduling and container orchestration. Bonree ONE can help assess this potential performance impact:

Kubernetes Resource Monitoring: Bonree ONE provides real-time monitoring of Kubernetes nodes and pods, allowing you to track whether Kubernetes management introduces any latency or resource inefficiencies.
Containerized Performance Assessment: By tracking CPU and memory usage within TiDB containers, Bonree ONE can help identify performance bottlenecks introduced by Kubernetes.
Efficient Scaling Management: If Kubernetes is auto-scaling TiDB pods, Bonree ONE can ensure that scaling is happening efficiently, ensuring optimal application performance under varying loads.
3. Using Existing Servers for TiDB Nodes

Using your existing MariaDB servers as TiDB nodes while continuing to run MariaDB during the migration is feasible. Bonree ONE can help monitor and optimize this shared resource setup:

Resource Utilization Monitoring: Bonree ONE tracks CPU, memory, and storage usage on shared servers to ensure that both MariaDB and TiDB can run efficiently without resource contention.
Migration Performance Monitoring: As you migrate data from MariaDB to TiDB, Bonree ONE can monitor both databases' performance, ensuring that the migration process doesn't impact production system stability.
Gradual Migration Support: By monitoring resource consumption on each database instance, Bonree ONE can ensure that the load is evenly distributed and help avoid overloading any single node during the migration.
4. Recommendations and Pitfalls to Avoid

Here are some additional recommendations Bonree ONE can help with:

Gradual Migration and Monitoring: As you gradually migrate data, Bonree ONE can provide continuous monitoring, ensuring that TiDB can handle the load without negatively impacting performance.
Performance Benchmarking: Before and after the migration, Bonree ONE can help benchmark the performance of both MariaDB and TiDB, ensuring that TiDB performs similarly to MariaDB in production.
Resource Scaling and Optimization: With Bonree ONE, you can monitor resource usage and identify any resource bottlenecks, allowing you to scale or optimize servers as needed to ensure a smooth migration process.