Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/en/news/blog/2008/lockdep-for-pthreads/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ tags:

Linux has a great tool called lockdep for identifying locking dependency problems.  Instead of waiting until an actual deadlock occurs (which may be extremely difficult when it is a timing-sensitive thing), lockdep keeps track of which locks are already held when any new lock is taken, and ensures that there are no cycles in the dependency graph.

The other day I was sifting through gdb backtraces decoding a deadlock bug in the OSD daemon when it occured to me that it would be nice to have a similar tool for user space applications using pthreads.  A quick search didn’t turn up anything promising, so I put together a simple dependency checker and hooked it into Ceph’s existing Mutex and RWLock wrappers. It was surprisingly quick to put together, and it works!  I was a little disappointed to only find two real dependency bugs.  But the project also motivated me to disable recursive locking (since my lockdep doesn’t cope with that), and that turned up a half dozen other instances of lazyness.
The other day I was sifting through gdb backtraces decoding a deadlock bug in the OSD daemon when it occurred to me that it would be nice to have a similar tool for user space applications using pthreads.  A quick search didn’t turn up anything promising, so I put together a simple dependency checker and hooked it into Ceph’s existing Mutex and RWLock wrappers. It was surprisingly quick to put together, and it works!  I was a little disappointed to only find two real dependency bugs.  But the project also motivated me to disable recursive locking (since my lockdep doesn’t cope with that), and that turned up a half dozen other instances of lazyness.

My lockdep code (C++) is [here](http://ceph.newdream.net/git/?p=ceph.git;a=blob;f=src/common/lockdep.cc;h=2c5f017421180a26cc7666e25c2288405bbb43b3;hb=unstable) and [here](http://ceph.newdream.net/git/?p=ceph.git;a=blob;f=src/common/lockdep.h;h=00026b30b972179f7f05600540d377e288a12370;hb=unstable), plus the hooks into the [mutex wrapper](http://ceph.newdream.net/git/?p=ceph.git;a=blob;f=src/common/Mutex.h;h=a6490cc975192d40dbdd0044f3d76f8e4fb04ee0;hb=unstable).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,4 +95,4 @@ We can now calculate the standard deviation of these positions: 6020910.93405966



> The purpose of this article was to demonstre and justify the second write penalty into Ceph. The second write is being called by `syncfs` which is writing all the objects to their respective PG directories. Understanding the PG placement of the object in addition to the physical mapping of each object of the filesystem on the block device might be a great helper while debugging perfomance issue. Unfortunately this problem is hard to solve because of the client concurrency writes and the distributed nature of Ceph. Obviously what was written here remains pure theory (it's likely true though :p) given that determining the real placement of a data on a disk is difficult. One more thing abou the block placement returned by xfs, this placement gives us values but we don't know how the mapping of these ranges really looks like on the device.
> The purpose of this article was to demonstre and justify the second write penalty into Ceph. The second write is being called by `syncfs` which is writing all the objects to their respective PG directories. Understanding the PG placement of the object in addition to the physical mapping of each object of the filesystem on the block device might be a great helper while debugging performance issue. Unfortunately this problem is hard to solve because of the client concurrency writes and the distributed nature of Ceph. Obviously what was written here remains pure theory (it's likely true though :p) given that determining the real placement of a data on a disk is difficult. One more thing abou the block placement returned by xfs, this placement gives us values but we don't know how the mapping of these ranges really looks like on the device.
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ In this blog, I'll share my journey of completing the project "Visual Regression

My GSoC experience has been amazing, from the initial days of having zero clue of how to navigate around the codebase to successfully completing my project. Along the way I learnt:

- Test Driven Developement (TDD).
- Test Driven Development (TDD).
- How to work on problems together.
- Reviewing other's code.
- Efficient communication.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ In some cases, RocksDB compaction threads can also periodically use CPU resource

Edit: *The previously reported cycles/OP numbers were significantly inflated due to a bug in the cycles/OP parsing code. This made the OSDs appear to be much less efficient than they really are. The corrected numbers have been verified to be within ~14-17% of the rough expected numbers when calculated using aggregate average CPU consumption and IOPS instead of aggregate cycles and OPS. We believe the remaining discrepancy is primarily due variations in clock speed over time (frequency scaling) and possibly how CPU consumption is being reported by collectl. Thanks go out to Dan van der Ster for spotting the issue.*

In both single and multi client tests, efficiency appears to have improved slightly from release to release which is great. A more interesting take-away however is that we are far more efficient at processing data under heavy load in the multi-client tests vs low-load in the single-client tests. Why is this the case? In the previous section we discussed how the the majority of the work in the OSD is done by the OSD worker and async messenger threads. When a new IO is recieved by an OSD, it is first processed and moved into user space by an async messenger thread that is associated with the corresponding network connection. It is then placed into a scheduler workqueue for a given OSD shard. If the queue was previously empty, all threads associated with that shard wake up and don't go back to sleep until the shard's workqueue is empty. When only 1 client is performing IO there is significantly less load on the cluster, and the per-shard queues will often be empty for short periods of time. Threads will constantly wake up and go back to sleep. When the cluster is heavily loaded, the queues are more likely to have work to do so the threads spend less of their time sleeping and waking up. Instead, they spend more of their time doing actual work. This is one of the areas [Clyso](https://clyso.com) is actively working on improving the ceph-osd code.
In both single and multi client tests, efficiency appears to have improved slightly from release to release which is great. A more interesting take-away however is that we are far more efficient at processing data under heavy load in the multi-client tests vs low-load in the single-client tests. Why is this the case? In the previous section we discussed how the the majority of the work in the OSD is done by the OSD worker and async messenger threads. When a new IO is received by an OSD, it is first processed and moved into user space by an async messenger thread that is associated with the corresponding network connection. It is then placed into a scheduler workqueue for a given OSD shard. If the queue was previously empty, all threads associated with that shard wake up and don't go back to sleep until the shard's workqueue is empty. When only 1 client is performing IO there is significantly less load on the cluster, and the per-shard queues will often be empty for short periods of time. Threads will constantly wake up and go back to sleep. When the cluster is heavily loaded, the queues are more likely to have work to do so the threads spend less of their time sleeping and waking up. Instead, they spend more of their time doing actual work. This is one of the areas [Clyso](https://clyso.com) is actively working on improving the ceph-osd code.

# Single and Multi Client Average Latency

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ On anything larger than a single-OSD clusters, Ceph places data determinsiticall

![](images/Pre-Firmware-Fix_Quincy-vs-Reef_-_4KB-Random-Write.svg)

Reef is using the new RocksDB tunings that were tested in the [RocksDB Tuning Deep Dive](https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive). We saw significantly better gains there when those tunings were paired with Quincy and we expected to see similar gains for Reef. In these tests, Reef is performing no better than Quincy and in fact not much better than Pacific. I confess that I ended up running quite a few tests trying to tease out any patterns or issues that might explain the discrepancy. At some point (far later than it should have occured) I realized that I should probably start looking at system metrics. CBT runs a copy of collectl for every single test and records a huge amount of system metric data. In fact between the RBD and RGW tests that were run, CBT recorded over 20GB worth of metrics data over the course of many hours of testing. I got to work and ended up looking at the underlying device behavior for every NVMe drive in the system. I started noticing a pattern. nvme4 in the the 10th cluster node was showing high device queue wait times in many of the write tests, but not in the read tests. Once I looked at 4KB random writes, the effect became obvious:
Reef is using the new RocksDB tunings that were tested in the [RocksDB Tuning Deep Dive](https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive). We saw significantly better gains there when those tunings were paired with Quincy and we expected to see similar gains for Reef. In these tests, Reef is performing no better than Quincy and in fact not much better than Pacific. I confess that I ended up running quite a few tests trying to tease out any patterns or issues that might explain the discrepancy. At some point (far later than it should have occurred) I realized that I should probably start looking at system metrics. CBT runs a copy of collectl for every single test and records a huge amount of system metric data. In fact between the RBD and RGW tests that were run, CBT recorded over 20GB worth of metrics data over the course of many hours of testing. I got to work and ended up looking at the underlying device behavior for every NVMe drive in the system. I started noticing a pattern. nvme4 in the the 10th cluster node was showing high device queue wait times in many of the write tests, but not in the read tests. Once I looked at 4KB random writes, the effect became obvious:

![](images/Pre-Firmware-Fix_Reef-RBD-4KB-RandWrite_-_Mako10-Queue-Sizes.svg)
![](images/Pre-Firmware-Fix_Reef-RBD-4KB-RandWrite_-_Mako10-Queue-Wait-Time.svg)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ With more objects comes longer bucket list times vs the previous tests. Reef is
![](images/Quincy-vs-Reef_-_4KB-Get_-_200M_Total-Objects_-_Mako01-RGW.svg)
![](images/Quincy-vs-Reef_-_4KB-Get_-_200M_Total-Objects_-_Mako01-OSD.svg)

First, get performance is both a little faster and more stable in Reef than it was in Quincy. Having said that, there are some really interesting behaviors here. In Quincy, the first 2 minutes of testing are dominated by extremely high CPU usage. Once that completes, the steady state CPU usage is quite a bit lower than it is in Reef. OSD CPU usage is likewise lower. Reef is using about 10-20% more CPU on the RGW side and about 17-23% more CPU on the OSD side in this test. This may ultimately be another regression that we need to bisect and track down. The good news is that again, Reef's behavior appears to be much more consistent. While RGW CPU usage is higher at the beginning of the test, it's no where near as high as what occured in the Quincy tests.
First, get performance is both a little faster and more stable in Reef than it was in Quincy. Having said that, there are some really interesting behaviors here. In Quincy, the first 2 minutes of testing are dominated by extremely high CPU usage. Once that completes, the steady state CPU usage is quite a bit lower than it is in Reef. OSD CPU usage is likewise lower. Reef is using about 10-20% more CPU on the RGW side and about 17-23% more CPU on the OSD side in this test. This may ultimately be another regression that we need to bisect and track down. The good news is that again, Reef's behavior appears to be much more consistent. While RGW CPU usage is higher at the beginning of the test, it's no where near as high as what occurred in the Quincy tests.

# 4KB DELETE

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Thanks to Mark Nelson for finding and fixing this issue. Thanks to Kefu Chai for

### So what does this mean?

RocksDB performance is sub-optimal when built without `RelWithDebInfo`. This can be mitigated by installing "peformance" package builds. The actual performance increase depends on the cluster, but the RocksDB compaction is reduced by a factor of three. In some cases random 4K write performance is doubled. See these links [1](https://bugs.gentoo.org/733316) and [2](https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/).
RocksDB performance is sub-optimal when built without `RelWithDebInfo`. This can be mitigated by installing "performance" package builds. The actual performance increase depends on the cluster, but the RocksDB compaction is reduced by a factor of three. In some cases random 4K write performance is doubled. See these links [1](https://bugs.gentoo.org/733316) and [2](https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/).

### How can I resolve this performance issue?

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,6 @@ Within this part we have used CBT to successfully compare Jerasure and CLAY for

## <a id="conclusion"></a>Conclusion

In conlusion this blog has demonstrated the seamless experience of how you can generate a CBT performance benchmark run from start to finish, generating performance reports along the way and enabling analysis/comparison of performance. We used **CLAY** and **Jerasure** as an example of how to easily do a performance benchmark but sometimes the results can be unexpected and lead to more questions arising than answers being received. This can lead to further experiments to deep-dive into why certain results occured, and this is what I'll be doing in **Part 4** of the blog that will be coming in the near future. **Part 4** will provide more detailed analysis and IO breakdown for CLAY and Jerasure to provide more clarity on why CLAY performance was worse!
In conclusion this blog has demonstrated the seamless experience of how you can generate a CBT performance benchmark run from start to finish, generating performance reports along the way and enabling analysis/comparison of performance. We used **CLAY** and **Jerasure** as an example of how to easily do a performance benchmark but sometimes the results can be unexpected and lead to more questions arising than answers being received. This can lead to further experiments to deep-dive into why certain results occurred, and this is what I'll be doing in **Part 4** of the blog that will be coming in the near future. **Part 4** will provide more detailed analysis and IO breakdown for CLAY and Jerasure to provide more clarity on why CLAY performance was worse!

[Links to previous parts of the blog series](#outline)