Recently, I have come across what seemed to be a big performance issue on my ZFS miror (the equivalent of a RAID 1). I would like to talk about the reasoning and the information I could have acquired on the topic in order to solve the issue I have encountered.

Note: I'm not a specialist about the inner workings of an SSD, especially when I'm describing possible functionning in details. Some parts can have approximative or slightly skewed explaination about how an SSD works, in details.

The symptoms

In terms of symptoms, writting on the SSD array was taking a long time, if we compare with the normal throughput of an SSD. The activity time of the problematic SSD was practically pinned at 100% accodring to Netdata. Since the write speed of a mirror corresponds to the slowest disk, I let you imagine the apparent performance.

What this lead to ? The RDBMS¹ was not responding very well, which in turn affects other pieces of software like my decentralized instantaneous messaging server whose requests were answered witht 504 Gateway Timeout.

Something was off and one could see/feel it.

The research

Obviously, I've inquired the Internet about this. First, looking by keywords. Then, in chat rooms right and left asking the same question. I have then asked the question further on Mastodon.

A dying SSD ?

I've been told the hypothesis of a dying SSD. In fact, this person indicated me that the S.M.A.R.T (that I will write SMART from now on because I'm too lazy to put dots everywhere) of the SSD said it was fine, but it gave up the ghost "shortly after". For those who don't know what are those SMART data, it's diagnostic information that the storage media collects about itself. This information allows to have an idea of the health state of the disk and, if needed, do actions like replacing the disk or orderning some in advance in order to prepare yourself for an imminent failure.

Unlike "classic" hard drives (HDD or Hard Disk Drive), an SSD doesn't have any mechanical parts and can go away "without saying a thing", just the kernel crying disconnected disk. A mechanical disk, even though it can go silently, it can make noises signifying that it will be broken soon.

So, I was ready to order an SSD when the one that bothers me decides to give up.

I have also searched a little it on the SSD in question. Even though it let me down twice and causes a few issues in ZFS, it should hold more than three years of power-on time and 55 TB of write.

Have you TRIMmed your SSD ?

Another thesis that I have been told is related with TRIMming. Unlike mechanical disks whose magnetic cells can be overwritten without any prior action, the SSDs have to "erase" their cells before they can be rewritten with other data. One command to give to SSDs exist so they can reset cells without penalizing the rest of the system because it's asynchronous. It is also translarent when a controller has enough space to move data around.

It turned out that one option was deactivated in the ZFS pool settings, "deleted" but not reset cells started to pile up until... there wasn't any free cells and performance took a hit. I'm asking myself: how the SSD is supposed to know what can be purged if it decides to purge by itself ? Does the file system have the cells reset before overwriting them, or the SSD resets the cells before writing ?

The solution

The solution to solve this issue has been, in a first step, give the right command to the SSD and then configure the ZFS pool.

To TRIM a ZFS pool, it is enough to run the command zpool trim <your pool>. On a more conventional file system, fstrim will do the job. The operation will give a clean sweep on the cells "deleted" at the file system level but still considered used by the controlle.

Then, for ZFS, you must activate the option autotrim with the command zpool set autotrim=on <pool>. Regarding classical file systems, you have to put the option discord on the right mount points in /etc/fstab/.

The difference between an untrimmed and trimmed SSD is remarkable. At least, I know what to expect if I don't trim automagically or regularly.

The word of the end

To finish, it wasn't a dying SSD that was bugging me; it was an SSD full with free cells on the file system but still busy according to the controller. This SSD made me know this with a rather unconventionnal way, or nearly.