LizardFS chunkserver removal

 

Comment the filesystem you want to remove in /etc/mfs/mfshdd.cfg

*/srv/mfs1
*/srv/mfs2

 

Tell the daemon to reload:

[root@zzz ~]# ps -ef | grep mfs
mfs        231     1 15 00:38 ?        00:03:58 /usr/sbin/mfschunkserver start
[root@zzz ~]# kill -1 231

 

 

 

Here it shows 7 chunks missing from a daemon restart. those aren't really missing, you can ignore them.

The question is with a lot of "overgoal" data, but also a lot of chunks with valid / needed copies 1/1 you can't actually tell when it's safe to remove a storage.

 

The same problem is if you look at the individual systems, it tells you with filesystems are marked for removal, but also shows usages (and that number won't ever decrease as long as the filesystem is still in mfshdd.conf)

 

 

Wait for the (needed 1 / valid 1) combination to drop to 0 before you make this filesystem unavailable!

 

 

What to do if it went wrong?

I once thought the system is already clear of chunks and removed the filesystem. The result is many files that had a goal of 1 are lost.

As usual in storage administration, the first step is to assess the exact damage and what data is really affected.

After a few hours the mfs check loop identifies all missing or unhappy chunks. It'll also tell you the number of affected files and file names. If you read carefully, you'll notice it lists a lot less file names than it reports affected files. It apparently only reports the *first* file affected by a missing chunk. I removed a large amount of data that wasn't really critical and then suddenly other files were reported as missing. This is because now the check loop finds the next ones.

To get a real overview you can use mfscheckfile on your root mfs datastore

mkdir /mfsmount
mfsmount /mfsmount
find /mfsmount -type f -exec mfscheckfile {} + | grep -B1 "0 copies"

 

With the full list available I decided I need some of this data. I removed the VG I had already created, rebuilt the old partition table and mounted the still alive filesystem back to it's place.

Restarted the mfs daemon and now I'm letting ALL the data sync off those two disks. After that I can finally do a dangerous OS update and yet feel safe. I need that to gain more stability with SSD caching.

 

In summary, my life could have been a lot easier by just having some specific data at goal 2. You get what you configure :-)