I use the vMware vCenter migration tool to copy virtual machines between different instances of ESXi on a daily basis and today I ran into a problem I have never seen before.
Checking email on my way into work I noticed two sites I deployed within the last few weeks were experiencing an inability for users to login. Each site consists of at minimum one Ubuntu 12.04 LTSP server that users login to via thin clients. Each LTSP server has an NFS mount for /home and a /shared folder that map to a server, also running Ubuntu 12.04, that is the equivalent to a NAS we refer to internally as a filestore.
Why do we do this? Two reasons, user file consistency and site scalability. No matter what LTSP server a user logs into at a site all their files are present. Additionally, we have found when there are more than 20 users on a single LTSP server it becomes unstable (which is primarily related to Firefox’s insatiable thirst for memory, but that’s a different post). Once we are close to 20 users we simply spin up a new LTSP virtual machine and mount its /home and /shared folder on the sites filestore.
Once, I arrived at work I logged in to each site over ssh and realized that each sites filestore was complaining about being out of space and the reason no one could login. This was surprising considering each filestore was provisioned for 150GB of hard drive space and was less than 2 weeks old. To confound the issue further when running the command df -h the /dev/sda1 partition on each filestore reported back as less than 1 percent full. After a bit of googling I discovered the df option -i that reports back the amount of free inode space of a hard drive when run. Once I ran the command on each drive I discover they were both completely out of inode space on the /dev/sda1 partition and even more puzzling the inode limit for each filestore was extremely small for a hard drive drive of 150GB in size at 155,000. To put things in perspective the max inode limit for a hard drive provisioned for 150GB should be in the millions and not in the hundred thousand range.
After a few hours of banging my head against the wall I figured out what happened. I used vCenter Migration Tool to clone each filestore vm from a vm that I prebuilt in our lab. To save space I built the original filestore vm with only an 8GB hard drive and during the cloning of each filestore to each site’s ESXi server I increased the hard drive size to 150GB. Which appeared to work as when I ran a df -h the size of the hard drive reported back as expected. However, as you may have guessed by now the inode size did not increase. After a few weeks in production each site maxed out its extremely low inode limit while still having plenty of disk space available.
In hindsight if I had built the initial filestore vm using LVM I could have added another virtual hard drive and used it to extend the filestores logical volume increasing its inode size. As that was not the case I had to spend the next few days completely rebuilding each site’s filestore.
Lesson Learned!