It wasn't my job so I didn't look into this fully, but the main issue we had was clients claiming that files didn't exist when they did. I just reread the NFS man page and I guess this is the issue:
> To detect when directory entries have been added or removed on the server, the Linux NFS client watches a directory's mtime. If the client detects a change in a directory's mtime, the client drops all cached LOOKUP results for that directory. Since the directory's mtime is a cached attribute, it may take some time before a client notices it has changed. See the descriptions of the acdirmin, acdirmax, and noac mount options for more information about how long a directory's mtime is cached.
> Caching directory entries improves the performance of applications that do not share files with applications on other clients. Using cached information about directories can interfere with applications that run concurrently on multiple clients and need to detect the creation or removal of files quickly, however. The lookupcache mount option allows some tuning of directory entry caching behavior.
People did talk about using Lustre or GPFS but apparently they are really complex to set up and maybe need fancier networking than ethernet, I don't remember.
I did set up GPFS tadam... almost exactly 20 years ago. I wouldn't say it absolutely required fancy networking (infiniband) or was extraordinary complex to set up, certainly on par with NFS when you hit its quirks (which was the reason we went off experimenting with gpfs and whatnot).
> To detect when directory entries have been added or removed on the server, the Linux NFS client watches a directory's mtime. If the client detects a change in a directory's mtime, the client drops all cached LOOKUP results for that directory. Since the directory's mtime is a cached attribute, it may take some time before a client notices it has changed. See the descriptions of the acdirmin, acdirmax, and noac mount options for more information about how long a directory's mtime is cached.
> Caching directory entries improves the performance of applications that do not share files with applications on other clients. Using cached information about directories can interfere with applications that run concurrently on multiple clients and need to detect the creation or removal of files quickly, however. The lookupcache mount option allows some tuning of directory entry caching behavior.
People did talk about using Lustre or GPFS but apparently they are really complex to set up and maybe need fancier networking than ethernet, I don't remember.