Lucene: “too many open files” explained

Been bitten by this once again during my SOLR tests (after seeing it a few times in Cocoon), and found a good explanation by Yonik Seeley in SOLR-4.

The problem is mainly caused by Lucene creating many segment files, I had 335 in my index directory after adding 100’000 documents in several operations to the index, without optimizing it (and I was running on a plain vanilla linux account with a low ulimit -n).

But go read the above mentioned page for details if you need them.

How did I fix it in my case? I ran SOLR as root with a higher ulimit, sent an <optimize/> command to it, and restored the correct file owner before restarting with my normal user account. Then made the necessary changes to the SOLR and/or Linux config to avoid future problems.

2 Responses to Lucene: “too many open files” explained

  1. You may also find this explained in one of my old Lucene articles on onjava.com

    Increase your open file limit with ulimit, use compound index format, and consider a lower mergeFactor.

%d bloggers like this: