The problem is mainly caused by Lucene creating many segment files, I had 335 in my index directory after adding 100’000 documents in several operations to the index, without optimizing it (and I was running on a plain vanilla linux account with a low ulimit -n).
But go read the above mentioned page for details if you need them.
How did I fix it in my case? I ran SOLR as root with a higher ulimit, sent an
<optimize/> command to it, and restored the correct file owner before restarting with my normal user account. Then made the necessary changes to the SOLR and/or Linux config to avoid future problems.