On disk, using nested directories - (00-zz)/(00-zz)/file.ext
The two levels above can either be based on the initial letters of either the filename of a hash of the file content, or use incremental names until each one is "full".
We tried exactly this approach, but browsing (ls) was really slow. If you would change the depth of the nesting in relation to the overall amount of files it might work better but then you would have to impl. a logic for moving files around if the depth changes.
Random access by filename was fast, since we used a hashing mechanism to locate the directory quickly. But (what I meant with "browsing") was slow backup and slow inspection using normal shell tools.
Yes I understood the issue, I specifically asked how many files you were storing per directory - if directory scanning is slow, you still have too many files per directory.
I can't really remember, but we had tens of millions of files and the directory depth was 2, so there were approx. 80000 files in each directory (we used an md5 checksum for the directories). And we used ext3. I understand that the speed with large directories was increased in ext4... maybe that was the problem back then.
Comments
On disk, using nested directories - (00-zz)/(00-zz)/file.ext
The two levels above can either be based on the initial letters of either the filename of a hash of the file content, or use incremental names until each one is "full".
Increase depth to increase maximum scale.
We tried exactly this approach, but browsing (ls) was really slow. If you would change the depth of the nesting in relation to the overall amount of files it might work better but then you would have to impl. a logic for moving files around if the depth changes.
How many files did you have per directory?
Normally something like this would be used with a simple metadata db - filename, type, bytes, filepath.
Random access by filename was fast, since we used a hashing mechanism to locate the directory quickly. But (what I meant with "browsing") was slow backup and slow inspection using normal shell tools.
Yes I understood the issue, I specifically asked how many files you were storing per directory - if directory scanning is slow, you still have too many files per directory.
I can't really remember, but we had tens of millions of files and the directory depth was 2, so there were approx. 80000 files in each directory (we used an md5 checksum for the directories). And we used ext3. I understand that the speed with large directories was increased in ext4... maybe that was the problem back then.
That's the way how git and git-annex work.