For a dataset larger than physical memory, using a memory filesystem like tmpfs for the merge stage will either swap (|tmpfs| < |ram|) or deadlock (|tmpfs| >= |ram|).
Instead, your best bet in that case is to give sort as much physical memory as you can spare:
sort -S 95% -k1 huge.tsv
Extra disk I/O is inevitable since your dataset doesn't fit in memory. At least during a merge sort your disk reads will be O(N) and sequentially ordered.
Note: in the special case that your dataset is slightly larger than physical memory, splitting it up in advance such that one of the `sort -m` input files lives on a tmpfs should indeed be faster.
Other things to check out if you need Very Fast Large Sorts:
- Use `sort --parallel=N` to use multiple cores. By default it only uses 1.
- Use `sort --batch-size=NMERGE` to increase the number of files merged at once. Otherwise you may be doing more mergesort stages than are necessary.
Comments
For a dataset larger than physical memory, using a memory filesystem like tmpfs for the merge stage will either swap (|tmpfs| < |ram|) or deadlock (|tmpfs| >= |ram|).
Instead, your best bet in that case is to give sort as much physical memory as you can spare:
Extra disk I/O is inevitable since your dataset doesn't fit in memory. At least during a merge sort your disk reads will be O(N) and sequentially ordered.Note: in the special case that your dataset is slightly larger than physical memory, splitting it up in advance such that one of the `sort -m` input files lives on a tmpfs should indeed be faster.
Other things to check out if you need Very Fast Large Sorts:
- Use `sort --parallel=N` to use multiple cores. By default it only uses 1.
- Use `sort --batch-size=NMERGE` to increase the number of files merged at once. Otherwise you may be doing more mergesort stages than are necessary.