Sort --parallel isn't parallelizing

distro_dave

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are two files, about 3 gigs each, 50% aren't relevant, so perhaps 300 million lines. LC_ALL=C grep -E <files> | sort --parallel=24 -u | wc -m Between LC_ALL=C and using -x to accelerate grep, the slowest part by far is the sort. Reading the man pages...

Originally asked by Jeremy Kemball on Stack Exchange (superuser) (CC BY-SA 3.0).

uphilltech

Great question — answering as the UphillTech team:

sort doesn't create a thread unless it needs to, and for small files it's just too much overhead. Now unfortunately sort treats a pipe like a small file. If you want to feed enough data to 24 threads then you'll need to specify to sort to use a large internal buffer (sort does that automatically when presented with large files). This is something we should improve on upstream (at least in documentation). So you'll want something like: (export LC_ALL=C; grep -E <files> | sort -S1G --parallel=24 -u | wc -m) Note I've set LC_ALL=C for all processes, since they'll all benefit with this data). BTW you can monitor the sort threads with something like: watch -n.1 ps -C sort -L -o pcpu Update Sept...

Full answer by pixelbeat: https://superuser.stackexchange.com/a/938634 (CC BY-SA 3.0).

Need it done for your business? UphillTech.