Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

parallel is awesome for when you want to want to run a number of jobs unknown in advance, such as using it as an xargs replacement:

  find . -type f -name | parallel --jobs 5 process_file
That will run "process_file foo" for ever file in a directory with 5 processes running in parallel.


Not being a jerk, but what does parallel bring to the table that xargs doesn't?

  find . -type f -name | xargs -I {} -n 1 -P 5 process_file {}


You can have remote workers with parallel (not sure if you can do that with xargs or not).

    parallel --progress --wd '.' -S '4/[email protected],8/[email protected],2/:'  "./complicated_command --param1={1} --param2={2} ::: $(some_shellcommand_to_generate_param1_values.sh) ::: $(some_shellcommand_to_generate_param2_values.sh)
It's very flexible, become my go-to for speeding up parallelizable work. Next step up would be Celery or Spark or something.





Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: