Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Is there a way to automatically set dd's ibs= and obs= depending on the block and/or stripe sizes of the disks involved? It's weird that optimizing a large transfer involves manually looking up and then setting parameters that the computer itself already knows.

(Though I suppose it only "knows" them through proprietary APIs that a program like dd might not want to build into itself. Still, that's what wrapper scripts are for, no?)

Speaking of, is this something sendfile(2) thinks about? Trying to optimize between time spent reading blocks from the disk and time spent writing packets to the NIC?



The better answer to your question is that you don't /want/ dd to use the filesystem block size because this isn't optimal.

Generally speaking, the larger bs= you use, the faster things will be as it minimizes the syscall overhead. Don't worry about the particulars of the underlying filesystem or block device. Just set the buffers as big as you can reasonably afford given available memory.

And yes, sendfile or io_uring are designed to avoid this.


That supposes a fast block device (i.e. one that can accept/queue reads/writes in its controller faster than you can context switch to the kernel to submit those reads/writes.)

For a slow block device, with no controller-side queue/buffering — say, a floppy disk, the original use-case for dd(1) — you really want to try to ensure that each read(2) or write(2) is for exactly a single disk sector; otherwise you'll end up suffering from read/write amplification which could give you a 2x-or-more penalty on your copy. (Or you rely on the kernel to buffer and coalesce your writes. But that presupposes you're writing to a filesystem, not directly to a disk.)

(Now that I think about it, I'm kind of surprised dd(1) doesn't default to doing "adaptive" block sizes, like TCP's window size — starting low and adjusting up until it finds the optimum, and adjusting back down if it ever notices writes doing patterned straddling slowdowns representing IO "packets" getting split.)


A lot of use cases of `dd` are better served by `head -c $bytes`. `dd` does provide a lot more control if you need it, but when you don't just use head.


I don't know what you're using dd(1) for where `head -c` would serve (reading the MBR off a disk?) Seems kind of a niche use of dd(1) to me, compared to e.g. mirroring one disk to another disk.

Personally, my own favorite use of dd(1) that I've stumbled upon recently (when making backups of database servers) is that dd(1) functions as a (many times) faster cp(1), when copying single large files between disks with large stripe sizes (e.g. when one or both volumes is backed by a RAID0ed NVMe pools with stripe≥256MiB.)

I think dd(1) is still below optimal throughput here, though, because it's not taking advantage of the huge number of IOPS these disks can put out. (I.e. it's not trying to saturate the IO queue by reading+writing the file's blocks as parallel operations.) Is there a tool that does do that? Like one of the many "parallel rsync" tools, but for single-file use? (Basically, BitTorrent, but for disk:disk rather than disk:network?)

(For my original use case, backing up a database, I had lots of individual 1GB heap files, so GNU parallel + dd(1) actually was saturating the IOPS capacity of the src+dest disk pools. But ever-so-slightly-different use-cases, e.g. moving a tarball of said database around, wouldn't be so amenable.)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: