[arm-allstar] faster SD card write

[Fred Moore]

Rich, you might not be aware that all SD cards have an internal buffer
of 128K.  So when you transfer a block of data at 4M blocks, the SD card
will re-block each 4M block to 128K blocks before it is written, this
re-buffering takes LOTS of time.  I have done extensive testing both
reading and writing of different block sizes and have found 128k to both
read and write faster using both OSX, and FreeBSD to usb devices.  Your
mileage may very, but mine seemed very constant, but only on my machine
and OS

Multiple things occur that need to be timed to come up with the best
figures for your system.   Lets look at what happens at the disk level
when you are reading, if the disk is blocked at 4096, it requires how
many blocks to be read if you tell it to get 4M you told dd to get, you
can look at the disk specifications and come close to that estimation,
but not so true if you don't have 4M consecutive of space in memory and
have to page something else out and interrupt the disk read, or unknown
to you your image file extends a cylinder on the drive.    Then what
happens with you have one of the drives that the specifications show it
to be 4096, but is fact reading and writing in 512k blocks using
internal buffering in the drive, it least one of the major drive
manufactures is doing this (seagate).   Did I mention that on most
systems the USB buffer's are 4096 again it takes time to re-block no
matter where it happens.  So if you dump 4M into consecutive memory, it
must then be read out using 4096 when writing to the USB driver.  
Everything that is happening is complex.

Then we have some people using Windows, some Linux, some OSX, and even
me who uses FreeBSD a lot.  These are all factors,  as well.  If you get
into the kernel code you find that different Linux kernels r/w memory in
different size blocks, requiring even further buffering (time).. 

I only bring this up, to show that there is nothing that is absolute
best blocking factor and sync vs non-sync on each machine, as an example
I can turn turn off buffering to my usb drive on osx by writing to the
un-buffered device ie the raw device, this reduces my image writing by a
factor of around 20.  No one even mentions this when talking about
writing an image. 

example  
of=/dev/rusb1  vs  of=/dev/usb1 

prefixing your disk id with an r makes it a raw writing device which is
about 20% faster than a buffered device.  This is only on the BSD's and OSX

There are lots of articles out there on optimizing hard disk and usb
attached storage devices, some are very deep reading..
I think it is very hard to make an absolute statement what what is best
with out testing.  In my case when I was wrestling with this it required
about 3 days of testing, and reading some kernel code to understand all
that was happening in my system.. There are scripts out that that help
you test..

regards... Fred




On 11/20/15 3:41 PM, Rich Prim wrote:
> Gentlemen
>
> If in the dd command you add bs=4M, sync it is faster
>
> I E of=/dev/sdb bs=4M, sync
>
> bs= block size writes the file faster, sync makes sure the code is
> fully written to the SD card.
>
> Rich Prim WA3IWT
>
> -- 
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
>
>
> _______________________________________________
>
> arm-allstar mailing list
> arm-allstar at hamvoip.org
> http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar
>
> Visit the BBB and RPi2 web page - http://hamvoip.org

-- 
Fred Moore
email: fred at fmeco.com
       fred at safes.com
phone:  321-217-8699

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hamvoip.org/pipermail/arm-allstar/attachments/20151121/760298d5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.hamvoip.org/pipermail/arm-allstar/attachments/20151121/760298d5/attachment.pgp>