[arm-allstar] faster SD card write

[R. Wayne]

This is very good reading! Thank you for sharing it. As a System Administrator for a known hosting company I regularly install or supervise such. We use CentOS, which is the chosen standard for hosting providers. I don’t really know why since Apache is limited. Nonetheless, we have to format for 4096 blocks so that we maximize disc use of Virtuozzo (OpenVZ; like VMWare). We are just beginning to use SD’s for booting machines and I am wondering if you tested CentOS? I am not asking for my job use but for Allstar, since I know CentOS and not much about FreeBSD or Debian.

Once again, a nice read. Thank you!

From: Fred Moore 
Sent: Saturday, November 21, 2015 2:43 PM
To: arm-allstar at hamvoip.org 
Subject: Re: [arm-allstar] faster SD card write

Rich, you might not be aware that all SD cards have an internal buffer of 128K.  So when you transfer a block of data at 4M blocks, the SD card will re-block each 4M block to 128K blocks before it is written, this re-buffering takes LOTS of time.  I have done extensive testing both reading and writing of different block sizes and have found 128k to both read and write faster using both OSX, and FreeBSD to usb devices.  Your mileage may very, but mine seemed very constant, but only on my machine and OS

Multiple things occur that need to be timed to come up with the best figures for your system.   Lets look at what happens at the disk level when you are reading, if the disk is blocked at 4096, it requires how many blocks to be read if you tell it to get 4M you told dd to get, you can look at the disk specifications and come close to that estimation, but not so true if you don't have 4M consecutive of space in memory and have to page something else out and interrupt the disk read, or unknown to you your image file extends a cylinder on the drive.    Then what happens with you have one of the drives that the specifications show it to be 4096, but is fact reading and writing in 512k blocks using internal buffering in the drive, it least one of the major drive manufactures is doing this (seagate).   Did I mention that on most systems the USB buffer's are 4096 again it takes time to re-block no matter where it happens.  So if you dump 4M into consecutive memory, it must then be read out using 4096 when writing to the USB driver.   Everything that is happening is complex. 

Then we have some people using Windows, some Linux, some OSX, and even me who uses FreeBSD a lot.  These are all factors,  as well.  If you get into the kernel code you find that different Linux kernels r/w memory in different size blocks, requiring even further buffering (time)..  

I only bring this up, to show that there is nothing that is absolute best blocking factor and sync vs non-sync on each machine, as an example I can turn turn off buffering to my usb drive on osx by writing to the un-buffered device ie the raw device, this reduces my image writing by a factor of around 20.  No one even mentions this when talking about writing an image.  

example   
of=/dev/rusb1  vs  of=/dev/usb1  

prefixing your disk id with an r makes it a raw writing device which is about 20% faster than a buffered device.  This is only on the BSD's and OSX

There are lots of articles out there on optimizing hard disk and usb attached storage devices, some are very deep reading.. 
I think it is very hard to make an absolute statement what what is best with out testing.  In my case when I was wrestling with this it required about 3 days of testing, and reading some kernel code to understand all that was happening in my system.. There are scripts out that that help you test.. 

regards... Fred





On 11/20/15 3:41 PM, Rich Prim wrote:

  Gentlemen

  If in the dd command you add bs=4M, sync it is faster

  I E of=/dev/sdb bs=4M, sync

  bs= block size writes the file faster, sync makes sure the code is fully written to the SD card.

  Rich Prim WA3IWT

  -- 
  This message has been scanned for viruses and 
  dangerous content by MailScanner, and is 
  believed to be clean. 
   

_______________________________________________

arm-allstar mailing list
arm-allstar at hamvoip.org
http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar

Visit the BBB and RPi2 web page - http://hamvoip.org

-- 
Fred Moore
email: fred at fmeco.com
       fred at safes.com
phone:  321-217-8699


--------------------------------------------------------------------------------
_______________________________________________

arm-allstar mailing list
arm-allstar at hamvoip.org
http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar

Visit the BBB and RPi2 web page - http://hamvoip.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hamvoip.org/pipermail/arm-allstar/attachments/20151121/4d455af4/attachment.html>