[arm-allstar] faster SD card write
David McGough
kb4fxc at inttek.net
Sun Nov 22 07:37:25 EST 2015
Hi Fred,
I'm not sure why the results I presented are a bad assumption (when using
recent Linux, at least)? ---Not for the case of -sequentially- writing a
LARGE file to a flash device...You have to remember that all these files
are tiny compared to the amount of data a modern CPU and memory can move
around in a split second--whether that data is block-aligned, or
not....The 5.8MB/sec barrier I hit is clearly the write-speed limit of 8GB
SanDisk microSD card I used, with my given USB2 card reader/writer, in a
sequential-write scenario where the data written is bigger than the device
buffer.
Okay, now I certainly also agree that huge buffers are NOT the way to go
if you're performing random reads and writes to a flash device. I agree,
you must optimize the buffer size to block size read/written in a
random-access scenario, for best performance....But, that wasn't what the
testing I did demonstrated---I was looking strictly how to optimally write
a firmware flash image to an SD card.
Anyhow, just for grins, I performed a "real life" test using a 4GB
firmware image and I get virtually identical thru-put results to the test
script output I presented earlier.
Here is the output:
--------------------------------------------------------------------------------------
First tests on a high-performance Linux PC with gobs of RAM, etc.
david-vb:/home/mcgough/ham/RasPi2# time zcat RPi2_V2.0_Allstar-beta2.img.gz | dd of=/dev/sdf bs=2048
1996800+0 records in
1996800+0 records out
4089446400 bytes (4.1 GB) copied, 1974.54 s, 2.1 MB/s
real 32m54.552s
user 1m35.858s
sys 2m16.093s
david-vb:/home/mcgough/ham/RasPi2# time zcat RPi2_V2.0_Allstar-beta2.img.gz | dd of=/dev/sdf bs=4096
998400+0 records in
998400+0 records out
4089446400 bytes (4.1 GB) copied, 732.403 s, 5.6 MB/s
real 12m12.408s
user 0m32.830s
sys 0m16.913s
david-vb:/home/mcgough/ham/RasPi2# time zcat RPi2_V2.0_Allstar-beta2.img.gz | dd of=/dev/sdf bs=8192
499200+0 records in
499200+0 records out
4089446400 bytes (4.1 GB) copied, 751.809 s, 5.4 MB/s
real 12m31.814s
user 0m31.518s
sys 0m17.293s
david-vb:/home/mcgough/ham/RasPi2# time zcat RPi2_V2.0_Allstar-beta2.img.gz | dd of=/dev/sdf bs=1M
0+120508 records in
0+120508 records out
4089446400 bytes (4.1 GB) copied, 730.851 s, 5.6 MB/s
real 12m10.856s
user 0m32.406s
sys 0m16.857s
NOTE: Notice that -NO- complete 1M buffers got read due to a combination
of the maximum pipe buffer limit and/or zcat buffer limits!
--------------------------------------------------------------------------------------
Now, on a RPi2 with a 8GB SD card, reading the source image file from the
SD card:
[root at RPi2-dev2 ~]# time zcat RPi2_V2.0_Allstar-beta2.img.gz | dd of=/dev/sda bs=1M
0+65107 records in
0+65107 records out
4089446400 bytes (4.1 GB) copied, 727.31 s, 5.6 MB/s
real 12m7.322s
user 3m3.620s
sys 0m52.900s
NOTE: Notice that -NO- complete 1M buffers got read due to a combination
of the maximum pipe buffer limit and/or zcat buffer limits!
--------------------------------------------------------------------------------------
So, just like the previous test script demonstrated, a minimum buffer size
of 4K is where you first see a major speed improvement, for the scenario
of writing a large firmware image to a SD card...Note that these tests
showed the same results on a RPi2. Since the source data file was gzipped
(and only about 600MB, compressed), the relatively slow SD card reads
on the RPi2 weren't a negative impact at all.
And, the recommendation of using a 1 megabyte buffer (bs=1M) is still
reasonable, since this -ALSO- should help maximize input buffer thru-put
from the data source (probably a file on the local hard drive).
If you'd got massively different results, PLEASE PLEASE post them!! All
I'm looking for is the best answer to this specific question: How to best
write the firmware to the flash card.
As always, YMMV....Time for me to move back to software development.
73, David KB4FXC
On Sun, 22 Nov 2015, Fred Moore wrote:
> While I certainly agree with the general thoughts of most people don't
> write enough cards for time to make much difference, some of us insist
> in being tech-weenies and want to understand everything that is going
> on. General settings are good starting points and will certainly get
> you there, and I don't have a problem with using them..
>
> David, I want to point out that the script you found is not a good test
> for what we do.. It has a very bad assumption, and is creating a test
> file to write that is the same blk size of the file it wants to write.
> This will invalidate any issues of buffering should the file your are
> reading is below disk, memory, usb channel, and sd card writing buffer
> size. In all cases that I can imagine the bs of the the disk (if=) is
> actually controlled by the disk buffer size, and the internal disk
> buffer size, where manufacturers are typically huge liars. Look at how
> much disk space is used up when we want to write a 100 byte file, in
> most cases the block size is 4096 depending on the drive manufacture.
> Only point this out as it is something that needs to be taken into account.
>
> In all cases when writing a image we are using a fixed file size.. and
> are not writing data zero blocks our data is random, and when this file
> got to our disk drive, the drive most likely decided how to block the
> file (also an issue between drive manufactures and OS's) and will very
> between OS's, drive manufacturer etc.. The peak you published could be
> from hitting a size limitations i.e we suddenly started buffering and
> re-blocking, in a buffer "to/from, from/to" disk, memory, usb channel,
> or internal usb writing buffers, anywhere in the system, we have not
> even mentioned what happens when we are trying to force writes outside
> of block boundaries which is a huge os hit.
>
> As others have pointed out the usb device you are plugged into and the
> card you are writing are also involved, and should also be considered
> also as part of the overall system you are testing.
>
> Anyway to get this thread back to normal, only brought this whole thing
> up because of generalities that have been stated COULD be poor choices
> depending on the hardware, OS, what you are doing and other stuff.. they
> certainly was in my case. If anyone is having problems do some testing..
> in my case I found that a bs=128k and writing to the usb drive
> un-buffered was my fastest times by a huge amount. How much time, I can
> write an image in 2+ minutes, and started out at 12+ minutes using the
> generalities stated on this list the first time I created a allstar image.
>
> From my perspective, different than most, here I backup some very
> critical files from my servers to the house, and then as a final archive
> I write them to a SD card, they are also cloud backed up. I am paranoid
> about data integrity so at each point in the transfer gets checksum's
> generated, and transfer(s) over the network are with zfs send/receives
> so I can make sure I have 100% good data at every step, did I say I hate
> disk drives, they are everyone's enemy. I only run zfs file systems as
> I am paranoid about bad data. So I do much more than just write allstar
> images with dd.. In my case testing was worth the effort but again I
> needed to understand what was going on so did research and testing.
> Just as I suggest everyone do what is thinking they are having a problem.
>
> regards.. Fred
>
>
> On 11/21/15 9:13 PM, David McGough wrote:
> > Hi Guys,
> >
> > I found this thread interesting, too. Over the last few years, I've
> > written firmware images to hundreds or thousands of flash cards. And,
> > years ago, I just increased the buffer sizes in dd (probably following
> > advise from some website). The increased size made sense to me, it was
> > much faster and I didn't study it any further. Typically, the
> > "recommended" command I mention to people is very similar to:
> >
> > dd if=<input file> of=/dev/sdX bs=1M
> >
> > or
> >
> > zcat <gzipped input image file> | dd of=/dev/sdX bs=1M
> >
> > ...where /dev/sdX is the linux SD card block interface.
> >
> > So, since there is some interest and I'm wondering about the optimal
> > settings, too, I did some googling and testing. I found this webpage that
> > has some scripts: http://blog.tdg5.com/tuning-dd-block-size/ ...Please
> > find the test script I used attached to bottom of this message (below).
> > For this test, I transferred a 64MB file of zeros from /dev/zero.
> >
> >
> > I tried 3 test setups, all Linux:
> >
> > First, my primary setup is a roaring 4-core high-speed 3.6GHz 64-bit linux
> > PC running Debian Wheezy, with a cheapo 8GB Sandisk SD card attached via a
> > good performing external USB 2.0 hub. Here are the results:
> >
> > david-vb:/home/mcgough/ham/RasPi2# ./dd_obs_test.sh /dev/sdf
> >
> > block size : transfer rate
> > 512 : 2.2 MB/s
> > 1024 : 2.2 MB/s
> > 2048 : 2.2 MB/s
> > 4096 : 5.8 MB/s
> > 8192 : 5.8 MB/s
> > 16384 : 5.8 MB/s
> > 32768 : 5.8 MB/s
> > 65536 : 5.8 MB/s
> > 131072 : 5.8 MB/s
> > 262144 : 5.8 MB/s
> > 524288 : 5.8 MB/s
> > 1048576 : 5.4 MB/s
> > 2097152 : 5.8 MB/s
> > 4194304 : 5.4 MB/s
> > 8388608 : 5.8 MB/s
> > 16777216 : 5.4 MB/s
> > 33554432 : 5.4 MB/s
> > 67108864 : 5.4 MB/s
> >
> > Second, the same setup as above, except the SD card reader is plugged
> > directly into a USB 2.0 port on the PC, rather than the external
> > hub...Virtually the same speed:
> >
> > david-vb:/home/mcgough/ham/RasPi2# ./dd_obs_test.sh /dev/sdf
> > block size : transfer rate
> > 512 : 2.4 MB/s
> > 1024 : 2.2 MB/s
> > 2048 : 2.2 MB/s
> > 4096 : 5.8 MB/s
> > 8192 : 5.8 MB/s
> > 16384 : 5.8 MB/s
> > 32768 : 5.8 MB/s
> > 65536 : 5.8 MB/s
> > 131072 : 5.8 MB/s
> > 262144 : 5.8 MB/s
> > 524288 : 5.8 MB/s
> > 1048576 : 5.8 MB/s
> > 2097152 : 5.4 MB/s
> > 4194304 : 5.8 MB/s
> > 8388608 : 5.8 MB/s
> > 16777216 : 5.4 MB/s
> > 33554432 : 5.3 MB/s
> > 67108864 : 5.4 MB/s
> >
> > Finally, the same test, using the same SD card reader, but now plugged
> > into a RPi2 USB port. Note that the RPi2 is running our latest & greatest
> > beta firmware with a 4.1.13 kernel. Also, the SD card block device is
> > now /dev/sda (it was /dev/sdf on the PC):
> >
> >
> > [root at RPi2-dev2 ~]# ./dd_obs_test.sh /dev/sda
> > block size : transfer rate
> > 512 : 2.3 MB/s
> > 1024 : 2.3 MB/s
> > 2048 : 2.4 MB/s
> > 4096 : 5.6 MB/s
> > 8192 : 5.6 MB/s
> > 16384 : 5.6 MB/s
> > 32768 : 5.7 MB/s
> > 65536 : 5.6 MB/s
> > 131072 : 5.6 MB/s
> > 262144 : 5.6 MB/s
> > 524288 : 5.6 MB/s
> > 1048576 : 5.6 MB/s
> > 2097152 : 5.6 MB/s
> > 4194304 : 5.6 MB/s
> > 8388608 : 5.6 MB/s
> > 16777216 : 5.6 MB/s
> > 33554432 : 5.6 MB/s
> > 67108864 : 5.5 MB/s
> >
> > Again, using the RPi2 also yielded the results I expected.
> >
> > So, on the positive side, using a 1 megabyte buffer (like I typically
> > tell people) is perfectly acceptable....But, a smaller buffer is fine
> > too--ANY buffer size at least 4K bytes seems to yield optimal thru-put.
> >
> > NOTE that another useful test would be to monitor CPU use while performing
> > each test and report this as well...I expect the larger buffer sizes will
> > win here, slightly, with quickly diminishing returns once past perhaps
> > 128K byte sizes....
> >
> > As with everything, your mileage may vary. I recommend trying this script
> > on -YOUR- system, too. And, please report the results!
> >
> > Here is the script:
> >
> > #------------------------------------------------------------------------------------
> > #!/bin/bash
> >
> > # Since we're dealing with dd, abort if any errors occur
> > set -e
> >
> > TEST_FILE=${1:-dd_obs_testfile}
> > [ -e "$TEST_FILE" ]; TEST_FILE_EXISTS=$?
> > ###TEST_FILE_SIZE=134217728
> > TEST_FILE_SIZE=67108864
> >
> > # Header
> > PRINTF_FORMAT="%8s : %s\n"
> > printf "$PRINTF_FORMAT" 'block size' 'transfer rate'
> >
> > # Block sizes of 512b 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M 8M 16M 32M 64M
> > for BLOCK_SIZE in 512 1024 2048 4096 8192 16384 32768 65536 131072 262144
> > 524288 1048576 2097152 4194304 8388608 16777216 33554432 67108864
> > do
> > # Calculate number of segments required to copy
> > COUNT=$(($TEST_FILE_SIZE / $BLOCK_SIZE))
> >
> > if [ $COUNT -le 0 ]; then
> > echo "Block size of $BLOCK_SIZE estimated to require $COUNT blocks, aborting further tests."
> > break
> > fi
> >
> > # Create a test file with the specified block size
> > DD_RESULT=$(dd if=/dev/zero of=$TEST_FILE bs=$BLOCK_SIZE count=$COUNT 2>&1 1>/dev/null)
> >
> > # Extract the transfer rate from dd's STDERR output
> > TRANSFER_RATE=$(echo $DD_RESULT | \grep --only-matching -E '[0-9.]+ ([MGk]?B|bytes)/s(ec)?')
> >
> > # Clean up the test file if we created one
> > [ $TEST_FILE_EXISTS -ne 0 ] && rm $TEST_FILE
> >
> > # Output the result
> > printf "$PRINTF_FORMAT" "$BLOCK_SIZE" "$TRANSFER_RATE"
> > done
> > #------------------------------------------------------------------------------------
> >
> >
> > Have fun!
> >
> > 73, David KB4FXC
> >
> > _______________________________________________
> >
> > arm-allstar mailing list
> > arm-allstar at hamvoip.org
> > http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar
> >
> > Visit the BBB and RPi2 web page - http://hamvoip.org
> >
>
>
More information about the arm-allstar
mailing list