OT: Flash Memory

Gentle comp.dsp People,

I've been involved on several projects over the past few years that
utilize flash memory in microSD form, and several times they have
inquired about problems with lifetime of such cards. My employer
recently tasked me with investigating the alleged write lifetime issue.

Frankly I was appalled at the numerous, significant issues I found in
current-day cards.

Does anyone know if there are any _economically feasible_ techniques or
approaches which either eliminate or greatly mitigate the write lifetime
issue? 

From my analysis, it seems that the biggest factors are: 

  1) The switch away from SLC memory (to MLC and TLC).

  2) Unknown write amplification factors, arising in part from
  lack of knowledge as to whether or not any specific microSD
  controller implements effective WA algorithms. 

  3) Unknown wear leveling algorithm for microSDs: static
  or just dynamic?

  4) Abrupt power-down issues.

Any insights, thoughts, etc. would be welcome.
-- 
Randy Yates, Embedded Firmware Developer
Garner Underground, Inc.
866-260-9040, x3901
http://www.garnerundergroundinc.com

On Thu, 13 Apr 2017 10:03:46 -0400, Randy Yates wrote:

> Gentle comp.dsp People,
> 
> I've been involved on several projects over the past few years that
> utilize flash memory in microSD form, and several times they have
> inquired about problems with lifetime of such cards. My employer
> recently tasked me with investigating the alleged write lifetime issue.
> 
> Frankly I was appalled at the numerous, significant issues I found in
> current-day cards.
> 
> Does anyone know if there are any _economically feasible_ techniques or
> approaches which either eliminate or greatly mitigate the write lifetime
> issue?
> 
> From my analysis, it seems that the biggest factors are:
> 
>   1) The switch away from SLC memory (to MLC and TLC).
> 
>   2) Unknown write amplification factors, arising in part from lack of
>   knowledge as to whether or not any specific microSD controller
>   implements effective WA algorithms.
> 
>   3) Unknown wear leveling algorithm for microSDs: static or just
>   dynamic?
> 
>   4) Abrupt power-down issues.
> 
> Any insights, thoughts, etc. would be welcome.


Random notes in no particular order:


Worse Flash retention and higher read error rates are the down side of 
Moore's law.  The smaller cells just don't work as well.
My products from a decade ago work better than my ones of today :(


MicroSD isn't really designed to be used for a file system for a serious 
computer, nor is it an archival medium.
Moral: don't use it for that!  Use a flash device that is designed for 
that purpose, e.g. raw nand (which means you control the ECC, write 
leveling and scrubbing) or eMMC.


The Raspberry Pi is probably the most popular small computer that uses 
microSD for file system storage.  Googling for "Raspberry Pi flash 
corruption" gives an awful lot of hits, some of which might give you some 
hints.


Better results may be obtained by overprovisioning.  Don't fill up your 
filesystems.  Write amplification increases as the free space is 
reduced.  Consider using a larger card and creating an unformatted 
partition just to act as spare space.


Secure your supply chain.  (This is harder than it sounds.)  Ensure you 
are using genuine parts rather than fakes.  You probably can't tell just 
by looking at the card.


There are "industrial" microSD cards, for about 10x the cost of a 
"normal" one.
E.g.
<https://www.digikey.com/product-detail/en/4d-systems-pty-ltd/USD-4GB-
INDUSTRIAL/1613-1117-ND/5725505>
I've never tried one, and I don't know if they are better in practice.


Not for microSD, but eMMC (which has a lot in common with microSD) can 
sometimes be put into "PSLC" or "pseudo SLC" mode.  This is done by 
setting a parameter in the on-chip controller, and it treats the MLC or 
TLC cells as single bit cells.  Basically it just writes (into, e.g. a MLC 
cell) the bit patterns "00" and "11" rather than the usual "00" "01" "10" 
and "11".
This is mostly done to improve the erase / write time rather than to 
improve the read reliability or retention, though it will do that as well.
Of course you lose at least half the card capacity, but that may be a 
good tradeoff.
In theory they could use soft decision ECC for even lower error rates, 
but I don't think the controllers do that in practice.


Something that I've considered (but not actually done) is change to a 
better file system.  We want the following features:
- better file integrity checks (e.g. full file CRC).
- Mirroring on a single device.
I believe that BTRFS supports both of those "out of the box" and doesn't 
have the high overhead of a file system such as ZFS.


Regards,
Allan

Randy Yates  <randyy@garnerundergroundinc.com> wrote:

>Does anyone know if there are any _economically feasible_ techniques or
>approaches which either eliminate or greatly mitigate the write lifetime
>issue? 

One variable is what sort of proprietary information you have
from the device manufacturers.  Otherwise it is difficult to
surround the devices with suitable FEC, bad sector mapping etc.
and really know the reliability one has obtained.

Steve

On Thu, 13 Apr 2017 10:03:46 -0400, Randy Yates
<randyy@garnerundergroundinc.com> wrote:

>Gentle comp.dsp People,

Crap, that leaves me out from the beginning.   ;)

>I've been involved on several projects over the past few years that
>utilize flash memory in microSD form, and several times they have
>inquired about problems with lifetime of such cards. My employer
>recently tasked me with investigating the alleged write lifetime issue.
>
>Frankly I was appalled at the numerous, significant issues I found in
>current-day cards.

Can you share what you found?

>Does anyone know if there are any _economically feasible_ techniques or
>approaches which either eliminate or greatly mitigate the write lifetime
>issue? 
>
>From my analysis, it seems that the biggest factors are: 
>
>  1) The switch away from SLC memory (to MLC and TLC).
>
>  2) Unknown write amplification factors, arising in part from
>  lack of knowledge as to whether or not any specific microSD
>  controller implements effective WA algorithms. 
>
>  3) Unknown wear leveling algorithm for microSDs: static
>  or just dynamic?
>
>  4) Abrupt power-down issues.
>
>Any insights, thoughts, etc. would be welcome.
>-- 
>Randy Yates, Embedded Firmware Developer
>Garner Underground, Inc.
>866-260-9040, x3901
>http://www.garnerundergroundinc.com

I think Allan Herriman captured pretty much everything useful I would
have said, except that all devices are not created equal even in
consumer cards.   If you're looking to put chips on a board, there are
decent options.

Also FWIW, I've got a bunch of Raspberry Pi 3s and CHIPs and
Beaglebones that sit around here running a lot doing various things
(satellite signal monitoring. ADS-B traffic collection, signal
analysis, etc., etc.), and so far haven't had any failures that I've
noticed that were attributable to FLASH failures.   The satellite
monitors (there are now two of them) pretty much run 24/7 and have
been fine.   That's a small sampling, but an anecdotal sample
nonetheless, fwiw.



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Randy Yates wrote:
> Gentle comp.dsp People,
>
> I've been involved on several projects over the past few years that
> utilize flash memory in microSD form, and several times they have
> inquired about problems with lifetime of such cards. My employer
> recently tasked me with investigating the alleged write lifetime issue.
>
> Frankly I was appalled at the numerous, significant issues I found in
> current-day cards.
>
> Does anyone know if there are any _economically feasible_ techniques or
> approaches which either eliminate or greatly mitigate the write lifetime
> issue?
>
> From my analysis, it seems that the biggest factors are:
>
>   1) The switch away from SLC memory (to MLC and TLC).
>
>   2) Unknown write amplification factors, arising in part from
>   lack of knowledge as to whether or not any specific microSD
>   controller implements effective WA algorithms.
>
>   3) Unknown wear leveling algorithm for microSDs: static
>   or just dynamic?
>
>   4) Abrupt power-down issues.
>
> Any insights, thoughts, etc. would be welcome.
>


Buy "industrial" SD cards. Which have that designaiton changes like the 
wind.

Expect to spend some money.

-- 
Les Cargill

On Thu, 13 Apr 2017 19:15:53 -0500, Les Cargill
<lcargill99@comcast.com> wrote:

>Randy Yates wrote:
>> Gentle comp.dsp People,
>>
>> I've been involved on several projects over the past few years that
>> utilize flash memory in microSD form, and several times they have
>> inquired about problems with lifetime of such cards. My employer
>> recently tasked me with investigating the alleged write lifetime issue.
>>
>> Frankly I was appalled at the numerous, significant issues I found in
>> current-day cards.
>>
>> Does anyone know if there are any _economically feasible_ techniques or
>> approaches which either eliminate or greatly mitigate the write lifetime
>> issue?
>>
>> From my analysis, it seems that the biggest factors are:
>>
>>   1) The switch away from SLC memory (to MLC and TLC).
>>
>>   2) Unknown write amplification factors, arising in part from
>>   lack of knowledge as to whether or not any specific microSD
>>   controller implements effective WA algorithms.
>>
>>   3) Unknown wear leveling algorithm for microSDs: static
>>   or just dynamic?
>>
>>   4) Abrupt power-down issues.
>>
>> Any insights, thoughts, etc. would be welcome.
>>
>
>
>Buy "industrial" SD cards. Which have that designaiton changes like the 
>wind.
>
>Expect to spend some money.


The typical way to extend the write-life of Flash memory uses "wear
leveling" techniques where data is written to different parts of the
flash rather than over-writing the already used locations.  After a
whiile, the older portions of memory are re-used again but this
significantly increases life time.

This, I think is done in the background so you don't have to think
about it but as far as flash hard drives go, I believe  they used
the unformatted area.  So, what you want to do is to use an SD card
that has lots more space than you will actually need, and keep it
unformatted.

That's my understanding.

boB Gudgel

boB K7IQ  <boB@everett.wa> wrote:

>The typical way to extend the write-life of Flash memory uses "wear
>leveling" techniques where data is written to different parts of the
>flash rather than over-writing the already used locations.  

How do you know you're doing this?  


Steve

On Sun, 16 Apr 2017 00:18:06 +0000 (UTC), spope33@speedymail.org
(Steve Pope) wrote:

>boB K7IQ  <boB@everett.wa> wrote:
>
>>The typical way to extend the write-life of Flash memory uses "wear
>>leveling" techniques where data is written to different parts of the
>>flash rather than over-writing the already used locations.  
>
>How do you know you're doing this?  
>
>
>Steve

For commercial Flash, I don't know.

Did do it for EEprom in an  embedded application long time ago though.

On 16.4.17 03:18, Steve Pope wrote:
> boB K7IQ  <boB@everett.wa> wrote:
> 
>> The typical way to extend the write-life of Flash memory uses "wear
>> leveling" techniques where data is written to different parts of the
>> flash rather than over-writing the already used locations.
> 
> How do you know you're doing this?
> 
> 
> Steve


The Flash memory sticks, cards and similar products contain
controllers which take care of the limited re-writability of
the base medium. The exact algorithms seem to be trade secrets
of the producers, but there is plenty of discussion on publications.
One starting point is to Google for 'flash file system'.

-- 

-TV

Tauno Voipio  <tauno.voipio@notused.fi.invalid> wrote:

>On 16.4.17 03:18, Steve Pope wrote:

>> boB K7IQ  <boB@everett.wa> wrote:

>>> The typical way to extend the write-life of Flash memory uses "wear
>>> leveling" techniques where data is written to different parts of the
>>> flash rather than over-writing the already used locations.

>> How do you know you're doing this?

>The Flash memory sticks, cards and similar products contain
>controllers which take care of the limited re-writability of
>the base medium. The exact algorithms seem to be trade secrets
>of the producers, but there is plenty of discussion on publications.
>One starting point is to Google for 'flash file system'.

Thanks.  So it's based on available, but not necessarily
complete or authoritative, information.

Steve