OT: Flash Memory

On 16.4.17 16:54, Steve Pope wrote:
> Tauno Voipio  <tauno.voipio@notused.fi.invalid> wrote:
> 
>> On 16.4.17 03:18, Steve Pope wrote:
> 
>>> boB K7IQ  <boB@everett.wa> wrote:
> 
>>>> The typical way to extend the write-life of Flash memory uses "wear
>>>> leveling" techniques where data is written to different parts of the
>>>> flash rather than over-writing the already used locations.
> 
>>> How do you know you're doing this?
> 
>> The Flash memory sticks, cards and similar products contain
>> controllers which take care of the limited re-writability of
>> the base medium. The exact algorithms seem to be trade secrets
>> of the producers, but there is plenty of discussion on publications.
>> One starting point is to Google for 'flash file system'.
> 
> Thanks.  So it's based on available, but not necessarily
> complete or authoritative, information.
> 
> Steve


Kind of. Maybe the best ones are still hidden.

It seems to me that the main aim of wear leveling is to
minimize the number of erase cycles on each erase block.

-- 

-TV

Tauno Voipio  <tauno.voipio@notused.fi.invalid> wrote:

>On 16.4.17 16:54, Steve Pope wrote:

>> Tauno Voipio  <tauno.voipio@notused.fi.invalid> wrote:
>> 
>>> On 16.4.17 03:18, Steve Pope wrote:
>> 
>>>> boB K7IQ  <boB@everett.wa> wrote:
>> 
>>>>> The typical way to extend the write-life of Flash memory uses "wear
>>>>> leveling" techniques where data is written to different parts of the
>>>>> flash rather than over-writing the already used locations.
>> 
>>>> How do you know you're doing this?
>> 
>>> The Flash memory sticks, cards and similar products contain
>>> controllers which take care of the limited re-writability of
>>> the base medium. The exact algorithms seem to be trade secrets
>>> of the producers, but there is plenty of discussion on publications.
>>> One starting point is to Google for 'flash file system'.
>> 
>> Thanks.  So it's based on available, but not necessarily
>> complete or authoritative, information.
>> 
>> Steve
>
>
>Kind of. Maybe the best ones are still hidden.
>
>It seems to me that the main aim of wear leveling is to
>minimize the number of erase cycles on each erase block.

Yes, I would also want an algorithm to migrate the data.
And since the word "economically" appeared in the original
post, it could not be a greedy algorithm.

So .. I think it would be tricky to surround a black box
containing semi-known algorithms with additional algorithms
that further improve (or at least, allow one to state) the
reliability.

Steve

On Sun, 16 Apr 2017 15:12:20 +0000 (UTC), spope33@speedymail.org
(Steve Pope) wrote:

>Tauno Voipio  <tauno.voipio@notused.fi.invalid> wrote:
>
>>On 16.4.17 16:54, Steve Pope wrote:
>
>>> Tauno Voipio  <tauno.voipio@notused.fi.invalid> wrote:
>>> 
>>>> On 16.4.17 03:18, Steve Pope wrote:
>>> 
>>>>> boB K7IQ  <boB@everett.wa> wrote:
>>> 
>>>>>> The typical way to extend the write-life of Flash memory uses "wear
>>>>>> leveling" techniques where data is written to different parts of the
>>>>>> flash rather than over-writing the already used locations.
>>> 
>>>>> How do you know you're doing this?
>>> 
>>>> The Flash memory sticks, cards and similar products contain
>>>> controllers which take care of the limited re-writability of
>>>> the base medium. The exact algorithms seem to be trade secrets
>>>> of the producers, but there is plenty of discussion on publications.
>>>> One starting point is to Google for 'flash file system'.
>>> 
>>> Thanks.  So it's based on available, but not necessarily
>>> complete or authoritative, information.
>>> 
>>> Steve
>>
>>
>>Kind of. Maybe the best ones are still hidden.
>>
>>It seems to me that the main aim of wear leveling is to
>>minimize the number of erase cycles on each erase block.
>
>Yes, I would also want an algorithm to migrate the data.
>And since the word "economically" appeared in the original
>post, it could not be a greedy algorithm.
>
>So .. I think it would be tricky to surround a black box
>containing semi-known algorithms with additional algorithms
>that further improve (or at least, allow one to state) the
>reliability.
>
>Steve

This is complicated by the fact that there are many different types of
flash memory, and even many different types of NAND flash memory.
They each have their own characteristics and varying degrees of
susceptibility to erase cycle quantities, etc.   This is partly why
vendors of things like USB sticks use proprietary algorithms to suit
their own implmentations and technologies and make them transparent to
the user.

And it's a moving target, since the implementation technologies are
still constantly changing.

So what works on one may not work on another or just be more or less
necessary on another.



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

On Thursday, April 13, 2017 at 5:17:10 PM UTC-4, eric.j...@ieee.org wrote:
> On Thu, 13 Apr 2017 10:03:46 -0400, Randy Yates
> <randyy@garnerundergroundinc.com> wrote:
> 
> >Gentle comp.dsp People,
> 
> Crap, that leaves me out from the beginning.   ;)

me too.

> >I've been involved on several projects over the past few years that
> >utilize flash memory in microSD form, and several times they have
> >inquired about problems with lifetime of such cards. My employer
> >recently tasked me with investigating the alleged write lifetime issue.
> >
> >Frankly I was appalled at the numerous, significant issues I found in
> >current-day cards.
> 
> Can you share what you found?
> 
> >Does anyone know if there are any _economically feasible_ techniques or
> >approaches which either eliminate or greatly mitigate the write lifetime
> >issue? 
> >
> >From my analysis, it seems that the biggest factors are: 
> >
> >  1) The switch away from SLC memory (to MLC and TLC).
> >
> >  2) Unknown write amplification factors, arising in part from
> >  lack of knowledge as to whether or not any specific microSD
> >  controller implements effective WA algorithms. 
> >
> >  3) Unknown wear leveling algorithm for microSDs: static
> >  or just dynamic?
> >
> >  4) Abrupt power-down issues.
> >
> >Any insights, thoughts, etc. would be welcome.
> >-- 
> >Randy Yates, Embedded Firmware Developer
> >Garner Underground, Inc.
> >866-260-9040, x3901
> >http://www.garnerundergroundinc.com
> 
> I think Allan Herriman captured pretty much everything useful I would
> have said, except that all devices are not created equal even in
> consumer cards.   If you're looking to put chips on a board, there are
> decent options.
> 
> Also FWIW, I've got a bunch of Raspberry Pi 3s and CHIPs and
> Beaglebones that sit around here running a lot doing various things
> (satellite signal monitoring. ADS-B traffic collection, signal
> analysis, etc., etc.), and so far haven't had any failures that I've
> noticed that were attributable to FLASH failures.   The satellite
> monitors (there are now two of them) pretty much run 24/7 and have
> been fine.   That's a small sampling, but an anecdotal sample
> nonetheless, fwiw.
> 

so i know absolute zip about the state of the art, but i have actually thought a little about what basic concepts would make for a good file system for FLASH. my pedantic understanding is that changing "1" to "0" is of little cost, but changing a "0" back to "1" requires erasing the entire "page", which is what is costly.  now it would seem to me that a decent file system would write updated files similarly as a new file (or, if file fragmentation is allowed, to write each updated allocation block as a newly-allocated block and tie the blocks together in the FAT table) and the filing system would not erase pages until all of the FLASH was filled and previously used pages (and allocation blocks) had to be reused.  if unused bits are always initialize to "1", then it should be possible to always append to a file without erasing existing data.  the idea is to reduce the number of times the worst-case page of FLASH is erased.

so couldn't it be that because Eric's application does fewer file updating (other than appending, which consumes memory but does not erase pages) that Randy's case where they are worried about overdoing the page erasure limit?

interesting (not terribly) "off-topic" question, Randy.

r b-j

On Sun, 16 Apr 2017 20:10:22 -0700, robert bristow-johnson wrote:

> On Thursday, April 13, 2017 at 5:17:10 PM UTC-4, eric.j...@ieee.org
> wrote:
>> On Thu, 13 Apr 2017 10:03:46 -0400, Randy Yates
>> <randyy@garnerundergroundinc.com> wrote:
>> 
>> >Gentle comp.dsp People,
>> 
>> Crap, that leaves me out from the beginning.   ;)
> 
> me too.
> 
>> >I've been involved on several projects over the past few years that
>> >utilize flash memory in microSD form, and several times they have
>> >inquired about problems with lifetime of such cards. My employer
>> >recently tasked me with investigating the alleged write lifetime
>> >issue.
>> >
>> >Frankly I was appalled at the numerous, significant issues I found in
>> >current-day cards.
>> 
>> Can you share what you found?
>> 
>> >Does anyone know if there are any _economically feasible_ techniques
>> >or approaches which either eliminate or greatly mitigate the write
>> >lifetime issue?
>> >
>> >From my analysis, it seems that the biggest factors are:
>> >
>> >  1) The switch away from SLC memory (to MLC and TLC).
>> >
>> >  2) Unknown write amplification factors, arising in part from lack of
>> >  knowledge as to whether or not any specific microSD controller
>> >  implements effective WA algorithms.
>> >
>> >  3) Unknown wear leveling algorithm for microSDs: static or just
>> >  dynamic?
>> >
>> >  4) Abrupt power-down issues.
>> >
>> >Any insights, thoughts, etc. would be welcome.
>> >--
>> >Randy Yates, Embedded Firmware Developer Garner Underground, Inc.
>> >866-260-9040, x3901 http://www.garnerundergroundinc.com
>> 
>> I think Allan Herriman captured pretty much everything useful I would
>> have said, except that all devices are not created equal even in
>> consumer cards.   If you're looking to put chips on a board, there are
>> decent options.
>> 
>> Also FWIW, I've got a bunch of Raspberry Pi 3s and CHIPs and
>> Beaglebones that sit around here running a lot doing various things
>> (satellite signal monitoring. ADS-B traffic collection, signal
>> analysis, etc., etc.), and so far haven't had any failures that I've
>> noticed that were attributable to FLASH failures.   The satellite
>> monitors (there are now two of them) pretty much run 24/7 and have been
>> fine.   That's a small sampling, but an anecdotal sample nonetheless,
>> fwiw.
>> 
>> 
> so i know absolute zip about the state of the art, but i have actually
> thought a little about what basic concepts would make for a good file
> system for FLASH. my pedantic understanding is that changing "1" to "0"
> is of little cost, but changing a "0" back to "1" requires erasing the
> entire "page", which is what is costly.  [snip]

Today's nand Flash has a non-zero raw read error rate and requires ECC to 
work reliably in a system.  Typically each page has 64 bytes of OOB (Out 
Of Band) storage for each 2048 bytes of user data.  The OOB area is used 
for a number of things, and some of it is used for ECC for the 2k page 
and a subset of the OOB.
The datasheet for a nand Flash part will specify the size in terms of the 
number of valid blocks that give the specified retention given a certain 
number of erase / write cycles, with a particular level of ECC.  
Typically the ECC is required to correct 4, 8, 16 or 24 errored bits per 
2k page, and will be a BCH code.  Historically the ECC might have been RS 
and could correct 1 bit per page, but Flash parts that are specified with 
1 bit ECC are hard(er) to buy now.


So back to rbj's comment regarding the different costs of writing 0 and 
writing 1: parts of the OOB are not protected by the ECC and are meant to 
be used as flags by the filesystem.

OTOH, to change just 1 bit in the user data part of the page, you might 
need to allocate a new page for the changed data, and mark the old page 
as expired.


Regards,
Allan

Whether or not OT, this thread is both interesting and potentially valuable.

The last time I was involved at chip level the 8085 was recent and 
non-volatile semiconductor RAM was erased with a UV lamp.

Now I own a laptop having:
  1. old rotating disk replaced by a larger solid state disk
  2. a 512 GB SD card plugged in.
  3. a choice of *MANY* USB flash drives.

What online sources be recommended for one wishing to be an intelligent 
consumer? If relevant my OS is Debian Jessie.
TIA

eric.jacobsen@ieee.org writes:

> On Thu, 13 Apr 2017 10:03:46 -0400, Randy Yates
> <randyy@garnerundergroundinc.com> wrote:
>
>>Gentle comp.dsp People,
>
> Crap, that leaves me out from the beginning.   ;)

You get a free pass, Eric! :)

>>I've been involved on several projects over the past few years that
>>utilize flash memory in microSD form, and several times they have
>>inquired about problems with lifetime of such cards. My employer
>>recently tasked me with investigating the alleged write lifetime issue.
>>
>>Frankly I was appalled at the numerous, significant issues I found in
>>current-day cards.
>
> Can you share what you found?

Other than what I already wrote, there's not much to share. I wish I
were an expert - this seems like a really cool technology to get
involved with.

This paper pretty much eliminated any faith I might have in the cards,
unless you can avoid abrupt power-offs:

  https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf  

--Randy

PS: I apologize to everyone for my radio silence.

PSS: Thanks for the datapoints Eric on your systems.

>
>>Does anyone know if there are any _economically feasible_ techniques or
>>approaches which either eliminate or greatly mitigate the write lifetime
>>issue? 
>>
>>From my analysis, it seems that the biggest factors are: 
>>
>>  1) The switch away from SLC memory (to MLC and TLC).
>>
>>  2) Unknown write amplification factors, arising in part from
>>  lack of knowledge as to whether or not any specific microSD
>>  controller implements effective WA algorithms. 
>>
>>  3) Unknown wear leveling algorithm for microSDs: static
>>  or just dynamic?
>>
>>  4) Abrupt power-down issues.
>>
>>Any insights, thoughts, etc. would be welcome.
>>-- 
>>Randy Yates, Embedded Firmware Developer
>>Garner Underground, Inc.
>>866-260-9040, x3901
>>http://www.garnerundergroundinc.com
>
> I think Allan Herriman captured pretty much everything useful I would
> have said, except that all devices are not created equal even in
> consumer cards.   If you're looking to put chips on a board, there are
> decent options.
>
> Also FWIW, I've got a bunch of Raspberry Pi 3s and CHIPs and
> Beaglebones that sit around here running a lot doing various things
> (satellite signal monitoring. ADS-B traffic collection, signal
> analysis, etc., etc.), and so far haven't had any failures that I've
> noticed that were attributable to FLASH failures.   The satellite
> monitors (there are now two of them) pretty much run 24/7 and have
> been fine.   That's a small sampling, but an anecdotal sample
> nonetheless, fwiw.
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>

-- 
Randy Yates, Embedded Firmware Developer
Garner Underground, Inc.
866-260-9040, x3901
http://www.garnerundergroundinc.com

On Tuesday, April 18, 2017 at 10:40:36 AM UTC-4, Randy Yates wrote:
> 
> This paper pretty much eliminated any faith I might have in the cards,
> unless you can avoid abrupt power-offs:
> 
>   https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf  
> 

geez, what does this mean?  that we need to, with a large capacitor, keep the CPU and flash alive after the appliance is switched OFF so that, before the CPU dies, it can gracefully complete it's flash card operations and go into an idle state so that no flash operations occur as the power dies?  is that what they're saying?

r b-j

On Wed, 19 Apr 2017 01:35:11 -0700, robert bristow-johnson wrote:

> On Tuesday, April 18, 2017 at 10:40:36 AM UTC-4, Randy Yates wrote:
>> 
>> This paper pretty much eliminated any faith I might have in the cards,
>> unless you can avoid abrupt power-offs:
>> 
>>   https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf
>> 
>> 
> geez, what does this mean?  that we need to, with a large capacitor,
> keep the CPU and flash alive after the appliance is switched OFF so
> that, before the CPU dies, it can gracefully complete it's flash card
> operations and go into an idle state so that no flash operations occur
> as the power dies?  is that what they're saying?
> 
> r b-j


I'm typing this post on an old Dell laptop that contains an Intel 320-
series SSD which supposedly contains a capacitor bank for that very 
purpose.  Intel call it "Enhanced Power Loss Data Protection".

I decided to fork out the extra $ after an earlier SSD (from OCZ) bricked 
itself going into (or coming out of) standby.  I guess it's not such a 
good idea to store the internal SSD firmware in the nand Flash array.

Regards,
Allan

Allan Herriman <allanherriman@hotmail.com> writes:

> On Wed, 19 Apr 2017 01:35:11 -0700, robert bristow-johnson wrote:
>
>> On Tuesday, April 18, 2017 at 10:40:36 AM UTC-4, Randy Yates wrote:
>>> 
>>> This paper pretty much eliminated any faith I might have in the cards,
>>> unless you can avoid abrupt power-offs:
>>> 
>>>   https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf
>>> 
>>> 
>> geez, what does this mean?  that we need to, with a large capacitor,
>> keep the CPU and flash alive after the appliance is switched OFF so
>> that, before the CPU dies, it can gracefully complete it's flash card
>> operations and go into an idle state so that no flash operations occur
>> as the power dies?  is that what they're saying?

Robert: yeah, that's my take-away.

>> r b-j
>
>
> I'm typing this post on an old Dell laptop that contains an Intel 320-
> series SSD which supposedly contains a capacitor bank for that very 
> purpose.  Intel call it "Enhanced Power Loss Data Protection".
>
> I decided to fork out the extra $ after an earlier SSD (from OCZ) bricked 
> itself going into (or coming out of) standby.  I guess it's not such a 
> good idea to store the internal SSD firmware in the nand Flash array.

Allan, one thing I wanted to ask you is this: what would you use
to replace flash? There are no more 2732 UV EPROMS, no more PROMs,
etc. 

Even if you went to the trouble (and expense) to design in a bunch of
DRAM and load the NV stuff on power-up, what type of device would you
load it FROM?
-- 
Randy Yates, DSP/Embedded Firmware Developer
Digital Signal Labs
http://www.digitalsignallabs.com