EEPROM library on logging principle (from STM32)

[GeorgeBobrov]

Hello dbuezas!
On LGT8F328p, EEPROM is emulated using space from the main Flash memory. The emulation uses a technic of page swapping: each time we write to the emulated EEPROM, the device erases the new page, copies the old data from the old page and updates the new data on the fly on the new page, then it swaps the old and new page.
The STM32 also does not have an EEPROM and is also emulated using the main Flash memory space. However, it uses a much more economical logging principle in terms of the number of page erasures.
The principle is described in the document http://www.st.com/web/en/resource/technical/document/application_note/CD00165693.pdf.
In short: all changes are written to the log, and only when the log takes up a full page (1KB), the compressed log
is copied to the second page (only the latest actual values are copied).
The actual library code is here:
https://github.com/rogerclarkmelbourne/Arduino_STM32/tree/master/STM32F1/libraries/EEPROM
It would be nice if this principle was also ported to LGT8F328p (perhaps as a second version of the EEPROM library).
This will greatly reduce the number of Flash memory overwrites, which is already limited. In addition, such porting does not look too complicated.
What do you think about this? Do you want to try this task?

[SuperUserNameMan]

If i remember correctly, the page-swap algorithm of the LGT8 is implemented in hardware.
A software write-buffer / log could be implemented, but there is only 2k of RAM on the LGT8. That's not a lot of room for a log / buffer.

Though, you can implement a write-buffer and use the serial-write functions to write it all at once using void lgt_eeprom_writeSWM( uint16_t address, uint32_t *pData, uint16_t length ). (The buffer has to be 32 bits, because that's the actual size of the flash memory cells.)

[GeorgeBobrov]

I have read the documentation. Indeed, on LGT8F328p, the emulation of EEPROM uses a technic of page swapping at the hardware level. This mode is enabled in the ECCR register. The emulation mode is called E2PROM.
However, as far as I understand, there is a direct access mode to Flash memory, without E2PROM and page swapping, and this mode is used, for example, by the bootloader to update the firmware.
Perhaps, in this case, it is possible to implement the EEPROM emulation algorithm on raw Flash memory, on its free section, with E2PROM turned off, exactly as it is done in the library for STM32.

[SuperUserNameMan]

I don't remember if it is possible to write directly to the flash once outside the bootloader :-/
Is it ?

Also, I have not read the STM32 pdf you provided yet, because I don't understand the benefit of storing a log in RAM given the fact that the LGT8 only has 2kB of RAM :

• If the log requires more than 1kB of RAM to write a 1kB of Flash, there would be not enough RAM left free for other usage. :-/

What are the minimal requirements of this algorithm ?

[SuperUserNameMan]

PS : it is possible to write to the flash without triggering the swap using the SWM functions and macros (SWM = Serial Write Mode).
However, at the minimum, it requires to write 4 bytes aligned on 32bits (because cells of the flash are 32bits).

[SuperUserNameMan]

PS2 : also note that the hardware swap is not triggered if you write to a cell that contains 0xFFffFFff (it's blank).

[GeorgeBobrov]

The algorithm from STM does not use RAM. It creates a log directly in Flash memory, appending it to the same Flash page without rewriting. And only when the log fills one page, the algorithm rewrites the data to another page, but not all, only the last actual values. To understand the meaning, you still need to read the document.

The point of this is that the number of Flash page rewrites is reduced by a factor of thousands. However, there is a drawback - this algorithm is effective only when a relatively small amount of settings is stored in the emulated EEPROM, much less than the full size of the emulated EEPROM. However, that is what usually happens.

[SuperUserNameMan]

Maybe you could try implementing it ?

The SWM functions and macro I mentioned would allow you to write the log into blank pages without triggering hardware page swaps.

Given the fact that the cells of the flash are 32bits wide, they are large enough to store a tiny struct containing a value, an address and a flag for each log entry (if this is the approach you choose).

Also, keep in mind that the last 4 bytes of each 1k pages are reserved to the hardware page-swap algorithm.
Which mean, only the first 1020 bytes (or 255 cells) are available consecutively in each page.

[GeorgeBobrov]

Porting itself will not be difficult if it is feasible in principle. And whether it is feasible or not depends on one fact: whether or not the LGT8F328 has the ability to directly access Flash memory pages, including arbitrary writing and erasing. There is definitely such an opportunity in the bootloader, the question is whether we can do this from the user program. If so, then the sequence of actions could be as follows:

1. Disabling the hardware EEPROM emulation mechanism (you can disable it in the ECCR register)
2. Using the same pages of Flash memory at the same addresses that were used in the hardware EEPROM emulation mode, but we need to use direct access to this memory (the page swapping mechanism must be disabled, since it will not allow you to manually transfer data from one page to another)
3. All mechanisms for actually creating a log in Flash memory, saving data there, reading data from there, rewriting the log to another page when the current one is filled, have already been implemented in the library https://github.com/rogerclarkmelbourne/Arduino_STM32/tree/master/STM32F1/libraries/EEPROM in the EEPROM.cpp file
4. The EEPROM.cpp file is written in a fairly high-level style, it does not work directly with Flash memory, but calls functions from the flash_stm32.c file. There are located the low-level functions for working with Flash memory:
   FLASH_ErasePage, FLASH_ProgramHalfWord, FLASH_WaitForLastOperation, FLASH_GetStatus
5. In general, all the functions that need to be ported are described in the flash_stm32.h file, and there are only 6 of them.

The main thing, going back to the beginning, is the question whether direct access to Flash memory possible from a user program.
Unfortunately, I don't have enough low-level programming experience to answer this question.

[LaZsolt]

Basically pgm_read_byte() is the direct access.

But LGT8Fx capable read the FLASH like RAM from address 0x4000. The complier can't produce an optimal code when acessing this address range.

[dbuezas]

This PR may be relevant in this discussion: #261
Particularly because it may be incompatible with the ideas discussed in this issue.