Add offset on file upload/download

[talfsaverone]

Hi,

Are you considering adding an offset to the FileUploader and file download?

We implemented this kind of feature with your library, in our project.

If you want to add this feature we can integrate it with you.

Best regards,
Tal

[philips77]

The FileManager already has an API to download or upload a packet from/to an offset:

Android-nRF-Connect-Device-Manager/mcumgr-core/src/main/java/io/runtime/mcumgr/managers/FsManager.java

Lines 169 to 175 in 52d62ba

	public void download(@NotNull String name, int offset,
	@NotNull McuMgrCallback<McuMgrFsDownloadResponse> callback) {
	HashMap<String, Object> payloadMap = new HashMap<>();
	payloadMap.put("name", name);
	payloadMap.put("off", offset);
	send(OP_READ, ID_FILE, payloadMap, SHORT_TIMEOUT, McuMgrFsDownloadResponse.class, callback);
	}

Android-nRF-Connect-Device-Manager/mcumgr-core/src/main/java/io/runtime/mcumgr/managers/FsManager.java

Lines 213 to 217 in 52d62ba

	public void upload(@NotNull String name, byte @NotNull [] data, int offset,
	@NotNull McuMgrCallback<McuMgrFsUploadResponse> callback) {
	HashMap<String, Object> payloadMap = buildUploadPayload(name, data, offset);
	send(OP_WRITE, ID_FILE, payloadMap, SHORT_TIMEOUT, McuMgrFsUploadResponse.class, callback);
	}

The FileUplader is more like a helper to upload a full file, not just parts.

How does your addition works and what are you using it for?

[talfsaverone]

We are uploading/Downloading a ~30MB file to/from our remote device at a slow rate (due to hardware limitations), so we need to continue the transfer of the file from the last offset.

I see from the documentation that the upload/download commands are for a single packet and not for the rest of the file from a specific offset as we need:

Android-nRF-Connect-Device-Manager/mcumgr-core/src/main/java/io/runtime/mcumgr/managers/FsManager.java

Lines 158 to 175 in 52d62ba

	/**
	* Read a packet of a file with given name from the specified offset from the device
	* (asynchronous).
	* <p>
	* Use {@link #fileDownload} to download the whole file asynchronously using one command.
	*
	* @param name the file name.
	* @param offset the offset, from which the chunk will be requested.
	* @param callback the asynchronous callback.
	* @see #fileDownload(String, DownloadCallback)
	*/
	public void download(@NotNull String name, int offset,
	@NotNull McuMgrCallback<McuMgrFsDownloadResponse> callback) {
	HashMap<String, Object> payloadMap = new HashMap<>();
	payloadMap.put("name", name);
	payloadMap.put("off", offset);
	send(OP_READ, ID_FILE, payloadMap, SHORT_TIMEOUT, McuMgrFsDownloadResponse.class, callback);
	}

Android-nRF-Connect-Device-Manager/mcumgr-core/src/main/java/io/runtime/mcumgr/managers/FsManager.java

Lines 197 to 217 in 52d62ba

	/**
	* Send a packet of given data from the specified offset to the device (asynchronous).
	* <p>
	* The chunk size is limited by the current MTU. If the current MTU set by
	* {@link #setUploadMtu(int)} is too large, the {@link InsufficientMtuException} error will be
	* thrown. Use {@link InsufficientMtuException#getMtu()} to get the current MTU and
	* pass it to {@link #setUploadMtu(int)} and try again.
	* <p>
	* Use {@link #fileUpload} to upload the whole file asynchronously using one command.
	*
	* @param name the file name.
	* @param data the file data.
	* @param offset the offset, from which the chunk will be sent.
	* @param callback the asynchronous callback.
	* @see #fileUpload(String, byte[], UploadCallback)
	*/
	public void upload(@NotNull String name, byte @NotNull [] data, int offset,
	@NotNull McuMgrCallback<McuMgrFsUploadResponse> callback) {
	HashMap<String, Object> payloadMap = buildUploadPayload(name, data, offset);
	send(OP_WRITE, ID_FILE, payloadMap, SHORT_TIMEOUT, McuMgrFsUploadResponse.class, callback);
	}

We added it to the FileUploader in a simple and clean implementation, and also to the fileUpload and fileDownload methods that are in the FsManager combined with the call to the status method before to get the current file offset.

[philips77]

OK, I get it.

However, I believe the current FileUploader may "resume" uploading just like the image uploader (they share logic):

1. send first packet with file name and offset 0
2. receive a notification with offset X
3. continue sending from offset X

Have a look here:

Android-nRF-Connect-Device-Manager/mcumgr-core/src/main/java/io/runtime/mcumgr/transfer/Uploader.kt

Line 104 in 52d62ba

next.send(newChunk(0))

The first chunk (offset 0) is sent as a single packet. It is only after a response is received when the uploader starts several parallel uploads of following chunks, but the next offset is known at that point.

You just need to make sure to reply with an offset equal to the length of "already received part" when you receive offset 0 of a file that you partially already have.

[talfsaverone]

Our use is to continue the upload on different days and connections, so we can't start sending the first packet from offset 0 every time. The same thing to downloading.

If it is something that you are willing to add to the library I can attach our code for the FileUploader, fileUpload and fileDownload methods.

[philips77]

Our use is to continue the upload on different days and connections, so we can't start sending the first packet from offset 0 every time. The same thing to downloading.

I understand, but that doesn't change anything. The packet with offset 0 is sent to get the actual offset to start sending from. For example, on day 0 the uploader sends 100 packets with offsets 0-9999. On the following day the uploader again starts by sending packet with offset 0, but the device replies with a notification with offset 99999, so the next packet sent from the uploader will have offset 10000 and upload will resume.
It is the device that keeps track of what it received so far, not the mobile app.

[talfsaverone]

From our experience with the fileUpload method of the FsManager it just overrides the file and starts uploading again from 0.

We first get the offset from the status method.

Our fix for the FileUploader is to add an offset for the constructor:

open class FileUploader(
    private val fsManager: FsManager,
    private val name: String,
    data: ByteArray,
    offset: Int = 0,
    windowCapacity: Int = 1,
    memoryAlignment: Int = 1
) : Uploader(
    data,
    windowCapacity,
    memoryAlignment,
    fsManager.mtu,
    fsManager.scheme,
    offset
) {

Then set the uploader's current offset to the passed offset:

abstract class Uploader(
    private val data: ByteArray,
    private val windowCapacity: Int,
    private val memoryAlignment: Int,
    internal var mtu: Int,
    private val protocol: McuMgrScheme,
    private var currentOffset: Int
) {

And send the initial chunk with the current offset:

next.send(newChunk(currentOffset))

We implemented similar things in the fileUpload and fileDownload methods.

[philips77]

From our experience with the fileUpload method of the FsManager it just overrides the file and starts uploading again from 0.

It depends on what offset it returns after receiving the packet with offset 0 again. It should check that it already has the file with this name and return the size instead of removing it and returning the size of the received packet.

[Yaniv-Benhaim]

What about downloading @philips77 i see that there is a new helper class for Uploading called FileUploader.kt
Without downloading and altering the source code for mcumgr-core is there a way to download from an offset previously
saved locally ?

[philips77]

Hmm.. I don't think there is. This is a good point.