Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subsys: net: download_client rework #17481

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

eivindj-nordic
Copy link
Contributor

@eivindj-nordic eivindj-nordic commented Sep 25, 2024

Work in progress. No need for thorough review yet.

  • Deprecate current download client
  • Download client restructuring:
    • Move socket features to a separate file.
    • Restructuring of socket functions.
    • Update to download client internal states.
    • Replace is_idle(), is_connecting, is_downloading(), is closed() and is_finished() with is_state(). This reduces the code size a bit.
    • Align download client variable name as dlc.
    • Add interface for transport (http, coap, ...). Goal is to make the library easier to maintain and ease the work to add more transports (mqtt, ...).
  • Use range requests for nRF91 TLS only
  • Use uri instead of separate host and filename as input parameters. This has ripple effects to other libraries.
  • Let application provide client buffer. This allows for multiple download clients with different buffer sizes.
  • Parse HTTP header line for line. This reduces the size requirement for the download client recv buffer.
  • Add new field for requesting new data to dlc.
  • Change TLS range override logic.

TODOs

  • Update libraries and samples.
  • Update tests.
  • Address changes suggested for CoAP
  • Migration guide
  • Fixes to align behavior with new (internal) tests.

@eivindj-nordic eivindj-nordic self-assigned this Sep 25, 2024
@github-actions github-actions bot added doc-required PR must not be merged without tech writer approval. changelog-entry-required Update changelog before merge. Remove label if entry is not needed or already added. labels Sep 25, 2024
@NordicBuilder
Copy link
Contributor

NordicBuilder commented Sep 25, 2024

CI Information

To view the history of this post, clich the 'edited' button above
Build number: 108

Inputs:

Sources:

sdk-nrf: PR head: 761cfb8d85fbd80059f100b174816fee76d2f4ff

more details

sdk-nrf:

PR head: 761cfb8d85fbd80059f100b174816fee76d2f4ff
merge base: 4a745e3c313c7e86a8f7643f9ff741d4ed69e1e3
target head (main): 155fd870faae7621b2825301f1b7cee6fb84f188
Diff

Github labels

Enabled Name Description
ci-disabled Disable the ci execution
ci-all-test Run all of ci, no test spec filtering will be done
ci-force-downstream Force execution of downstream even if twister fails
ci-run-twister Force run twister
ci-run-zephyr-twister Force run zephyr twister
List of changed files detected by CI (116)
CODEOWNERS
applications
│  ├── asset_tracker_v2
│  │  ├── boards
│  │  │  │ native_sim.conf
│  │  ├── doc
│  │  │  │ asset_tracker_v2_description.rst
│  │  ├── overlay-carrier.conf
│  │  │ prj.conf
│  ├── serial_lte_modem
│  │  ├── doc
│  │  │  │ slm_description.rst
│  │  ├── overlay-carrier.conf
│  │  ├── prj.conf
│  │  ├── src
│  │  │  │ slm_at_fota.c
doc
│  ├── nrf
│  │  ├── app_dev
│  │  │  ├── device_guides
│  │  │  │  ├── nrf91
│  │  │  │  │  │ nrf91_building.rst
│  │  ├── libraries
│  │  │  ├── bin
│  │  │  │  ├── lwm2m_carrier
│  │  │  │  │  ├── app_integration.rst
│  │  │  │  │  │ requirements.rst
│  │  │  ├── networking
│  │  │  │  ├── aws_fota.rst
│  │  │  │  ├── azure_fota.rst
│  │  │  │  ├── download_client.rst
│  │  │  │  ├── downloader.rst
│  │  │  │  ├── fota_download.rst
│  │  │  │  ├── nrf_cloud.rst
│  │  │  │  │ nrf_cloud_pgps.rst
│  │  ├── releases_and_maturity
│  │  │  ├── known_issues.rst
│  │  │  ├── migration
│  │  │  │  │ migration_guide_2.9.rst
│  │  │  ├── releases
│  │  │  │  ├── release-notes-2.4.0.rst
│  │  │  │  │ release-notes-changelog.rst
include
│  ├── net
│  │  ├── download_client.h
│  │  ├── downloader.h
│  │  ├── downloader_transport.h
│  │  ├── fota_download.h
│  │  │ nrf_cloud_pgps.h
lib
│  ├── bin
│  │  ├── lwm2m_carrier
│  │  │  ├── Kconfig
│  │  │  ├── include
│  │  │  │  │ lwm2m_os.h
│  │  │  ├── os
│  │  │  │  │ lwm2m_os.c
samples
│  ├── cellular
│  │  ├── http_update
│  │  │  ├── application_update
│  │  │  │  ├── overlay-carrier.conf
│  │  │  │  ├── prj.conf
│  │  │  │  ├── src
│  │  │  │  │  │ main.c
│  │  │  ├── modem_delta_update
│  │  │  │  ├── prj.conf
│  │  │  │  ├── src
│  │  │  │  │  │ main.c
│  │  │  ├── modem_full_update
│  │  │  │  ├── prj.conf
│  │  │  │  ├── src
│  │  │  │  │  │ main.c
│  │  ├── location
│  │  │  │ overlay-pgps.conf
│  │  ├── lwm2m_carrier
│  │  │  │ prj.conf
│  │  ├── lwm2m_client
│  │  │  ├── prj.conf
│  │  │  │ sample_description.rst
│  │  ├── modem_shell
│  │  │  ├── overlay-carrier.conf
│  │  │  ├── overlay-modem_fota_full.conf
│  │  │  ├── prj.conf
│  │  │  ├── src
│  │  │  │  ├── fota
│  │  │  │  │  │ fota_shell.c
│  │  │  │  ├── gnss
│  │  │  │  │  │ gnss.c
│  │  ├── nrf_cloud_multi_service
│  │  │  ├── Kconfig
│  │  │  │ prj.conf
│  │  ├── nrf_cloud_rest_fota
│  │  │  │ prj.conf
│  ├── net
│  │  ├── aws_iot
│  │  │  ├── boards
│  │  │  │  ├── nrf7002dk_nrf5340_cpuapp_ns.conf
│  │  │  │  ├── nrf9151dk_nrf9151_ns.conf
│  │  │  │  ├── nrf9160dk_nrf9160_ns.conf
│  │  │  │  ├── nrf9161dk_nrf9161_ns.conf
│  │  │  │  ├── thingy91_nrf9160_ns.conf
│  │  │  │  │ thingy91x_nrf9151_ns.conf
│  │  ├── azure_iot_hub
│  │  │  ├── README.rst
│  │  │  ├── boards
│  │  │  │  ├── nrf7002dk_nrf5340_cpuapp_ns.conf
│  │  │  │  ├── nrf9151dk_nrf9151_ns.conf
│  │  │  │  ├── nrf9160dk_nrf9160_ns.conf
│  │  │  │  │ nrf9161dk_nrf9161_ns.conf
│  │  ├── download
│  │  │  ├── README.rst
│  │  │  ├── prj.conf
│  │  │  ├── sample.yaml
│  │  │  ├── src
│  │  │  │  │ main.c
scripts
│  │ quarantine_integration.yaml
subsys
│  ├── dfu
│  │  ├── dfu_target
│  │  │  │ Kconfig
│  ├── net
│  │  ├── lib
│  │  │  ├── CMakeLists.txt
│  │  │  ├── Kconfig
│  │  │  ├── aws_fota
│  │  │  │  ├── src
│  │  │  │  │  │ aws_fota.c
│  │  │  ├── azure_fota
│  │  │  │  │ azure_fota.c
│  │  │  ├── download_client
│  │  │  │  ├── Kconfig
│  │  │  │  ├── src
│  │  │  │  │  │ download_client.c
│  │  │  ├── downloader
│  │  │  │  ├── CMakeLists.txt
│  │  │  │  ├── Kconfig
│  │  │  │  ├── dl_transports.ld
│  │  │  │  ├── include
│  │  │  │  │  ├── dl_parse.h
│  │  │  │  │  │ dl_socket.h
│  │  │  │  ├── src
│  │  │  │  │  ├── dl_parse.c
│  │  │  │  │  ├── dl_socket.c
│  │  │  │  │  ├── downloader.c
│  │  │  │  │  ├── sanity.c
│  │  │  │  │  ├── shell.c
│  │  │  │  │  ├── transports
│  │  │  │  │  │  ├── coap.c
│  │  │  │  │  │  │ http.c
│  │  │  ├── fota_download
│  │  │  │  ├── CMakeLists.txt
│  │  │  │  ├── Kconfig
│  │  │  │  ├── src
│  │  │  │  │  ├── fota_download.c
│  │  │  │  │  ├── util
│  │  │  │  │  │  ├── fota_download_delta_modem.c
│  │  │  │  │  │  ├── fota_download_full_modem.c
│  │  │  │  │  │  │ fota_download_util.c
│  │  │  ├── mcumgr_smp_client
│  │  │  │  ├── Kconfig
│  │  │  │  ├── src
│  │  │  │  │  │ mcumgr_smp_client_shell.c
│  │  │  ├── nrf_cloud
│  │  │  │  ├── Kconfig.nrf_cloud_fota
│  │  │  │  ├── Kconfig.nrf_cloud_pgps
│  │  │  │  ├── include
│  │  │  │  │  ├── nrf_cloud_download.h
│  │  │  │  │  │ nrf_cloud_pgps_utils.h
│  │  │  │  ├── src
│  │  │  │  │  ├── nrf_cloud_codec_internal.c
│  │  │  │  │  ├── nrf_cloud_download.c
│  │  │  │  │  ├── nrf_cloud_fota.c
│  │  │  │  │  ├── nrf_cloud_fota_poll.c
│  │  │  │  │  ├── nrf_cloud_pgps.c
│  │  │  │  │  │ nrf_cloud_pgps_utils.c
tests
│  ├── subsys
│  │  ├── net
│  │  │  ├── lib
│  │  │  │  ├── aws_fota
│  │  │  │  │  ├── aws_fota_json
│  │  │  │  │  │  │ CMakeLists.txt
│  │  │  │  ├── downloader
│  │  │  │  │  ├── CMakeLists.txt
│  │  │  │  │  ├── boards
│  │  │  │  │  │  ├── native_sim.conf
│  │  │  │  │  │  │ qemu_cortex_m3.conf
│  │  │  │  │  ├── prj.conf
│  │  │  │  │  ├── src
│  │  │  │  │  │  │ main.c
│  │  │  │  │  │ testcase.yaml
│  │  │  │  ├── fota_download
│  │  │  │  │  ├── CMakeLists.txt
│  │  │  │  │  ├── src
│  │  │  │  │  │  │ test_fota_download.c
│  │  │  │  ├── lwm2m_client_utils
│  │  │  │  │  │ CMakeLists.txt
│  │  │  │  ├── lwm2m_fota_utils
│  │  │  │  │  │ CMakeLists.txt
│  │  │  │  ├── mcumgr_smp_client
│  │  │  │  │  │ CMakeLists.txt
│  │  │  │  ├── nrf_cloud
│  │  │  │  │  ├── cloud
│  │  │  │  │  │  │ CMakeLists.txt

Outputs:

Toolchain

Version:
Build docker image:

Test Spec & Results: ✅ Success; ❌ Failure; 🟠 Queued; 🟡 Progress; ◻️ Skipped; ⚠️ Quarantine

  • 🟠 Toolchain
  • 🟠 Build twister
  • 🟠 Integration tests
    • 🟠 test-fw-nrfconnect-boot
    • 🟠 test-fw-nrfconnect-chip
    • 🟠 test-fw-nrfconnect-nrf-iot_serial_lte_modem
    • 🟠 test-fw-nrfconnect-nrf-iot_zephyr_lwm2m
    • 🟠 test-fw-nrfconnect-nrf-iot_samples
    • 🟠 test-fw-nrfconnect-nrf-iot_lwm2m
    • 🟠 test-fw-nrfconnect-nrf-iot_thingy91
    • 🟠 test-fw-nrfconnect-zigbee
    • 🟠 test-sdk-find-my
    • 🟠 test-fw-nrfconnect-nrf-iot_mosh
    • 🟠 test-fw-nrfconnect-nrf-iot_positioning
    • 🟠 test-sdk-sidewalk
    • 🟠 test-sdk-mcuboot
    • ⚠️ test-fw-nrfconnect-fw-update
    • ⚠️ test-fw-nrfconnect-nrf-iot_cloud
Disabled integration tests
    • desktop52_verification
    • doc-internal
    • test_ble_nrf_config
    • test-fw-nrfconnect-apps
    • test-fw-nrfconnect-ble_mesh
    • test-fw-nrfconnect-ble_samples
    • test-fw-nrfconnect-fem
    • test-fw-nrfconnect-nfc
    • test-fw-nrfconnect-nrf-iot_libmodem-nrf
    • test-fw-nrfconnect-nrf-iot_nrf_provisioning
    • test-fw-nrfconnect-nrf_crypto
    • test-fw-nrfconnect-ps
    • test-fw-nrfconnect-rpc
    • test-fw-nrfconnect-rs
    • test-fw-nrfconnect-tfm
    • test-fw-nrfconnect-thread
    • test-low-level
    • test-sdk-audio
    • test-sdk-dfu
    • test-sdk-pmic-samples
    • test-sdk-wifi
    • test-secdom-samples-public

Note: This message is automatically posted and updated by the CI

Comment on lines 133 to 137
struct download_client_host_cfg {
/** Server hosting the file, null-terminated.
* The host name must be kept in scope while download is going on.
*/
const char *hostname;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you are so heavily refactoring, I would propose that we get rid of this separation of host and path.
Just use URI on the API.
It is actually very rare that any interfaces would give us host and path separately. I only know nRF Cloud to do so on some REST APIs. It is more common that we receive full URI where to fetch the data.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh.. and also now host is provided inside a struct and path is provided as a function parameter. This emphasizes the separation even further.

It feels like cleanest API would just take URI as one parameter, or field inside this struct. Let the HTTP parser to split protocol, host, port and path.

@github-actions github-actions bot removed the changelog-entry-required Update changelog before merge. Remove label if entry is not needed or already added. label Oct 17, 2024
@eivindj-nordic eivindj-nordic force-pushed the 25700_download_client_rework branch 6 times, most recently from 6573c1d to 2666f2d Compare October 24, 2024 13:07
@eivindj-nordic eivindj-nordic force-pushed the 25700_download_client_rework branch 9 times, most recently from 272b2de to c058985 Compare November 1, 2024 09:41
@eivindj-nordic
Copy link
Contributor Author

Marking ready for review to trigger CI.

@eivindj-nordic eivindj-nordic marked this pull request as ready for review November 1, 2024 12:21
@eivindj-nordic eivindj-nordic requested a review from a team as a code owner November 1, 2024 12:21
Comment on lines 106 to 108
.. code-block:: C
#include <net/download_client.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.. code-block:: C
#include <net/download_client.h>
.. code-block:: C
#include <net/download_client.h>

Comment on lines 112 to 114
.. code-block:: C
#include <net/downloader.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.. code-block:: C
#include <net/downloader.h>
.. code-block:: C
#include <net/downloader.h>

Comment on lines 118 to 125
* Remove:

.. code-block:: C
static struct download_client dlc;
static int callback(const struct download_client_evt *event);
download_client_init(&dlc, callback)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Remove:
.. code-block:: C
static struct download_client dlc;
static int callback(const struct download_client_evt *event);
download_client_init(&dlc, callback)
* Remove:
.. code-block:: C
static struct download_client dlc;
static int callback(const struct download_client_evt *event);
download_client_init(&dlc, callback)

Comment on lines 172 to 183
* Remove:

.. code-block:: C
int err;
const struct download_client_cfg dlc_config = {
...
};
err = download_client_set_host(&dlc, dl_host, &dlc_config);
err = download_client_start(&dlc, dl_file, offset);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Remove:
.. code-block:: C
int err;
const struct download_client_cfg dlc_config = {
...
};
err = download_client_set_host(&dlc, dl_host, &dlc_config);
err = download_client_start(&dlc, dl_file, offset);
* Remove:
.. code-block:: C
int err;
const struct download_client_cfg dlc_config = {
...
};
err = download_client_set_host(&dlc, dl_host, &dlc_config);
err = download_client_start(&dlc, dl_file, offset);

Comment on lines 185 to 205
* Add:

.. code-block:: C
/* Note: All configuration of the downloader is done through the config structs.
* The downloader struct should not be modified by the application.
*/
static struct downloader_host_cfg host_config = {
...
/* Note:
* .frag_size_override is replaced by .range_override.
* .set_tls_hostname is replaced by .set_native_tls.
* dlc.close_when_done is moved here and inverted(.keep_connection).
* Set .cid if CONFIG_DOWNLOAD_CLIENT_CID was enabled in download client.
*/
};
int err = downloader_get_with_host_and_path(&dl, &host_config, dl_host, dl_file, offset);
.. note:: The new downloader has an API to download the file using the URI directly.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Add:
.. code-block:: C
/* Note: All configuration of the downloader is done through the config structs.
* The downloader struct should not be modified by the application.
*/
static struct downloader_host_cfg host_config = {
...
/* Note:
* .frag_size_override is replaced by .range_override.
* .set_tls_hostname is replaced by .set_native_tls.
* dlc.close_when_done is moved here and inverted(.keep_connection).
* Set .cid if CONFIG_DOWNLOAD_CLIENT_CID was enabled in download client.
*/
};
int err = downloader_get_with_host_and_path(&dl, &host_config, dl_host, dl_file, offset);
.. note:: The new downloader has an API to download the file using the URI directly.
* Add:
.. code-block:: C
/* Note: All configuration of the downloader is done through the config structs.
* The downloader struct should not be modified by the application.
*/
static struct downloader_host_cfg host_config = {
...
/* Note:
* .frag_size_override is replaced by .range_override.
* .set_tls_hostname is replaced by .set_native_tls.
* dlc.close_when_done is moved here and inverted(.keep_connection).
* Set .cid if CONFIG_DOWNLOAD_CLIENT_CID was enabled in download client.
*/
};
int err = downloader_get_with_host_and_path(&dl, &host_config, dl_host, dl_file, offset);
.. note::
The new downloader has an API to download the file using the URI directly.

Comment on lines 212 to 216
* Add:

.. code-block:: C
err = downloader_deinit(&dl);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Add:
.. code-block:: C
err = downloader_deinit(&dl);
* Add:
.. code-block:: C
err = downloader_deinit(&dl);

@@ -3512,7 +3512,7 @@ Jobs not received after reset

.. rst-class:: v2-6-2 v2-6-1 v2-6-0 v2-5-3 v2-5-2 v2-5-1 v2-5-0

NCSDK-24305: fota_download library sends FOTA_DOWNLOAD_EVT_FINISHED when unable to connect
NCSDK-24305: fota_download and fota_download libraries sends FOTA_DOWNLOAD_EVT_FINISHED and FOTA_DOWNLOAD_EVT_FINISHED events, respectively, when unable to connect
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 2 entries for the same library and same event? Isn't this about the FOTA download library that is not changed as part of the PR?

@@ -3521,9 +3521,9 @@ NCSDK-24305: fota_download library sends FOTA_DOWNLOAD_EVT_FINISHED when unable
.. rst-class:: v1-1-0

Stalled download
:ref:`lib_fota_download` does not resume a download if the device loses the connection.
:ref:`lib_fota_download` and :ref:`lib_fota_download` does not resume a download if the device loses the connection.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 2 entries for the same library? Isn't this about the FOTA download library that is not changed as part of the PR?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, I missed to review this file during the initial review. Also, line nos: 3501, 3502, and 3516 would require an update to :ref:lib_downloader, which mentions :ref:lib_download_client.


**Workaround:** Call :cpp:func:`fota_download_start` again with the same arguments when the connection is re-established to resume the download.
**Workaround:** Call :cpp:func:`fota_download_start` or :cpp:func:`fota_download_start` again with the same arguments when the connection is re-established to resume the download.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate entry for the same function

doc/nrf/releases_and_maturity/known_issues.rst Outdated Show resolved Hide resolved
doc/nrf/releases_and_maturity/known_issues.rst Outdated Show resolved Hide resolved
@eivindj-nordic eivindj-nordic force-pushed the 25700_download_client_rework branch 6 times, most recently from aee34a9 to 8f2d185 Compare November 25, 2024 12:36
@eivindj-nordic eivindj-nordic force-pushed the 25700_download_client_rework branch 3 times, most recently from ad19a95 to 8441121 Compare November 26, 2024 11:57
Copy link
Contributor

@MirkoCovizzi MirkoCovizzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good job, Eivind. First round of comments, a lot more to come.


#include "dl_parse.h"

static int swallow(const char **str, const char *swallow)
Copy link
Contributor

@MirkoCovizzi MirkoCovizzi Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest trim (or strip), I had to go into the code to see what it actually does because it wasn't obvious from the name. Also, generally not recommended to have names that shadow each other (the function swallow and the parameter swallow in the same scope) even though the compiler can figure it out.

Suggested change
static int swallow(const char **str, const char *swallow)
static int trim(const char **str, const char *substr)

{
switch (state) {
case DOWNLOADER_IDLE:
return "IDLE";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could use the Zephyr macro STRINGIFY here instead of hardcoding a string, same for the other ones.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, though I would like to avoid logging the full name. For some logs there are multiple states in the same line, so it will be quite long.

}
}

int downloader_init(struct downloader *const dl,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int downloader_init(struct downloader *const dl,
int downloader_init(struct downloader *const dl, struct downloader_cfg *config)

/* The thread is spawned now, but it will suspend itself;
* it is resumed when the download is started via the API.
*/
dl->tid =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strange line formatting here.

return 0;
}

int downloader_deinit(struct downloader *const dl)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if we pass an uninitialized downloader?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I've added some checks for the state, event_sem and thread to be set.

int downloader_cancel(struct downloader *const dl)
{
if (dl == NULL ||
is_state(dl, DOWNLOADER_IDLE) ||
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to be explicit about which states are permitted rather than the opposite. From looking at the code here I wouldn't know which states are permitted and would have to see the list of all states and eliminate the ones here. Also, this permits the state DOWNLOADER_STOPPING to be canceled, which seems strange. If it's on purpose, why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

{
int rc;

if (dl == NULL || uri == NULL) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

host_config not checked. Yes, it's checked later in downloader_start but now you have coupling between the two functions.

return -EINVAL;
}

k_mutex_lock(&dl->mutex, K_FOREVER);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On one hand host_config is not checked because it's done in downloader_start, but then the same should apply to the mutex lock: there is no need to lock here because it is locked inside downloader_start. I think that this lock/unlock can be removed.

return rc;
}

int downloader_get_with_host_and_path(struct downloader *dl,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

host_and_file perhaps since the other argument is file?

return -EINVAL;
}

sprintf(dl->config.buf, "%s/%s", host, file);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer the more secure snprintf.

@@ -56,7 +55,9 @@ SHELL_STATIC_SUBCMD_SET_CREATE(
sub_fota,
SHELL_CMD_ARG(
download, NULL,
"<server> <filename>\nDownload and install a FOTA update. Available servers are \"eu\", \"us\", \"jpn\" and \"au\".",
"[<server> <filename>]\nDownload and install a FOTA update. "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"[<server> <filename>]\nDownload and install a FOTA update. "
"<server> <filename>\nDownload and install a FOTA update. "

@eivindj-nordic eivindj-nordic force-pushed the 25700_download_client_rework branch 3 times, most recently from 53bde3d to 9b9b914 Compare November 27, 2024 08:02
@@ -3520,7 +3520,7 @@ Jobs not received after reset
.. rst-class:: v2-6-2 v2-6-1 v2-6-0 v2-5-3 v2-5-2 v2-5-1 v2-5-0

NCSDK-24305: fota_download library sends FOTA_DOWNLOAD_EVT_FINISHED when unable to connect
The :ref:`lib_download_client` library does not resume a download if the device cannot connect to a target server.
The :ref:`lib_download_client` and :ref:`lib_downloader` libraries does not resume a download if the device cannot connect to a target server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The :ref:`lib_download_client` and :ref:`lib_downloader` libraries does not resume a download if the device cannot connect to a target server.
The :ref:`lib_download_client` and :ref:`lib_downloader` libraries do not resume a download if the device cannot connect to a target server.

@@ -38,7 +38,7 @@ HTTPS downloads
The FOTA download library is used in the :ref:`http_application_update_sample` sample.

By default, the FOTA download library uses HTTP for downloading the firmware file.
To use HTTPS, apply the changes described in :ref:`the HTTPS section of the download client documentation <download_client_https>` to the library.
To use HTTPS, apply the changes described in :ref:`the HTTPS section of the download client documentation <downloader_https>` to the library.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To use HTTPS, apply the changes described in :ref:`the HTTPS section of the download client documentation <downloader_https>` to the library.
To use HTTPS, apply the changes described in :ref:`the HTTPS section of the Downloader library documentation <downloader_https>` to the library.

.. note::
The new downloader has an API to download the file using the URI directly.

#. [optional] Deinitlaize the downloader after use:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#. [optional] Deinitlaize the downloader after use:
#. [optional] Deinitialize the downloader after use:

@@ -3520,7 +3520,7 @@ Jobs not received after reset
.. rst-class:: v2-6-2 v2-6-1 v2-6-0 v2-5-3 v2-5-2 v2-5-1 v2-5-0

NCSDK-24305: fota_download library sends FOTA_DOWNLOAD_EVT_FINISHED when unable to connect
The :ref:`lib_download_client` library does not resume a download if the device cannot connect to a target server.
The :ref:`lib_download_client` and :ref:`lib_downloader` libraries does not resume a download if the device cannot connect to a target server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The :ref:`lib_download_client` and :ref:`lib_downloader` libraries does not resume a download if the device cannot connect to a target server.
The :ref:`lib_download_client` and :ref:`lib_downloader` libraries do not resume a download if the device cannot connect to a target server.

@@ -3505,10 +3505,10 @@ NCSDK-11432: DFU: Erasing secondary slot returns error response
.. rst-class:: v1-7-1 v1-7-0 v1-6-1 v1-6-0 v1-5-2 v1-5-1 v1-5-0 v1-4-2 v1-4-1 v1-4-0

NCSDK-6238: Socket API calls might hang when using Download client
When using the :ref:`lib_download_client` library with HTTP (without TLS), the application might not process incoming fragments fast enough, which can starve the :ref:`nrfxlib:nrf_modem` buffers and make calls to the Modem library hang.
When using the :ref:`lib_download_client` and :ref:`lib_downloader` libraries with HTTP (without TLS), the application might not process incoming fragments fast enough, which can starve the :ref:`nrfxlib:nrf_modem` buffers and make calls to the Modem library hang.
Samples and applications that are affected include those that use :ref:`lib_download_client` to download files through HTTP, or those that use :ref:`lib_fota_download` with modem updates enabled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Samples and applications that are affected include those that use :ref:`lib_download_client` to download files through HTTP, or those that use :ref:`lib_fota_download` with modem updates enabled.
Samples and applications that are affected include those that use :ref:`lib_download_client` and :ref:`lib_downloader` libraries to download files through HTTP, or those that use :ref:`lib_fota_download` with modem updates enabled.

Samples and applications that are affected include those that use :ref:`lib_download_client` to download files through HTTP, or those that use :ref:`lib_fota_download` with modem updates enabled.

**Workaround:** Set :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS`.
**Workaround:** Set :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS` with the :ref:`lib_download_client` library. Set ``range_override`` in :c:struct:`downloader_host_cfg` with the :ref:`lib_downloader` library.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Workaround:** Set :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS` with the :ref:`lib_download_client` library. Set ``range_override`` in :c:struct:`downloader_host_cfg` with the :ref:`lib_downloader` library.
**Workaround:** Set :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_RANGE_REQUESTS` for the :ref:`lib_download_client` library.
Set ``range_override`` in :c:struct:`downloader_host_cfg` for the :ref:`lib_downloader` library.

REQUIRED = 2,
};

verify = REQUIRED;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The size of enum is system dependent, I would stick to int which is the type expected by setsockopt.

Instead use:

int verify = TLS_PEER_VERIFY_REQUIRED

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, updated.

return 0;
}

static int socket_tls_hostname_set(int fd, const char * const host)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
static int socket_tls_hostname_set(int fd, const char * const host)
static int socket_tls_hostname_set(int fd, const char * const hostname)

For consistency since there are other functions that have a hostname parameter named hostname.

if (err) {
err = -errno;
/* Make sure that ECONNRESET is not returned as it has a special meaning
* in the download client API
Copy link
Contributor

@MirkoCovizzi MirkoCovizzi Nov 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps?

Suggested change
* in the download client API
* in the downloader API

return -EINVAL;
}

timeo.tv_sec = (timeout_ms / 1000);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is shared with send_timeout_set, maybe it can be abstracted?

*/

BUILD_ASSERT(
STACK_SIZE - (HOSTNAME_SIZE + FILENAME_SIZE) >= 512,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this 512 defined somewhere?

}

static int cmd_host_config_sec_tag(const struct shell *shell, size_t argc,
char **argv)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting

}

static int cmd_host_config_pdn_id(const struct shell *shell, size_t argc,
char **argv)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting

SHELL_STATIC_SUBCMD_SET_CREATE(host_config_options,
SHELL_CMD(sec_tag, NULL, "Set security tag", cmd_host_config_sec_tag),
SHELL_CMD(pdn_id, NULL, "Set PDN ID", cmd_host_config_pdn_id),
SHELL_CMD(sec_tag, NULL, "Set security tag", cmd_host_config_sec_tag),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears twice


static int dl_callback(const struct downloader_evt *event);

static char dl_buf[2048];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on the 2048 here too like in transports/http.c?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not the same.

http->sock.proto = IPPROTO_TCP;
http->sock.type = SOCK_STREAM;

if (strncmp(uri, "https://", 8) == 0 ||
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strlen


static bool dl_http_proto_supported(struct downloader *dl, const char *uri)
{
if (strncmp(uri, "https://", 8) == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two strings and their length are used in multiple places, I would define them at the top and use strlen.

* Length of data payload left to process on success
* Negative errno on error.
*/
static int http_parse(struct downloader *dl, size_t len)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you checked if there is already an http parser in Zephyr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is, I'll leave it as a future task to see if that can be used.
https://docs.zephyrproject.org/apidoc/latest/parser_8h.html

return -EBADMSG;
}

expected_status = (http->ranged || dl->progress) ? 206 : 200;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two http status codes can be defined too, mostly for their name/use. For example, I know from the top of my head that 200 is success, but don't remember 206.

if (err) {
switch (http->sock.proto) {
case IPPROTO_TLS_1_2:
http->sock.port = 443;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would define these two too

@eivindj-nordic eivindj-nordic force-pushed the 25700_download_client_rework branch 3 times, most recently from 12781ef to 2db381f Compare November 27, 2024 09:51
The new downloader is replacing the download_client library and is
based on that.

Internal restructuring:

* Restructuring of socket functions and files.
* Parse HTTP header line for line. This reduces the size requirement
  for the download client recv buffer.
* Change TLS range override logic.
* Use range requests for nRF91 TLS only, and when specified by app.

API updates:

* Let application provide client buffer. This allows for multiple
  download clients with different buffer sizes.
* Add downloader_deinit()
* Add downloader_stop()
* Remove downloader_disconnect()
* Changed signature of downloader_init(), downloader_start()
  and downloader_get() to take a URI.
* Added downloader_get_with_host_and_path() for downloads where
  host and path are separate arguments to keep backwards compatibility.
* The transports (http, coap) are now separated out of the download
  client with its own API.

Future work:

* Take uri as input param to fota_download library and use URI in
  other relevant libaries and structures.
* Curent download client is deprecated and will be removed later.

Signed-off-by: Eivind Jølsgard <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DNM doc-required PR must not be merged without tech writer approval.
Projects
None yet
Development

Successfully merging this pull request may close these issues.