Replies: 13 comments 33 replies
-
Can you please attach the crash logs, decoded ? |
Beta Was this translation helpful? Give feedback.
-
|
You also point to an old version |
Beta Was this translation helpful? Give feedback.
-
|
yes, that was the version that I have used until now in combination with Arduino 2.x |
Beta Was this translation helpful? Give feedback.
-
|
ok! "Which" problem ? |
Beta Was this translation helpful? Give feedback.
-
|
The changes are listed in the release notes: https://github.com/mathieucarbou/ESPAsyncWebServer/releases The api has been cleaned up a little, for example, On ESP32, you also do not need to use the Again, the deprecation warnings when compiling will tell you that. |
Beta Was this translation helpful? Give feedback.
-
|
yes I am aware of this, but even when I change this with the new Version, it is not working. |
Beta Was this translation helpful? Give feedback.
-
|
when I tried it this morning, it kept crashing with backtrace on AsyncTCP. 
I'll have to take another look at it this evening. Maybe I have also to change something regarding the gzip coded content (css, js, html) |
Beta Was this translation helpful? Give feedback.
-
|
I still get there errors: after some research it seems that I have to use the new middleware function. but still the same error. Am I on the right way and did I still miss something? |
Beta Was this translation helpful? Give feedback.
-
|
You have to configure the CORS middleware according to your needs. Above this is just an example, but the origin should be set to What I do not understand is that if you were not using cors before, why do you need to add it now ? ESPAsyncWS, whether now or before, never did anything regarding cors. The new version allows to add a middleware to answer cors preflight and add cors headers but this is if you need it. Something has changed in your setup also. |
Beta Was this translation helpful? Give feedback.
-
|
I don't understand it either. I've just checked it again. with this version it works fine and I get no errors: then I only change the version in platformio.ini to: full-clean, download and then it is not working and I see this error messages in my web browser: The google research then brought me to the hint that it could be due to CORS. The web server itself is running, but my SSE connection is not working anymore |
Beta Was this translation helpful? Give feedback.
-
|
So, It took me a while because the effects were a bit confusing, but I think I found the reason for my main problem. events.onConnect([](AsyncEventSourceClient *client) {
// check if it´s a new client or reconnect
if (client->lastId()) {
MY_LOGI(TAG, "Client reconnected with lastId %lu\n", client->lastId());
} else {
MY_LOGI(TAG, "New Client connected");
}
client->send("ping", "ping", millis(), 5000);
onLoadRequest(); // send all data to the client - !!! this is the problem !!!!
clientConnected = true;
});This has worked so far, but has now led to strange effects. |
Beta Was this translation helpful? Give feedback.
-
|
rule of thumb: NEVER do a long operation in a async_tcp callback. You are within an I/O tcp callback, so your code should execute as fast as possible. All the things that could be delegated to the loop task have to be done from the loop task. |
Beta Was this translation helpful? Give feedback.
-
|
Now that it is basically running with the latest EspAsyncWebserver version and Arduino 3, I would like to come back to my general questions. Since users of my project have recently increasingly reported that the ESP crashes when they use the WebUI, especially if the WebUI is regularly open or more than one client is connected, I have just made a few more tests. And I can sometimes provoke a crash if, for example, I open several tabs in the browser (>6). This also fits with the specification of SSE where a maximum number of 6 client connections is defined. I don't think that users have more than 6 clients open, but I can imagine that connections are not always closed cleanly and this can lead to such crashes even if only 1 or 2 clients are connected at the end. Do you share my concerns here? The second point is the question of the use of AsycTCP vs AsyncTcpSock. I would therefore be pleased to receive tips and advice from you experts |
Beta Was this translation helpful? Give feedback.
-
|
Hello,
Both have pros and cons, it really depends on the app., if you need cors, etc
That warrants some investigation... Definitely send me complete stack traces (with the registers above) when you see that. If you can reproduce that would be even better, and if you can reproduce in a specific handler, you could share some code, print also the heap, etc.
I use both. I have some little apps running since days with AsyncTCP and AsyncTcpSock, but found that AsyncTcpSock is definitely better for concurrent connexions. I found also that is creates more heap consumption (spikes). But it is a known fact that AsyncTCP does handle well concurrent connexions. You can see the benchmarks here: https://github.com/mathieucarbou/ESPAsyncWebServer?tab=readme-ov-file#performance
SSE is used a lot, and I even suspect this is what the ESPHome team is using for their UI since they only focus on improving SSE in their fork (that is why I cherry picked their improvements).
It should much more be an application-driven decision. Personally I have a preference over the flexibility of Websocket but it all depends on the app. |
Beta Was this translation helpful? Give feedback.
-
|
if it is interesting, here are two examples of crash reports that happen if I open more than 6 connections ——— |
Beta Was this translation helpful? Give feedback.
-
|
I was also able to force some crashes with your simpleServer.ino code two times this one: and this one with some more info: |
Beta Was this translation helpful? Give feedback.
-
|
I had seven Tabs of And with your example code I have also used AsyncTCP and not TcpSock one more remark: here my full example that I have used: //
// A simple server implementation showing how to:
// * serve static messages
// * read GET and POST parameters
// * handle missing pages / 404s
//
#include <Arduino.h>
#include <AsyncTCP.h>
#include <WiFi.h>
#include <ESPAsyncWebServer.h>
#if __has_include("ArduinoJson.h")
#include <ArduinoJson.h>
#include <AsyncJson.h>
#include <AsyncMessagePack.h>
#endif
#include <LittleFS.h>
const char *sseContent PROGMEM = R"(
function setupSSE() {
var source = new EventSource("/events");
source.addEventListener(
"open",
function (e) {
console.log("Events Connected");
},
false
);
source.addEventListener(
"error",
function (e) {
if (e.target.readyState != EventSource.OPEN) {
console.log("Events Disconnected");
}
},
false
);
source.addEventListener(
"message",
function (e) {
console.log("message", e.data);
},
false
);
source.addEventListener(
"heartbeat",
function (e) {
console.log("heartbeat", e.data);
},
false
);
}
setupSSE();
)";
AsyncWebServer server(80);
AsyncEventSource events("/events");
/////////////////////////////////////////////////////////////////////////////////////////////////////
const char *PARAM_MESSAGE PROGMEM = "message";
const char *SSE_HTLM PROGMEM = R"(
<!DOCTYPE html>
<html>
<head>
<title>Server-Sent Events</title>
<script defer src="sse.js" defer></script>
</head>
<body>
<h1>Open your browser console!</h1>
</body>
</html>
)";
void notFound(AsyncWebServerRequest *request) {
request->send(404, "text/plain", "Not found");
}
#if __has_include("ArduinoJson.h")
AsyncCallbackJsonWebHandler *jsonHandler =
new AsyncCallbackJsonWebHandler("/json2");
AsyncCallbackMessagePackWebHandler *msgPackHandler =
new AsyncCallbackMessagePackWebHandler("/msgpack2");
#endif
static const char characters[] = "0123456789ABCDEF";
static size_t charactersIndex = 0;
void setup() {
Serial.begin(115200);
#ifndef CONFIG_IDF_TARGET_ESP32H2
// WiFi.mode(WIFI_STA);
// WiFi.begin("YOUR_SSID", "YOUR_PASSWORD");
// if (WiFi.waitForConnectResult() != WL_CONNECTED) {
// Serial.printf("WiFi Failed!\n");
// return;
// }
// Serial.print("IP Address: ");
// Serial.println(WiFi.localIP());
WiFi.mode(WIFI_AP);
WiFi.softAP("esp-captive");
#endif
events.onConnect([](AsyncEventSourceClient *client) {
if (client->lastId()) {
Serial.printf(
"SSE Client reconnected! Last message ID that it gat is: %" PRIu32
"\n",
client->lastId());
} else {
Serial.println("NEW SSE Client connected!");
}
client->send("hello!", NULL, millis(), 1000);
});
events.onDisconnect([](AsyncEventSourceClient *client) {
Serial.println("SSE Client disconnected!");
});
server.on("/sse.js", HTTP_GET, [](AsyncWebServerRequest *request) {
AsyncWebServerResponse *response =
request->beginResponse(200, "text/js", sseContent);
request->send(response);
});
server.on("/sse", HTTP_GET, [](AsyncWebServerRequest *request) {
request->send(200, "text/html", SSE_HTLM);
});
// go to http://192.168.4.1/sse
server.addHandler(&events);
#if __has_include("ArduinoJson.h")
server.addHandler(jsonHandler);
server.addHandler(msgPackHandler);
#endif
server.onNotFound(notFound);
server.begin();
}
uint32_t lastSSE = 0;
uint32_t deltaSSE = 5;
void loop() {
uint32_t now = millis();
if (now - lastSSE >= deltaSSE) {
events.send(String("ping-") + now, "heartbeat", now);
lastSSE = millis();
}
} |
Beta Was this translation helpful? Give feedback.
-
|
@dewenni : the use case is not valid. The way the example is done, it sends a SSE event each 100 ms, and here in your version each 5 ms. You are just exhausting the SSE message queue and also the board dies and runs out of heap. The code is exhausting all the ressources available. If you need such a high message rate, don't use SSE: use binary WebSocket. And even with WebSocket, you'll have a hard time keeping up a rate of 200 messages per second! |
Beta Was this translation helpful? Give feedback.
-
|
yes sure, this is not the use case of my project, but it is your original example ;-) deltaWS=100 but deltaSSE =5 but I didn't realise that either |
Beta Was this translation helpful? Give feedback.
-
|
This ino file is used to test the features of ESpAsyncWS. The 5ms is set because this is the limit I found on my esp32dev board in ap mode when connecting directly to it. I am using that to test the limit of SSE in good conditions and see if there is some impact when modifying the code. Don't take the examples as something to reproduce: you need to understand also what you are doing because blindly copying the examples for sure won't work. There are a lot of things in this file done for testing purposes. |
Beta Was this translation helpful? Give feedback.
-
|
just got a glimpse to your code, @dewenni, nice project indeed. It has ways to improve though. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting project indeed, and nice to see @dewenni we're both trying to build systems around the complicated Buderus protocols. In my project I went for a simple approach, just REST API to fetch/write data and pagination for larger data sets, and msgpack. I only use SSE for the real-time log. It works well for now, and scales, but may need to reconsider a better socket (or http/2) approach in the future. I've never had reports of crashes, but will test if my project still functions with multiple client sessions. Would love to learn some of your tricks too. I'm also curious why a user would open more than 2 tabs - what's the use case here? |
Beta Was this translation helpful? Give feedback.
-
|
thanks a lot for joining the conversation @vortigont and @proddy !
...and I would love to hear them!
I have now changed the implementation from SSE to WS to see if this is more stable for my use case
My intention was to avoid String class because of the dynamic memory allocation and I had the same thoughts about the use of ArduinoJson. But perhaps these thoughts are also unnecessary or wrong. I have also limited the maximum web socket connections to 2 and redirect all other clients to a page with a corresponding note My use case is generally as follows:
Two open Tabs is for sure not a real use case. However, I have heard from users that they use a tablet as a permanent dashboard and that a client is then permanently open there. A second additional connection on a PC or mobile phone is then of course possible. By the way, your project is also very impressive. I discovered this a long time ago and also got some inspiration. So I would be very happy if you would take a look at my two main webUI tasks and give me some constructive criticism. |
Beta Was this translation helpful? Give feedback.
-
I tend to disagree: when used correctly, this class is useful and does not really add any overhead. For exemple in your code here: IMO, you can get rid of snprintf and directly use memcpy or the String class: String s;
s.reserve(128);
s.concat(elementID);
s.concat('#');
s.concat(typSuffix);
jsonBuf[key] = s; |
Beta Was this translation helpful? Give feedback.
-
|
@dewenni, |
Beta Was this translation helpful? Give feedback.
-
|
agree! |
Beta Was this translation helpful? Give feedback.
-
|
@vortigont My quick test was: Perhaps my current settings are not suitable for this scenario, but I would expect that this will not lead to a crash. If a buffer is too small, then messages may be lost or sent with a delay, but it shouldn't crash. crash report from origin version: and here with your rework: However, users of my project also report crashes or hang-ups when connecting one or two clients. |
Beta Was this translation helpful? Give feedback.
-
|
@dewenni pls point me to your latest SSE branch, I'll try to build and run it too. Now sure that toggling WLAN is good thing to do tests :) Opening/closing tabs should be a more clean way to recycle memory resources |
Beta Was this translation helpful? Give feedback.
-
|
@vortigont thanks that you like to test it with my project. I have updated and pushed the SSE branch. |
Beta Was this translation helpful? Give feedback.
-
|
@dewenni ws implementation looks nice, you could optimize some repetitive string literals into a |
Beta Was this translation helpful? Give feedback.
-
|
@vortigont thanks a lot for the additional hints! constexpr const char *STR_AUTOMATIC[] = {"Automatik", "Automatic"};
constexpr const char *STR_MANUAL[] = {"Handbetrieb", "Manual"};
typedef struct {
const char *const *AUTOMATIC = STR_AUTOMATIC;
const char *const *MANUAL = STR_MANUAL;
}I did it as a test for a few essential declarations and got the following result before: after: So in my project, I would have to reduce the size required for the flash memory :-) Regarding the other point. Yes, so far I've actually implemented my code in C because I'm not really experienced in C++. |
Beta Was this translation helpful? Give feedback.
-
|
@dewenni I was able to build and flash it. Though have problems connecting it to my WiFi AP. Anyway, after connection I've enabled sim mode as you mentioned, then opened 8 tabs (two browsers, 4 tabs each) - looks like it works. |
Beta Was this translation helpful? Give feedback.
-
|
@vortigont ok, interesting. To get it into AP Mode again, you can use a double click of the Reset Button. 
The values are only generated and sent once - they do not change. But they are all sent after refreshing the browser or reconnecting. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dewenni, Here is what I see from serial log with a bit of debuging added in just few milliseconds heap usage spikes for more than 50k and it's only two clients connected. Here is browser's console so it periodically received 10-20k of message data in json's, plus static web resources. Leave alone small text updates, not makes much difference. you run this part VERY frequently, each This is not good design.
Another considerations:
Maybe websocket approach would be better for you because it does not queue, but anyway there are thing that might benefit a lot from simplifying and reducing useless duplicated data transmissions. |
Beta Was this translation helpful? Give feedback.
-
|
@vortigont Thank you very much for taking the time to analyze my code in more detail and also the effort of your answer. I'm also quite happy that it's my design fault, because then I can improve it myself and learn. One more question on the subject of “static web files” I am already looking forward to taking care of the points you mentioned. |
Beta Was this translation helpful? Give feedback.
-
It is also possible to serve the huge PROGMEM content with chunk encoding by limiting the amount of bytes sent for each chunk: server.on("/chunked.html", HTTP_GET, [](AsyncWebServerRequest* request) {
String len = String(htmlContentLength);
if (request->header(asyncsrv::T_INM) == len) {
request->send(304);
return;
}
AsyncWebServerResponse* response = request->beginChunkedResponse("text/html", [](uint8_t* buffer, size_t maxLen, size_t index) -> size_t {
Serial.printf("%u / %u\n", index, htmlContentLength);
// finished ?
if (htmlContentLength <= index) {
Serial.println("finished");
return 0;
}
// serve a maximum of 1024 or maxLen bytes of the remaining content
const int chunkSize = min((size_t)1024, min(maxLen, htmlContentLength - index));
Serial.printf("sending: %u\n", chunkSize);
memcpy(buffer, htmlContent + index, chunkSize);
return chunkSize;
});
response->addHeader(asyncsrv::T_Cache_Control, "public,max-age=60");
response->addHeader(asyncsrv::T_ETag, len);
request->send(response);
});outputs: Note: I didn't test the cache part, I will let you do that because etag might not be what you want - maybe you want to control the caching another way. To test: # sends the whole content with the cache headers
> curl -N -v -X GET http://192.168.4.1/chunked.html --output -
# sends back 304 - not modified
> curl -N -v -X GET -H "if-none-match: 4272" http://192.168.4.1/chunked.html --output - |
Beta Was this translation helpful? Give feedback.
-
|
@dewenni you are welcome! Anyway was testing my recent PR, found a small issue by the way. So was for mutual benefit :) As for static content you could use, for example, git commit hash as ETAG or build date-time. See Mathieu's reply for inspiration. IMS (if-modified-since) is implemented only for static content from FS 'cause it takes file modification timestamps. Also forget the term constexpr const char *STR_AUTOMATIC[] = {"Automatik", "Automatic"};
constexpr const char *STR_MANUAL[] = {"Handbetrieb", "Manual"};
typedef struct {
const char *const *AUTOMATIC = STR_AUTOMATIC;
const char *const *MANUAL = STR_MANUAL;
}This is one of the options, but you could avoid If you need this for translations |
Beta Was this translation helpful? Give feedback.
-
Thank you for the praise. To be honest, this is the first time I have programmed a web frontend with html, js and css. I only had limited knowledge and experience here before and now. But I am also satisfied with the result. I use the Pico.css framework. |
Beta Was this translation helpful? Give feedback.
-
|
short update: @vortigont the advice to use namespace for the const strings is great and was easy to change - thanks! Since I already save the settings that can be made in the webUI in a config.json anyway, I have now written a js function that parses the config.json and also identifies and updates the elementID of the web elements generically based on the json elements. This means that I no longer have to send this ~2kB of data via WS. I could now also include the ETag check. Regarding the general web content (html, js, css) I will probably have to stick to serving the content directly from the normal Flash. I already have the problem with my application that goes out with the Flash. I already use min_spiffs.csv as partition scheme to have as much space as possible for the application. I can't afford to generate the web content again in LittleFS. The next step will be to think about how I can reduce or optimize the transmission of the remaining WS data (2x ≈ 4 kB values read by the connected heating system). |
Beta Was this translation helpful? Give feedback.
-
Have a look at MycilaSafeBoot: this is a recovery partition that you can use in order to increase your app partition size. Tasmota is using the same mechanism (they also have a safeboot recovery partition). The way it works, you need to boot into this small recovery image to do the firmware update and there is only one app partition, which then can be more than 3Mb for a 4Mb flash and more than 7Mb for a 8Mb flash. That's a library I did and I am using it in 2 of my biggest projects. Like you, I don't want to put the burden on users to update 2 partitions : app and fs. This is a pain.
You can use other cache headers - look at the HTTP specs or google search how to do it. You can use If modified since, or no etag and just cache and do not revalidate. Anyway... Up to you to use the headers you want to control the caching based on your app needs. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, thank you. That's something I can still deal with and experiment with myself now. |
Beta Was this translation helpful? Give feedback.
-
|
(I edited my message above while you were answering to add some info about safeboot recovery partition if you need) |
Beta Was this translation helpful? Give feedback.
-
|
that is also great and I think I have to use it! At the moment I have implemented OTA Update directly in my webUI that is comfortable. |
Beta Was this translation helpful? Give feedback.
-
|
At least you have the code so you should be able to craft something closer to your needs / UI if you need to go this route. |
Beta Was this translation helpful? Give feedback.
-
Hi Mathieu, Hi Community,
first of all I want to thank you for being so active in developing EspAsyncWebserver and all your other Libraries.
Especially EspAsyncWebserver that was not maintained for a long time but even used in so many projects!
I have developed a project that is based on EspAsyncWebserver and can be used to monitor and control an older "Oil Heating System" from Buderus.
The Project can be found here: ESP_Buderus_KM271
The actual version of the projects works fine for me and in my environment. But more and more user are using my Software and in the last time they reporting more and more issues regarding the WebUI and general stability.
For example it seems to be a problem for some users if they open more than one client at the same time.
Because of a feature request for Ethernet Support I am also trying to migrate the project to Arduino 3 and implement support for W5500. I hoped that this migration and also the update to the newest version von EspAsyncWebserver can solve some problems?!
But unfortunately I am struggling to get it running with the latest versions.
Migration to Arduino 3.0.7 works for me, but if I want to upgrade also your EspAsyncWebserver it crashes all the time.
Therefore I wanted to ask you for some help, because I don’t know how to handle this.
In the meantime, I'm also unsure whether what I've programmed is actually OK, or whether your experience tells you that it should be done differently.
In the past I used a Library ESPUI but there was also more and more a limitation of what I can do, and with my need for more web elements, this solution also became unstable when a client connected and all data had to be sent to the client.
In general:
The oil heating system delivers a lot of values that has to send to the web client.
When I decided to build my own solution that is not based on ESPUI-Library, I did some tests with web sockets vs. EventSource+SSE and it seems to me that EventSource+SSE is more stable when I try to send a lot of data to the client.
So my actual solution is based on EventSource+SSE
In the past, I used to send every single value from the ESP to the client. Meanwhile I have changed this and initially send the data in larger JSON packets during a client connect and then only the changes as individual commands. Size of this JSON is from 2048-5120 bytes.
I would therefore be delighted if you could take a look at my project and give me some advice on the following topics:
I have uploaded the branch „arduino3“ in my GitHub repository that can be used for tests, commits or PR
https://github.com/dewenni/ESP_Buderus_KM271/tree/arduino3
If you want to get it up and running, here is a little description
once you have downloaded the software on a ESP32 you should get this log message:
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
I (APP-SETUP): > WiFi Mode: AccessPoint <
I (APP-SETUP): 1. connect your device to SSID: ESP-Buderus-km271
I (APP-SETUP): 2. open Browser and go to Address: http://192.168.4.1/
I (APP-SETUP):
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
So you can connect to the AP called ESP-Buderus-km271 und then open http://192.168.4.1/
after that you should be able to see the webUI and edit the WiFi credentials
I would then also recommend to set also this options to get a realistic setup with more web elements:
Then press restart at the bottom of the settings page.
So again, I would be really really happy if you could support me here.
Beta Was this translation helpful? Give feedback.
All reactions