更新华尔街日报的源就内存不足

[toyo2333]

路由地址

/wsj/:lang/:category?

完整路由地址

/wsj/zh-cn

相关文档

https://docs.rsshub.app/zh/routes/traditional-media#the-wall-street-journal-(wsj)-hua-er-jie-ri-bao-news

预期是什么？

正常更新

实际发生了什么？

info: /wsj/zh-cn, user IP: ::ffff:172.21.0.4

<--- Last few GCs --->

[18:0x6df5fc0] 1093794 ms: Mark-sweep (reduce) 355.5 (364.3) -> 355.3 (363.1) MB, 1465.0 / 0.0 ms (+ 106.3 ms in 11 steps since start of marking, biggest step 26.0 ms, walltime since start of marking 1631 ms) (average mu = 0.242, current mu = 0.177) fin[18:0x6df5fc0] 1094990 ms: Mark-sweep 356.3 (363.1) -> 356.3 (367.1) MB, 1192.2 / 0.0 ms (average mu = 0.142, current mu = 0.005) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0xb83f50 node::Abort() [node]
2: 0xa94834 [node]
3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xf42265 [node]
6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x17035b9 [node]
Aborted (core dumped)

部署

自建

部署相关信息

No response

额外信息

使用了默认镜像和chromium-bundled镜像，都会出现同样的问题。

只有少数情况下，更新会正常。每天都会有几次出现爆内存，然后检查日志都是在执行华尔街日报这条的时候，就爆了。

我是700M内存。没有使用redis。

docker配置如下
    rsshub:
        # two ways to enable puppeteer:
        # * comment out marked lines, then use this image instead: diygod/rsshub:chromium-bundled
        # * (consumes more disk space and memory) leave everything unchanged
        image: diygod/rsshub:chromium-bundled
        container_name: rsshub
        restart: always
        logging:
            options:
                max-size: 10m        
        # ports:
        #     - '1200:1200'
        environment:
            NODE_ENV: production

这不是重复的 issue

• 我已经搜索了 现有 issue，以确保该错误尚未被报告。

[github-actions]

Searching for maintainers:

• /wsj/zh-cn: Route not found

To maintainers: if you are not willing to be disturbed, list your username in scripts/workflow/test-issue/call-maintainer.js. In this way, your username will be wrapped in an inline code block when tagged so you will not be notified.

如果所有路由都无法匹配，issue 将会被自动关闭。如果 issue 和路由无关，请使用 NOROUTE 关键词，或者留下评论。我们会重新审核。
If all routes can not be found, the issue will be closed automatically. Please use NOROUTE for a route-irrelevant issue or leave a comment if it is a mistake.

[github-actions]

Searching for maintainers:

• /wsj/:lang/:category?: @oppilate

To maintainers: if you are not willing to be disturbed, list your username in scripts/workflow/test-issue/call-maintainer.js. In this way, your username will be wrapped in an inline code block when tagged so you will not be notified.

如果所有路由都无法匹配，issue 将会被自动关闭。如果 issue 和路由无关，请使用 NOROUTE 关键词，或者留下评论。我们会重新审核。
If all routes can not be found, the issue will be closed automatically. Please use NOROUTE for a route-irrelevant issue or leave a comment if it is a mistake.

[toyo2333]

我刚刚试着把redis也打开了。swap也有768MB 。但是更新源的时候瞬间CPU百分百，然后内存急剧上到接近700MB，还是内存爆了。日志报的错误和上面一样。

[toyo2333]

vercel上部署的，第一次能打开，过一会再打开就报这个错误了

This Serverless Function has timed out.

Your connection is working correctly.

Vercel is working correctly.

504: GATEWAY_TIMEOUT
Code: FUNCTION_INVOCATION_TIMEOUT

[toyo2333]

官网也打不了 https://rsshub.app/wsj/en-us/opinion

如果返回超时，能理解。但超时爆了内存，就导致后面的feed也更新不了了。连锁反应就不合理。（超时为什么爆内存？）

而且几乎每次都是wsj

看了下昨晚的日志

Tue, 29 Aug 2023 06:41:00 +0800] [warning] --- cURL error 28: Operation timed out after 20000 milliseconds with 0 bytes received [http://rsshub:1200/wsj/zh-cn


Tue, 29 Aug 2023 08:01:36 +0800] [warning] --- cURL error 52: Empty reply from server [http://rsshub:1200/wsj/zh-cn

Tue, 29 Aug 2023 09:21:21 +0800] [warning] --- cURL error 52: Empty reply from server [http://rsshub:1200/wsj/zh-cn

[toyo2333]

info: /weibo/user/XXXXXXXXXXXXXXX, user IP: ::ffff:172.18.0.5
info: /weibo/user/XXXXXXXXXXX, user IP: ::ffff:172.18.0.5
info: /weibo/user/SSSSSS, user IP: ::ffff:172.18.0.5
info: /weibo/user/XXXXXXXX, user IP: ::ffff:172.18.0.5
info: /wsj/zh-cn, user IP: ::ffff:172.18.0.5

<--- Last few GCs --->

[18:0x74f3fc0]  4802721 ms: Mark-sweep (reduce) 361.8 (371.3) -> 361.7 (370.0) MB, 1161.7 / 0.0 ms  (+ 30.3 ms in 10 steps since start of marking, biggest step 18.0 ms, walltime since start of marking 1229 ms) (average mu = 0.155, current mu = 0.035) fina[18:0x74f3fc0]  4803909 ms: Mark-sweep 362.7 (370.0) -> 362.7 (374.0) MB, 1182.6 / 0.0 ms  (average mu = 0.084, current mu = 0.007) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb83f50 node::Abort() [node]
 2: 0xa94834  [node]
 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf42265  [node]
 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x17035b9  [node]
Aborted (core dumped)

每个小时的自动更新都是WSJ爆了内存，这真不是内存管理的bug么？

[toyo2333]

试了官网https://rsshub.app/wsj/zh-cn ，也是一样

Route requested: /zh-cn
Error message: Proxy connection timed out: target website might be blocking our access, you can host your own RSSHub instance for a better usability.
Helpful Information to provide when opening issue:
Path: /zh-cn
Node version: v18.17.1
Git Hash: 3ac90ba

这个路由已经几乎用不了了。

[toyo2333]

试了官网https://rsshub.app/wsj/zh-cn ，也是一样

Route requested: /zh-cn Error message: Proxy connection timed out: target website might be blocking our access, you can host your own RSSHub instance for a better usability. Helpful Information to provide when opening issue: Path: /zh-cn Node version: v18.17.1 Git Hash: 3ac90ba

这个路由已经几乎用不了了。

再测试了官网，可以了，但是自建的就是打不开，套了Warp也不行。

[la3rence]

我使用 Docker 部署的 RSSHub 在尝试访问此 Feed 时短时间额外请求了 150 MB 内存。但随着请求结束很快就被 GC 了。问题不大。我最早分配给容器 256 MB，也经常 OOM （不是此 Feed 导致），现在给到 512 MB 是足够的。

[toyo2333]

我使用 Docker 部署的 RSSHub 在尝试访问此 Feed 时短时间额外请求了 150 MB 内存。但随着请求结束很快就被 GC 了。问题不大。我最早分配给容器 256 MB，也经常 OOM （不是此 Feed 导致），现在给到 512 MB 是足够的。

我的vps一共就700MB 内存...