Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LogbookClientHandler leads OutOfMemoryError on large Flux<DataBuffer> response body #1983

Open
d-a-gerashenko opened this issue Dec 18, 2024 · 2 comments
Labels

Comments

@d-a-gerashenko
Copy link

org.zalando.logbook.netty.LogbookClientHandler tries to allocate ByteBuf with size of Content-Length and it leads to OutOfMemoryError in case of large (more than JVM memory limit) body size which is transferred via "Flux<DataBuffer>".

Steps to Reproduce

This method leads to OutOfMemoryError:

...
    public long test() throws Exception {
        Flux<DataBuffer> dataBufferFlux = WebClient.builder()
            .clientConnector(
                new ReactorClientHttpConnector(
                    HttpClient
                        .create()
                        .doOnConnected(conn -> {
                            //* <-- remove slash to make it work
                            conn
                                .addHandlerLast(
                                    new LogbookClientHandler(
                                        Logbook.builder()
                                            .condition(Conditions.requestTo("/**"))
                                            .requestFilter(RequestFilters.replaceBody(message -> ""))
                                            .responseFilter(ResponseFilters.replaceBody(message -> ""))
                                            .requestFilter(RequestFilters.replaceBody(BodyReplacers.binary()))
                                            .responseFilter(ResponseFilters.replaceBody(BodyReplacers.binary()))
                                            .requestFilter(RequestFilters.replaceBody(BodyReplacers.stream()))
                                            .responseFilter(ResponseFilters.replaceBody(BodyReplacers.stream()))
                                            .bodyFilter((contentType, body) -> "***")
                                            .build()
                                    )
                                );
                            //*/
                        })
                )
            )
            .build()
            .get()
            .uri("https://mirror.team-host.ru/ubuntu-cdimage/24.04.1/ubuntu-24.04.1-desktop-amd64.iso")
            .retrieve()
            .bodyToFlux(DataBuffer.class);

        Path path = Paths.get("large.dat");
        DataBufferUtils.write(dataBufferFlux, path)
            .block();

        return Files.size(path);
    }
...

Environment

Windows 11.
pom.xml

...
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <properties>
        <java.version>17</java.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.zalando</groupId>
                <artifactId>logbook-bom</artifactId>
                <version>3.10.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-tomcat</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>
        <dependency>
            <groupId>org.zalando</groupId>
            <artifactId>logbook-spring-boot-webflux-autoconfigure</artifactId>
        </dependency>
    </dependencies>
...
@d-a-gerashenko
Copy link
Author

d-a-gerashenko commented Dec 19, 2024

Perhaps is happens when there is a content-length header in streaming response.
https://github.com/zalando/logbook/blob/main/logbook-netty/src/main/java/org/zalando/logbook/netty/Offering.java#L24

@d-a-gerashenko
Copy link
Author

This fixed my OutOfMemoryError problem:

Logbook.builder()
    .strategy(new WithoutBodyStrategy())
    .build()

But I'm not sure if it's the right decision to use a buffer size equal to the length of the content because it could be too long. If everything is ok, please close this issue.

@d-a-gerashenko d-a-gerashenko changed the title org.zalando.logbook.netty.LogbookClientHandler leads OutOfMemoryError on large Flux<DataBuffer> response body LogbookClientHandler leads OutOfMemoryError on large Flux<DataBuffer> response body Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant