IndexOutOfBoundsException with Emoji in YAML

[twonirwana]

Characters that are represented by more then one byte in UTF-8 can produce java.lang.IndexOutOfBoundsException: Range [1024, 1024 + 1) out of bounds for length 1024 if there are at place where they are split up in two buffers.
This can be reproduced with the code below and jackson-dataformat-yaml-2.18.1

 @Test
    void test() throws JsonProcessingException {
       String p = """
                   ---
                   value: "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   000000000000000000000000000000000000000000000000000000000000000000000🪺"
                """;
        ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
        mapper.readValue(p, Object.class);
    }

[cowtowncoder]

@twonirwana Thank you for reporting this.

I wonder if this is same as #497 (or rather has same root cause)

@twonirwana What is the Unicode value for the Emoji here? Would be good to use \u escape in source, just to make sure editors won't mess it.

[cowtowncoder]

Ok I guess it's:

https://unicodeplus.com/U+1FABA

which I think then must become 2 char surrogate pair:

0xD83E 0xDEBA 

so something like

"00000....\\uD83E\\uDEBA"

in sources

[twonirwana]

@twonirwana Thank you for reporting this.

I wonder if this is same as #497 (or rather has same root cause)

@twonirwana What is the Unicode value for the Emoji here? Would be good to use \u escape in source, just to make sure editors won't mess it.

yes, that looks like same problem as #497. It always happens if a character needs more than, that is the case by a lot of the newer emojis and asian characters, and they are split by the buffer. If the buffer splits the character in halve then the exception is throw. Maybe it is possible to check while filling the buffer that no multi byte character is split.

The \u version of my example is:

String p = """
                   ---
                   value: "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
                   000000000000000000000000000000000000000000000000000000000000000000000\uD83E\uDEBA"
                """;

[cowtowncoder]

Ok yes, good to know it is very likely the same issue. Things are only tricky wrt performance; should not have to scan multiple times and so on. But mostly I need to find time to figure out a good fix now that the problem itself is understood.

[twonirwana]

The problem is with SnakeYAML and they will fix it with 2.4: https://bitbucket.org/snakeyaml/snakeyaml/issues/1098/openapi-file-that-crashes-snakeyaml-when . Sorry for opening a issue here.

[cowtowncoder]

I am actually not 100% sure it's SnakeYAML that does decoding; exception stack trace can confirm that. What I know is that yaml/src/main/java/com/fasterxml/jackson/dataformat/yaml/UTF8Reader.java has an issue that would explain this.

But I can reopen this when I find time to dig into the earlier issue (#497).