Tolerate Unicode Escape Sequences introduced by Pact JVM

[timvahlbrock]

Pre issue-raising checklist

I have already (please mark the applicable with an x):

• Upgraded to the latest version of the relevant libraries
• Checked to see if the issue has already been raised
• Created an executable example that demonstrates the issue using either:
  • a Dockerfile
  • a fork of https://github.com/pact-foundation/pact-ruby-standalone-e2e-example
  • a Git repository with a Travis or Appveyor (or similar) build
  • a gist with all the relevant code and full instructions on how to run it

Software versions

• pact library: pact-cli:1.4.0.3 docker image
• pact-ruby-standalone: ?
• OS: Container

Expected behaviour

The CLI is able to handle unicode escape sequences introduced by Pact JVM into example values.

Actual behaviour

Pact CLI throws error message for some escaped unicode characters.

For "\uDAA5" pact cli may throws

/usr/lib/ruby/gems/3.3.0/gems/json-2.9.0/lib/json/common.rb:221:in `parse': incomplete surrogate pair at '\uDAA5", (JSON::ParserError)
          "created": "2'
	from /usr/lib/ruby/gems/3.3.0/gems/json-2.9.0/lib/json/common.rb:221:in `parse'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/pact_file.rb:28:in `pact_hash'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/pact_file.rb:20:in `consumer_name'
	from /usr/lib/ruby/gems/3.3.0/gems/rake-13.2.1/lib/rake/file_list.rb:70:in `collect'
	from /usr/lib/ruby/gems/3.3.0/gems/rake-13.2.1/lib/rake/file_list.rb:70:in `collect'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:1[46](https://git.tech.rz.db.de/ifp/app/arbeitssteuerung/-/jobs/285704110#L46):in `consumer_names'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:61:in `publish_pacts'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:34:in `call'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:16:in `call'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/cli/pact_commands.rb:74:in `publish_pacts'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/cli/pact_commands.rb:33:in `publish'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/command.rb:28:in `run'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor.rb:[53](https://git.tech.rz.db.de/ifp/app/arbeitssteuerung/-/jobs/285704110#L53)8:in `dispatch'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/base.rb:584:in `start'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/cli/custom_thor.rb:34:in `start'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/bin/pact-broker:10:in `<top (required)>'
	from /usr/bin/pact-broker:25:in `load'
	from /usr/bin/pact-broker:25:in `<main>'

For others it throws the following, but I'm not sure which escape sequences are responsible for that yet (the ones mentioned in the log aren't exactly in the contract file).

/usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:127:in `to_json': source sequence is illegal/malformed utf-8 (JSON::GeneratorError)
Invalid object: "\xED\xB2\x94"
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:127:in `block in contracts_for'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:119:in `collect'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:119:in `contracts_for'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:[56](https://git.tech.rz.db.de/ifp/app/arbeitssteuerung/-/jobs/285563406#L56):in `request_body_for'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:62:in `block in publish_pacts'
	from /usr/lib/ruby/gems/3.3.0/gems/rake-13.2.1/lib/rake/file_list.rb:70:in `collect'
	from /usr/lib/ruby/gems/3.3.0/gems/rake-13.2.1/lib/rake/file_list.rb:70:in `collect'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:61:in `publish_pacts'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:34:in `call'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/publish_pacts.rb:16:in `call'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/cli/pact_commands.rb:74:in `publish_pacts'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/cli/pact_commands.rb:33:in `publish'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/command.rb:28:in `run'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor.rb:538:in `dispatch'
	from /usr/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/base.rb:[58](https://git.tech.rz.db.de/ifp/app/arbeitssteuerung/-/jobs/285563406#L58)4:in `start'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/lib/pact_broker/client/cli/custom_thor.rb:34:in `start'
	from /usr/lib/ruby/gems/3.3.0/gems/pact_broker-client-1.77.0/bin/pact-broker:10:in `<top (required)>'
	from /usr/bin/pact-broker:25:in `load'
	from /usr/bin/pact-broker:25:in `<main>'

Steps to reproduce

Insert "\uDAA5" into an example value in the contract and try to upload it using the pact CLI. Can provide a reproduction repo if required, "pact-ruby-standalone-e2e-example" doesn't seem to work well on windows.

Relevent log files

see above

Edit: fixed wrong usage of keyword 'surrogate' in description

[timvahlbrock]

Reproduction Dockerfile:

FROM pactfoundation/pact-cli:latest

RUN mkdir pacts
RUN wget https://raw.githubusercontent.com/pact-foundation/pact-5-minute-getting-started-guide/refs/heads/main/pacts/GettingStartedOrderWeb-GettingStartedOrderApi.json
RUN sed -i -e 's/burger/b\\uDAA5rger/g' GettingStartedOrderWeb-GettingStartedOrderApi.json
RUN mv GettingStartedOrderWeb-GettingStartedOrderApi.json ./pacts

# This will fail
CMD pact-broker publish ./pacts --consumer-app-version 1.0.0

[timvahlbrock]

To verify that the unicode escape is actually the problem, I replaced it with ', so the symbol that it is encoding - which did not run into any problems.

[timvahlbrock]

Just saw, that the pact file parsing is done in https://github.com/pact-foundation/pact_broker-client, not here, so this issue might be better off there?

[timvahlbrock]

Dug a little bit further: the problem seems to be known at ruby json module, but the maintainers don't consider this sufficiently relevant to fix: ruby/json#383

[timvahlbrock]

So, after revising how UTF-16 works I think one could come to the conclusion that ruby-json (and the pact cli) is behaving as expected. \uDAA5 is a high surrogate, which are used to represent less frequent characters but require a two byte sequence (a low surrogate) to specify which specific character is represented. By definition, \uDAA5 and any other high surrogate needs to be followed by a low surrogate or is invalid unicode otherwise.