RFC: Isomorphic response

[kettanaito]

Goals

The isomorphic response aims to achieve two primary things.

Standardized response representation

An HTTP response can be represented in multiple ways depending on the environment, request type, or even the framework you’re using. For example:

• Node.js http uses IncomingMessage.
• The browser uses Response from Fetch API.
• The browser can also use an enclosed response abstraction in XMLHttpRequest (never exposed directly to the consumer).

But in actuality, the intentions behind any response are the same: describe an HTTP response. I agree that responses may include additional information but we’re speaking about a barebone response representation:

• Status code/text.
• Response headers.
• Response body.

The biggest complexity and, as a result, the biggest divergence between response representations in different tools come from the fact that those tools also execute responses. That’s where sockets, agents, and friends come into play, adding the execution capabilities to otherwise static response representation.

Stringified response representation

Apart from that, any response should be serializable. This way the intention behind the response can be represented as plain text so it’s possible to signal that intention to other tools. For example, to a Service Worker via a message channel, or between Node processes via IPC.

That is not possible with any object-based response representation because you cannot serialize functions. Such implementations often describe certain properties like getters (i.e. IncomingMessage.headers), so their values cannot be serialized unless explicitly read.

Implementation

Below, is a draft of what such an isomoprhic response API may look like from the consumer’s (MSW) perspective.

import { IsomorphicResponse } from 'isomorphic-response'

// Ideally, keep the same constructor signature
// as the "fetch.Response" class.
const res = new IsomorphicResponse('hello world', {
	status: 200,
	headers: {
	  'X-Powered-By': 'Express'
	}
})

// Serialize the response object to a string
// that can be sent over IPC and such, and then
// parsed back to the same response instance.
const serialized = res.toString()

// Create a new response instance based on the
// serialized (stringified) response representation.
IsomorphicResponse.from(serialized) // IsomorphicResponse {}

This way, during the request handling, MSW could create an IsomorphicResponse instance in the res() composition chain, and send that instance to either the Service Worker or to the Node.js interceptor.

// Service Worker response mocking.
const response = await handleRequest(...)

worker.sendMessage({
  // Send stringified representation of the mocked response
  // to the running Service Worker instance.
  response: response.toString()
})

// ...later on, in the worker.
const message = await fromTheClient()
const { response } = message

// Respond to the intercepted request in the "fetch"
// event listener.
event.respondWith(IsomorphicResponse.from(response))

// Node.js response mocking.
interceptor.on('request', async (event) => {
  const response = await handleRequest(...) // IsomorphicResponse {}

  // Technically, it's not necessary to stringify
  // the response when running in Node.js as MSW
  // and your tests often run in the same process.
  // This may be an issue once we consider supporting
  // inter-process interception.
  return event.respondWith(response)
})

Response serialization

In order to be reconstructed, a serialized response must contain the following information:

• ResponseInit (status, statusText, headers).
• Stringified representation of the original response body.
• Response body type (i.e. text, json, Uint8Array, FormData, etc.). This way it’s apparent how to parse the string body representation.

Serialized format

Proposal 1: a plain string

type SerializedResponse = 
  `${status}|${statusText}|${headersJson}|${bodyText}`

This initial format proposal operates on the assumption that any response can be represented as text: plain text, stringified JSON string, read and stringified ReadableStream data, serialized FormData, stringified ArrayBuffer, etc.

Proposal 2: formatted string

200 OK
Content-Type: application/octet-stream
X-Powered-By: msw

hello world

Something similar to chat CURL prints out.

Benefits:

• Human-readable. Can be used in logs directly.

Downsides:

• Needs additional parsing as opposed to .split(',').

Questions

• Should IsomorphicResponse be compatible with fetch.Response?
  • Should it contain the properties like type, ok(), url, etc.
  • Should it act as an input to various systems that expect fetch.Response? Like the Service Worker’s event.respondWith() in the “fetch” event listener?
• How can one represent stream-like data (i.e. ReadableStream) with text?
  • Initial idea: read the original response stream, serialize its data (because it’s a text anyway once read), then re-create the stream in .from().
• Should the worker script ship with the IsomorphicResponse implementation injected? If it needs to de-serialize the stringified response, it needs the logic to do that based on the response type (i.e. to properly emulate ReadableStream mocked responses).

[kettanaito]

Responding with ReadableStream

I've kicked off a proof of concept that uses ReadableStream constructed in the worker to respond to intercepted requests from the client-side response definition. The benefit of using ReadableStream is that both non-stream and stream responses can be issued by the worker (currently, MSW doesn't support readable streams).

I think this is the way to go in general: create a stream, connect it to the MessageChannel between the client and the worker, and pipe the mocked response to the worker.

This doesn't solve the stringification issue, as one can stream non-textual data like binaries or buffers. Some sort of response normalization is still necessary, and an isomorphic response may become that layer.

[kettanaito]

The idea here was that instead of normalizing any response to a text, an isomorphic response could normalize it internally and stream it anywhere the consumer wants. This way the streaming contract is defined by the consumer, and it can be an IPC in Node or a MessageChannel between the client and the worker. An isomorphic response would still normalize the response and provide a streaming interface to it.

[PeterlitsZo]

UPDATED

Although I do not know ReadableStream very much, I do think it is a good idea:

1. ReadableStream can be turned into ArrayBuffer.
2. ReadableStream is used in fetch.

---

I think using strings to present data is not the best idea.

In JS, its string is a list of UTF-16 char codes. More Information.

Each char code holds two bytes of space. But we cannot use it as a bytes array --- some char code list is not a valid string. JS will replace invalid character to be REPLACEMENT CHARACTER(�)(?: Do not find the source of the infomation)

So if we want to store binary data into a string, we will:

const a = new Uint8Array([12, 255, 56])
let str = ''
for (let i = 0; i < a.length; i++) {
  str += String.fromCharCode(a[i])
}

We can make sure each bi-bytes' range is from 0x00 to 0xFF to let JS know it is a valid string. But it is still not a good idea... Make just using ArrayBuffer or Uint8Array a better idea?

I do not know ReadableStream. I will study it to get it. Maybe will be edited in the future.

[kettanaito]

This may be made obsolete by #1404.