[Proof of concept] HttpClient Improvements

[Robbie-Microsoft]

Core Library

MSAL Node (@azure/msal-node)

Wrapper Library

Not Applicable

Public or Confidential Client?

Confidential

Description

The current HttpClient is located here. It's overly complicated, and many developers are still having issues with proxies.

Summary of Key Improvements:

• Simplified configuration with agents and proxy support.
• Improved proxy tunneling via the CONNECT method.
• Adoption of fetch for sending requests, offering a more modern and flexible approach.
• Error handling improvements with structured error responses.
• Better code organization and modular design, making the class easier to maintain and extend.

The following assumes NodeJS version 18+, where a native fetch method has been added. We will still be manually making http requests and creating sockets for proxy tunneling - we want to avoid third-party libraries.

New HttpClient class:

/*
 * Copyright (c) Microsoft Corporation. All rights reserved.
 * Licensed under the MIT License.
 */

import { INetworkModule, NetworkRequestOptions, NetworkResponse, HttpStatus } from "@azure/msal-common/node";
import { HttpMethod, Constants } from "../utils/Constants.js";
import { NetworkUtils } from "../utils/NetworkUtils.js";
import https, { Agent as HttpsAgent } from "https";
import http, { Agent as HttpAgent } from "http";

/**
 * HttpClient is a class that allows sending HTTP requests with optional proxy support.
 * It supports sending GET and POST requests and can tunnel requests through a proxy server using the `CONNECT` method.
 * This class uses Node's built-in `https` and `http` modules to handle requests, and it supports proxy authentication using the `Proxy-Authorization` header.
 * 
 * @remarks
 * This class is designed to work in environments using Node.js 18 or later.
 */
export class HttpClient implements INetworkModule {
    private proxyUrl?: string;  // Optional proxy URL to route requests through a proxy server.
    private proxyAuth?: string; // Optional Base64-encoded username and password for proxy authentication.
    private fetchAgent: HttpAgent | HttpsAgent;  // Agent used for direct fetch requests (without a proxy).
    private tunnelAgent?: HttpsAgent; // Agent used for tunneling requests through a proxy, only if proxyUrl is provided.

    /**
     * Constructs an instance of HttpClient.
     * 
     * @param fetchAgent - Optional. The agent used for direct fetch requests (without proxy). Default is `http.Agent`.
     * @param tunnelAgent - Optional. The agent used for tunneling through a proxy. Default is `https.Agent`.
     * @param proxyUrl - Optional. URL of the proxy server (e.g., `https://proxy.example.com:8080`). 
     * @param proxyAuth - Optional. Base64-encoded string for proxy authentication (e.g., `dXNlcm5hbWU6cGFzc3dvcmQ=` for `username:password`).
     */
    constructor(
        fetchAgent?: HttpAgent | HttpsAgent,   // Agent for fetch requests (default is `http.Agent`).
        tunnelAgent?: HttpsAgent,              // Agent for proxy tunneling (default is `https.Agent`).
        proxyUrl?: string,                     // Proxy URL for tunneling (optional).
        proxyAuth?: string                     // Proxy authentication (Base64 'username:password').
    ) {
        // If no fetchAgent is provided, default to HttpAgent (used for normal requests).
        this.fetchAgent = fetchAgent || new http.Agent();

        // If no proxyUrl is provided, ensure that tunnelAgent and proxyAuth are not set.
        if (proxyUrl) {
            // If proxyUrl is provided, both tunnelAgent and proxyAuth can be provided.
            this.tunnelAgent = tunnelAgent || new https.Agent(); // Default to https.Agent for tunneling.
            this.proxyAuth = proxyAuth; // Set proxyAuth only if proxyUrl is provided.
        } else {
            // If proxyUrl is not provided, make sure tunnelAgent and proxyAuth are undefined.
            if (tunnelAgent || proxyAuth) {
                throw new Error("tunnelAgent and proxyAuth can only be provided if proxyUrl is provided.");
            }
        }

        // Set the optional proxyUrl (if provided).
        this.proxyUrl = proxyUrl;
    }

    /**
     * Sends an HTTP GET request to the specified URL.
     * 
     * @param url - The URL to send the GET request to.
     * @param options - Optional. Additional request options, such as headers or body (if needed for certain requests).
     * 
     * @returns A promise that resolves to the response of the request.
     */
    async sendGetRequestAsync<T>(
        url: string,
        options?: NetworkRequestOptions
    ): Promise<NetworkResponse<T>> {
        // Forward the GET request to the generic request handler method.
        return this.sendRequest<T>(url, HttpMethod.GET, options);
    }

    /**
     * Sends an HTTP POST request to the specified URL.
     * 
     * @param url - The URL to send the POST request to.
     * @param options - Optional. Additional request options, such as headers or body.
     * 
     * @returns A promise that resolves to the response of the request.
     */
    async sendPostRequestAsync<T>(
        url: string,
        options?: NetworkRequestOptions
    ): Promise<NetworkResponse<T>> {
        // Forward the POST request to the generic request handler method.
        return this.sendRequest<T>(url, HttpMethod.POST, options);
    }

    /**
     * A generic method to handle both GET and POST requests.
     * 
     * @param url - The URL for the request.
     * @param method - The HTTP method (e.g., GET, POST).
     * @param options - Optional. Additional request options like headers or body.
     * 
     * @returns A promise that resolves to the network response.
     */
    private async sendRequest<T>(
        url: string,
        method: string,
        options?: NetworkRequestOptions
    ): Promise<NetworkResponse<T>> {
        // Construct the request options, including method, headers, and body.
        const fetchOptions: RequestInit = {
            method,  // The HTTP method (GET/POST).
            headers: options?.headers,  // Optional headers.
            body: method === HttpMethod.POST ? options?.body : undefined,  // Body only for POST requests.
        };

        // If a proxy URL is provided, route the request via the proxy; otherwise, use direct fetch.
        return this.proxyUrl
            ? this.sendRequestViaProxy<T>(url, fetchOptions)
            : this.sendRequestViaFetch<T>(url, fetchOptions);
    }

    /**
     * Sends the request through a proxy server using a tunnel.
     * This method establishes a tunnel using the `CONNECT` HTTP method to forward the request to the destination.
     * 
     * @param url - The URL to send the request to.
     * @param fetchOptions - The options for the fetch request (headers, body, etc.).
     * 
     * @returns A promise that resolves to the network response after the tunnel is established and the request is sent.
     */
    private async sendRequestViaProxy<T>(
        url: string,
        fetchOptions: RequestInit
    ): Promise<NetworkResponse<T>> {
        const destinationUrl = new URL(url);  // Parse the destination URL.
        const proxyUrl = new URL(this.proxyUrl!);  // Parse the proxy URL.

        // Step 1: Construct the request options for the `CONNECT` method to create the proxy tunnel.
        const tunnelRequestOptions: https.RequestOptions = {
            hostname: proxyUrl.hostname,
            port: proxyUrl.port,
            method: "CONNECT",  // Use CONNECT to establish the tunnel.
            path: destinationUrl.hostname,  // Destination host to tunnel to.
            headers: fetchOptions.headers as Record<string, string>,
            ...(this.proxyAuth ? {
                // Add Proxy-Authorization header if authentication is provided.
                "Proxy-Authorization": `Basic ${this.proxyAuth}`
            } : {}),
        };

        // Step 2: Establish the tunnel to the proxy and wait for the tunnel response.
        const tunnelResponse = await this.createTunnel({ ...tunnelRequestOptions, agent: this.tunnelAgent });

        // Step 3: Once the tunnel is established, pass the socket and send the request via fetch.
        return this.sendRequestViaFetch<T>(url, fetchOptions, tunnelResponse.socket);
    }

    /**
     * Establishes a TCP tunnel through the proxy server using the `CONNECT` method.
     * This is the first step in sending a request via a proxy.
     * 
     * @param tunnelRequestOptions - The options for the tunnel request, including the proxy server and destination.
     * 
     * @returns A promise that resolves with the socket used to send the request after the tunnel is established.
     */
    private createTunnel(tunnelRequestOptions: https.RequestOptions): Promise<{ socket: https.Socket }> {
        return new Promise((resolve, reject) => {
            // Create the HTTPS request to establish the tunnel.
            const request = https.request(tunnelRequestOptions);

            // Listen for the 'connect' event, which signals that the tunnel has been established.
            request.on("connect", (res, socket) => {
                // Use the 'ok' pattern to check if the proxy response is successful (status code in 2xx range).
                if (!res.ok) {
                    // If the status code is not successful, destroy the request and socket, and reject the promise.
                    request.destroy();
                    socket.destroy();
                    reject(new Error(`Error connecting to proxy. Status: ${res.statusCode}`));
                } else {
                    // Resolve with the established socket if the proxy response is successful.
                    resolve({ socket });
                }
            });

            // Handle request errors (e.g., network errors).
            request.on("error", (err) => {
                request.destroy();
                reject(err);
            });

            // End the tunnel request to begin establishing the connection.
            request.end();
        });
    }

    /**
     * Sends the HTTP request using fetch once the proxy tunnel is established (or directly if no proxy).
     * 
     * @param url - The URL for the request.
     * @param fetchOptions - Options for the fetch request (headers, body, etc.).
     * @param socket - Optional. The socket used for tunneling, if a proxy is involved.
     * 
     * @returns A promise that resolves to the network response.
     */
    private async sendRequestViaFetch<T>(
        url: string,
        fetchOptions: RequestInit,
        socket?: https.Socket
    ): Promise<NetworkResponse<T>> {
        // Create a new Agent for fetch that uses the provided socket (if available) or falls back to the default agent.
        fetchOptions.agent = new https.Agent({
            socket: socket || undefined,  // Explicitly set the socket for tunneling.
            ...this.fetchAgent.options,   // Include options from the provided fetch agent.
        });

        // Handle timeouts if specified in the fetch agent's options.
        if (this.fetchAgent.options.timeout) {
            const controller = new AbortController();
            fetchOptions.signal = controller.signal;
            setTimeout(() => controller.abort(), this.fetchAgent.options.timeout);
        }

        try {
            // Send the HTTP request using the fetch API.
            const response = await fetch(url, fetchOptions);

            // Parse the response body based on the Content-Type header (e.g., JSON).
            const responseBody = await this.parseResponseBody<T>(response);

            /*
             * Check for non-2xx status codes, which indicate failure.
             * Device Code Flow is the exception, as it returns a non 2xx status code when authorization is pending.
             */
            if (!response.ok && (responseBody["error"] !== Constants.AUTHORIZATION_PENDING)) {
                throw new Error(
                    `Request failed with status ${response.status}: ${response.statusText}`
                );
            }

            // Build and return a network response object that contains the parsed response data.
            const networkResponse = NetworkUtils.getNetworkResponse(
                response.headers,
                responseBody,
                response.status
            );

            return networkResponse;
        } catch (error) {
            // If an error occurs during the request, throw a new error with the message.
            throw new Error(`Network request failed: ${error.message}`);
        }
    }

    /**
     * Parses the response body. If the body is valid JSON, it returns the parsed JSON.
     * If the body isn't valid JSON or if the status code is an error (4xx or 5xx), 
     * it returns a detailed error object.
     * 
     * @param response {Response} - The response object from the fetch call.
     * @returns {Promise<T | object>} - A parsed JSON body or an error object.
     */
    private async parseResponseBody<T>(response: Response): Promise<T | object> {
        const contentType = response.headers.get("Content-Type");

        // Check if the response content type is JSON
        if (contentType?.includes("application/json")) {
            const body = await response.text(); // Get the body as text

            // Try to parse the JSON body
            try {
                return JSON.parse(body);
            } catch (error) {
                // If JSON parsing fails, build a detailed error response
                return this.createErrorResponse(
                    response.status,
                    response.statusText,
                    response.headers,
                    body
                );
            }
        } else {
            // If the content type isn't JSON, just return the body as text
            const body = await response.text();
            return this.createErrorResponse(
                response.status,
                response.statusText,
                response.headers,
                body
            );
        }
    }

    /**
     * Creates a detailed error object when JSON parsing fails or when the response status code 
     * indicates an error (4xx or 5xx).
     * 
     * @param statusCode {number} - The HTTP status code of the response.
     * @param statusMessage {string} - The status message of the response.
     * @param headers {Headers} - The headers of the response.
     * @param body {string} - The raw body of the response.
     * @returns {object} - A structured error object with detailed information.
     */
    private createErrorResponse(
        statusCode: number,
        statusMessage: string,
        headers: Headers,
        body: string
    ): object {
        let errorType: string;
        let errorDescriptionHelper: string;

        // Determine error type based on status code
        if (statusCode >= HttpStatus.CLIENT_ERROR_RANGE_START &&
            statusCode <= HttpStatus.CLIENT_ERROR_RANGE_END) {
            errorType = "client_error";
            errorDescriptionHelper = "A client";
        } else if (statusCode >= HttpStatus.SERVER_ERROR_RANGE_START &&
            statusCode <= HttpStatus.SERVER_ERROR_RANGE_END) {
            errorType = "server_error";
            errorDescriptionHelper = "A server";
        } else {
            errorType = "unknown_error";
            errorDescriptionHelper = "An unknown";
        }

        return {
            error: errorType,
            error_description: `${errorDescriptionHelper} error occurred.\nHttp status code: ${statusCode}\nHttp status message: ${statusMessage || "Unknown"}\nHeaders: ${JSON.stringify(headers)}\nBody: ${body}`,
        };
    }
}

Examples

Examples 1: Using a Custom Fetch Agent (Without Proxy)

const nodeSystemOptions: NodeSystemOptions = {
    fetchAgent: new http.Agent({ keepAlive: true, timeout: 5000 }), // optional custom agent for regular network requests
}

Example 2: Proxy Usage

const nodeSystemOptions: NodeSystemOptions = {
    proxyUrl: "https://proxy.example.com:8080",        // optional, indicates a tunnel will need to be opened to a proxy before network requests can occur
    tunnelAgent: new https.Agent({ keepAlive: true }), // optional custom tunnel agent, for the proxy connection
    proxyAuth: "dXNlcm5hbWU6cGFzc3dvcmQ=",             // optional Base64 encoded "username:password" (assumes developer's proxy can handle this format. This is standard, per [RFC 7617](https://datatracker.ietf.org/doc/html/rfc7617#section-2.1)
}

Example 3: Using Proxy (with Authentication) with a Custom Tunnel Agent and a Custom Fetch Agent

const customFetchAgent = new http.Agent({ keepAlive: true });
const customTunnelAgent = new https.Agent({ timeout: 10000 });

const nodeSystemOptions: NodeSystemOptions = {
    fetchAgent: customFetchAgent,               // Custom fetchAgent with keepAlive
    tunnelAgent: customTunnelAgent,             // Custom tunnelAgent with timeout
    proxyUrl: "https://proxy.example.com:8080", // Proxy URL
    proxyAuth: "dXNlcm5hbWU6cGFzc3dvcmQ=",      // Base64 "username:password"
}

[xirzec]

Some unstructured thoughts:

• Would it make more sense to use http/https.globalAgent instead of creating a new one without options?
• Be careful with classes with private methods, they're not truly private in TS but they still affect class identity in strange ways. In Azure SDKs outside of service clients we avoid classes and prefer interfaces + a factory method so that state can live privately in modules
• Love the use of async functions
• Methods with suffix async feel weird to me since JS is naturally async
• having three versions of 'sendRequest' feels unnecessary, fetch itself only has a single entrypoint for making a new request and the convenience of setting the method feels minimal
• I wasn't aware global fetch supported traditional Node agents, did that change recently? Also, global fetch is experimental until Node 21+ and as of our last testing had some performance issues.
• What is the idea behind making the sendRequest methods generic? It seems this is used to type the response payload, but it feels more natural to me to have it return the dynamic portion as unknown or any and narrow based on assignment rather than forcing the developer to pass the structure of the response through the generic directly.
• object is almost never the return value you want for something, prefer unknown or an interface
• Do you have any scenarios for binary/stream responses? Sometimes these can get corrupted by .text()
• snake_case names in your error object feel strange to me, should the error object be an actual Error since async functions will have to use try/catch with this anyway? You can still set a custom name and code, see what we did for RestError in core-rest-pipeline

[maorleger]

@xirzec knows a lot more about Azure SDKs httpClient than myself but I can offer a few more thoughts / questions:

Constructor changes

1. The constructor changes would be considered breaking, assuming the HttpClient class is publicly exposed - would you major version this? If not, may make sense to keep the public interface compatible if at all possible
2. The constructor here takes multiple optional positional parameters. That can get awkward to use and unwieldly to evolve. Consider using a single options parameter to capture all optional parameters to a function instead.

Methods

• sendRequestViaFetch: consider https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal/timeout_static for managing timeouts instead of a setTimeout function
• Agree with having three versions of 'sendRequest' feels unnecessary, fetch itself only has a single entrypoint for making a new request and the convenience of setting the method feels minimal 👍