[JWT] Notify components about changes in Fleet Server credentials

[jsoriano]

There are some components managed by Agent like Endpoint that use API keys to interact directly with Fleet Server.

Once #4277 is implemented, we will use JWT instead of API keys, and credentials will be periodically rotated, so we need to communicate this change to managed components.

Ensure that components always use valid credentials.

[jlind23]

@pierrehilbert @cmacknz the target here is to make this happened before end of April 2024, do you foresee any implementation issue/technical challenge?

@pierrehilbert could you please triage this?

[pierrehilbert]

I will let @cmacknz reply but from my perspective, it should be okay to implement.
I will check which sprint will fit the best to do so.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

We provide the Fleet configuration to both endpoint and apm-server, see

elastic-agent/internal/pkg/agent/application/fleet_server_bootstrap.go

Lines 113 to 134 in bdd885c

	for i, comp := range comps {
	if comp.InputSpec != nil && (comp.InputSpec.InputType == endpoint || comp.InputSpec.InputType == apmServer) {
	for j, unit := range comp.Units {
	if unit.Type == client.UnitTypeInput && (unit.Config.Type == endpoint || unit.Config.Type == apmServer) {
	unitCfgMap, err := toMapStr(unit.Config.Source.AsMap(), map[string]interface{}{"fleet": fleetCfg})
	if err != nil {
	return nil, err
	}
	// Set host.id for the host, assign the host from the top level config
	// Endpoint expects this
	// "host": {
	// "id": "b62e91be682a4108bbb080152cc5eeac"
	// },
	if v, ok := unitCfgMap["fleet"]; ok {
	if m, ok := v.(map[string]interface{}); ok {
	m["host"] = cfg["host"]
	m["hosts"] = fleetHosts
	// Inject agent log level
	injectAgentLoggingLevel(m, agentInfo)
	}
	}

In both cases we are injecting the entire Fleet configuration directly into the input configuration for both endpoint and apm-server. Whatever change we make the agent's internal Fleet configuration will be communicated directly to both of them. Today this configuration is defined as:

elastic-agent/internal/pkg/agent/configuration/fleet.go

Lines 12 to 21 in bdd885c

	// FleetAgentConfig is the internal configuration of the agent after the enrollment is done,
	// this configuration is not exposed in anyway in the elastic-agent.yml and is only internal configuration.
	type FleetAgentConfig struct {
	Enabled bool `config:"enabled" yaml:"enabled"`
	AccessAPIKey string `config:"access_api_key" yaml:"access_api_key"`
	Client remote.Config `config:",inline" yaml:",inline"`
	Info *AgentInfo `config:"agent" yaml:"agent"`
	Server *FleetServerConfig `config:"server" yaml:"server,omitempty"`
	}

The biggest missing piece here is that we currently assume the Fleet API key never changes without being re-enrolled. That assumption is invalidated by refreshable JWTs. The current implementation assumes the credentials are static so we'll have to build a way to resend the Fleet configuration whenever the JWT changes. This could be done using the current method of injecting the Fleet configuration or we could add proper support for the Fleet configuration into the control protocol. Either way this needs to be implemented on both side of the protocol, and we have to coordinate this with affected components.

[cmacknz]

@axw I see apm-server defines the entire Fleet configuration including the AccessAPIKey but I wasn't able to quickly determine if it is still used.

Will APM server be affected if we switch from using API keys to JWTs for Fleet authentication? The initial implementation would only be for Serverless Fleet server instances.

[cmacknz]

Also CC @nfritts from the endpoint side of this, they'll have work to do here as well for us to support this.

[axw]

@cmacknz it's not used any more. We used to make requests to the Fleet Server artifacts API, but no longer. Since we don't specifically interact with Fleet Server in the APM Server code, I don't think we would be affected.

[intxgo]

The biggest missing piece here is that we currently assume the Fleet API key never changes without being re-enrolled. That assumption is invalidated by refreshable JWTs. The current implementation assumes the credentials are static so we'll have to build a way to resend the Fleet configuration whenever the JWT changes. This could be done using the current method of injecting the Fleet configuration or we could add proper support for the Fleet configuration into the control protocol. Either way this needs to be implemented on both side of the protocol, and we have to coordinate this with affected components.

@cmacknz I've been reviewing the protocol from Endpoint side. Ideally I would prefer to have the JWT token passed in the CheckinExpected structure, similar to Features, with fleet config just having a new flag to indicate the use of JWT instead of API key. This is for optimization. Perhaps Tokens having now just the fleet token.

I would prefer that the idx of the configuration unit does not change when only a new token is sent. Otherwise, we will have to do a full comparison to discover the situation and avoid false application of the policy. As you pointed out, currently the fleet configuration is not even a proper unit. Embedding the token in the fleet configuration would require comparing the entire policy. If we make the fleet configuration the proper unit, it would be better than what we have now, but in general the best solution would be to transfer the token outside a configuration unit, and this would make the current location of the fleet configuration irrelevant.

[cmacknz]

I would prefer that the idx of the configuration unit does not change when only a new token is sent. Otherwise, we will have to do a full comparison to discover the situation and avoid false application of the policy.

+1, adding adding a Fleet configuration object to the Features section of the policy which has a separate revision number is the easiest path. You'd still have to compare which features have changed, but for now I don't think endpoint supports any of the existing ones.

Defining the Fleet configuration as a separate unit so you can report on its health is probably the best outcome for a user, but it's more work. Perhaps this would make sense for agent itself as well. We already have a concept of a separate fleet connection state, but it isn't a proper unit either.

[intxgo]

Having it in Features section looks good for me.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)