[charts][sda-mq] Issue with connecting to CEGA

[josefhandl]

Hello,

I would like to ask you for help with our SDA installation and configuration. I am deploying the SDA but encountered a problem with MQ component. The MQ Pod fails to connect to CEGA due to invalid options:

Shovel 'to_cega' failed to connect (URI: amqps://rabbitmq.test.ega-archive.org:5671/czech): {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
Shovel 'to_cega' has no more URIs to try for connection
Shovel 'to_cega' could not connect to destination: error failed_to_connect_using_provided_uris
...
Federation queue 'from_cega' in vhost 'sda' did not connect to queue 'to_fega' in vhost 'czech' on amqps://rabbitmq.test.ega-archive.org:5671/czech. Reason: {error,{options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}}

Unfortunately, event though I set global.tls.caCert and global.tls.issuer (k8s-local self-signed CA), the setting is not propagated to the Pod. I verified that the var RABBITMQ_SERVER_CACERT: /etc/rabbitmq/tls/ca.crt is set and the file exists, but the problem with Reason: {error,{options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}} is still present.

I also tried to use global.tls.clusterIssuer: letsencrypt-prod instead of the self-signed issuer, but the sda-mq-certs Certificate resource seems to be incompattible with public CA:

Warning  Failed     82s   cert-manager-certificates-issuing          The certificate request has failed to complete and will be retried: Failed to wait for order resource "sda-mq-1-2066896948" to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:unsupportedIdentifier: NewOrder request included invalid non-DNS type identifier: type "ip", value "127.0.0.1"

There is a part of the values.yaml file:

global:
  adminUser: user
  adminPassword: aaaaaaaa
  ingress:
    # extra annotations for the ingress
    annotations:
      kubernetes.io/tls-acme: "true"
    hostName: {} # "mq.fega.cloud.e-infra.cz"
    ingressClassName: "nginx"
    issuer: ""
    clusterIssuer: "letsencrypt-prod"
    # If the certificates is generated by external providers
    # the secrets containing them needs to be created manually.
    secretName: "mq-fega-cloud-e-infra-cz"
  tls:
    enabled: true
    issuer: "ega-ca-ns-issuer"
    #clusterIssuer: "letsencrypt-prod"
    secretName: ""
    certName: tls.crt
    keyName: tls.key
    caCert: ca.crt
    verifyPeer: true
  vhost: "sda"

# Upstream shovel recipient
# only used in a federated setup
  shovel:
    host: rabbitmq.test.ega-archive.org
    pass: "aaaaaaaa"
    port: 5671
    user: user
    vhost: czech

Thank you

[jbygdell]

Interesting issue, let's start with clarifying parts of the TLS settings.

The block global.tls.XXX relates to communication inside the k8s cluster you are using, so the clusterIssuer or Issuer here are the same one that all components of the sensitive-data-archive stack should use, these are usually selfsigned certificates managed by cert-manager.
This has noting to do with external communication to CEGA or other sites outside of your k8s cluster.

The block global.ingress.XXX is for external access to the application, i.e. if you want to use the web UI available for RabbitMQ.
The clusterIssuer or Issuer here are for requesting public certificates from Let's Encrypt or similar institution, to secure the external endpoint.
This has noting to do with external communication to CEGA or other sites outside of your k8s cluster.

CA certificates for validating public certificates already exists in the container and shouldn't be needed.

You can test if there are some network related issues by executing wget from inside the RabbitMQ container, this query should work unless there are some network policies that blocks external access.

kubectl exec RABBITMQ-POD-NAME -- wget -O- www.example.org

[josefhandl]

Thank you for clarification, I was confused about this global.tls option.

I use the same internal Issuer with self-signed cert for all components. So issuer seems to be the right choice for this as well. For now and for testing purposes I don't need external access to the MQ.

I deployed the SDA without any network policy for testing purposes. There is no network policy in the namespace, so no restriction are applied. wget works:

Connecting to www.example.org ([2606:2800:220:1:248:1893:25c8:1946]:80)
writing to stdout
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }
    </style>    
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
-                    100% |*******************************************************************************************************************************************************************************************************|  1256  0:00:00 ETA
written to stdout

[blankdots]

maybe you did already, but have you tried with the address you are trying to connect kubectl exec RABBITMQ-POD-NAME -- wget -O- https://rabbitmq.test.ega-archive.org/ ?

also to double check what version of sda-mq are you using, sometimes erlang has these issues (the error message seems related to erlang/otp#8066 ) and i see the CEGA test env is using 3.11.10 and if the sda-mq is at 3.13.0 wondering if that might cause some issues (had those in the past)

[pontus]

And for extra clarity - if you operate in a "secure" environment that hijacks outgoing connections (e.g. for security scanning or similar), you'd need to make sure to inject the corresponding root CA for that (should likely be made visible by the wget suggested above, but I don't recall if wget and rabbitmq will use the same certificate store).

[josefhandl]

wget works as expected. A few hours ago it was returning 502, but maybe it was just temporary server unavailability, because my laptop was working the same.

Pontus, I am sorry, but maybe I don't understand the note with wget. I keep in mind that for internal communication to services signed by internal CA , the clients must have access to the CA's root certificate.

Based on the RABBITMQ_VERSION variable the RabbitMQ version is 3.13.0. Here is the MQ chart version:

version: 0.7.29
appVersion: v0.3.25

[blankdots]

maybe a shot in the dark, but try downgrading the sda-mq charts to something that has 3.12.x as Rabbitmq version, maybe a version from December 🤔 might be good

[jbygdell]

@josefhandl try with MQ chart version 0.7.13 that should be the latest 3.12.X version.
If that works we'll have to rollback the MQ version to 3.12.X until CEGA decides to update the version they are running.

[josefhandl]

I have tried MQ chart v0.7.13 with untagged image and latest MQ chart with image tag v0.2.31 from 19 Dec. Both variants seem work the same. However, there are many other errors:

...
[error] ** Generic server <0.1074.0> terminating
[error] ** Last message in was {'$gen_cast',
[error]                            {method,
[error]                                {'basic.consume',0,<<"completed_stream">>,
[error]                                    <<>>,false,false,false,false,[]},
[error]                                none,noflow}}
[error] ** When Server state == {ch,{conf,running,rabbit_framing_amqp_0_9_1,1,
[error]                                 <0.1067.0>,<0.1067.0>,<0.988.0>,
[error]                                 <<"<[email protected]>">>,
[error]                                 undefined,
[error]                                 {user,<<"none">>,[],
[error]                                     [{rabbit_auth_backend_dummy,none}]},
[error]                                 <<"sda">>,<<>>,<0.1059.0>,
[error]                                 [{<<"publisher_confirms">>,bool,true},
[error]                                  {<<"exchange_exchange_bindings">>,bool,true},
[error]                                  {<<"basic.nack">>,bool,true},
[error]                                  {<<"consumer_cancel_notify">>,bool,true},
[error]                                  {<<"connection.blocked">>,bool,true},
[error]                                  {<<"authentication_failure_close">>,bool,
[error]                                   true}],
[error]                                 none,1000,134217728,14400000,#{},1000000000,
[error]                                 false},
[error]                             {lstate,<0.1073.0>,false},
[error]                             none,1,
[error]                             {0,[],[]},
[error]                             {state,#{},erlang},
[error]                             #{},#{},
[error]                             {state,fine,5000,
[error]                                 #Ref<0.1607705670.487587843.39159>},
[error]                             false,1,
[error]                             {rabbit_confirms,undefined,#{}},
[error]                             [],[],none,flow,[],
[error]                             {rabbit_queue_type,#{}},
[error]                             #Ref<0.1607705670.487587843.39150>,false}
[error] ** Reason for termination ==
[error] ** {{stream_not_found,{resource,<<"sda">>,queue,<<"completed_stream">>}},
[error]     [{rabbit_queue_type,get_ctx_with,3,
[error]                         [{file,"rabbit_queue_type.erl"},{line,605}]},
[error]      {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,390}]},
[error]      {rabbit_channel,'-basic_consume/8-fun-0-',10,
[error]                      [{file,"rabbit_channel.erl"},{line,1747}]},
[error]      {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,511}]},
[error]      {rabbit_channel,basic_consume,8,
[error]                      [{file,"rabbit_channel.erl"},{line,1744}]},
[error]      {rabbit_channel,handle_method,3,
[error]                      [{file,"rabbit_channel.erl"},{line,1423}]},
[error]      {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,633}]},
[error]      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}
...

[warning] Connection (<0.988.0>) closing: internal error in channel (<0.1067.0>): {{stream_not_found,
[warning]                                                                           {resource,
[warning]                                                                            <<"sda">>,
[warning]                                                                            queue,
[warning]                                                                            <<"completed_stream">>}},
...

[error] Shovel 'completed' in virtual host 'sda' is stopping, reason: {{{stream_not_found,
[error]                                                                  {resource,
[error]                                                                   <<"sda">>,
[error]                                                                   queue,
[error]                                                                   <<"completed_stream">>}},
...

[info] <0.1169.0> osiris_writer:init/1: name: sda_mapping_stream_1712750783982457234 last offset: -1 committed chunk id: -1 epoch: 1
[info] <0.781.0> Ready to start client connection listeners
[info] <0.1326.0> started TLS (SSL) listener on [::]:5671
 completed with 8 plugins.
[info] <0.781.0> Server startup complete; 8 plugins started.
[info] <0.781.0>  * rabbitmq_prometheus
[info] <0.781.0>  * rabbitmq_shovel_management
[info] <0.781.0>  * rabbitmq_federation_management
[info] <0.781.0>  * rabbitmq_federation
[info] <0.781.0>  * rabbitmq_shovel
[info] <0.781.0>  * rabbitmq_management
[info] <0.781.0>  * rabbitmq_management_agent
[info] <0.781.0>  * rabbitmq_web_dispatch
[info] <0.9.0> Time to start RabbitMQ: 14586856 us
[info] <0.1339.0> accepting AMQP connection <0.1339.0> (x.x.230.255:33582 -> x.x.49.81:5671)
[info] <0.1339.0> connection <0.1339.0> (x.x.230.255:33582 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1355.0> accepting AMQP connection <0.1355.0> (x.x.230.255:33594 -> x.x.49.81:5671)
[info] <0.1355.0> connection <0.1355.0> (x.x.230.255:33594 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1371.0> accepting AMQP connection <0.1371.0> (x.x.230.255:33596 -> x.x.49.81:5671)
[info] <0.1371.0> connection <0.1371.0> (x.x.230.255:33596 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1411.0> accepting AMQP connection <0.1411.0> (x.x.56.57:48826 -> x.x.49.81:5671)
[info] <0.1411.0> connection <0.1411.0> (x.x.56.57:48826 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1427.0> accepting AMQP connection <0.1427.0> (x.x.56.41:39412 -> x.x.49.81:5671)
[info] <0.1427.0> connection <0.1427.0> (x.x.56.41:39412 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1443.0> accepting AMQP connection <0.1443.0> (x.x.56.113:58468 -> x.x.49.81:5671)
[info] <0.1443.0> connection <0.1443.0> (x.x.56.113:58468 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1459.0> accepting AMQP connection <0.1459.0> (x.x.56.149:36418 -> x.x.49.81:5671)
[info] <0.1459.0> connection <0.1459.0> (x.x.56.149:36418 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1475.0> accepting AMQP connection <0.1475.0> (x.x.237.100:57566 -> x.x.49.81:5671)
[info] <0.1475.0> connection <0.1475.0> (x.x.237.100:57566 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1491.0> accepting AMQP connection <0.1491.0> (x.x.56.60:34574 -> x.x.49.81:5671)
[info] <0.1491.0> connection <0.1491.0> (x.x.56.60:34574 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[info] <0.1507.0> accepting AMQP connection <0.1507.0> (x.x.56.249:39430 -> x.x.49.81:5671)
[info] <0.1507.0> connection <0.1507.0> (x.x.56.249:39430 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[error] <0.1516.0> Channel error on connection <0.1507.0> (x.x.56.249:39430 -> x.x.49.81:5671, vhost: 'sda', user: 'admin'), channel 1:
[error] <0.1516.0> operation basic.consume caused a channel exception not_found: no queue 'from_cega' in vhost 'sda'
[info] <0.1507.0> closing AMQP connection <0.1507.0> (x.x.56.249:39430 -> x.x.49.81:5671, vhost: 'sda', user: 'admin')
[info] <0.1528.0> accepting AMQP connection <0.1528.0> (x.x.56.249:39438 -> x.x.49.81:5671)
[info] <0.1528.0> connection <0.1528.0> (x.x.56.249:39438 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[error] <0.1536.0> Channel error on connection <0.1528.0> (x.x.56.249:39438 -> x.x.49.81:5671, vhost: 'sda', user: 'admin'), channel 1:
[error] <0.1536.0> operation basic.consume caused a channel exception not_found: no queue 'from_cega' in vhost 'sda'
[warning] <0.1528.0> closing AMQP connection <0.1528.0> (x.x.56.249:39438 -> x.x.49.81:5671, vhost: 'sda', user: 'admin'):
[warning] <0.1528.0> client unexpectedly closed TCP connection
Importing definitions in JSON from a file at "/var/lib/rabbitmq/federation.json" ...
[info] <0.1557.0> Asked to import definitions. Acting user: rmq-internal
[info] <0.1557.0> Importing concurrently 1 exchanges...
[info] <0.1557.0> Importing sequentially 1 policies...
[info] <0.1557.0> Importing sequentially 6 runtime parameters...
[warning] <0.1557.0> Connection (<0.1557.0>): certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
[warning] <0.1557.0> Connection (<0.1557.0>): certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
[info] <0.1557.0> Importing concurrently 1 queues...
[warning] <0.1750.0> Connection (<0.1750.0>): certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
[warning] <0.1750.0> Connection (<0.1750.0>): certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
[warning] <0.1806.0> Connection (<0.1806.0>): certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
Successfully started definition import. This process is asynchronous and can take some time. Watch target node logs for completion.
[warning] <0.1806.0> Connection (<0.1806.0>): certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
[warning] <0.1799.0> Description: "Server authenticity is not verified since certificate path validation is not enabled"
[warning] <0.1799.0>      Reason: "The option {verify, verify_peer} and one of the options 'cacertfile' or 'cacerts' are required to enable this."
[warning] <0.1799.0> 
[warning] <0.1825.0> Description: "Server authenticity is not verified since certificate path validation is not enabled"
[warning] <0.1825.0>      Reason: "The option {verify, verify_peer} and one of the options 'cacertfile' or 'cacerts' are required to enable this."
[warning] <0.1825.0> 
[info] <0.1806.0> Federation queue 'from_cega' in vhost 'sda' connected to queue 'to_fega' in vhost 'czech' on amqps://rabbitmq.test.ega-archive.org:5671/czech
[info] <0.1882.0> accepting AMQP connection <0.1882.0> (x.x.56.249:54388 -> x.x.49.81:5671)
[info] <0.1882.0> connection <0.1882.0> (x.x.56.249:54388 -> x.x.49.81:5671): user 'admin' authenticated and granted access to vhost 'sda'
[warning] <0.1799.0> Connection (<0.1799.0>) closing: received hard error {'connection.close',320,
[warning] <0.1799.0>                                                       <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,
[warning] <0.1799.0>                                                       0,0} from server
[error] <0.1750.0> Shovel 'to_cega' in virtual host 'sda' detected source Shovel (or link,  or channel) failure: <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>
[warning] <0.1825.0> Connection (<0.1825.0>) closing: received hard error {'connection.close',320,
[warning] <0.1825.0>                                                       <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,
[warning] <0.1825.0>                                                       0,0} from server
[error] <0.1750.0> Shovel 'to_cega' in virtual host 'sda' is stopping, reason: <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>
[info] <0.1806.0> Federation queue 'from_cega' in vhost 'sda' disconnected from queue 'to_fega' in vhost 'czech' on amqps://rabbitmq.test.ega-archive.org:5671/czech
[info] <0.1806.0> {upstream_channel_down,
[info] <0.1806.0>     {connection_closing,
[info] <0.1806.0>         {server_initiated_close,320,
[info] <0.1806.0>             <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>}}}
...

[pontus]

I'm not sure I understand if it exists by itself or if you kill it. If it runs until you kill it, does it seem to establish the connections needed?

You may need to throw away the PVC used if downgrading. Do you still the same if you do that?

[josefhandl]

I have a script to uninstall all charts, delete data-sda-mq-0 PVC and install charts again. So I already deleted the PVC.

[josefhandl]

The latest version you released works for me. But I still have some problems with the rabbit.

38;5;160m2024-04-29 12:28:13.102789+00:00 [error] <0.1124.0> Channel error on connection <0.1082.0> (<[email protected]>, vhost: 'sda', user: 'none'), channel 2:0m
38;5;160m2024-04-29 12:28:13.102789+00:00 [error] <0.1124.0> operation queue.declare caused a channel exception not_found: no queue 'completed' in vhost 'sda'0m
38;5;87m2024-04-29 12:28:13.112464+00:00 [notice] <0.1061.0> rabbit_stream_coordinator: candidate -> leader in term: 1 machine version: 40m
2024-04-29 12:28:13.112682+00:00 [info] <0.666.0> ra: started cluster rabbit_stream_coordinator with 1 servers0m
2024-04-29 12:28:13.112682+00:00 [info] <0.666.0> 0 servers failed to start: []0m
2024-04-29 12:28:13.112682+00:00 [info] <0.666.0> Leader: {rabbit_stream_coordinator,'rabbit@sda-mq-0'}0m
2024-04-29 12:28:13.113176+00:00 [info] <0.1061.0> Stream coordinator machine version changes from 0 to 4, applying incremental upgrade.0m
2024-04-29 12:28:13.113229+00:00 [info] <0.1061.0> Stream coordinator machine version changes from 0 to 1, no state changes required.0m
2024-04-29 12:28:13.113345+00:00 [info] <0.1061.0> Stream coordinator machine version changes from 1 to 2, updating state.0m
2024-04-29 12:28:13.113394+00:00 [info] <0.1061.0> Stream coordinator machine version changes from 2 to 3, updating state.0m
2024-04-29 12:28:13.113420+00:00 [info] <0.1061.0> Stream coordinator machine version changes from 3 to 4, updating state.0m
2024-04-29 12:28:13.113701+00:00 [info] <0.1175.0> rabbit_stream_coordinator: started writer sda_mapping_stream_1714393693079475453 on 'rabbit@sda-mq-0' in 10m
2024-04-29 12:28:13.113913+00:00 [info] <0.1183.0> rabbit_stream_coordinator: started writer sda_completed_stream_1714393693079137784 on 'rabbit@sda-mq-0' in 10m
2024-04-29 12:28:13.113834+00:00 [info] <0.1176.0> rabbit_stream_coordinator: started writer sda_error_stream_1714393693079325143 on 'rabbit@sda-mq-0' in 10m
2024-04-29 12:28:13.114139+00:00 [info] <0.1179.0> Stream: sda_mapping_stream_1714393693079475453 will use /var/lib/rabbitmq/mnesia/rabbit@sda-mq-0/stream/sda_mapping_stream_1714393693079475453 for osiris log data directory0m
2024-04-29 12:28:13.114153+00:00 [info] <0.1182.0> Stream: sda_error_stream_1714393693079325143 will use /var/lib/rabbitmq/mnesia/rabbit@sda-mq-0/stream/sda_error_stream_1714393693079325143 for osiris log data directory0m
2024-04-29 12:28:13.114149+00:00 [info] <0.1184.0> Stream: sda_completed_stream_1714393693079137784 will use /var/lib/rabbitmq/mnesia/rabbit@sda-mq-0/stream/sda_completed_stream_1714393693079137784 for osiris log data directory0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0> ** Generic server <0.1070.0> terminating0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0> ** Last message in was {'$gen_cast',0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                            {method,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                {'basic.consume',0,<<"completed_stream">>,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                    <<>>,false,false,false,false,[]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                none,noflow}}0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0> ** When Server state == {ch,{conf,running,rabbit_framing_amqp_0_9_1,1,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 <0.1065.0>,<0.1065.0>,<0.998.0>,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 <<"<[email protected]>">>,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 undefined,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 {user,<<"none">>,[],0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                     [{rabbit_auth_backend_dummy,none}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 <<"sda">>,<<>>,<0.1054.0>,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 [{<<"publisher_confirms">>,bool,true},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                  {<<"exchange_exchange_bindings">>,bool,true},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                  {<<"basic.nack">>,bool,true},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                  {<<"consumer_cancel_notify">>,bool,true},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                  {<<"connection.blocked">>,bool,true},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                  {<<"authentication_failure_close">>,bool,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                   true}],0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 none,1000,134217728,14400000,#{},1000000000,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 false},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             {lstate,<0.1069.0>,false},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             none,1,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             {0,[],[]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             {state,#{},erlang},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             #{},#{},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             {state,fine,5000,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                                 #Ref<0.879396256.705167361.20791>},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             false,1,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             {rabbit_confirms,undefined,#{}},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             [],[],none,flow,[],0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             {rabbit_queue_type,#{}},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                             #Ref<0.879396256.705167464.16921>,false}0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0> ** Reason for termination ==0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0> ** {{stream_not_found,{resource,<<"sda">>,queue,<<"completed_stream">>}},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>     [{rabbit_queue_type,get_ctx_with,3,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                         [{file,"rabbit_queue_type.erl"},{line,605}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,390}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {rabbit_channel,'-basic_consume/8-fun-0-',10,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                      [{file,"rabbit_channel.erl"},{line,1747}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,511}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {rabbit_channel,basic_consume,8,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                      [{file,"rabbit_channel.erl"},{line,1744}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {rabbit_channel,handle_method,3,0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>                      [{file,"rabbit_channel.erl"},{line,1423}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,633}]},0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0>      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}0m
38;5;160m2024-04-29 12:28:13.113636+00:00 [error] <0.1070.0> 0m

Please could you help me with this issue? I connect to the test archive server, with the country as user and vhost. For the local FEGA I have used the default sda value as vhost. I can't get anything out of these logs that would help me, but I am sure the credentials are valid.

[jbygdell]

What are the actual issue here?
I can see similar entries in the logs when starting the MQ in fedraeted mode, but those are due to the async startup of the different components.
After about 35 seconds this line show in the logs Federation queue 'from_cega' in vhost 'sda' connected to queue 'to_fega' in vhost 'cega' on amqp://... at which point no more errors are displayed.

[josefhandl]

Thank you for clarification. I wasn't sure if it is expected behavior caused by the startup process or something to be concerned about.

[jbygdell]

Is this a fresh installation or a upgrade where the old storage volume where re used?

[josefhandl]

Yes, it is a fresh installation.