RMQ input error points to localhost when remote host is specified

[JeMunos]

when connecting to a remote RMQ server for input data, if the connection fails the error points to localhost:5671 even though the remote host may be totally different.

[2018-06-19T16:59:26,078][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5671 refused", :exception=>"MarchHare::ConnectionRefused", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:505:in `converting_rjc_exceptions_to_ruby'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:541:in `new_connection_impl'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:534:in `build_new_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:157:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:127:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare.rb:24:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-rabbitmq_connection-5.0.2-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:201:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-rabbitmq_connection-5.0.2-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:138:in `connect!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-rabbitmq-6.0.3/lib/logstash/inputs/rabbitmq.rb:183:in `setup!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-rabbitmq-6.0.3/lib/logstash/inputs/rabbitmq.rb:177:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:512:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:505:in `block in start_input'"]}
[2018-06-19T16:59:26,955][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2018-06-19T16:59:26,958][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
[2018-06-19T16:59:26,959][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/test.conf"}
[2018-06-19T16:59:26,960][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2018-06-19T16:59:27,083][DEBUG][logstash.inputs.rabbitmq ] Connecting to RabbitMQ. Settings: {:vhost=>"/", :hosts=>["export-streaming.com"], :port=>443, :user=>"<UN>", :automatic_recovery=>true, :pass=>"<PW>", :timeout=>0, :heartbeat=>30, :tls=>"TLSv1.2", :tls_certificate_path=>nil, :tls_certificate_password=>nil}
[2018-06-19T16:59:27,085][WARN ][com.rabbitmq.client.TrustEverythingTrustManager] This trust manager trusts every certificate, effectively disabling peer verification. This is convenient for local development but prone to man-in-the-middle attacks. Please see http://www.rabbitmq.com/ssl.html#validating-cerficates to learn more about peer certificate validation.

[dbason]

We are also experiencing this issue using the rabbitmq output plugin.

It seems to be specific to Centos7. This issue doesn't occur on Centos6

[neilc2016]

I am having this issue when upgrading logstash from 6.2.2 to 6.4.1 on Redhat 7. Is there any resolution?

[neilc2016]

I found a workaround. If the port is specified in the host configuration like below the logstash agent connects to rabbitmq correctly.

rabbitmq {
host => "ves-elkmq-test.ebiz.verizon.com:35672"
vhost => "vapews"
exchange => "vapews_exchange"
queue => "vapews_queue"
key => "vapews_key"
user => "vapews"
password => "xxxxxx"
ack => false
threads => 1
durable => true
#port => 35672
ssl => true
ssl_version => "TLSv1.2"
}

[keitalbame]

Having the same issue with the container docker.elastic.co/logstash/logstash:6.6.2 when setting up TLS. Without TLS everything works fine.
Applying the workaround mentioned by @neilc2016 did not helped because if no port is setup, it will default to 5672.

logstash_1  | [2019-03-29T11:27:17,422][DEBUG][logstash.outputs.rabbitmq] Connecting to RabbitMQ. Settings: {:vhost=>"/", :hosts=>["rabbitmq:5671"], :port=>5672, :user=>"xxx", :automatic_recovery=>true, :pass=>"xxx", :timeout=>0, :heartbeat=>0, :tls=>"TLSv1.2", :tls_certificate_path=>xxx, :tls_certificate_password=>xxx}
logstash_1  | Using TLS/SSL version TLSv1.2
logstash_1  | [2019-03-29T11:27:17,422][WARN ][com.rabbitmq.client.TrustEverythingTrustManager] This trust manager trusts every certificate, effectively disabling peer verification. This is convenient for local development but prone to man-in-the-middle attacks. Please see http://www.rabbitmq.com/ssl.html#validating-cerficates to learn more about peer certificate validation.
logstash_1  | [2019-03-29T11:27:17,426][ERROR][logstash.outputs.rabbitmq] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5671 failed", :exception=>"MarchHare::ConnectionRefused", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:511:in `converting_rjc_exceptions_to_ruby'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:541:in `new_connection_impl'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:534:in `build_new_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:157:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:127:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare.rb:24:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-rabbitmq_connection-5.0.2-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:201:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-rabbitmq_connection-5.0.2-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:138:in `connect!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-rabbitmq-5.1.1-java/lib/logstash/outputs/rabbitmq.rb:47:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:102:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:46:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:611:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:280:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"]}

Keeping port => 5671 also does not help solving the localhost issue.

Possibly related with ruby-amqp/march_hare#133

[ymougenel]

Hello, ☺️
If it is ok with you, I can upgrade the march dependency (4.1.0 fixes the output) and test it with my local set up?

[vurso]

I have this issue with the following setup:

docker ELK+RabbitMQ stack
Windows 10 Pro 16GB ram
vmem fix applied to both docker-desktop and the WSL 2 Ubuntu distro

Have tried various ports, changed the hostname but that didn't help the other containers spin up fine I can view Kibana and get to the RMQ management interface.

My logstash.conf:

`input {
tcp {
port => 5000
}
rabbitmq {
host => "rabbit"
port => 5672
queue => "my-test-queue-001"
durable => true
passive => true
exchange => "amq.default"
user => "testuser001"
password => "s0merandomP@ss"
}
beats {
port => "5044"
}
}

output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}`

Thanks in advance.

[vurso]

I have this issue with the following setup:

docker ELK+RabbitMQ stack
Windows 10 Pro 16GB ram
vmem fix applied to both docker-desktop and the WSL 2 Ubuntu distro

Have tried various ports, changed the hostname but that didn't help the other containers spin up fine I can view Kibana and get to the RMQ management interface.

My logstash.conf:

`input {
tcp {
port => 5000
}
rabbitmq {
host => "rabbit"
port => 5672
queue => "my-test-queue-001"
durable => true
passive => true
exchange => "amq.default"
user => "testuser001"
password => "s0merandomP@ss"
}
beats {
port => "5044"
}
}

output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}`

Thanks in advance.

Hey folks I found that changing the port for RMQ to say 5674 works even though the log out will say connection refused just ignore it, daft I know but it get me past this problem (see ref: https://gitmemory.com/issue/logstash-plugins/logstash-output-rabbitmq/76/456807342)