BadFormatException in ElementManagementSystem.java

[wittling]

Use Case:

1. Load VIM (OpenStack Generic Driver)

2. Load NSD (I can provide this if you need it but it launches 1 VM and 2 dependent VMs)

3. Launch NSD

4. OpenStack Launches the VMs

5. If we click the VM and examine the syslog, we see it start to do a series of yum installs for Perl

6. Meanwhile, openbaton-generic prints a looped series of messages about waiting for hostnames...

7. All of a sudden, the generic log begins printing the following exception in a loop (which fills up my log and my root file system if I am not paying attention to it).
   ainer.java:1119)
   at java.lang.Thread.run(Thread.java:745)
   Caused by: org.openbaton.common.vnfm_sdk.exception.BadFormatException: Hostname does not fit the expected format. Must fit: '.*-[1-9]+$'
   at org.openbaton.vnfm.generic.core.ElementManagementSystem.extractIdFromHostname(ElementManagementSystem.java:159)
   at org.openbaton.vnfm.generic.core.ElementManagementSystem.unregisterFromMsg(ElementManagementSystem.java:105)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)
   at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:327)
   ... 12 common frames omitted

2017-09-15 19:00:43.136 DEBUG 7629 --- [SimpleAsyncTaskExecutor-1] o.o.v.g.core.ElementManagementSystem : EMSRegister received: {"hostname":"dsxprimary-8282333.novalocal"}
2017-09-15 19:00:43.136 DEBUG 7629 --- [SimpleAsyncTaskExecutor-1] o.o.v.g.core.ElementManagementSystem : extraIdFromHostname: Attempting to find id in hostname = dsxprimary-8282333.novalocal
2017-09-15 19:00:43.136 WARN 7629 --- [SimpleAsyncTaskExecutor-1] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed.

org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method 'unregisterFromMsg' threw exception
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:335)
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:253)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:756)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)

8. Back at the VM, it prints a kernel message (too much work for irq4) because of an overload of messaging from the serial port, I presume. It then tells me "Unit ems.service entered failed state".

[wittling]

As you can see from the debug I posted, the hostname has ".novalocal" appended as a suffix. It is this suffix that is causing the BadFormatException.

(I added debug to the ElementManagementSystem.java class to see this (no logic changes, just debug to see what was being passed in).

I realized that you could edit a parameter in nova.conf and change it from "nova.local" to an empty string of "", and it would suppress this suffix, which is my temporary workaround.

I am still getting errors on another VM, and I think it is because neutron adds a suffix also, to the NICs it creates. But at least for now, I know what is causing the issue.

[gc4rella]

@wittling you are right. The issue is the suffix. It was never happening so far, as most of the time the hostname assigned to a VM via the openstack APIs matches the one later on set in the VM (so not modified by openstack).

The hostname is used by the generic VNFM to uniquely identify a VM<-->EMS agent.

However, I see that this maybe the case for potential bugs also in the future, therefore we'll evaluate a different mechanism for the EMS identification.

Last but not least, the infinite loop is definitively a bug, due to the fact that the BadFormatException is not catched correctly. I'll fix this during next week.

PS: are you using the source code (develop) or binary (release) version?

[gc4rella]

7a7bde7 should be a fix for the infinite loop.

[wittling]

@gc4rella For OpenStack, I'm using the packages. I used the Newton Install doc procedure on OpenStack to get that deployed initially, and then had to debug things and fix them to get it all working. One major issue I ran into with OpenStack is libguestfs. Because OpenStack yum install requires libguestfs as a dependency, I wound up having to remove it from the box, and installing the OpenStack packages as individual rpms with the --nodeps flag. Libguestfs would cause the CPU to go through the roof, and prevent VMs from deploying properly. Everything seems to work fine now without libguestfs.

For OpenBaton, I initially ran your bootstrap scripts to install things. The major issue I ran into with the Bootstrap script is that Oracle had removed JDK1.7 from their website, which would cause your scripts to complain, leaving me wondering if anything would work or not. Unable to find Oracle's 1.7, I downloaded the OpenJDK 1.7, and then started downloading your modules one by one from GIT, and compiling and deploying those using the established /usr/lib/openbaton framework (env, systemd, et al).

BTW - there is another bug in those SystemD unit file env scripts where you have a sed command in there that attempts to figure out the right version-based jar, and this does not work if you recompile the code.

I would then start sprinkling some debug statements into the code when I needed to examine something that did not appear to be working correctly. I also downloaded all of the integration tests, so I am familiar with that as well. I was able to run the iPerf tests, for example.

So this is how I managed to find some of these issues I've been reporting. Rolling up the sleeves and getting into it. :)