Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After Pod coming up, it is not Ignition is not coming in logs and endpoint is not showing up #40

Closed
chaets opened this issue Oct 28, 2020 · 4 comments
Closed

After Pod coming up, it is not Ignition is not coming in logs and endpoint is not showing up #40

chaets opened this issue Oct 28, 2020 · 4 comments

Comments

@chaets
Copy link

chaets commented Oct 28, 2020

Igntion Edge version : 7.9.16

Deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name:ignitionedge
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ignitionedge
  template:
    metadata:
      labels:
        app: ignitionedge
    spec:
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      restartPolicy: Always
      imagePullSecrets:
      containers:
      - name: "ignitionedge"
        image: "kcollins/ignition:7.9.16"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 8088
        - containerPort: 8043
        - containerPort: 8060
        - containerPort: 8000
        volumeMounts:
        - mountPath: /var/lib/ignition/
          name: ignition-data
        resources:
            limits:
              cpu: 1500m
              memory: 4096Mi
            requests:
              cpu: 1200m
              memory: 1024Mi
      volumes:
      - name: ignition-data
        persistentVolumeClaim:
          claimName: ignition-data

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ignitionedge
spec:
  type: ClusterIP

  externalIPs:
    - 10.201.10.145
  ports:
  - port: 8088
    targetPort: 8088
    protocol: TCP
    name: user-ui
  - port: 8043
    targetPort: 8043
    protocol: TCP
    name: ssl
  - port: 8060
    targetPort: 8060
    protocol: TCP
    name: metro-ssl
  - port: 8000
    targetPort: 8000
    protocol: TCP
    name: sip

  selector:
    app: ignitionedge

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  name: ignition-data
  namespace: longhorn-system
  labels:
    type: ignition-data-pv
spec:
  storageClassName: manual
  claimRef:
    name: ignition-data
    namespace: prepord
  volumeMode: Filesystem
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
      path: "/usr/local/ignition/"

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ignition-data
  labels:
    type: ignition-data-pvc
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteMany

Error logs

Starting Ignition Gateway...
10/28/2020 12:19:43 PM wrapper  | --> Wrapper Started as Console
10/28/2020 12:19:43 PM �]0;Ignition Gateway�wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.35
10/28/2020 12:19:43 PM wrapper  |   Copyright (C) 1999-2018 Tanuki Software, Ltd. All Rights Reserved.
10/28/2020 12:19:43 PM wrapper  |     http://wrapper.tanukisoftware.com
10/28/2020 12:19:43 PM wrapper  |   Licensed to Inductive Automation for Inductive Automation
10/28/2020 12:19:43 PM wrapper  |
10/28/2020 12:19:43 PM wrapper  | Launching a JVM...
10/28/2020 12:19:43 PM jvm 1    | WrapperManager: Initializing...
10/28/2020 12:19:43 PM jvm 1    | 17:19:41,977 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [data//logback.xml] at [file:/usr/local/share/ignition/data/logback.xml]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,037 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,041 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAppender]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,049 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,089 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [com.inductiveautomation.logging.SQLiteAppender]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,096 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DB]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,230 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@153ef8cb - Driver name=SQLite JDBC
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,230 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@153ef8cb - Driver version=3.23.1
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,230 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@153ef8cb - supportsGetGeneratedKeys=true
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,268 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,270 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAsync]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,270 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAppender] to ch.qos.logback.classic.AsyncAppender[SysoutAsync]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,270 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Attaching appender named [SysoutAppender] to AsyncAppender.
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Setting discardingThreshold to 51
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DBAsync]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DB] to ch.qos.logback.classic.AsyncAppender[DBAsync]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Attaching appender named [DB] to AsyncAppender.
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Setting discardingThreshold to 51
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,271 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAsync] to Logger[ROOT]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,272 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DBAsync] to Logger[ROOT]
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,272 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
10/28/2020 12:19:43 PM jvm 1    | 17:19:42,273 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@534d6567 - Registering current configuration as safe fallback point
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.u.log                   ] [17:19:42]: Logging initialized @808ms to org.eclipse.jetty.util.log.Slf4jLog
10/28/2020 12:19:43 PM jvm 1    | I [Jetpad                        ] [17:19:42]: Could not generate subject alt names, the self generated certificate will not have any metadata. More details in debug log
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.s.Server                ] [17:19:43]: jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 1.8.0_265-b01
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.w.StandardDescriptorProcessor] [17:19:43]: NO JSP Support for /main, did not find org.apache.jasper.servlet.JspServlet
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.s.session               ] [17:19:43]: DefaultSessionIdManager workerName=node0
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.s.session               ] [17:19:43]: No SessionScavenger set, using defaults
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.s.session               ] [17:19:43]: node0 Scavenging every 600000ms
10/28/2020 12:19:43 PM jvm 1    | I [o.e.j.u.TypeUtil              ] [17:19:43]: JVM Runtime does not support Modules
10/28/2020 12:19:43 PM jvm 1    | W [o.e.j.w.WebAppContext         ] [17:19:43]: Failed startup of context o.e.j.w.WebAppContext@8ff6ce8{Ignition,/main,file:///usr/local/share/ignition/webserver/webapps/main/,UNAVAILABLE}
10/28/2020 12:19:43 PM jvm 1    | javax.servlet.ServletException: Unable to start up context. Context temp folder "/var/lib/ignition/temp/" does not exist.
10/28/2020 12:19:43 PM jvm 1    | 	at com.inductiveautomation.ignition.gateway.bootstrap.SRFilter.init(SRFilter.java:50)
10/28/2020 12:19:43 PM jvm 1    | 	at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:314)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751)
10/28/2020 12:19:43 PM jvm 1    | 	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
10/28/2020 12:19:43 PM jvm 1    | 	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
10/28/2020 12:19:43 PM jvm 1    | 	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
10/28/2020 12:19:43 PM jvm 1    | 	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.server.Server.start(Server.java:407)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.server.Server.doStart(Server.java:371)
10/28/2020 12:19:43 PM jvm 1    | 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
10/28/2020 12:19:43 PM jvm 1    | 	at com.inductiveautomation.catapult.Jetpad.start(Jetpad.java:414)
10/28/2020 12:19:43 PM jvm 1    | 	at com.inductiveautomation.catapult.Catapult.start(Catapult.java:138)
10/28/2020 12:19:43 PM jvm 1    | 	at com.inductiveautomation.catapult.Catapult.main(Catapult.java:63)
10/28/2020 12:19:43 PM jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10/28/2020 12:19:43 PM jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10/28/2020 12:19:43 PM jvm 1    | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10/28/2020 12:19:43 PM jvm 1    | 	at java.lang.reflect.Method.invoke(Method.java:498)

Engpoint is look like
image
@thirdgen88
Copy link
Owner

One quick aspect here to start with (I'll need to take a deeper look at this, as I've not done much Kubernetes yet) is that you should be targeting your volume persistence against /var/lib/ignition/data, not /var/lib/ignition, otherwise the modules and pylib will be persisted in your volume and you'll run into trouble with upgrades. I think there might be some deeper problems here, but I wanted to drop that note initially..

@thirdgen88
Copy link
Owner

thirdgen88 commented Oct 28, 2020 •
edited
Loading

Also, can you confirm your image hash for kcollins/ignition:7.9.16? I had to make some updates to that image earlier this month, see #38

@chaets
Copy link
Author

chaets commented Oct 29, 2020

@thirdgen88 When I am using /var/lib/ignition/data its showing following error logs
sed: can't read /usr/local/share/ignition/data/ignition.conf: No such file or directory
and here is the image hash for ignition

image: kcollins/ignition:7.9.16
imageID: docker-pullable://kcollins/ignition@sha256:9636fcffb7fe74d3bf8290f03a5fe3eb04d1e172b5ace885d56fcd0ded329c2c

I tried again today with kcollins/ignition:7.9.16 it is coming fine with /var/lib/ignition but something is fishy which is why it not coming swiftly again and again. Let me do some more testing and will update you with more insights

@thirdgen88
Copy link
Owner

Any updates on this issue @chaets ?

@thirdgen88 thirdgen88 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thirdgen88 @chaets