sift function has no tolerance for malformed JSON and fails tests

[cmwylie19]

Sometimes logs come in without complete JSON. It could be related to Istio deployed and pods are logged they are much larger and the log is not always complete json

console.log
    "000,\"protocol\":\"TCP\"}],\"env\":[{\"name\":\"PEPR_WATCH_MODE\",\"value\":\"false\"},{\"name\":\"PEPR_PRETTY_LOG\",\"value\":\"false\"},{\"name\":\"LOG_LEVEL\",\"value\":\"debug\"}],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}},\"volumeMounts\":[{\"name\":\"tls-certs\",\"readOnly\":true,\"mountPath\":\"/etc/certs\"},{\"name\":\"api-token\",\"readOnly\":true,\"mountPath\":\"/app/api-token\"},{\"name\":\"module\",\"readOnly\":true,\"mountPath\":\"/app/load\"},{\"name\":\"kube-api-access-8k54s\",\"readOnly\":true,\"mountPath\":\"/var/run/secrets/kubernetes.io/serviceaccount\"}],\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":3000,\"scheme\":\"HTTPS\"},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":3000,\"scheme\":\"HTTPS\"},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\",\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"runAsUser\":65532,\"runAsGroup\":65532,\"runAsNonRoot\":true,\"allowPrivilegeEscalation\":false}}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":5,\"dnsPolicy\":\"ClusterFirst\",\"serviceAccountName\":\"pepr-soak-ci\",\"serviceAccount\":\"pepr-soak-ci\",\"nodeName\":\"k3d-pexex-hello-pepr-soak-ci-e2e-server-0\",\"securityContext\":{\"runAsUser\":65532,\"runAsGroup\":65532,\"runAsNonRoot\":true,\"fsGroup\":65532},\"schedulerName\":\"default-scheduler\",\"tolerations\":[{\"key\":\"node.kubernetes.io/not-ready\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":300},{\"key\":\"node.kubernetes.io/unreachable\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":300}],\"priorityClassName\":\"system-node-critical\",\"priority\":2000001000,\"enableServiceLinks\":true,\"preemptionPolicy\":\"PreemptLowerPriority\"},\"status\":{\"phase\":\"Running\",\"conditions\":[{\"type\":\"Initialized\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2024-08-14T19:55:45Z\"},{\"type\":\"Ready\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2024-08-14T19:55:55Z\"},{\"type\":\"ContainersReady\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2024-08-14T19:55:55Z\"},{\"type\":\"PodScheduled\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2024-08-14T19:55:45Z\"}],\"hostIP\":\"192.168.144.3\",\"podIP\":\"10.42.0.9\",\"podIPs\":[{\"ip\":\"10.42.0.9\"}],\"startTime\":\"2024-08-14T19:55:45Z\",\"containerStatuses\":[{\"name\":\"server\",\"state\":{\"running\":{\"startedAt\":\"2024-08-14T19:55:46Z\"}},\"lastState\":{},\"ready\":true,\"restartCount\":0,\"image\":\"docker.io/library/pepr:dev\",\"imageID\":\"sha256:7544e479deffb4d5db33ae55933a44d28eec06981040c9fa4ad4aed59242ae5f\",\"containerID\":\"containerd://42c865f568a1a5eccff75f2b3a57bd28007005fd63bd78cb20b96661d16fff91\",\"started\":true}],\"qosClass\":\"Burstable\"},\"msg\":\"Watch event MODIFIED received\"}" SyntaxError: Unexpected number in JSON at position 1 (line 1 column 2)

[cmwylie19]

    "ame\":\"server\",\"state\":{\"running\":{\"startedAt\":\"2024-08-14T19:59:47Z\"}},\"lastState\":{},\"ready\":true,\"restartCount\":0,\"image\":\"docker.io/library/pepr:dev\",\"imageID\":\"sha256:7544e479deffb4d5db33ae55933a44d28eec06981040c9fa4ad4aed59242ae5f\",\"containerID\":\"containerd://918226d93a797e672f88aaae7ebca07fe2688a001f09e7f354ba3a76ac757cd2\",\"started\":true}],\"qosClass\":\"Burstable\"},\"msg\":\"Watch event MODIFIED received\"}" SyntaxError: Unexpected token 'a', "ame":"serv"... is not valid JSON

[cmwylie19]

not sure if this is an issue still. need to actually perform the test in CI and will re-evaluate

[cmwylie19]

this is biting me

 SyntaxError: Unterminated string in JSON at position 65536

[cmwylie19]

  console.log
    error parsing: "apiVersion\":\"v1\",\"fieldPath\":\"metadata.name\"}}},{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"INSTANCE_IP\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"status.podIP\"}}},{\"name\":\"SERVICE_ACCOUNT\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath

[btlghrants]

The error messages should be much more explicit now since #84 merged.

Still doesn't do fix the incomplete streams but should now show offending / unparseable lines directly (rather than burying them in huge, sometimes-ellipsized log dump).