Search of documents in cloud-native (Google Kubernetes Cluster) fails

[muruguru]

We are trying to use Phalanx as a search engine and we see that Search of documents after inserting in cloud-native (GKE) as well as stand-along docker steps does not allow for the documents to be returned after a search POST call.

I am seeing the following error during initialization of Phalanx after few documents are inserted:
{"_level_":"error","_timestamp_":"2023-06-21T23:05:42.632-0700","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:144","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-4x76b7O1"}

While doing the search, I am seeing the following warning and no documents returned back:
{"_level_":"warn","_timestamp_":"2023-06-21T23:16:02.958-0700","_name_":"phalanx","_caller_":"server/index_service.go:1241","_message_":"no index readers are assigned","index_name":"example","shard_names":["shard-1TrQbp3v","shard-BALwDsbc","shard-M9UIob2J","shard-linNfk1K","shard-cgllS7mr","shard-Rjjugv2u","shard-4x76b7O1","shard-0AY1YYWM","shard-p5JJAniy","shard-ns5Tb3lf"]}

Our setup is as follows and we have tried both in GKE as well as standalone docker:

• Google Kubernetes Cluster - running etcd, minIO and phalanx as Pods

When running it in a docker mode, I see that the Search is actually crashing with SIGSEGV:

• Steps:
  ** run the docker command that is posted in the documentation above
  ** add the index with index_uri=file:///tmp/phalanx/metadata
  ** add few documents using curl -XPUT which I see those are added in the index and I could see that in the index metastore
  ** Try to search using curl POST call and I see SIGSEGV error

curl -X POST http://localhost:8000/v1/indexes/example/_search -d `{
    "query": "text:example",
    "boost": 1.0,
    "start": 0,
    "num": 10,
    "sort_by": "-_score",
    "fields": [
        "id",
        "text"
    ]
}`

api_1 | {"_level_":"info","_timestamp_":"2023-06-22T20:33:01.158Z","_name_":"phalanx","_caller_":"server/index_service.go:103","_message_":"shard metadata has been created","metastore_event":{"Type":3,"Index":"example","Shard":"shard-pWJ2XGkd"}} api_1 | {"_level_":"info","_timestamp_":"2023-06-22T20:33:01.158Z","_name_":"phalanx","_caller_":"server/index_service.go:198","_message_":"opening index writers"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.161Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.161Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.165Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.165Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.168Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.168Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.171Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.171Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.174Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.174Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.177Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.177Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.180Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.180Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | panic: runtime error: invalid memory address or nil pointer dereference api_1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x122640b] api_1 | api_1 | goroutine 75 [running]: api_1 | github.com/mosuka/phalanx/server.(*IndexService).Search.func1() api_1 | /go/src/github.com/mosuka/phalanx/server/index_service.go:1108 +0x120b api_1 | golang.org/x/sync/errgroup.(*Group).Go.func1() api_1 | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x67 api_1 | created by golang.org/x/sync/errgroup.(*Group).Go api_1 | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x92 phalanx_api_1 exited with code 2
Any help is appreciated.