Searches does not return results and freezes MongoDB

[djensen47]

This is for the latest version 3.0.0

As you know I have a large index of posts: 378,189 (which is down from 1.2M). Despite fewer posts in the index, search rarely returns results. When it does it take 30+ seconds to return.

[djensen47]

Oh, and to add insult to injury, search eventually freezes up MongoDB causing it to be non-responsive, in turn crashing NodeBB.

[djensen47]

Does search acquire a global read lock? I hope I'm reading that wrong. Update: I think the lock is just for counts?

[djensen47]

I'm running on a 4GB instance. XFS disk and THP turned off.

> db.runCommand({ dbstats: 1, scale: 1048576 });
{
        "db" : "nodebb",
        "collections" : 6,
        "views" : 0,
        "objects" : 13574638,
        "avgObjSize" : 1405.2458239402038,
        "dataSize" : 18192.00836277008,
        "storageSize" : 4902.0234375,
        "numExtents" : 0,
        "indexes" : 12,
        "indexSize" : 2760.5546875,
        "fsUsedSize" : 8191.59765625,
        "fsTotalSize" : 20470,
        "ok" : 1
}

> db.runCommand({ dbstats: 1, scale: 1048576 });
{
        "db" : "nodebb",
        "collections" : 6,
        "views" : 0,
        "objects" : 10812543,
        "avgObjSize" : 501.215345455736,
        "dataSize" : 5168.354487419128,
        "storageSize" : 5232.03515625,
        "numExtents" : 0,
        "indexes" : 12,
        "indexSize" : 2804.99609375,
        "fsUsedSize" : 8574.328125,
        "fsTotalSize" : 20470,
        "ok" : 1
}

[barisusakli]

Can you post the output of db.objects.stats() db.searchpost.stats() db.searchtopic.stats()

[barisusakli]

Also compared to our community forum your avg object size is 10x bigger not sure why that is.

> db.runCommand({ dbstats: 1, scale: 1048576 });
{
        "db" : "0",
        "collections" : 3,
        "objects" : 3071782,
        "avgObjSize" : 111.21774071206876,
        "dataSize" : 325.81010246276855,
        "storageSize" : 174.46484375,
        "numExtents" : 0,
        "indexes" : 8,
        "indexSize" : 250.046875,
        "ok" : 1
}

What are the other 3 collections on that database?

[djensen47]

> show collections
objects
pubsub
searchpost
searchtopic
sessions
socket.io

[djensen47]

Can you post the output of db.objects.stats() db.searchpost.stats() db.searchtopic.stats()

The output is massive do you want everything or just a subset?

[djensen47]

Also compared to our community forum your avg object size is 10x bigger not sure why that is.

I still have the __imported_original_data__ fields. I would imagine that would only make the average object size 2-3x.

But, we do have a lot of long posts in our system. Especially the game topics, which comprise about 800k out of 1.2M posts. This topic is an example of that:
https://www.axisandallies.org/forums/topic/29768/ichabod-17-vs-cow-x

---

I'll snapshot my database and run the script to remove the imported data.

[djensen47]

I deleted all the old import data and the data size went down significantly. I updated the dbstats with the "after" data.

[barisusakli]

Yeah post the enitre thing.