Get time lag of monstache sync

[abhishekmathada]

I would like to determine the precise delay between when a document is inserted in MongoDB and the time it takes for it to sync in Elasticsearch. While inserting the document in mongo, I am inserting createdAt field. I want another field in elastic search document which shows the timestamp when it got inserted/updated in elastic search.
Can anyone please help how this can be achieved?

[iBrunox11]

Hi! I’m not sure if this fully addresses your question, but you can manage the "delay" for new document updates in Elasticsearch by adjusting the refresh_interval setting.

For example, you can set refresh_interval to values like "30s" or "1d" depending on how often you need Elasticsearch to refresh and make new changes searchable. Lower values make documents searchable faster but may increase resource usage.

Additionally, you can use a script in Monstache to add a timestamp field to each document in Elasticsearch, indicating when it was synced. Here’s an example of how to set this up in Monstache:

monstache config (config.toml)

[[script]]
namespace = "yourMongoDB.collectionName"
script = """
module.exports = function(doc, ns) {
doc.syncedAt = new Date(); // Add the synced timestamp field
return doc;
}
"""

Ref:
https://opster.com/guides/elasticsearch/glossary/elasticsearch-refresh-interval/
https://rwynn.github.io/monstache-site/config/#script_2