Fastest supported approach to sentence tokenization?

[srubin]

I've found that nlp.world().methods.one.tokenize.splitSentences(text, nlp.world()) is the fastest approach to tokenization in Compromise when I only care about sentence boundaries. Given that nlp.world() seems to be for internal use only, I assume that this isn't intended to be a supported API. I can have unit tests run in our project that verifies this behavior is still working, but, regardless: what's the fastest "supported" way to tokenize sentences in Compromise?

[spencermountain]

hey Steve, this is a good question.
The main takeaway i guess, is that the tokenizer is just a bunch of regexes, and regexes are all fast-enough to be free.

You can just pluck the methods out of here, if you wanted to completely bail. You're also welcome to call that method you found. It's been stable for years.

The slow part by far, is the tagger. so you're encouraged to avoid that. There is also a one-time overhead loading compromise/three, because it builds a gazillion word forms. Sounds like you're avoiding both of those, so yours sounds like a good solution to me.

fan of descript
cheers