Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search results with statistics #2061

Draft
wants to merge 50 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
5a0d132
Save the last search term and added a button to clear text
agrogers Dec 26, 2021
8376a93
Added option to disable search text recall
agrogers Dec 30, 2021
1b1195f
Added fuzzy search
agrogers Dec 30, 2021
8f12d36
.
agrogers Dec 31, 2021
856333a
Improved enabling of controls under different scenarios
agrogers Dec 31, 2021
15f6d79
Merge branch 'develop' into Search-Include-extra-search-options
agrogers Dec 31, 2021
c799b52
Rearranged the form and added Strongs search option
agrogers Jan 1, 2022
2fd5daf
Merge branch 'Search-Include-extra-search-options' into Search-Result…
agrogers Jan 10, 2022
1813f7b
Merge branch 'Search-Strongs-Term-Highlighting' into Search-Results-w…
agrogers Jan 10, 2022
429825f
Initial
agrogers Jan 10, 2022
42fa539
Opens to selected verse
agrogers Jan 10, 2022
07e674c
Buttons displayed using Flow method
agrogers Jan 12, 2022
210ca69
Mostly working with progress bars
agrogers Jan 13, 2022
2966e8b
Implemented Word Stats
agrogers Jan 14, 2022
1764f3b
Implemented filtering by word
agrogers Jan 15, 2022
3f1a7c6
Ensure that strongs number match on a word boundary
agrogers Jan 22, 2022
68bd008
Added verse count string
agrogers Jan 22, 2022
fd39490
Fixed highlighting of strongs words combining spans of adjacent words
agrogers Jan 22, 2022
d899246
Undid some code changes that broke normal highlighting
agrogers Jan 22, 2022
92c32dd
Renamed Files
agrogers Jan 25, 2022
faea5de
Fixed the clearing of the activity when it drops out of view.
agrogers Jan 25, 2022
7217619
Update MySearchResults.kt
agrogers Jan 25, 2022
61333bb
.
agrogers Jan 27, 2022
2bc8c70
Results display finally keeps up to date
agrogers Jan 27, 2022
1401604
Refreshes the book fragment 'correctly'
agrogers Jan 29, 2022
5b0b478
Mostly working now. Need to sort out performance still
agrogers Jan 29, 2022
f10c554
Add some benchmarking to try to work our what is slow
agrogers Jan 29, 2022
7995ad1
Merge branch 'develop' into Search-Results-with-Statistics
agrogers Feb 5, 2022
e3b2f61
Improved performance and added improved highlighting
agrogers Feb 5, 2022
267621e
Added some debugging to check times
agrogers Feb 7, 2022
4c49984
Merge branch 'develop' into Search-Results-with-Statistics
tuomas2 Feb 10, 2022
46c8592
Update build.gradle.kts
agrogers Apr 8, 2022
97aaa48
Merge branch 'develop' into Search-Results-with-Statistics
agrogers Apr 8, 2022
4ee12de
Highlighting bug and code cleanup.
agrogers Apr 19, 2022
b3751c8
Merge branch 'develop' into Search-Results-with-Statistics
tuomas2 May 5, 2022
d0bdd5a
Merge branch 'develop' into Search-Results-with-Statistics
tuomas2 May 6, 2022
8be3724
First batch of code improvements
tuomas2 May 7, 2022
478e418
Merge branch 'develop' into Search-Results-with-Statistics
tuomas2 May 19, 2022
b530111
Merge branch 'develop' into Search-Results-with-Statistics
tuomas2 May 26, 2022
7b84629
Merge branch 'develop' into Search-Results-with-Statistics
agrogers Oct 8, 2022
f923084
Fixed due to changes in the master
agrogers Oct 8, 2022
ceb74e5
Update strings.xml
agrogers Oct 8, 2022
e5478f9
Merge branch 'develop' into Search-Results-with-Statistics
agrogers Nov 12, 2022
20c0e44
Handle strongs numbers that have a letter appended
agrogers Nov 12, 2022
d7fa48a
Implemented the 'Show key words only' option
agrogers Nov 12, 2022
abdd0de
Cleaned the code
agrogers Nov 12, 2022
f987dd2
Exclude short words at the beginning and one punctuation
agrogers Nov 13, 2022
46589d5
Update MySearchResults.kt
agrogers Nov 19, 2022
61a8530
Merge branch 'develop' into Search-Results-with-Statistics
agrogers Dec 16, 2022
48aa5a9
Fixed issues resulting from Tuomas' changes
agrogers Dec 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions app/src/main/AndroidManifest.xml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,11 @@
android:configChanges="keyboardHidden|orientation|screenSize|locale"
android:label="@string/search"
/>
<activity
android:name="net.bible.android.view.activity.search.MySearchResults"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this temporary (2 search results activities)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe. I don't understand much about the manifest :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, we have now SearchResults, and MySearchResults. I think SearchResults is deprecated by MySearchResults. End result should be that we remove old SearchResults and rename MySearchResults -> SearchResults.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sure i had answered this somewhere else. Yes, MySearchResults is a duplicate. The old one could be removed.

android:configChanges="keyboardHidden|orientation|screenSize|locale"
android:label="@string/search"
/>
<activity
android:name="net.bible.android.view.activity.search.SearchIndex"
android:configChanges="keyboardHidden|orientation|screenSize|locale"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import net.bible.android.control.search.SearchControl.SearchBibleSection
import net.bible.android.view.activity.base.CurrentActivityHolder
import net.bible.android.view.activity.base.Dialogs
import net.bible.android.view.activity.page.BibleView
import net.bible.android.view.activity.search.MySearchResults
import net.bible.android.view.activity.search.SearchIndex
import net.bible.android.view.activity.search.SearchResults
import net.bible.service.common.CommonUtils.settings
Expand Down Expand Up @@ -279,7 +280,7 @@ class LinkControl @Inject constructor(
intent = if (needToIndex) {
Intent(activity, SearchIndex::class.java)
} else { //If an indexed Strong's module is in place then do the search - the normal situation
Intent(activity, SearchResults::class.java)
Intent(activity, MySearchResults::class.java)
}
intent.putExtras(searchParams)
activity.startActivity(intent)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,7 @@ import net.bible.android.control.page.CurrentPageManager
/**
* @author Martin Denham [mjdenham at gmail dot com]
*/

//interface ActiveWindowPageManagerProvider {
// val activeWindowPageManager: CurrentPageManager
//}
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,14 @@ import net.bible.android.control.versification.Scripture
import net.bible.android.view.activity.search.Search
import net.bible.android.view.activity.search.SearchIndex
import net.bible.android.view.activity.search.SearchResultsDto
import net.bible.service.common.CommonUtils.limitTextLength
import net.bible.service.sword.SwordContentFacade.getPlainText
import net.bible.service.sword.SwordContentFacade.readOsisFragment
import net.bible.service.sword.SwordContentFacade.search
import net.bible.service.sword.SwordDocumentFacade
import org.apache.commons.lang3.StringUtils
import org.crosswire.jsword.book.Book
import org.crosswire.jsword.book.BookCategory
import org.crosswire.jsword.book.BookException
import org.crosswire.jsword.book.basic.AbstractPassageBook
import org.crosswire.jsword.book.sword.SwordBook
Expand All @@ -38,6 +42,7 @@ import org.crosswire.jsword.index.lucene.LuceneIndex
import org.crosswire.jsword.index.search.SearchType
import org.crosswire.jsword.passage.Key
import org.crosswire.jsword.passage.Verse
import org.jdom2.Element
import javax.inject.Inject

/** Support for the document search functionality
Expand All @@ -52,6 +57,7 @@ class SearchControl @Inject constructor(
)
{
private val isSearchShowingScripture = true
var isStrongsSearch: Boolean = false

enum class SearchBibleSection {
OT, NT, CURRENT_BOOK, ALL
Expand Down Expand Up @@ -95,11 +101,33 @@ class SearchControl @Inject constructor(
"-"
}

fun decorateSearchString(searchString: String, searchType: SearchType, bibleSection: SearchBibleSection, currentBookName: String?): String {
val cleanSearchString = cleanSearchString(searchString)
fun decorateSearchString(searchString: String, searchType: SearchType, bibleSection: SearchBibleSection, currentBookName: String?,
includeAllEndings: Boolean=false, fuzzySearchAccuracy: Double? = null, proximityWords: Int? = null,
strongs: Char? = null): String {
var cleanSearchString = cleanSearchString(searchString)
isStrongsSearch = strongs != null
if (includeAllEndings || isStrongsSearch || fuzzySearchAccuracy != null) {
var newSearchString = ""
val wordArray: List<String> = cleanSearchString.split(" ")
for (it in wordArray) {
var decoratedWord = it
if (includeAllEndings) decoratedWord += "* "
if (isStrongsSearch) decoratedWord = "strong:$strongs$decoratedWord "
if (fuzzySearchAccuracy != null) {
val fuzzySearchAccuracyAdjusted = if (fuzzySearchAccuracy.equals(1.0)) 0.99 else fuzzySearchAccuracy
decoratedWord += "~%.2f ".format(fuzzySearchAccuracyAdjusted)
}
newSearchString += decoratedWord
}
cleanSearchString = newSearchString.trim()
}

if (proximityWords != null) {
cleanSearchString = "\"$cleanSearchString\"~$proximityWords"
}

// add search type (all/any/phrase) to search string
var decorated: String = searchType.decorate(cleanSearchString)
var decorated = searchType.decorate(cleanSearchString)
originalSearchString = decorated

// add bible section limitation to search text
Expand Down Expand Up @@ -139,6 +167,43 @@ class SearchControl @Inject constructor(
return searchResults
}

fun getSearchResultVerseText(key: Key?): String {
agrogers marked this conversation as resolved.
Show resolved Hide resolved
// There is similar functionality in BookmarkControl
// This is much slower than 'getSearchResultVerseElement'. Why? In the old version this was VERY fast.
var verseText = ""
try {
val doc = windowControl.activeWindowPageManager.currentPage.currentDocument
val cat = doc!!.bookCategory
verseText = if (cat == BookCategory.BIBLE || cat == BookCategory.COMMENTARY) {
getPlainText(doc, key)
} else {
val bible = windowControl.activeWindowPageManager.currentBible.currentDocument!!
getPlainText(bible, key)
}
verseText = limitTextLength(verseText)!!
} catch (e: Exception) {
Log.e(TAG, "Error getting verse text", e)
}
return verseText
}

fun getSearchResultVerseElement(key: Key?): Element {
// There is similar functionality in BookmarkControl
var xmlVerse:Element? = null
try {
val doc = windowControl.activeWindowPageManager.currentPage.currentDocument
val cat = doc!!.bookCategory
xmlVerse = if (cat == BookCategory.BIBLE || cat == BookCategory.COMMENTARY) {
readOsisFragment(doc, key)
} else {
val bible = windowControl.activeWindowPageManager.currentBible.currentDocument!!
readOsisFragment(bible, key)
}
} catch (e: Exception) {
Log.e(TAG, "Error getting verse text", e)
}
return xmlVerse!!
}
/** double spaces, :, and leading or trailing space cause lucene errors
*/
private fun cleanSearchString(search: String): String {
Expand Down Expand Up @@ -212,7 +277,7 @@ class SearchControl @Inject constructor(
const val TARGET_DOCUMENT = "TargetDocument"
private const val STRONG_COLON_STRING = LuceneIndex.FIELD_STRONG + ":"
private const val STRONG_COLON_STRING_PLACE_HOLDER = LuceneIndex.FIELD_STRONG + "COLON"
const val MAX_SEARCH_RESULTS = 1000
const val MAX_SEARCH_RESULTS = 10000
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite a lot?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think we need to limit it at all. The most hits i got was about 4000 for some common words. The search itself is very fast. Even big searches in its current form were acceptable (but slow) for me. The complete search results are valuable now because of the stats. My suggestions it make it very large.

private const val TAG = "SearchControl"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ import net.bible.android.view.activity.page.screen.DocumentViewManager
import net.bible.android.view.activity.readingplan.DailyReading
import net.bible.android.view.activity.readingplan.DailyReadingList
import net.bible.android.view.activity.readingplan.ReadingPlanSelectorList
import net.bible.android.view.activity.search.MySearchResults
import net.bible.android.view.activity.search.Search
import net.bible.android.view.activity.search.SearchIndex
import net.bible.android.view.activity.search.SearchIndexProgressStatus
Expand Down Expand Up @@ -102,6 +103,7 @@ interface ActivityComponent {
fun inject(w: SpeakTransportWidget)
fun inject(search: Search)
fun inject(searchResults: SearchResults)
fun inject(mySearchResults: MySearchResults)
fun inject(history: History)

// Services
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -320,10 +320,10 @@ class GridChoosePassageBook : CustomTitlebarActivityBase(R.menu.choose_passage_b
private val HISTORY_COLOR = Color.rgb(0xFE, 0xCC, 0x9B)
private val WISDOM_COLOR = Color.rgb(0x99, 0xFF, 0x99)
private val MAJOR_PROPHETS_COLOR = Color.rgb(0xFF, 0x99, 0xFF)
private val MINOR_PROPHETS_COLOR = Color.rgb(0xFF, 0xFE, 0xCD)
private val MINOR_PROPHETS_COLOR = Color.rgb(0xE6, 0xE5, 0xB8)
private val GOSPEL_COLOR = Color.rgb(0xFF, 0x97, 0x03)
private val ACTS_COLOR = Color.rgb(0x00, 0x99, 0xFF)
private val PAULINE_COLOR = Color.rgb(0xFF, 0xFF, 0x31)
private val PAULINE_COLOR = Color.rgb(0xF5, 0xf5, 0x21)
private val GENERAL_EPISTLES_COLOR = Color.rgb(0x67, 0xCC, 0x66) // changed 99 to CC to make a little clearer on dark background
private val REVELATION_COLOR = Color.rgb(0xFE, 0x33, 0xFF)
private val OTHER_COLOR = ACTS_COLOR
Expand Down
Loading