Build version 0.8.6

weixsong · Feb 28, 2016 · c60724e · c60724e
1 parent 34a30ce
commit c60724e
Show file tree

Hide file tree

Showing 15 changed files with 54 additions and 174 deletions.
diff --git a/CHANGELOG.mdown b/CHANGELOG.mdown
@@ -1,5 +1,10 @@
 # 
 
+## 0.8.6
+
+* update README
+* remove queryNorm compute, it make the score computation complex and useless.[Elasticsearch queryNorm](https://www.elastic.co/guide/en/elasticsearch/guide/current/practical-scoring-function.html#query-norm)
+
 ## 0.8.5
 
 * add trimmer parameter check, throw error if parameter is null or undefined.

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-0.8.5
+0.8.6
diff --git a/bower.json b/bower.json
@@ -1,6 +1,6 @@
 {
   "name": "elasticlunr.js",
-  "version": "0.8.5",
+  "version": "0.8.6",
   "main": "elasticlunr.js",
   "ignore": [
     "tests/",

diff --git a/component.json b/component.json
@@ -1,7 +1,7 @@
 {
   "name": "elasticlunr",
   "repo": "weixsong/elasticlunr.js",
-  "version": "0.8.5",
+  "version": "0.8.6",
   "description": "Lightweight full-text search engine in Javascript for browser search and offline search.",
   "license": "MIT",
   "main": "elasticlunr.js",

diff --git a/docs/index.html b/docs/index.html
@@ -92,7 +92,15 @@ <h1>elasticlunr</h1>
         </div>
         <div class="span9">
           <section id="Main"><h1>Elasticlunr.js</h1>
-<p><a href="https://travis-ci.org/weixsong/elasticlunr.js"><img src="https://travis-ci.org/weixsong/elasticlunr.js.svg?branch=master" alt="Build Status"></a></p><p>Elasticlunr.js is a lightweight full-text search engine developed in JavaScript for browser search and offline search.<br />Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time boosting, field search, more rational scoring/ranking methodology, fast computation speed and so on.<br />Elasticlunr.js is a bit like Solr, but much smaller and not as bright, but also provide flexible configuration, query-time boosting, field search and other features.</p><h1>Key Features Comparing with Lunr.js</h1>
+<p><a href="https://travis-ci.org/weixsong/elasticlunr.js"><img src="https://travis-ci.org/weixsong/elasticlunr.js.svg?branch=master" alt="Build Status"></a></p><p>Elasticlunr.js is a lightweight full-text search engine developed in JavaScript for browser search and offline search.<br />Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time boosting, field search, more rational scoring/ranking methodology, fast computation speed and so on.<br />Elasticlunr.js is a bit like Solr, but much smaller and not as bright, but also provide flexible configuration, query-time boosting, field search and other features.</p><h1>Why You Need Lightweight Offline Search?</h1>
+<ol>
+<li>In some system, you don&#39;t want to deploy any <strong>complex full-text search engine</strong>(such as Lucence, Elasticsearch, Sphinx, etc.), you only want to provide some static web pages and provide search functionality , then you could build index in previous and load index in client side(such as Browser).</li>
+<li>Provide offline search functionality. For some documents, user usually download these documents, you could build index and put index in the documents package, then provide offline search functionality.</li>
+<li>For some limited or restricted network, such WAN or LAN, offline search is a better choice.</li>
+<li>For mobile device, Iphone or Android phone, network traffic maybe very expensive, then provide offline search is a good choice.</li>
+<li>If you want to provide search functionality in your Node.js system, and you don&#39;t want to use a complex system, or you only need to support thousands of documents, then Elasticlunr.js is what you want to use.</li>
+</ol>
+<h1>Key Features Comparing with Lunr.js</h1>
 <ol>
 <li><strong>Query-Time Boosting</strong>, you don&#39;t need to setup boosting weight in index building procedure, Query-Time Boosting make it more flexible that you could try different boosting scheme.</li>
 <li><strong>More Rational Scoring Mechanism</strong>, Elasticlunr.js use quite the same scoring mechanism as Elasticsearch, and also this scoring mechanism is used by lucene. </li>
@@ -150,14 +158,7 @@ <h2>Example</h2>
 });
 </code></pre>
 <p>Then elasticlunr.js will not store the JSON documents, this will reduce the index size, but also bring some inconvenience such as update a document or delete a document by document id or reference. Actually most of the time user will not udpate or delete a document from index. </p><p><a href="http://elasticlunr.com/docs/index.html">API documentation</a> is available, as well as a <a href="http://elasticlunr.com/example/index.html">full working example</a>.</p><h2>Description</h2>
-<p>Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time Boosting, Field Search, more rational scoring/ranking methodology, flexible configuration and so on.<br />A bit like Solr, but much smaller and not as bright, but also provide flexible configuration, query-time boosting, field search, etc.</p><h2>Why</h2>
-<ol>
-<li>In some system, you don&#39;t want to deploy any Web Server(such as Apache, Nginx, etc.), you only provide some static web pages and provide search function in client side. Then you could build index in previous and load index in client side.</li>
-<li>Provide offline search functionality. For some documents, user usually download these documents, you could build index and put index in the documents package, then provide offline search functionality.</li>
-<li>For some limited or restricted network, such WAN or LAN, offline search is a better choice.</li>
-<li>For mobile device, Iphone or Android phone, network traffic maybe very expensive, then provide offline search is a good choice.</li>
-</ol>
-<h2>Installation</h2>
+<p>Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time Boosting, Field Search, more rational scoring/ranking methodology, flexible configuration and so on.<br />A bit like Solr, but much smaller and not as bright, but also provide flexible configuration, query-time boosting, field search, etc.</p><h2>Installation</h2>
 <p>Simply include the elasticlunr.js source file in the page that you want to use it.  Elasticlunr.js is supported in all modern browsers.</p><p>Browsers that do not support ES5 will require a JavaScript shim for Elasticlunr.js to work. You can either use <a href="https://github.com/olivernn/augment.js">Augment.js</a>, <a href="https://github.com/kriskowal/es5-shim">ES5-Shim</a> or any library that patches old browsers to provide an ES5 compatible JavaScript environment.</p><h1>Documentation</h1>
 <p>This part only contain important apects of elasticlunr.js, for the whole documentation, please go to <a href="http://elasticlunr.com/docs/index.html">API documentation</a>.</p><h2>1. Build Index</h2>
 <p>When you first create a index instance, you need to specify which field you want to index. If you did not specify which field to index, then no field will be searchable for your documents.<br />You could specify fields by:</p><pre><code class="lang-javascript">var index = elasticlunr(function () {
@@ -290,16 +291,14 @@ <h2>6. Add customized stop words</h2>
 <li>...</li>
 </ul>
 <p>Defaultly elasticlunr.js contains <strong>120</strong> stop words, user could decide not use these default stop words or add customized stop words.</p><h3>6.1 Remove default stop words</h3>
-<p>You could remove default stop words simply as:</p><pre><code class="lang-javascript">elasticlunr.clearStopWords()
+<p>You could remove default stop words simply as:</p><pre><code class="lang-javascript">elasticlunr.clearStopWords();
 </code></pre>
 <h3>6.2 Add customized stop words</h3>
 <p>User could add a list of customized stop words.</p><pre><code class="lang-javascript">var customized_stop_words = [&#39;an&#39;, &#39;hello&#39;, &#39;xyzabc&#39;];
-
 elasticlunr.addStopWords(customized_stop_words);
 </code></pre>
 <h1>Contributing</h1>
-<p>See the <a href="CONTRIBUTING.mdown"><code>CONTRIBUTING.mdown</code> file</a>.</p><h1>Donate</h1>
-<p>Alipay: [email protected]</p></section>
+<p>See the <a href="CONTRIBUTING.mdown"><code>CONTRIBUTING.mdown</code> file</a>.</p></section>
         </div>
       </div>
     </div>

diff --git a/docs/index.js.html b/docs/index.js.html
@@ -117,8 +117,6 @@ <h1>elasticlunr</h1>
               </li>
               <li><a href="#fieldSearchStats"><i class="alert alert-info"></i><span>fieldSearchStats</span></a>
               </li>
-              <li><a href="#computeSquaredWeight"><i class="alert alert-info"></i><span>computeSquaredWeight</span></a>
-              </li>
               <li><a href="#intersect"><i class="alert alert-info"></i><span>intersect</span></a>
               </li>
               <li><a href="#coordNorm"><i class="alert alert-info"></i><span>coordNorm</span></a>
@@ -608,15 +606,13 @@ <h5 class="subheader"></h5>
   var queryTokens = this.pipeline.run(elasticlunr.tokenizer(query));
 
   var queryResults = {};
-  var squaredWeight = this.computeSquaredWeight(queryTokens, config);
 
   for (var field in config) {
     var fieldSearchResults = this.fieldSearch(queryTokens, field, config);
     var fieldBoost = config[field].boost;
-    var queryNorm = 1 / Math.sqrt(1 / (fieldBoost * fieldBoost) * squaredWeight);
 
     for (var docRef in fieldSearchResults) {
-      fieldSearchResults[docRef] = fieldSearchResults[docRef] * queryNorm;
+      fieldSearchResults[docRef] = fieldSearchResults[docRef] * fieldBoost;
     }
 
     for (var docRef in fieldSearchResults) {
@@ -693,9 +689,9 @@ <h5 class="subheader"></h5>
       for (var docRef in docs) {
         var tf = this.index[fieldName].getTermFrequency(key, docRef);
         var fieldLength = this.documentStore.getFieldLength(docRef, fieldName);
-        var norm = 1;
+        var fieldLengthNorm = 1;
         if (fieldLength != 0) {
-          norm = 1 / Math.sqrt(fieldLength);
+          fieldLengthNorm = 1 / Math.sqrt(fieldLength);
         }
 
         var penality = 1;
@@ -713,7 +709,7 @@ <h5 class="subheader"></h5>
           this.fieldSearchStats(docTokens, key, docs);
         }
 
-        var score = tf * idf * norm * penality;
+        var score = tf * idf * fieldLengthNorm * penality;
 
         if (docRef in scores) {
           scores[docRef] += score;
@@ -774,54 +770,6 @@ <h5 class="subheader"></h5>
       docTokens[doc] = [token];
     }
   }
-};</code></pre>
-          <section id="computeSquaredWeight">
-            <h1>computeSquaredWeight</h1>
-            <h5 class="subheader"></h5>
-            <p>
-              <div class="label label-info radius ctx-type">method</div><span>elasticlunr.Index.prototype.computeSquaredWeight()</span>
-            </p>
-          </section>
-          <table class="table table-bordered table-striped">
-            <thead>
-              <tr>
-                <th style="width:20%">Option name</th>
-                <th style="width:20%">Type</th>
-                <th>Description</th>
-              </tr>
-            </thead>
-            <tbody>
-              <tr>
-                <td>queryTokens</td>
-                <td>Array</td>
-                <td><p>query tokens.</p></td>
-              </tr>
-              <tr>
-                <td>config</td>
-                <td>elasticlunr.Configuration</td>
-                <td><p>The user query config, JSON format.</p></td>
-              </tr>
-              <tr>
-                <td>return</td>
-                <td>Float</td>
-                <td></td>
-              </tr>
-            </tbody>
-          </table>
-          <div class="description"><p>compute squared weight of query tokens.</p> </div>
-          <pre><code class="language-javascript">elasticlunr.Index.prototype.computeSquaredWeight = function (queryTokens, config) {
-  var weight = 0.0;
-  queryTokens.forEach(function (token) {
-    var fieldWeight = 0.0;
-    for (var field in config) {
-      var fieldBoost = config[field].boost;
-      var idf = this.idf(token, field);
-      fieldWeight += idf * idf * fieldBoost * fieldBoost;
-    }
-    weight += fieldWeight;
-  }, this);
-
-  return weight;
 };</code></pre>
           <section id="intersect">
             <h1>intersect</h1>

diff --git a/docs/utils.js.html b/docs/utils.js.html
@@ -172,7 +172,7 @@ <h5 class="subheader"></h5>
   }
 
   return obj.toString();
-}</code></pre>
+};</code></pre>
         </div>
       </div>
     </div>

diff --git a/elasticlunr.js b/elasticlunr.js
@@ -1,6 +1,6 @@
 /**
  * elasticlunr - http://weixsong.github.io
- * Lightweight full-text search engine in Javascript for browser search and offline search. - 0.8.5
+ * Lightweight full-text search engine in Javascript for browser search and offline search. - 0.8.6
  *
  * Copyright (C) 2016 Oliver Nightingale
  * Copyright (C) 2016 Wei Song
@@ -83,7 +83,7 @@ var elasticlunr = function (config) {
   return idx;
 };
 
-elasticlunr.version = "0.8.5";
+elasticlunr.version = "0.8.6";
 /*!
  * elasticlunr.utils
  * Copyright (C) 2016 Oliver Nightingale
@@ -126,7 +126,7 @@ elasticlunr.utils.toString = function (obj) {
   }
 
   return obj.toString();
-}
+};
 /*!
  * elasticlunr.EventEmitter
  * Copyright (C) 2016 Oliver Nightingale
@@ -818,15 +818,13 @@ elasticlunr.Index.prototype.search = function (query, userConfig) {
   var queryTokens = this.pipeline.run(elasticlunr.tokenizer(query));
 
   var queryResults = {};
-  var squaredWeight = this.computeSquaredWeight(queryTokens, config);
 
   for (var field in config) {
     var fieldSearchResults = this.fieldSearch(queryTokens, field, config);
     var fieldBoost = config[field].boost;
-    var queryNorm = 1 / Math.sqrt(1 / (fieldBoost * fieldBoost) * squaredWeight);
 
     for (var docRef in fieldSearchResults) {
-      fieldSearchResults[docRef] = fieldSearchResults[docRef] * queryNorm;
+      fieldSearchResults[docRef] = fieldSearchResults[docRef] * fieldBoost;
     }
 
     for (var docRef in fieldSearchResults) {
@@ -873,9 +871,9 @@ elasticlunr.Index.prototype.fieldSearch = function (queryTokens, fieldName, conf
       for (var docRef in docs) {
         var tf = this.index[fieldName].getTermFrequency(key, docRef);
         var fieldLength = this.documentStore.getFieldLength(docRef, fieldName);
-        var norm = 1;
+        var fieldLengthNorm = 1;
         if (fieldLength != 0) {
-          norm = 1 / Math.sqrt(fieldLength);
+          fieldLengthNorm = 1 / Math.sqrt(fieldLength);
         }
 
         var penality = 1;
@@ -893,7 +891,7 @@ elasticlunr.Index.prototype.fieldSearch = function (queryTokens, fieldName, conf
           this.fieldSearchStats(docTokens, key, docs);
         }
 
-        var score = tf * idf * norm * penality;
+        var score = tf * idf * fieldLengthNorm * penality;
 
         if (docRef in scores) {
           scores[docRef] += score;
@@ -932,28 +930,6 @@ elasticlunr.Index.prototype.fieldSearchStats = function (docTokens, token, docs)
   }
 };
 
-/**
- * compute squared weight of query tokens.
- *
- * @param {Array} queryTokens query tokens.
- * @param {elasticlunr.Configuration} config The user query config, JSON format.
- * @return {Float}
- */
-elasticlunr.Index.prototype.computeSquaredWeight = function (queryTokens, config) {
-  var weight = 0.0;
-  queryTokens.forEach(function (token) {
-    var fieldWeight = 0.0;
-    for (var field in config) {
-      var fieldBoost = config[field].boost;
-      var idf = this.idf(token, field);
-      fieldWeight += idf * idf * fieldBoost * fieldBoost;
-    }
-    weight += fieldWeight;
-  }, this);
-
-  return weight;
-};
-
 /**
  * find documents contain all the query tokens.
  * only for inner use.
-Original file line number
+Diff line change
@@ Expand Up / @@ -172,7 +172,7 @@ <h5 class="subheader"></h5> @@
       }
       return obj.toString();
-    }</code></pre>
+    };</code></pre>
             </div>
           </div>
         </div>
@@ Expand Down @@