Skip to content

Commit

Permalink
feat: discovery powered search and sitemap gen
Browse files Browse the repository at this point in the history
  • Loading branch information
virkt25 committed May 17, 2018
1 parent 7594152 commit bf98bc6
Show file tree
Hide file tree
Showing 17 changed files with 609 additions and 292 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,5 @@ dist
.jekyll-metadata
vendor
.bundle
pages/en/lb4/
pages/**/*/readmes/
3 changes: 2 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ branches:
- gh-pages

before_deploy:
- npm run fetch-readmes
- npm run build
- npm run discovery-upload

deploy:
provider: pages
Expand Down
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ gem 'github-pages'
gem 'jekyll'
gem 'jekyll-redirect-from'
gem 'jekyll-relative-links'
gem 'jekyll-sitemap'
1 change: 1 addition & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,7 @@ DEPENDENCIES
jekyll
jekyll-redirect-from
jekyll-relative-links
jekyll-sitemap
rouge

BUNDLED WITH
Expand Down
87 changes: 78 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,25 @@
# loopback.io

LoopBack community site, http://loopback.io. This website now includes the LoopBack documentation.
LoopBack community site, http://loopback.io. This website now includes the LoopBack documentation.

NOTE: The website is served from the `gh-pages` branch.

This repository is provided under the [MIT License](LICENSE).

## Setup
## Setup

To preview the website locally:

1. Install [Ruby and Bundler](https://help.github.com/articles/setting-up-your-pages-site-locally-with-jekyll/) if you don't have them already.
1. Install [Ruby and Bundler](https://help.github.com/articles/setting-up-your-pages-site-locally-with-jekyll/) if you don't have them already.

2. Clone this repo (you might use the SSH URL instead of HTTPS).:

2. Clone this repo (you might use the SSH URL instead of HTTPS).:
```
git clone https://github.com/strongloop/loopback.io.git
```
3. `cd` to the repository directory and run the following command:

3. `cd` to the repository directory and run the following command:

```
$ cd loopback.io
$ bundle install
Expand All @@ -34,7 +37,7 @@ $ npm start

Then, load [http://localhost:4001/](http://localhost:4001/) on your browser.

NOTE: The docs part will be at [http://localhost:4001/doc](http://localhost:4001/doc). It's not yet linked from the main "overview" part of the site, but will be once we launch (RSN).
NOTE: The docs part will be at [http://localhost:4001/doc](http://localhost:4001/doc). It's not yet linked from the main "overview" part of the site, but will be once we launch (RSN).

## Formatting

Expand All @@ -46,10 +49,10 @@ Jekyll uses the [Liquid template engine](http://liquidmarkup.org/) for templatin

The documentation incorporates README files from a number of LoopBack example repositories.
We use the [get-readmes](https://github.com/strongloop/get-readmes) utility to fetch
the README files directly from GitHub. Here is how to update the READMEs
the README files directly from GitHub. Here is how to update the READMEs

1. `npm install` (*first time/setup only*)
2. `npm run fetch-readmes`
1. `npm install` (_first time/setup only_)
2. `npm run fetch-readmes`

From there, the README markdown files are incorporated into documentation articles
using the standard Jekyll "include" syntax as follows (for example):
Expand Down Expand Up @@ -90,3 +93,69 @@ You can run this script thus:
```js
$ npm run lint-readmes
```

## LoopBack.io Docs Search

Docs Search is powered by [IBM Watson Discovery](https://www.ibm.com/watson/services/discovery/).

### How it works

LoopBack Docs are uploaded to Discovery by Travis CI (before deploying a new
site -- after changes have been merged into `gh-pages` branch) automatically.
The script creates a new [collection](https://console.bluemix.net/docs/services/discovery/data-crawler-qs.html#create-a-collection)
and uploads each doc to that. The content of the docs along with certain
metadata can be found in `_site/posts.json`. Travis uploads each Object in that
file as a separate document (only uploads `documentation` pages).

Travis CI has encrypted variables which sets the environment variables for
Discovery username / password. Core team can reach out to Taranveer Virk /
Diana Lau for these credentials if needed.

* Once docs have been uploaded, Travis will delete the previous collection as
the Search URL uses the oldest collection available to serve results.

If you want to try this out locally, you can create an account for Watson
Discovery (Use the Lite plan which is free). Set the following environment
variables locally:

```
export DISCOVERY_USERNAME="username"
export DISCOVERY_PASSWORD="password"
```

You can now upload docs to your Watson Discovery instance by running

```
npm run discovery-upload
```

### Front End

Each documentation page has a search bar the top. The search will redirect the
user to a `/search` page to show the results. The results are retrieved from a
[IBM Cloud Function](https://www.ibm.com/cloud/functions) that acts as a proxy
to protect Watson Discovery credentials. The search bar also contains a hidden
form input that sets the `sidebar` value and Watson Discovery filters content
based on this value to return context aware results. Ex: Searching for the word
`controller` from a LoopBack 4 documentation page will return LoopBack 4 results.
To search all documents you can search from the `/search` page. Community Docs
and Contribution docs are included in all results.

### Cloud Function

The code for the Cloud function can be found [here](https://github.com/strongloop-internal/loopback-search-function).
It is deployed to the same account and must be edited directly in IBM Cloud
(formerly BlueMix). The repository exists to document code changes and have peer
reviews. Credentials for accessing the Cloud Function can be obtained from
Taranveer Virk / Diana Lau OR you can ask them to re-deploy changes.

#### Your own version

You can deploy the code on your own BlueMix account and upload the URL for
Discovery in [search.html](https://github.com/strongloop/loopback.io/blob/gh-pages/_layouts/search.html).
For the function to work, you must set the following parameters in the function settings:

```
discovery_username: username
discovery_password: password
```
1 change: 1 addition & 0 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ whitelist:
plugins:
- jekyll-redirect-from
- jekyll-relative-links
- jekyll-sitemap

topnav_title: LoopBack
# this appears on the top navigation bar next to the home button
Expand Down
28 changes: 12 additions & 16 deletions _includes/topnav.html
Original file line number Diff line number Diff line change
Expand Up @@ -51,23 +51,19 @@
{% endfor %}
{% if site.feedback_disable == null or site.feedback_disable == false %}
{% include feedback.html %}
{% endif %}
<!-- comment out this block if you want to hide search -->
{% endif %}
<li>
<div class="lb-search-container"><!-- Google Custom Search -->
<script>
(function() {
var cx = '008208300112466859893:btbpudzecwg';
var gcse = document.createElement('script');
gcse.type = 'text/javascript';
gcse.async = true;
gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(gcse, s);
})();
</script>
<gcse:search></gcse:search>
</div><!-- Google Custom Search -->
<form name="search" class="input-group lb-search" action="/search" method="GET">
<input type="text" class="form-control search-bar" placeholder="Search (powered by Watson)" name="q">
{% if page.sidebar contains "lb" %}
<input type="hidden" name="sidebar" value="{{page.sidebar}}">
{% endif %}
<div class="input-group-btn">
<button type="submit" class="btn btn-default" aria-label="search">
<span class="glyphicon glyphicon-search" aria-hidden="true"></span>
</button>
</div>
</form>
</li>
</ul>
</div>
Expand Down
127 changes: 127 additions & 0 deletions _layouts/search.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
<!DOCTYPE html>
<head>
{% include head.html %}

<script>
$(function () {
$('[data-toggle="tooltip"]').tooltip()
})
</script>
{% if page.datatable == true %}
<link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.5/css/jquery.dataTables.css">
<script type="text/javascript" charset="utf8" src="//cdn.datatables.net/1.10.5/js/jquery.dataTables.js"></script>
<script>
$(document).ready(function(){

$('table.datatable').DataTable( {
paging: false,
stateSave: true
}
);
</script>
{% endif %}

</head>
<body>
{% include topnav.html %}
<!-- Page Content -->
<div class="container">
<!-- Content Row -->
<div class="row">
<!-- Content Column -->
<div class="col-md-8 col-md-offset-2 col-sm-12">
<h1>Search Results</h1>
<div class="results">
<img src="/images/LB_Preloader.gif" alt="Loading">
</div>
</div>
<!-- /.row -->
</div>
<!-- /.container -->
</div>

<script>
// List of HTML entities for escaping.
var htmlEscapes = {
'&': '&amp;',
'<': '&lt;',
'>': '&gt;',
'"': '&quot;',
"'": '&#x27;',
'/': '&#x2F;'
};

// Regex containing the keys listed immediately above.
var htmlEscaper = /[&<>"'\/]/g;

// Escape a string for HTML interpolation.

function escapeHTML(string) {
string = ('' + string).replace(htmlEscaper, function(match) {
return htmlEscapes[match];
});
// Undo <em></em> -- used to highlight matches
string = string.replace(/&lt;em&gt;/g, '<em>');
return string.replace(/&lt;&#x2F;em&gt;/g, '</em>');
};

$(document).ready(function() {
var query = window.location.href.slice(window.location.href.indexOf('?') + 1);

if (query && query.length > 0) {
var split = query.split('&');
var queryObj = {};
split.forEach(function(s) {
queryObj[s.split('=')[0]] = s.split('=')[1];
});
}

$('.search-bar').val(queryObj.q.replace('+', ' '));

var urlBase = "https://openwhisk.ng.bluemix.net/api/v1/web/slbld%40ca.ibm.com_loopback/default/search.json";
var url = urlBase + query;

$.get(url).done(function(data) {
if (!data.status) {
$('.results').html('<h3>' + escapeHTML(data.msg) + '</h3>');
} else {
var html = '<ul class="resultList">'
data.results.forEach(function(res) {
if (!res.highlight.text) {
res.highlight.text = []
}
res.highlight.text.forEach(function(t) {
t = escapeHTML(t);
});

html += '<li><a class="title" href="' + escapeHTML(res.metadata.url) + '">' + escapeHTML(res.metadata.title) + '</a><br/>';
html += '<a class="link" href="' + escapeHTML(res.metadata.url) + '">' + window.location.protocol + '//' + window.location.host + escapeHTML(res.metadata.url) + '</a>';
html += '<p>' + res.highlight.text.splice(0, 3).join('<br/>') + '</p></li>';
});
html += '</ul>';

// Page Index
var pages = Math.floor(data.matching_results / 10);
var offset = 0;
if (queryObj.offset) {
offset = Number(queryObj.offset);
}

html += '<ul class="pagination">'

for (var i=0; i<pages; i++) {
queryObj.offset = i*10;
var page = i + 1;
var active = Math.floor(offset / 10) == i ? 'active' : '';

html += '<li class="' + active + '"><a href="/search?' + $.param(queryObj) + '">' + page + '</a></li>';
}

$('.results').html(html);
}
});
})
</script>
</body>

</html>
58 changes: 58 additions & 0 deletions css/customstyles.css
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,69 @@ margin-top: -40px

body h1 {margin-top:40px;}

/* LoopBack Search CSS */

.lb-search-container {
padding: 0 10px;
background-color: #72ac4a;
}

.lb-search {
display: inline-flex;
margin-top: 5px;
margin-left: 5px;
}

.lb-search input {
color: #72ac4a;
}

.search-bar::placeholder {
color: rgba(114, 172, 74, 0.5);
}

.resultList {
list-style: none;
}

.resultList a.title {
font-size: 18px;
font-weight: bold;
color: #72ac4a;
}

.resultList a.link {
color: #72ac4a;
display: block;
margin-top: -5px;
margin-bottom: 5px;
}

.resultList em {
font-weight: bold;
}

.resultList br {
line-height: 175%;
}

.resultList li {
border-bottom: 1px solid #72ac4a;
margin-bottom: 5px;
}

.pagination {
display: table;
margin-left: auto;
margin-right: auto;
}

.pagination a {
color: #72ac4a;
}

/* End of Search CSS */

@media only screen and (min-width: 768px) {
.lb-search-container {
padding-right: 0;
Expand Down
Loading

0 comments on commit bf98bc6

Please sign in to comment.