Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed zlibjs and switched dictionaries to be uncompressed by default. #33

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Directory tree is as follows:
build/
kuromoji.js -- JavaScript file for browser (Browserified)
demo/ -- Demo
dict/ -- Dictionaries for tokenizer (gzipped)
dict/ -- Dictionaries for tokenizer (uncompressed)
example/ -- Examples to use in Node.js
src/ -- JavaScript source
test/ -- Unit test
Expand Down Expand Up @@ -57,7 +57,7 @@ You can prepare tokenizer like this:

### Browser

You only need the build/kuromoji.js and dict/*.dat.gz files
You only need the build/kuromoji.js and dict/*.dat files

Install with Bower package manager:

Expand All @@ -77,6 +77,10 @@ In your JavaScript:
console.log(path);
});

On your server:

Host the dictionary files and compress them using either gzip or brotli. Make sure to
set the `Content-Type` header so they are automatically decompressed by the browser.

API
---
Expand Down
24 changes: 12 additions & 12 deletions bower.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,18 @@
"description": "JavaScript implementation of Japanese morphological analyzer",
"main": [
"build/kuromoji.js",
"dict/base.dat.gz",
"dict/cc.dat.gz",
"dict/check.dat.gz",
"dict/tid.dat.gz",
"dict/tid_map.dat.gz",
"dict/tid_pos.dat.gz",
"dict/unk.dat.gz",
"dict/unk_char.dat.gz",
"dict/unk_compat.dat.gz",
"dict/unk_invoke.dat.gz",
"dict/unk_map.dat.gz",
"dict/unk_pos.dat.gz"
"dict/base.dat",
"dict/cc.dat",
"dict/check.dat",
"dict/tid.dat",
"dict/tid_map.dat",
"dict/tid_pos.dat",
"dict/unk.dat",
"dict/unk_char.dat",
"dict/unk_compat.dat",
"dict/unk_invoke.dat",
"dict/unk_map.dat",
"dict/unk_pos.dat"
],
"keywords": [
"morphological analyzer",
Expand Down
91 changes: 30 additions & 61 deletions build/kuromoji.js

Large diffs are not rendered by default.

Binary file added dict/base.dat
Binary file not shown.
Binary file removed dict/base.dat.gz
Binary file not shown.
Binary file added dict/cc.dat
Binary file not shown.
Binary file removed dict/cc.dat.gz
Binary file not shown.
Binary file added dict/check.dat
Binary file not shown.
Binary file removed dict/check.dat.gz
Binary file not shown.
Binary file added dict/tid.dat
Binary file not shown.
Binary file removed dict/tid.dat.gz
Binary file not shown.
Binary file added dict/tid_map.dat
Binary file not shown.
Binary file removed dict/tid_map.dat.gz
Binary file not shown.
Binary file added dict/tid_pos.dat
Binary file not shown.
Binary file removed dict/tid_pos.dat.gz
Binary file not shown.
Binary file added dict/unk.dat
Binary file not shown.
Binary file removed dict/unk.dat.gz
Binary file not shown.
267 changes: 267 additions & 0 deletions dict/unk_char.dat

Large diffs are not rendered by default.

Binary file removed dict/unk_char.dat.gz
Binary file not shown.
Binary file added dict/unk_compat.dat
Binary file not shown.
Binary file removed dict/unk_compat.dat.gz
Binary file not shown.
Binary file added dict/unk_invoke.dat
Binary file not shown.
Binary file removed dict/unk_invoke.dat.gz
Binary file not shown.
Binary file added dict/unk_map.dat
Binary file not shown.
Binary file removed dict/unk_map.dat.gz
Binary file not shown.
Binary file added dict/unk_pos.dat
Binary file not shown.
Binary file removed dict/unk_pos.dat.gz
Binary file not shown.
9 changes: 1 addition & 8 deletions gulpfile.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ const merge = require('event-stream').merge;
const jshint = require("gulp-jshint");
const browserify = require("browserify");
const source = require("vinyl-source-stream");
const gzip = require("gulp-gzip");
const mocha = require("gulp-mocha");
const istanbul = require("gulp-istanbul");
const webserver = require('gulp-webserver');
Expand Down Expand Up @@ -132,18 +131,12 @@ gulp.task("create-dat-files", (done) => {
});
});

gulp.task("compress-dict", () => {
return gulp.src("dict/*.dat")
.pipe(gzip())
.pipe(gulp.dest("dict/"));
});

gulp.task("clean-dat-files", (done) => {
return del([ "dict/*.dat" ], done);
});

gulp.task("build-dict", [ "build", "clean-dict" ], () => {
sequence("create-dat-files", "compress-dict", "clean-dat-files");
sequence("create-dat-files", "clean-dat-files");
});

gulp.task("test", [ "build" ], () => {
Expand Down
Loading