JSPM – flexsearch@0.1.28

Package Exports

flexsearch

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (flexsearch) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Search Library

Fastest and most memory efficient full text search library with zero dependencies.

When it comes to raw search speed FlexSearch outperforms every single searching library out there and also provides flexible search capabilities like multi-word matching, phonetic transformations or partial matching. It also has the most memory-efficient index. Keep in mind that updating items from the index has a significant cost. When your index needs to be updated continuously then BulkSearch may be a better choice. FlexSearch also provides you a non-blocking asynchronous processing model as well as web workers to perform any updates on the index as well as queries through dedicated threads.

Benchmark:

Library Comparison: https://jsperf.com/compare-search-libraries
BulkSearch vs. FlexSearch: https://jsperf.com/flexsearch

Supported Platforms:

Browser
Node.js

Supported Module Definitions:

AMD (RequireJS)
CommonJS (Node.js)
Closure (Xone)
Global (Browser)

All Features:

Web-Worker Support (not available in Node.js)
Partial Matching
Multiple Words
Phonetic Search
Relevance-based Scoring
Limit Results
Caching
Asynchronous Mode
Custom Matchers
Custom Encoders

Web-Worker Support

Workers get its own dedicated memory. Especially for larger indexes, web worker improves speed and available memory a lot. FlexSearch index was tested with a 250 Mb text file including 10 Million words. The indexing was done silently in background by multiple parallel running workers in about 7 minutes. The final index reserves ~ 8.2 Mb memory/space. The search result took ~ 0.25 ms.

Note: It is slightly faster to use no web worker when the index isn't too big (< 100,000 words).

Compare BulkSearch vs. FlexSearch

Description	BulkSearch	FlexSearch
Access	Read-Write optimized index	Read-Memory optimized index
Memory	Large: ~ 5 Mb per 100,000 words	Tiny: ~ 100 Kb per 100,000 words
Usecase	Limited content Index needs to be updated continously (add, update, remove) Boost query time through pagination	Fastest possible search Rare updates on index Max out memory capabilities
Pagination	Yes	No
Ranked Searching	No	Yes
WebWorker	No	Yes

Installation

HTML / Javascript

<html>
<head>
    <script src="js/flexsearch.min.js"></script>
</head>
...

Note: Use flexsearch.min.js for production and flexsearch.js for development.

Use latest from CDN:

<script src="https://cdn.rawgit.com/nextapps-de/flexsearch/master/flexsearch.min.js"></script>

Node.js

npm install flexsearch

In your code include as follows:

var FlexSearch = require("flexsearch");

Or pass in options when requiring:

var index = require("flexsearch").create({/* options */});

AMD

var FlexSearch = require("./flexsearch.js");

API Overview

Global methods:

FlexSearch.create(<options>)
FlexSearch.addMatcher({KEY: VALUE})
FlexSearch.register(name, encoder)
FlexSearch.encode(name, string)

Index methods:

Index.add(id, string)
Index.search(string, <limit>, <callback>)
Index.update(id, string)
Index.remove(id)
Index.reset()
Index.destroy()
Index.init(<options>)
Index.info()
Index.addMatcher({KEY: VALUE})
Index.encode(string)

Usage

Create a new index

FlexSearch.create(<options>)

var index = new FlexSearch();

alternatively you can also use:

var index = FlexSearch.create();

Create a new index with custom options

var index = new FlexSearch({

    // default values:

    encode: "icase",
    mode: "forward",
    multi: false,
    async: false,
    cache: false
});

Add items to an index

Index.add_(id, string)

index.add(10025, "John Doe");

Search items

Index.search(string|options, <limit>, <callback>)

index.search("John");

Limit the result:

index.search("John", 10);

Perform queries asynchronously:

index.search("John", function(result){
    
    // array of results
});

Update item to the index

Index.update(id, string)

index.update(10025, "Road Runner");

Remove item to the index

Index.remove(id)

index.remove(10025);

Reset index

index.reset();

Destroy the index

index.destroy();

Re-Initialize index

Index.init(<options>)

Note: Re-initialization will also destroy the old index!

Initialize (with same options):

index.init();

Initialize with new options:

index.init({

    /* options */
});

Add custom matcher

FlexSearch.addMatcher({REGEX: REPLACE})

Add global matchers for all instances:

FlexSearch.addMatcher({

    'ä': 'a', // replaces all 'ä' to 'a'
    'ó': 'o',
    '[ûúù]': 'u' // replaces multiple
});

Add private matchers for a specific instance:

index.addMatcher({

    'ä': 'a', // replaces all 'ä' to 'a'
    'ó': 'o',
    '[ûúù]': 'u' // replaces multiple
});

Add custom encoder

Define a private custom encoder during creation/initialization:

var index = new FlexSearch({

    encode: function(str){
    
        // do something with str ...
        
        return str;
    }
});

Register a global encoder to be used by all instances

FlexSearch.register(name, encoder)

FlexSearch.register('whitespace', function(str){

    return str.replace(/ /g, '');
});

Use global encoders:

var index = new FlexSearch({ encode: 'whitespace' });

Call encoders directly

Private encoder:

var encoded = index.encode("sample text");

Global encoder:

var encoded = FlexSearch.encode("whitespace", "sample text");

Mixup/Extend multiple encoders

FlexSearch.register('mixed', function(str){
  
    str = this.encode("icase", str);  // built-in
    str = this.encode("whitespace", str); // custom
    
    return str;
});

FlexSearch.register('extended', function(str){
  
    str = this.encode("custom", str);
    
    // do something additional with str ...

    return str;
});

Get info

index.info();

Returns information about the index, e.g.:

{
    "bytes": 3600356288,
    "id": 0,
    "matchers": 0,
    "size": 10000,
    "status": false
}

Chaining

Simply chain methods like:

var index = FlexSearch.create()
                      .addMatcher({'â': 'a'})
                      .add(0, 'foo')
                      .add(1, 'bar');

index.remove(0).update(1, 'foo').add(2, 'foobar');

Options

Option	Values	Description
mode	"strict" "foward" "inverse" "full"	The indexing mode.
encode	false "icase" "simple" "advanced" "extra" function(string):string	The encoding type. Choose one of the built-ins or pass a custom encoding function.
cache	true false	Enable/Disable caching.
async	true false	Enable/Disable asynchronous processing.
worker	false {number}	Enable and also count of running worker threads.

Phonetic Encoding

Option	Description	False-Positives	Compression
false	Turn off encoding	no	no
"icase"	Case in-sensitive encoding	no	no
"simple"	Phonetic normalizations	no	~ 7%
"advanced"	Phonetic normalizations + Literal transformations	no	~ 35%
"extra"	Phonetic normalizations + Soundex transformations	yes	~ 60%
"ngram"	Phonetic normalizations + N-Gram sequencing	yes	~ 40%

Compare Phonetic Encoder

Reference String: "Björn-Phillipp Mayer"

Query	iCase	Simple	Advanced	Extra	N-Gram
björn	yes	yes	yes	yes	yes
björ	yes	yes	yes	yes	no
bjorn	no	yes	yes	yes	yes
bjoern	no	no	yes	yes	yes
philipp	no	no	yes	yes	yes
filip	no	no	yes	yes	yes
björnphillip	no	yes	yes	yes	no
meier	no	no	yes	yes	yes
björn meier	no	no	yes	yes	yes
meier fhilip	no	no	yes	yes	yes
byorn mair	no	no	no	no	yes
(false positives)	no	no	no	yes	no

Compare Memory Usage

Note: The required memory depends on several options.

Encoding	Memory usage of every ~ 100,000 indexed words
"icase" (default) / false	210 kb
"simple"	190 kb
"advanced"	150 kb
"extra"	90 kb
"ngram"	110 kb
Mode	Multiplied with:
"strict"	x 1
"forward" (default)	x 1.5
"inverse"	x 2
"full"	x 2.3

Author FlexSearch: Thomas Wilkerling
License: Apache 2.0 License