Package Exports

detect-file-encoding-and-language

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (detect-file-encoding-and-language) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Detect-File-Encoding-and-Language

npm npm npm bundle size

Functionality

Determine the encoding and language of any text file!

Detects 40 languages as well as the appropriate encoding
Works best with large inputs
Completely free, no API key required

For reliable encoding and language detection, use files containing 500 words or more. Smaller inputs can work as well but the results might be less accurate and in some cases incorrect.

Feel free to test the functionality of this NPM package here. Upload your own files and see if the encoding and language are detected correctly!

Index

Detect-File-Encoding-and-Language

Usage

There are several ways in which you can use this NPM package. You can use it as a command-line interface, server-side with Node.js or client-side in the browser.

In the browser

In the body section of your html file, create an input element of type file and give it an id.

// index.html
<body>
  <input type="file" id="my-input-field" />
  <script src="app.js"></script>
</body>

Next, load the module either by using the script tag or by using a bundler!

Using the script tag

When loading it via the <script> tag, you can either use the CDN version or download the code itself and include it in your project. For a quickstart use the CDN version. If you want to be able to use it offline, download and include it!

Via CDN

// index.html

<body>
  <input type="file" id="my-input-field" />
  <script src="https://unpkg.com/detect-file-encoding-and-language/umd/language-encoding.min.js"></script>
  <script src="app.js"></script>
</body>

Now that you've loaded the module, you can start using it.

Via download

Create a new folder called lib inside your root directory
Inside lib create a new file and call it language-encoding.min.js
Make sure the encoding of your newly created file is either UTF-8 or UTF-8 with BOM before proceeding!
Go to https://unpkg.com/detect-file-encoding-and-language/umd/language-encoding.min.js and copy the code
Paste it into language-encoding.min.js and save it
Use the code below to load language-encoding.min.js via the <script> tag.

// index.html

<body>
  <input type="file" id="my-input-field" />
  <script src="lib/language-encoding.min.js"></script>
  <script src="app.js"></script>
</body>

Usage

The <script> tag exposes the languageEncoding function to everything in the DOM located beneath it. You should have no trouble accessing it with Javascript by calling the languageEncoding function and passing in the file that you want to analyze as the only argument. As you can see in the example below, languageEncoding returns a Promise that you can use to retrieve the encoding, language and confidenc score of any text file as long as your text file is large enough.

// app.js

document.getElementById("my-input-field").addEventListener("change", inputHandler);

function inputHandler(e) {
    const file = e.target.files[0];

    languageEncoding(file).then(fileInfo => console.log(fileInfo));
    // Possible result: { language: english, encoding: UTF-8, confidence: 0.97}
}

Using a bundler

Installation

$ npm install detect-file-encoding-and-language

Usage

// app.js

const languageEncoding = require("detect-file-encoding-and-language");

document.getElementById("my-input-field").addEventListener("change", inputHandler);

function inputHandler(e) {
    const file = e.target.files[0];

    languageEncoding(file).then(fileInfo => console.log(fileInfo));
    // Possible result: { language: english, encoding: UTF-8, confidence: 0.97}
}

Note: This works great with frameworks such as React because they are doing the bundling for you. However, if you're using pure vanilla Javascript you will have to bundle it yourself!

In Node.js

Installation

$ npm install detect-file-encoding-and-language

Usage

// index.js

const languageEncoding = require("detect-file-encoding-and-language");

const pathToFile = "/home/username/documents/my-text-file.txt"

languageEncoding(pathToFile).then(fileInfo => console.log(fileInfo));
// Possible result: { language: japanese, encoding: Shift-JIS, confidence: 1 }

In the terminal (CLI)

Installation

$ npm install -g detect-file-encoding-and-language

Usage

Once installed you'll be able to use the command dfeal to retrieve the encoding and language of your text files.

$ dfeal "/home/user name/Documents/subtitle file.srt"
# Possible result: { language: french, encoding: CP1252, confidence: 0.99 }

or without quotation marks, using backslashes to escape spaces:

$ dfeal /home/user\ name/Documents/subtitle\ file.srt
# Possible result: { language: french, encoding: CP1252, confidence: 0.99 }

Supported Languages

Polish
Czech
Hungarian
Romanian
Slovak
Slovenian
Albanian
Russian
Ukrainian
Bulgarian
English
French
Portuguese
Spanish
German
Italian
Danish
Norwegian
Swedish
Dutch
Finnish
Serbo-Croatian
Estonian
Icelandic
Malay-Indonesian
Greek
Turkish
Hebrew
Arabic
Farsi-Persian
Lithuanian
Chinese-Simplified
Chinese-Traditional
Japanese
Korean
Thai
Bengali
Hindi
Urdu
Vietnamese

Used Encodings

UTF-8
CP1250
CP1251
CP1252
CP1253
CP1254
CP1255
CP1256
CP1257
GB18030
BIG5
Shift-JIS
EUC-KR
TIS-620

Confidence Score

The confidence score ranges from 0 to 1. It is based on the amount of matches that were found for a particular language and the frequency of those matches. If you want to learn more about how it all works, check out the Wiki entry!

Known Issues

Unable to detect Shift-JIS encoded Japanese text files when using Node.js. Solutions are welcome!
Unable to detect UTF-16-LE encoded files when using Node.js. Solutions are welcome!

License

This project is licensed under the MIT License