Home Reference Source Test
public class | source

Tokenizer

You can directly use an instance of this class. tokenizer

This Tokenizer class is for tokenizer sentence

Example:

[EXAMPLE  ERROR! /home/huynhnguyen/github/causality/packages/causality-preprocessing/src/Tokenizer/tokenizer.js, /home/huynhnguyen/github/causality/packages/causality-preprocessing/nlpPreprocessing.babel.js]

Constructor Summary

Public Constructor
public

Member Summary

Public Members
public

model: *

public

vocab: *[]

Method Summary

Public Methods
public

async connect(link: *)

public

encode(text: *): *

public

tokenize(text: *, asEncode: boolean): *

Public Constructors

public constructor() source

Public Members

public model: * source

public vocab: *[] source

Public Methods

public async connect(link: *) source

Params:

NameTypeAttributeDescription
link *

public encode(text: *): * source

Params:

NameTypeAttributeDescription
text *

Return:

*

public tokenize(text: *, asEncode: boolean): * source

Params:

NameTypeAttributeDescription
text *
asEncode boolean
  • optional
  • default: true

Return:

*