Package opennlp.tools.tokenize

# Class ThreadSafeTokenizerME

java.lang.Object
opennlp.tools.tokenize.ThreadSafeTokenizerME

All Implemented Interfaces:
  `AutoCloseable`, `Tokenizer`

@ThreadSafe public class ThreadSafeTokenizerME extends Object implements Tokenizer, AutoCloseable
A thread-safe version of `TokenizerME`. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

See Also:
  - `Tokenizer`
  - `TokenizerME`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ThreadSafeTokenizerME(String language)`
  Initializes a `ThreadSafeTokenizerME` by downloading a default model for a given `language`.
  `ThreadSafeTokenizerME(TokenizerModel model)`
  Initializes a `ThreadSafeTokenizerME` with the specified `model`.
  `ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)`
  Instantiates a `ThreadSafeTokenizerME` with an existing `TokenizerModel`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `double[]`
  `getProbabilities()`
  `String[]`
  `tokenize(String s)`
  Splits a string into its atomic parts.
  `Span[]`
  `tokenizePos(String s)`
  Finds the boundaries of atomic parts in a string.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### ThreadSafeTokenizerME

    public ThreadSafeTokenizerME(String language) throws IOException
    Initializes a `ThreadSafeTokenizerME` by downloading a default model for a given `language`.
    Parameters:
      `language` - An ISO conform language code.
    Throws:
      `IOException` - Thrown if the model could not be downloaded or saved.

  - ### ThreadSafeTokenizerME

    public ThreadSafeTokenizerME(TokenizerModel model)
    Initializes a `ThreadSafeTokenizerME` with the specified `model`.
    Parameters:
      `model` - A valid `TokenizerModel`.

  - ### ThreadSafeTokenizerME

    public ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)
    Instantiates a `ThreadSafeTokenizerME` with an existing `TokenizerModel`.
    Parameters:
      `model` - The `TokenizerModel` to be used.
      `abbDict` - The `Dictionary` to be used. It must fit the language of the `model`.

- ## Method Details

  - ### tokenize

    public String[] tokenize(String s)
    Description copied from interface: `Tokenizer`
    Splits a string into its atomic parts.
    Specified by:
      `tokenize` in interface `Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The String[] with the individual tokens as the array elements.

  - ### tokenizePos

    public Span[] tokenizePos(String s)
    Description copied from interface: `Tokenizer`
    Finds the boundaries of atomic parts in a string.
    Specified by:
      `tokenizePos` in interface `Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The `spans (offsets into `s`)` for each token as the individuals array elements.

  - ### getProbabilities

    public double[] getProbabilities()

  - ### close

    public void close()
    Specified by:
      `close` in interface `AutoCloseable`