Package opennlp.tools.tokenize

# Class TokenizerStream

java.lang.Object
opennlp.tools.tokenize.TokenizerStream

All Implemented Interfaces:
  `AutoCloseable`, `ObjectStream<TokenSample>`

public class TokenizerStream extends Object implements ObjectStream<TokenSample>
The `TokenizerStream` uses a `Tokenizer` to tokenize the input string and output `samples`.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)`
  Initializes a `instance`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  Closes the `ObjectStream` and releases all allocated resources.
  `TokenSample`
  `read()`
  Returns the next `ObjectStream` object.
  `void`
  `reset()`
  Repositions the stream at the beginning and the previously seen object sequence will be repeated exactly.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### TokenizerStream

    public TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)
    Initializes a `instance`.
    Parameters:
      `tokenizer` - A working `Tokenizer` instance.
      `input` - A plain text `line stream`.

- ## Method Details

  - ### read

    public TokenSample read() throws IOException
    Description copied from interface: `ObjectStream`
    Returns the next `ObjectStream` object. Calling this method repeatedly until it returns `null` will return each object from the underlying source exactly once.
    Specified by:
      `read` in interface `ObjectStream<TokenSample>`
    Returns:
      The next object or `null` to signal that the stream is exhausted.
    Throws:
      `IOException` - Thrown if there is an error during reading.

  - ### close

    public void close() throws IOException
    Description copied from interface: `ObjectStream`
    Closes the `ObjectStream` and releases all allocated resources. After close was called, it's not allowed to call `ObjectStream.read()` or `ObjectStream.reset()`.
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `ObjectStream<TokenSample>`
    Throws:
      `IOException` - Thrown if there is an error during closing the stream.

  - ### reset

    public void reset() throws IOException, UnsupportedOperationException
    Description copied from interface: `ObjectStream`
    Repositions the stream at the beginning and the previously seen object sequence will be repeated exactly. This method can be used to re-read the stream if multiple passes over the objects are required.

    The implementation of this method is optional.

    Specified by:
      `reset` in interface `ObjectStream<TokenSample>`
    Throws:
      `IOException` - Thrown if there is an error during resetting the stream.
      `UnsupportedOperationException` - Thrown if the `reset()` is not supported. By default, this is the case.