SnmpKit.MIB.SnmpTokenizer (snmpkit v2.0.1)

True 1:1 Elixir port of Erlang SNMP tokenizer (snmpc_tok.erl).

This is a direct translation of the official Erlang SNMP tokenizer from OTP lib/snmp/src/compile/snmpc_tok.erl

Original copyright: Ericsson AB 1996-2025 (Apache License 2.0)

Summary

Types

state()

token()

A token is {category, line} for reserved words and punctuation, or {category, line, value} for :integer, :string, :variable, :atom and :quote. Identifier values (:variable, :atom) are binaries, never runtime-created atoms, so an untrusted MIB cannot grow the atom table. :quote carries the quoted characters as a reversed charlist, matching the OTP snmpc_tok convention the grammar expects.

warning()

A lexical warning: {line, message}. Modelled on libsmi's lexer diagnostics.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

format_error(error)

Format error message. Equivalent to snmpc_tok:format_error/1

get_all_tokens(pid)

Get all remaining tokens. Equivalent to snmpc_tok:get_all_tokens/1

get_token(pid)

Get next token from tokenizer. Equivalent to snmpc_tok:get_token/1

null_get_line()

Null get_line function. Equivalent to snmpc_tok:null_get_line/0

scan(input)

Tokenize MIB text and return lexical warnings alongside the tokens.

start_link(chars, get_line_pid)

Start tokenizer gen_server. Equivalent to snmpc_tok:start_link/2

stop(pid)

Stop tokenizer. Equivalent to snmpc_tok:stop/1

test()

Test function. Equivalent to snmpc_tok:test/0

tokenize(chars, get_line_fun)

Tokenize a string directly. Equivalent to snmpc_tok:tokenize/2

Types

state()

@type state() :: %SnmpKit.MIB.SnmpTokenizer{
  chars: charlist(),
  get_line_fun: function() | nil,
  line: pos_integer(),
  warnings: term()
}

token()

@type token() :: {atom(), pos_integer()} | {atom(), pos_integer(), any()}

A token is {category, line} for reserved words and punctuation, or {category, line, value} for :integer, :string, :variable, :atom and :quote. Identifier values (:variable, :atom) are binaries, never runtime-created atoms, so an untrusted MIB cannot grow the atom table. :quote carries the quoted characters as a reversed charlist, matching the OTP snmpc_tok convention the grammar expects.

warning()

@type warning() :: {non_neg_integer(), String.t()}

A lexical warning: {line, message}. Modelled on libsmi's lexer diagnostics.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

format_error(error)

@spec format_error(term()) :: charlist()

Format error message. Equivalent to snmpc_tok:format_error/1

get_all_tokens(pid)

@spec get_all_tokens(pid()) :: {:ok, [token()]} | {:error, term()}

Get all remaining tokens. Equivalent to snmpc_tok:get_all_tokens/1

get_token(pid)

@spec get_token(pid()) :: {:ok, token()} | {:error, term()}

Get next token from tokenizer. Equivalent to snmpc_tok:get_token/1

null_get_line()

@spec null_get_line() :: :eof

Null get_line function. Equivalent to snmpc_tok:null_get_line/0

scan(input)

@spec scan(binary() | charlist()) :: {:ok, [token()], [warning()]} | {:error, term()}

Tokenize MIB text and return lexical warnings alongside the tokens.

Accepts a binary or a charlist. A binary that is not valid UTF-8 is decoded as Latin-1 with a warning instead of failing, since vendor MIBs are often saved in legacy code pages. Warnings follow libsmi's lexer checks: underscores or trailing hyphens in identifiers, hex strings with an odd digit count, binary strings whose length is not a multiple of eight, and non-radix characters inside those literals.

start_link(chars, get_line_pid)

@spec start_link(
  charlist(),
  pid()
) :: {:ok, pid()} | {:error, term()}

Start tokenizer gen_server. Equivalent to snmpc_tok:start_link/2

stop(pid)

@spec stop(pid()) :: :ok

Stop tokenizer. Equivalent to snmpc_tok:stop/1

test()

Test function. Equivalent to snmpc_tok:test/0

tokenize(chars, get_line_fun)

@spec tokenize(
  charlist(),
  function()
) :: {:ok, [token()]} | {:error, term()}

Tokenize a string directly. Equivalent to snmpc_tok:tokenize/2