You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
8 lines
313 B
8 lines
313 B
// Package tokenizer implements file tokenization used by the enry content
|
|
// classifier. This package is an implementation detail of enry and should not
|
|
// be imported by other packages.
|
|
package tokenizer
|
|
|
|
// ByteLimit defines the maximum prefix of an input text that will be tokenized.
|
|
const ByteLimit = 100000
|