A valid corpus tokens object is (possibly named) list of character vectors. The character vectors, as well as names, should be in UTF-8 encoding. No other attributes should be present in either the list or any of its elements.
Arguments
- tokens
a tokens object to test for validity
- warn
logical. Should the function produce a verbose warning for the condition for which the validation fails. Useful for testing.
Details
The tests are run sequentially and the function returns, with a warning if the warn flag is set, on the first test that fails. We use this implementation because some tests may fail entirely or be meaningless if the prior ones are note passed.