Title
TOKENIZATION BASED ON THE METHOD OF FUNCTIONAL PATTERNS
Journal
Systems and Means of Informatics
Volume 32, Issue 4, pp 59-68
Cover Date
2022-30-11
DOI
10.14357/08696527220406
Print ISSN
0869-6527
Publisher
Institute of Informatics Problems, Russian Academy of Sciences
Additional Links
Key words
tokenization; segmentation; graphematic analysis; computational linguistics; patterns; substitution; token
Authors
Yu. V. Nikitin
![](1.gif)
, A. A. Khoroshilov
![](1.gif)
,
![](2.gif)
,
![](3.gif)
, and A. E. Makarova
Author Affiliations
![](1.gif)
Federal Research Center "Computer Science and Control", Russian Academy of Sciences, 44-2 Vavilov Str., Moscow 119333, Russian Federation
![](2.gif)
Moscow State Aviation Institute (National Research University), 4 Volokolamskoe Shosse, Moscow 125933, Russian Federation
![](3.gif)
27th Central Research Institute of the Ministry of Defence of the Russian Federation, 5, 1st Khoroshevsky Passage, Moscow 123007, Russian Federation
![](4.gif)
Scientific Industrial Joint Stock Company "High Technology and Strategic Systems," 27-9 Elektrozavodskaya Str., Moscow 107023, Russian Federation