Keywords
Abstract
Maqolada o'zbek tilida tabiiy tilni qayta ishlash (NLP) sohasidagi asosiy muammolar tahlil qilinadi va ularni hal etish yo'llari ko'rib chiqiladi. O'zbek tili agglutinativ tuzilishga ega, ikki yozuv (lotin va kirill) tizimida qo'llanadi va raqamli resurslar jihatidan "kam resursli" tillar qatoriga kiradi. Bu holat katta til modellari, mashina tarjimasi, nutqni tanish va matnni tahlil qilish tizimlarini yaratishda jiddiy qiyinchiliklar tug'diradi. Tadqiqotda morfologik murakkablik, annotatsiyalangan korpuslarning yetishmasligi, yozuvlar aralashuvi va standartlashtirish muammolari ajratib ko'rsatiladi. Yechim sifatida morfologik tahlilga asoslangan tokenizatsiya, ko'p tilli oldindan o'qitilgan modellar (transfer learning), korpus yig'ish va tozalash, kirill–lotin transliteratsiyasi hamda ochiq ma'lumotlar bazalarini rivojlantirish tavsiya etiladi. Maqola oxirida o'zbek tili NLP infratuzilmasini rivojlantirish bo'yicha amaliy takliflar beriladi.
References
Vaswani A., et al. Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
2. Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 2019.
3. Joshi P., Santy S., Budhiraja A., Bali K., Choudhury M. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. ACL, 2020.
4. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units. ACL, 2016.
5. Kudo T., Richardson J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP (System Demonstrations), 2018.
6. Conneau A., et al. Unsupervised Cross-lingual Representation Learning at Scale. ACL, 2020.
7. NLLB Team. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672, 2022.
8. Ardila R., et al. Common Voice: A Massively-Multilingual Speech Corpus. LREC, 2020.