stanza multilingual pipeline for mlp
Description: Update to Stanza 1.3 multilingual pipeline for MLP.
Actions:
-
- Update Stanza package to 1.3 -
- Use langid processor instead of langdetect to detect input language -
- Use MultilingualPipeline to maintain a cache of pipelines for each language
Tests:
-
- Check that Stanza package 1.3 is stable in MLP -
- Check langid processor is detecting correct language -
- Tests for MultilangualPipelines
Documentation: https://stanfordnlp.github.io/stanza/langid.html