Improving language understanding with unsupervised learning

Researchers achieved state-of-the-art results on diverse language tasks using a scalable system combining transformers and unsupervised pre-training. The approach demonstrates that pairing supervised learning with unsupervised pre-training is highly effective for language understanding tasks.

• →State-of-the-art results achieved on multiple language understanding tasks using a scalable, task-agnostic system.
• →The approach combines two existing concepts: transformer architecture and unsupervised pre-training.
• →Results provide strong evidence that supervised learning paired with unsupervised pre-training works very well.
• →The research team is releasing their system to the public.
• →Findings may motivate further research into applying this approach on larger and more diverse datasets.