Four Thoughts on Deep Learning AI in 2022

This article is part of a special issue of VB. Read the full series here: How Data Privacy is Transforming Marketing.

We leave behind another year of exciting developments in the field of deep learning artificial intelligence (AI), a year filled with remarkable progress, controversy and, of course, dispute. As we wrap up 2022 and prepare to embrace what 2023 has in store, here are some of the most notable global trends that have marked this year in deep learning.

1. Scale remains an important factor

One theme that has remained consistent in deep learning over the past few years is the drive to create larger neural networks. The availability of computing resources makes it possible to scale neural networks, as well as specialized AI hardware, large datasets, and the development of scalable architectures like the transformer model. /p>

Right now, companies are getting better results by scaling neural networks to larger sizes. Last year DeepMind announced Gopher, a 280 billion parameter Large Language Model (LLM); Google announced Pathways Language Model (PaLM), with 540 billion parameters, and Generalist Language Model (GLaM), with up to 1.2 trillion parameters; and Microsoft and Nvidia launched the Megatron-Turing NLG, a 530 billion parameter LLM.

One of the interesting aspects of the scale is the emergent capabilities, where the larger models manage to accomplish tasks that were impossible with the smaller ones. This phenomenon has been particularly intriguing in LLMs, where models show promising results on a wider range of tasks and benchmarks as they grow.

Event

Low-Code/No-Code vertex

Join today's top leaders at the Low-Code/No-Code Summit virtually on November 9. Sign up for your free pass today.

register here

It should be noted, however, that some of the fundamental problems of deep learning remain unsolved even in the largest models (more on that in a moment).

2. Unsupervised learning continues to yield results

Many successful deep learning applications require humans to label training examples, also known as supervised learning. But most of the data available on the Internet does not come with the proper labels needed for supervised learning. And data annotation is expensive and slow, creating bottlenecks. That's why researchers have long sought advances in unsupervised learning, where deep learning models are trained without the need for human-annotated data.

There has been tremendous progress in this area in recent years, especially in LLMs, which are mostly trained on large sets of raw data gathered from the internet. As LLMs continued to grow in 2022, we also saw other trends in unsupervised learning techniques gain traction.

For example, there has been tremendous progress in text-to-image conversion models this year. Models like DALL-E 2 from OpenAI, Imagen from Google and Stable Diffusion from Stability AI have shown the power of unsupervised learning. Unlike older text-image models, which required well-annotated pairs of images and descriptions, these models use large datasets of loosely captioned images that already exist on the Internet. The sheer size of their training datasets (which...