Defining the Future of AI

Q: What is parallel computing?

Parallel computing is both straightforward and fascinating. At its core, it involves using more than one processor to perform a task. Think of it like organizing a group effort among a group of people. Take, for instance, the task of sorting a thousand numbers. This task is challenging for one person but can be made easier by having 100 people sort 10 numbers each. Parallel computing operates on a similar principle, where you coordinate multiple execution units—like our human sorters—to complete a single task.

Essentially, you could say that deep learning is enabled by the availability of massively parallel devices that can train massively parallel models. Today, the workload of an AI system is extremely parallel, allowing it to be distributed across thousands, or even millions, of processing components.

Q: What are some key components for enabling, deploying and advancing AI applications?

The AI revolution we’re seeing today is basically driven by three different components. First, the algorithmic component, which determines the training methods such as stochastic gradient descent. The second is data availability, crucial for feeding models. The third is the compute component, essential for number-crunching.

To build an effective system, we engage in a co- design process. This involves tailoring HPC hardware to fit the specific workload, algorithm and data requirements. One such component is the tensor core.

It’s a specialized matrix multiplication engine integral to deep learning. These cores perform matrix multiplications, a central deep-learning task, at blazingly fast speeds.

Another crucial aspect is the use of specialized, small data types. Deep learning aims to emulate the brain, which is essentially a biological circuit. Our brain, this dark and mushy thing in our heads, is teeming with about 86 billion neurons, each with surprisingly low resolution.

Neuroscientists have shown that our brain differentiates around 24 voltage levels, equivalent to just a bit more than 4 bits. Considering that traditional HPC systems operate at 64 bits, that’s quite an overkill for AI. Today, most deep-learning systems train with 16 bits and can run with 8 bits—sufficient for AI, though not for scientific computing.

Lastly, we look at sparsity, another trait of biological circuits. In our brains, each neuron isn’t connected to every other neuron. This sparse connectivity is mirrored in deep learning through sparse circuits. In NVIDIA hardware, for example, we see 2-to-4 sparsity, meaning out of every four elements, only two are connected. This approach leads to another level of computational speed-up.

Overall, these developments aim to improve computational efficiency—a crucial factor given that companies invest millions, if not billions, of dollars to train deep neural networks.

Q What are some of the most exciting applications of AI?

One of the most exciting prospects is in the weather and climate sciences. Currently some deep-learning models can predict weather at a cost 1,000 times lower than traditional simulations, with comparable accuracy. While these models are still in the research phase, several centers are moving toward production. I anticipate groundbreaking advancements in forecasting extreme events and long-term climate trends. For example, predicting the probability and intensity of typhoons hitting places like Singapore in the coming decades. This is vital for long-term planning, like deciding where to build along coastlines or whether stronger sea defenses are necessary.

Another exciting area is personalized medicine which tailors medical care based on individual genetic differences. With the advent of deep learning and big data systems, we can analyze treatment data from hospitals worldwide, paving the way for customized, effective healthcare based on each person’s genetic makeup.

Finally, most people are familiar with generative AI chatbots like ChatGPT or Bing Chat by now. Such bots are based on large language models with capabilities that border on basic reasoning. They also show primitive forms of logical reasoning. They’re learning concepts like “not cat”, a simple form of negation but a step toward more complex logic. It’s a glimpse into how these models might evolve to compress knowledge and concepts, like how humans developed mathematics as a simplification of complex ideas. It’s a fascinating direction, with potential developments we can only begin to imagine.

Q: What challenges can come up in these areas?

In weather and climate research, the primary challenge is managing the colossal amount of data generated. A single high-resolution, ensemble kilometer-scale climate simulation can produce over an exabyte of data. Handling this data deluge is a significant task and requires innovative strategies for data management and processing.

The shift toward cloud computing has broadened access to supercomputing resources, but this also means handling sensitive data like healthcare records on a much larger scale. Thus, in precision medicine, the main hurdles are security and privacy. There’s a need for careful anonymization to ensure that people can contribute their health records without fear of misuse.

Previously, supercomputers processed highly secure data only in secure facilities that can only be accessed by a limited number of individuals. Now, with more people accessing these systems, ensuring data security is vital. My team recently proposed a new algorithm at the Supercomputing Conference 2023 for security in deep-learning systems using homomorphic encryption which received both the best student paper and the best reproducibility advancement awards. This is a completely new direction that could contribute to solving security in healthcare computing.

For large language models, the challenge lies in computing efficiency, specifically in terms of communication within parallel computing systems. These models require connecting thousands of accelerators through a fast network, but current networks are too slow for these demanding workloads. To address this, we’ve helped to initiate the Ultra Ethernet Consortium, to develop a new AI network optimized for large-scale workloads.

These are just some preliminary solutions in these areas—industry and computing centers need to explore these for implementation and further refine them to make them production-ready.

Q: How can HPC help address AI bias and privacy concerns?

Tackling AI bias and privacy involves two main challenges: ensuring data security and maintaining privacy. The move to digital data processing, even in how secure and private our data is. The challenge is twofold: protecting infrastructure from malicious attacks and ensuring that personal data doesn’t inadvertently become part of training datasets for AI models.

With large language models, the concern is that data fed into systems like ChatGPT might be used for further model training. Companies offer secure, private options, but often at a cost. For example, Microsoft’s retrieval-augmented generation technique ensures data is used only during the session and not embedded in the model permanently.

Regarding AI biases, they often stem from the data itself, reflecting existing human biases. HPC can aid in “de-biasing” these models by providing the computational power needed. De-biasing is a data- intensive process that requires substantial computing resources to emphasize less represented data aspects. It’s mostly on data scientists to identify and rectify biases, a task that requires both computational and ethical considerations.

Q How crucial is international collaboration when it comes to regulating AI?

International collaboration is absolutely crucial. It’s like weapons regulation—if not everyone agrees and abides by the rules, the regulations lose their effectiveness. AI, being a dual-use technology, can be used for beneficial purposes but also has the potential for harm. Technology designed for personalized healthcare, for instance, can be employed in creating biological weapons or harmful chemical compounds.

However, unlike weapons which are predominantly harmful, AI is primarily used for good—enhancing productivity, advancing healthcare, improving climate science and much more. The variety of uses introduces a significant grey area.

Proposals to limit AI capabilities, like those suggested by Elon Musk and others, and the recent US Executive Order requiring registration of large AI models based on compute power, highlight the challenges in this area. This regulation, interestingly defined by computing power, underscores the role of supercomputing in both the potential and regulation of AI.

For regulation to be effective, it absolutely must be a global effort. If only one country or a few countries get on board, it just won’t work. International collaboration is probably the most important thing when we talk about effective AI regulation.