Parallel implementation of Gradient Descent

This is the parallel version of the gradient descent algorithm, implemented in C and C++. It is expected to use OpenMP, MPI, oneAPI and CUDA to implement the algorithm. serial.c includes the serial version of the algorithm.

Gradient Descent

The gradient descent algorithm is an iterative optimization algorithm used to minimize a function by iteratively moving in the direction of the steepest descent as defined by the negative of the gradient. The algorithm is commonly used in machine learning and artificial intelligence to optimize models and minimize loss functions.

The gradient descent algorithm is defined by the following equation:

where is the learning rate and

is the gradient of the loss function.

Parallelism

This implementation of the gradient descent algorithm is parallelized using multiple programming models to take advantage of the parallel processing capabilities of modern CPUs and GPUs. The OpenMP and MPI versions are designed for shared-memory and distributed-memory systems, respectively. The oneAPI version is designed to take advantage of the heterogeneous computing capabilities of modern CPUs and GPUs. The CUDA version is designed specifically for NVIDIA GPUs.

This project is intended for educational purposes and to demonstrate the performance benefits of parallel programming. Enjoy!

Usage

Environment

I used an Intel DevCloud environment for this project. The link for it is in the references section. The Intel DevCloud has a Linux environment with the Intel C/C++ compiler and other necessary tools for parallel programming. You can create a free account there.

If you are using VS Code, there is a way to code in the Intel DevCloud using VS Code. They have instructions for that. If you follow their instructions you will find a way to easily code in the DevCloud environment using VS Code.

I also have created some VS Code tasks to compile and run some C/C++ files for VS Code users. Tasks' names are self-explanatory. Check them out.

Serial

serial.c contains the serial version of the program. This program can be compiled and run on any system with a C compiler. To compile the serial version of the program, run the following command:

gcc -o serial serial.c

To run the program, run the following command:

./serial

OpenMP

omp.c contains the OpenMP version of the program. Compile the code running the following command:

gcc -fopenmp -o omp omp.c

To run the program, run the following command:

qsub omp.pbs

MPI

mpi.c contains the MPI version of the program. Compile it by running:

mpicc -o mpi mpi.c

To run the program, run the following command:

qsub mpi.pbs

oneAPI

not yet implemented

CUDA

not yet implemented

References

• Gradient Descent

• OpenMP

• MPI

• oneAPI

• CUDA

• Intel DevCloud