Hidden parallelism is parallelism that is not explicitly visible to the programmer. It is parallelism that is automatically exploited by the compiler or runtime system. This type of parallelism can be found in many different areas of computer science, including:
- Data parallelism: This is the most common type of hidden parallelism. It occurs when the same operation is performed on different data elements. For example, the `for` loop in the Python code you provided is an example of data parallelism.
- Task parallelism: This type of parallelism occurs when different tasks can be executed independently of each other. For example, the `map()` function in Python can be used to execute a function on each element of a list in parallel.
- I/O parallelism: This type of parallelism occurs when different I/O operations can be executed independently of each other. For example, the `open()` function in Python can be used to open multiple files in parallel.
Hidden parallelism can be a great way to improve the performance of your code. However, it is important to be aware of it so that you can understand how it works and how it can affect your code.
Benefits
- Increased performance: Hidden parallelism can help to increase the performance of your code by allowing it to be executed on multiple cores or CPUs simultaneously.
- Reduced development time: Hidden parallelism can help to reduce the development time of your code by allowing you to focus on the logic of your code without having to worry about the details of parallelization.
- Simplified code: Hidden parallelism can help to simplify your code by reducing the need for explicit parallelization constructs.
Drawbacks
- Increased complexity: Hidden parallelism can increase the complexity of your code by introducing new dependencies and synchronization issues.
- Reduced control: Hidden parallelism can reduce your control over the execution of your code, making it more difficult to debug and profile.
- Potential for race conditions: Hidden parallelism can introduce the possibility of race conditions, which are errors that can occur when multiple threads or processes access the same data at the same time.
Parallel BLAS
Parallel BLAS is a set of linear algebra subroutines that are optimized for specific CPUs or chipsets. Using an optimized BLAS can dramatically improve the performance of linear algebra computations in R.
Benefits
- Increased performance: Optimized BLAS libraries are often much faster than the default BLAS library that comes with R.
- Parallelization: Many optimized BLAS libraries are multi-threaded, which means that they can split work off into chunks and run them in parallel on multi-core machines.
- Customization: Optimized BLAS libraries are often customized for specific CPUs or chipsets, which can further improve performance.
If you are doing a lot of linear algebra computations in R, then it is a good idea to install an optimized BLAS library. There are a number of different optimized BLAS libraries available, so you can choose one that is optimized for your specific CPU or chipset.
Popular optimized BLAS libraries
- AMD Core Math Library (ACML)
- Intel Math Kernel Library (MKL)
- Accelerate framework on the Mac
- Automatically Tuned Linear Algebra Software (ATLAS)
Detailed instructions on how to use R with optimized BLAS libraries can be found in the R Installation and Administration manual. Here is an example of how to use parallel BLAS in R:
library(Rcpp)
library(RcppParallel)
# Create a matrix of 1 million observations by 100 predictors
X <- matrix(rnorm(1e6 * 100), 1e6, 100)
# Compute the least squares estimates of the linear regression coefficients
# when regressing the response y on the predictor matrix X.
b <- solve(crossprod(X), crossprod(X, y))
This code will use the RcppParallel package to parallelize the matrix inversion step. If you have an optimized BLAS library installed, then this code will run much faster than if you use the default BLAS library.
