Sparse research can boost deep learning efficiency. To use sparsity in practical applications, hardware, software, and algorithm development must work together. A versatile toolkit is needed for rapid concept generation and testing against dynamic benchmarks in such partnerships.
Sparsity may occur in neural network activations or parameters. JaxPruner prioritizes sparsity. Previous studies showed that it outperforms dense models with the same amount of parameters.
Recently, scientists have used JAX more. JAX differs from PyTorch and TensorFlow by separating functions and states. Due to its data independence, parameter sparsity can be hardware accelerated.
This research examines pruning, which creates sparse networks from dense networks for efficient inference, and sparse training, which creates sparse networks from scratch while reducing training costs.
This simplifies function transformations like gradients, hessian calculations, and vectorization, reducing implementation time. When its state is in one place, changing a function is easy. As they will see, these properties make it easier to create shared procedures across pruning and sparse training approaches.
JAX needs a sparsity research library despite N: M sparse training and quantization. Google Research created JaxPruner.
They want JaxPruner to help us answer key questions like “Which sparsity pattern achieves a desired trade-off between accuracy and performance?” “Can sparse networks be trained without first training a large dense model?” When creating the library, they followed three principles:
Fast Integration Machine learning advances rapidly. Due to the variety of ML applications, many codebases evolve. New research concepts’ adaptability and usability are linked. They wanted to simplify JaxPruner integration into existing codebases.
JaxPruner uses the well-known Optax optimization library, requiring modest tweaks for interaction with current libraries. Pruning and sparse training state variables are preserved with the optimization state, simplifying parallelization and checkpointing.
Study First Rapid prototyping helps research teams execute various algorithms and baselines. JaxPruner commits to a generic API used by numerous algorithms, making switching algorithms easy. They implement popular baselines and make their algorithms easy to adapt. Switching between popular sparsity structures is easy.
CPU acceleration, activation sparsity, and other methods for neural network sparsity acceleration are growing. Integration with current frameworks is often inadequate, making it difficult to exploit these developments in research.
Since binary masks enable research, JaxPruner uses them to introduce sparsity, which requires more operations and storage. Their research focused on lowering this minimal overhead. GitHub hosts lessons and open-source code.