Written by Juncheng Liu; Translated by Xiaozhen Liu, Chenyang Xu
In this article, the engineer of OneFlow shares the way of setting grid_size and block_size when writing CUDA kernels.
In general, we might see the following code that launches a CUDA kernel:
Welcome to use OneFlow v0.6.0, we would love to hear feedback!
This version mainly updates three parts: framework, model and OneFlow-ONNX
Written by Luyang Zhao; Translated by Xiaozhen Liu, Chenyang Xu
This article introduces the principles and workflow of DataLoader in OneFlow/PyTorch, as well as how multiprocessing dataloader works.
In simple terms, DataLoader is a data loader for processing batch data and labels in each iteration of a dataset. …
Keywords: OneFlow, GPT-3
OneFlow Made Training GPT-3 Easier (Part 1)
Correct Level of Abstraction for Distributed Deep Learning Frameworks (Part2)
Runtime of OneFlow Based On Boxing and Actor Model (Part 3)
The Limitations of Existing Deep Learning Frameworks: Resource Dependency
The Limitations of Existing Deep Learning Frameworks: Data Movement
Written by Zekang Zheng, Chi Yao, Ran Guo, Juncheng Liu; Translated by Xiaozhen Liu, Hengrui Zhang
Elementwise operation refers to applying a function transformation to every element of a tensor. …
Written by Ran Guo, Chi Yao, Zekang Zheng, Juncheng Liu; Translated by Xiaozhen Liu, Hengrui Zhang
In a previous article, we discussed OneFlow’s techniques for optimizing the Softmax CUDA Kernel: How to implement an efficient Softmax CUDA kernel — — OneFlow Performance Optimization. …
Written by Jinhui Yuan; Translated by Kaiyan Wang, Xiaozhen Liu
In the last article “What an Optimal Point-to-Point Communication Library Should Be? (Part 1)”, we introduced what a point-to-point communication library is, and discussed some of the general characteristics of the optimal P2P communication library. …
Written by Jinhui Yuan; Translated by Kaiyan Wang, Xiaozhen Liu
In the previous article “Combating Software System Complexity:Appropriate Abstraction Layer”, we divided several levels of abstraction when discussing the network transmission requirements of distributed deep learning frameworks, and one of the lower levels was point-to-point (P2P).
In this series, we…
Written by Jinhui Yuan; Translated by Kaiyan Wang, Xiaozhen Liu
This article lists papers on GShard, presents background information and inspiration from the papers, and finally evaluates what else can be done to improve GShard from similar work that has been done in OneFlow.
Written by Ran Guo; Translated by Kaiyan Wang
The Softmax operation is one of the most common operations in deep learning models. In deep learning classification tasks, the network’s final classifier is often a combination of Softmax and CrossEntropy:
Although the mathematical derivation can be reduced when Softmax and CrossEntropy…
