gradient-checkpointing - Make huge neural nets fit in memory

  •        17

Training very deep neural networks requires a lot of memory. Using the tools in this package, developed jointly by Tim Salimans and Yaroslav Bulatov, you can trade off some of this memory usage with computation to make your model fit into memory more easily. For feed-forward models we were able to fit more than 10x larger models onto our GPU, at only a 20% increase in computation time. The memory intensive part of training deep neural networks is computing the gradient of the loss by backpropagation. By checkpointing nodes in the computation graph defined by your model, and recomputing the parts of the graph in between those nodes during backpropagation, it is possible to calculate this gradient at reduced memory cost. When training deep feed-forward neural networks consisting of n layers, we can reduce the memory consumption to O(sqrt(n)) in this way, at the cost of performing one additional forward pass (see e.g. Training Deep Nets with Sublinear Memory Cost, by Chen et al. (2016)). This repository provides an implementation of this functionality in Tensorflow, using the Tensorflow graph editor to automatically rewrite the computation graph of the backward pass.

https://github.com/openai/gradient-checkpointing

Tags
Implementation
License
Platform

   




Related Projects

vim-olive - Vim Mode Line Verifier

  •    VimL

Vim Mode Line Verifier

vim-mash - Motion Activated Search Highlighter for Vim

  •    VimL

Motion Activated Search Highlighter for Vim

vim-grillz - Flash your wicked grillz!

  •    VimL

Flash your wicked grillz!

vim-foist - Complete whole lines from any partial therein

  •    VimL

Complete whole lines from any partial therein

vim-efmc - Vim Error Format Compiler

  •    VimL

Vim Error Format Compiler


uzbl-utrs - mkng uzbl tlrbl

  •    Javascript

mkng uzbl tlrbl

toycsv - A toy CSV parser written in ruby + lexr + racc

  •    Ruby

A toy CSV parser written in ruby + lexr + racc

tiktok - TikTok provides a simple asynchronous timer object for VimL.

  •    VimL

TikTok provides a simple asynchronous timer object for VimL.

tabby - Using Vim's Tabs the Right Way

  •    VimL

Using Vim's Tabs the Right Way

SohiVila - GNU source-highlight Vim language

  •    

GNU source-highlight Vim language

SinTax - A DSL for generating Vim syntax highlighting files

  •    VimL

A DSL for generating Vim syntax highlighting files

rkdots - Generate a graphviz visualisation of the given javascript statement.

  •    Ruby

Generate a graphviz visualisation of the given javascript statement.

RelNumBar - Show relativenumbers alongside normal numbers in Vim

  •    VimL

Show relativenumbers alongside normal numbers in Vim

Punisher - Punisher hurts you where it hurts most - your time.

  •    VimL

Punisher hurts you where it hurts most - your time.

noisy - Don't chat quietly (weechat channel noises)

  •    Perl

Don't chat quietly (weechat channel noises)

newlisp-manual - Asciidoc version of the newLISP manual

  •    Common

Asciidoc version of the newLISP manual

Marathon-Wow - Wow Even Longer

  •    Common

Wow Even Longer

Land-of-newLISP - Selected snippets from Land of Lisp rewritten in newLISP

  •    Common

Selected snippets from Land of Lisp rewritten in newLISP

Fossilise - Poor man's collaborative editing in Vim

  •    VimL

Poor man's collaborative editing in Vim

firstly - Convert Between Numeric, Spelt, and Short & Long Ordinal Forms of Numbers

  •    VimL

Convert Between Numeric, Spelt, and Short & Long Ordinal Forms of Numbers