How does it handle recurrent connections. I really like Microsoft's approach with CNTK - they use a DSL called 'BrainScript' to define the network, and it's basically a computational network (hence then name), but there are special operations like
x = PastValue(y)
and
y = FutureValue(x)
for forward and backward recurrent connections. It even automatically only unrolls the recurrent part of the network.
In contrast all of the other NN systems I've used have premade layers like LSTM and GRU, and to be honest their methods are so hacky I haven't really been able to work out how to do custom RNNs in any other system.
func (m *model) costFn() (cost *Node, n int, err error) {
var prev *recurrentOutput
for ...{
// build your recurrent graph here
}
}
func (m *model) runOne() (err error) {
cost, n err := costFn()
g := m.g.SubgraphRoots(cost)
machine := NewLispMachine(g)
if err = machine.RunAll(); err != nil{
return
}
}
The LispMachine type allows for rapid prototyping. Then once that's all done and stuff, you would probably have figured out what the size of your graph is and you can then create a TapeMachine that have the sizes you defined (though I have removed all the bits of the TapeMachine that made it turing complete, so it may be a bit hard to do that).
Manual unrolling fails as soon as the number of repetitions is not fixed (e.g. NLP tasks where it matches number of tokens in a sentence); so you want the looping to happen after the graph has been made, preferably on the GPU.
For Gorgonia it's the other way around. You build the graph by manually unrolling it. It'll be a big goddamn graph, which is why you subgraph it and run it.
source: actually have various LSTMs running. Some with attention, some with ADHD
Is performance acceptable if you're required to re-build/adjust the graph for every single minibatch, because the number of unrolled items is different every time?
That is what I mean by "Manual unrolling fails as soon as the number of repetitions is not fixed"; you can do manual unrolling in every framework but generally repeated graph building makes it unusably slow if you're unable to do the loop "in the system" with a runtime-chosen number of repetitions.
Padding is also not really a solution, since your average sequence length is likely 5 or 10 times less than the maximum sequence length that you want to support, so just padding to a fixed size will mean 5-10 times slower processing.
I just checked out BrainScript. Looks pretty cool. Hacking up a parser for that into Gorgonia should be fairly trivial. Could be a fun weekend hacking project.
In contrast all of the other NN systems I've used have premade layers like LSTM and GRU, and to be honest their methods are so hacky I haven't really been able to work out how to do custom RNNs in any other system.
How does Gorgonia handle it?