To be fair static graphs has its own advantages, specifically figuring out how to distribute computation over heterogeneous units. I think TF will still have upper hand when distributing computation over 10K CPUs/GPus but for everything else it’s a major pain to tolerate. The major issue with TF is it’s dissopointing API design- especially Estimator APIs and in many cases even lower level APIs. I wants to say WTF like at every line. Nothing is intuitive in that land. TF has jumped the shark here. In 1.5, they replaced documentation from old MNIST quick start to dumbed down Iris using Estimator API. I can only say WTF? Hopefully this disaster would divert enough of new comers to better things like PyCharm or even CNTK.
I wholeheartedly agree about the TF API becoming very cluttered. The entire way contrib is handled seems confusing.
While I think TF probably has a better distributed story, it's still not great (several months of experience getting this to work), and very very few people can justify distributing computation across 10K CPUs/GPUs - in my experience, it turns out most async SGD algorithms don't really work very well.
One place I see this actually working is using SVI/EM and storing local parameters across the network - something maybe 1 GPU can't handle for extremely large models.
The Estimator and Dataset API docs is so bad I'm planning on starting a blog with posts on running MNIST and a few other common datasets so others don't have to suffer.
[Disclosure: I designed and wrote most of the docs for TensorFlow’s Dataset API.]
I’m sorry to hear that you’ve not had a pleasant experience with tf.data. One of the doc-related criticisms we’ve heard is that they aim for broad coverage, rather than being examples you can drop in to your project and run straight away. We’re trying to address that with more tutorials and blog posts, and it’d be great if you started a blog on that topic to help out the community!
If there are other areas where we could improve, I’d be delighted to hear suggestions (and accept PRs).
No offense but TensorFlow's Dataset API documentation also sucks. This combined with bad API design (that actually can be used as case study for bad design in classrooms) is disaster in making. For example, shuffle() takes a mysterious argument. Why? It's not to be found in docs except that it should be more than items in dataset. Why can't shuffle() just be shuffle() and why do I now always have to remember passing correct parameter for rest of my life? Whatever. I still don't get what exactly repeat() does. Does it rewinds back to start when you reach past end? Why you need it? Why not just stick to epochs? Why make things complicated with steps vs epochs anyway? Docs gives zero clue. Then there are whole bunch of mysteriously named unexplained methods like make_one_shot_iterator() or from_tensor_slices(). Why is make_one_shot_iterator() not just iterator()? Why do I have to rebuild dataset using from_tensor_slices()? The docs are designed with a point of view "take all these code calling mysteriously designed APIs, copy-paste and don't bother too much about understanding what those APIs really do". It really sucks.
IMO, shuffle is something they did really fine. Unlike PyTorch datasets, TF allows streaming unbounded data. For something like this work with shuffle, it must cache some data before passing it down the pipeline. You specify how much in the argument.
This may not seem useful this conventional training, where you usually work with a fixed amount of samples you know beforehand. But there may be cases where this is not true (for instance, in some special cases of augmentation) - the streaming part is useful but then you must use this caching trick.
But I agree API naming is not stellar, or at least should come with better documentation.
I think tf.data is amazing, its far far better than the previous queue and string_input_producer style approach.
More than documentation, I would argue that TF especially tf.data lacks a tracing tool that would let a user quickly debug how data is being transformed and if there are any obvious ways to speed up. E.g. image_load -> cast -> resize vs image_load -> resize -> cast had different behavior and lead to hard to identify bugs. For tf.data prefetch which ends up being key to improving speed yet its is not documented, the only way I actually found out about it was by reading your TF.Data presentation.
I guess you misread me. The Dataset API is somewhat fine, much better than queues for instance. However not clear from documentation how to do more complex stuff, or how to integrate it with the rest of TF stack, specially the new Estimator API.
Yes, totally agree. TF is solid and it’s still my goto tool, but the API has a very low elegance score, to put it nicely. Clearly the team has a lot of brillian engineers, but they’re missing the high level sense of design (yes, APIs do need good design as well) that puts everything together beautifully.
