Two exciting NLP papers at ICML 2018! ICML 2018 accepts are out, and I am excited about two papers that I will briefly outline here. I think both papers are phenomenally good and will bring back structured prediction in NLP to modern deep learning architectures. Differentiable Dynamic Programming for Structured Prediction and Attention Arthur Mensch […]
When Men and Women talk to Siri
Just adding a short note to elaborate on the Twitter conversation I recently had on differential error rates in speech products for male and female speakers. This is not a commentary on Siri; I am using Siri in the title as eponymous to voice assistants in general. TLDR: different gender error rates for speech products […]
Everything is a Model
TLDR: I review a recent systems paper from Google, why it is a wake-up call to the industry, and the recipe it provides for nonlinear product thinking. Here, I will be enumerating my main takeaways from a recent paper, “The Case for Learned Index Structures” by Tim Kraska, Alex Beutel, Ed Chi, Jeffrey Dean, and […]
Two Recent Results in Transfer Learning for Music and Speech
In this post, I want to highlight two recent complementary results on transfer learning applied to audio — one related to music, another related to speech. The first paper, related to music, is by Keunwoo Choi and friends, which incidentally also won the best paper award at ISMIR 2017. But that’s not the only reason […]
When (not) to use Deep Learning for NLP
We are preparing for the second edition of our PyTorch-based Deep Learning for NLP training. It’s a two-day affair, crammed with a lot of learning and hands-on model building where we get to play the intricate dance of introducing the topics from the ground up while still making sure folks are not far from the […]
The Two Tribes of Language Researchers
TL;DR not-a-rant rant When I talk to friends who work on human language (#nlproc), I notice two tribes of people. These are folks who do Natural Language Processing and folks who do Computational Linguistics. This distinction is not mine and is blurry, but I think it explains some of the differences in values different researchers […]
A Billion Words and The Limits of Language Modeling
In this post, I will talk about Language Models, when (and when not) to use LSTMs for language modeling, and some state of the art results. While I mostly discuss the “Exploring Limits” paper, I’m adding a few things elementary (for some) here for completeness sake. The Exploring Limits paper is not new, but I think it’s a good illustration […]
Is BackPropagation Necessary?
In the previous post, we saw how the backprop algorithm itself is a bottleneck in training, and how the Synthetic Gradient approach proposed by DeepMind reduces/avoids network locking during training. While very clever, there is something unsettling about the solution. It seems very contrived, and definitely resource intensive. For example, a simple feed forward network under the […]
Synthetic Gradients .. Cool or Meh?
Synthetic what now? DeepMind recently published about Synthetic Gradients. This post is about that — what they are, and does it make sense for your average Deep Joe to use it. A Computational Graph is the best data structure to represent deep networks. (D)NN training and inference algorithms are examples of data flow algorithms, and […]
Turi Acquisition
(This post is about Turi, but I will occasionally refer to it by its older names Dato and Graphlab.) Turi got acquired by Apple for $200M (or so it is rumored). Reactions on the internet go from hearty congratulations to folks saying this is another example of the “AI bubble”. And of course, there are ill-informed, get-rich-quick […]