This is the second part of a two-part series on deconstructing BERT. In part 1, Deconstructing BERT: Distilling 6 Patterns from 100 Million Parameters, I described how BERT’s attention mechanism can take on many different forms. For example, one attention head focused nearly all of the attention on the next word in the sequence; another focused on the previous … [Read more...] about Deconstructing BERT, Part 2: Visualizing the Inner Workings of Attention
Technical Guide
Deconstructing BERT: Distilling 6 Patterns from 100 Million Parameters
The year 2018 marked a turning point for the field of Natural Language Processing, with a series of deep-learning models achieving state-of-the-art results on NLP tasks ranging from question answering to sentiment classification. Most recently, Google’s BERT algorithm has emerged as a sort of “one model to rule them all,” based on its superior performance over a wide variety of … [Read more...] about Deconstructing BERT: Distilling 6 Patterns from 100 Million Parameters
How To Solve 90% Of NLP Problems: A Step-By-Step Guide
Text data is everywhere Whether you are an established company or working to launch a new service, you can always leverage text data to validate, improve, and expand the functionalities of your product. The science of extracting meaning and learning from text data is an active topic of research called Natural Language Processing (NLP). NLP … [Read more...] about How To Solve 90% Of NLP Problems: A Step-By-Step Guide
How Do AutoML Solutions Find The Optimal Machine Learning Models?
AutoML refers to techniques and tools which automate parts of the machine learning process, ranging from data collection and cleaning, to model development and testing, to production deployment and scaling. While the umbrella term does refer to a wide array of functionality, it’s most commonly used to refer to automated model selection and / or hyperparameter … [Read more...] about How Do AutoML Solutions Find The Optimal Machine Learning Models?
Data Science Project Flow for Startups
I was recently asked by a startup I’m consulting to give my opinion about the structure and flow of data science projects, which made me think about what makes them unique. Both managers and the different teams in a startup might find the differences between a data science project and a software development one unintuitive and confusing. If not stated and accounted for … [Read more...] about Data Science Project Flow for Startups
Google’s AutoML: Cutting Through The Hype
This is part 3 in a series. Part 1 is here and Part 2 is here. To announce Google’s AutoML, Google CEO Sundar Pichai wrote, “Today, designing neural nets is extremely time intensive, and requires an expertise that limits its use to a smaller community of scientists and engineers. That’s why we’ve created an approach called AutoML, showing that it’s possible for neural nets … [Read more...] about Google’s AutoML: Cutting Through The Hype
An Opinionated Introduction To AutoML And Neural Architecture Search
This is part 2 in a series. Check out part 1 here and part 3 here. Researchers from CMU and DeepMind recently released an interesting new paper, called Differentiable Architecture Search (DARTS), offering an alternative approach to neural architecture search, a very hot area of machine learning right now. Neural architecture search has been heavily hyped in the last year, … [Read more...] about An Opinionated Introduction To AutoML And Neural Architecture Search
What Do Machine Learning Practitioners Actually Do?
This post is part 1 of a series. Part 2 is an opinionated introduction to AutoML and neural architecture search, and Part 3 looks at Google’s AutoML in particular. There are frequent media headlines about both the scarcity of machine learning talent (see here, here, and here) and about the promises of companies claiming their products automate machine learning and eliminate … [Read more...] about What Do Machine Learning Practitioners Actually Do?