Deep Learning: Challenges and Excitements

Since 2013, I have started looking into Deep Learning (DL) technique. In this article I only touch the surface of the teqnique hoping to highlight some opportunities along with challenges this technique provides to ML community. This is a work in-progress and I'll be adding more stuffs here over time.

  1. Death of the expert? The rise of algorithms and decline of domain experts
  2. An Introduction to Variable and Feature Selection
  3. Motivations for Deep Learning

A) Death of the expert? The rise of algorithms and decline of domain experts

I went to one interesting talk presented by Jeremy Howarwho founded Kaggle and worked there as Chief Data Scientists. In that talk he was arguing that since ML community has access to powerful algorithms like deep learning and the fact that these algorithms perform automatic feature engineering for us, we do not need anymore domain experts.

This is really interesting considering its huge impact on machine learning space as well as its business impact. In 2013, we submitted a paper to EMNLP conference. We tried to improve performance of NER algorithms (i.e. Name Entity Recognition) in the presence of rarity issue. Unfortunately, the paper got rejected. However, in the paper we highlighted the importance of feature engineering in a machine learning problem where we discovered that morphological features were the most informative ones for an NER task to tag "DISEASE" and "TREATMENT" entities in a medical document. As a time consuming activity, we started employing important techniques for feature selection in machine learning domain.

B) An Introduction to Variable and Feature Selection

The main challenge in building an accurate ML model is amount of time you have to spend going through data and using ranking/statistical methods to extract and rank informative features. This is the focus of the "An Introduction to Variable and Feature Selection" paper.

However, by using deep learning the goal is to get away from the manual/tedious data engineering phase. Building a big deep learning networks by feeding a large amount of data will allow us to push such a time consuming task to machine. Another strong point of deep learning is its unsupervised nature in feature engineering. However, none of these benefits come for free. Mastering a technique like deep learning needs efforts to fully understand the technique and find out how we can successfully apply it to different machine learning and natural language processing tasks.

Also, another big challenge in working with deep learning is the fact that its very computational intensive. This requires us to start thinking and building distributed systems for feeding a large amount of data and building a big deep learning model. The reader should take a look at work done by Google last year in using deep learning to label images.

C) Motivations for Deep Learning

All have been said, I think it would be beneficial to investigate this domain which has its roots in neural networks. This investigation is very important considering the success of deep learning technique in areas like image recognition, speech recognition and natural language processing. Deep learning was invented by Geoffrey E. Hinton at UoT. Last year I read an interesting article in NY Times about a graduate student from UoT who attended one of Kaggle competition in chemistry domain in the last two weeks of the competition and could win the compeition without any domain knowledge by employing deep learning technique.

Resources for Deep Learning: