Before we start discussing the topic of a hybrid NLP (Natural Language Processing) system, let us look at the concept of hybrid from our life experiences.  

I was driving a classical Camry for years and had never thought of a change to other brands because as a vehicle, there was really nothing to complain.  Yes, style is old but I am getting old too, "who beats whom"?

Until one day a few years ago when we needed to buy a new car to retire my damaged Camry.  My daughter suggested hybrid, following the trend of going green.  So I ended up driving a Prius ever since and fallen in love with it.  It is quiet, with bluetooth and line-in, ideal for my iPhone music enjoyment.  It has low emission and I finally can say bye to smog tests.  It at least saves 1/3 gas.  We could have gained all these benefits by purchasing an expensive all-electronic car but I want the same feel of power at freeway and dislike the concept of having to charge the car too frequently.  Hybrid gets the best of both worlds for me now, and is not that more expensive.  

Now back to NLP.

There are two major approaches to NLP, namely machine learning and grammar engineering (or hand-crafted rule system).  As mentioned in previous posts, each has its own strengths and limitations, as summarized below.

1. In general, a rule system is good at capturing a specific language phenomenon (trees) while machine learning is good at representing the general picture of the phenomena (forest).  

2. As a result, it is easier for rule systems to reach high precision but it takes a long time to develop enough rules to gradually raise the recall.  Machine learning, on the other hand, has much higher recall, usually with compromise in precision or with a precision ceiling.  

3. Machine learning is good at simple, clear and coarse-grained task while rules are good at fine-grained tasks.  One example is sentiment extraction.  The coarse-grained task there is sentiment classification of documents (thumbs-up thumbs down), which can be achieved fast by a learning system.  The fine-grained task for sentiment extraction involves extraction of sentiment details and the related actionable insights, including association of the sentiment with an object, differentiating positive/negative emotions from positive/negative behaviors, capturing the aspects or features of the object involved, decoding the motivation or reasons behind the sentiment,etc.  In order to perform sophisticated tasks of extracting such details and actionable insights, rules are a better fit.    

4. The strength for machine learning lies in its retraining ability.  In theory, the algorithm, once developed and debugged, remains stable and the improvement of a learning system can be expected once a larger and better quality corpus is used for retraining (in practice, retraining is not always easy: I have seen famous learning systems deployed in client basis for years without being retrained for various reasons). Rules, on the other hand, need to be manually crafted and enhanced.

5. Supervised machine learning is more mature for applications but it requires a large labelled corpus.  Unsupervised machine learning only needs raw corpus, but it is research oriented and more risky in application.  A promising approach is called semi-supervised learning which only needs a small labelled corpus as "seeds" to guide the learning.  We can also use rules to generate the initial corpus or "seeds" for semi-supervised learning.

6. Both approaches involve knowledge bottlenecks.  Rule systems's bottleneck is the skilled labor, it requires linguists or knowledge engineers to manually encode each rule in NLP, much like a software engineer in the daily work of coding.  The biggest challenge to machine learning is the "sparse data" problem, which requires a very large  labelled corpus to help overcome.  The knowledge bottleneck for supervised machine learning is the labor required for labeling such a large corpus.

7. We can build a system to combine the two approaches to complement each other.  There are different ways of combining the two approaches in a hybrid system.  One example is the practice we use in our product, where the results of insights are structured in a back-off model: high precision results from rules are ranked higher than the medium precision results returned by statistical systems or machine learning. This helps the system to reach configurable balance between precision and recall.

8. When labelled data are available (e.g. the community has already built the corpus, or for some tasks, the public domain has the data, e.g. sentiment classification of movie reviews can use the review data with users' feedback on 5-star scale), and when the task is simple and clearly defined, using machine learning will greatly speed up the development of a capability.

9. Not every task is suitable for both approaches.  (Note that "suitability" is in the eyes of beholder: I have seen many passionate ML specialists willing to try everything in ML irrespective of the nature of the task: as an old saying goes, when you have a hammer, everything looks like a nail.) For example, machine learning is good at document classification whilerules are mostly powerless for such tasks.  But for complicated tasks such as deep parsing, rules constructed by linguists usually achieve better performance than machine learning.  Rules also perform better for tasks which have clear patterns, for example, identifying data items like time,weight, length, money, address etc.  This is because clear patterns can be directly encoded in rules to be logically complete in coverage while machine learning based on samples still has a sparse data challenge.  When designing a system,  in addition to using a hybrid approach for some tasks, for other tasks, we should choose the most suitable approach depending on the nature of the tasks.

10. Other aspects of comparison between the two approaches involve the modularization and debugging in industrial development.  A rule system can be structured as a pipeline of modules fairly easily so that a complicated task is decomposed into a series of subtasks handled by different levels of modules.  In such an architecture, a reported bug is easy to localize and fix by adjusting the rules in the related module.  Machine learning systems are based on the learned model trained from the corpus.  The model itself, once learned, is often like a black-box (even when the model is represented by a list of symbolic rules as results of learning, it is risky to manually mess up with the rules in fixing a data quality bug).  Bugs are supposed to be fixable during retraining of the model based on enhanced corpus and/or adjusting new features.  But re-training is a complicated process which may or may not solve the problem.  It is difficultto localize and directly handle specific reported bugs in machine learning.  

To conclude, due to the complementary nature for pros/cons of the two basic approaches to NLP, a hybrid system involving both approaches is desirable, worth more attention and exploration.  There are different ways of combining the two approaches in a system, including a back-off model using rulles for precision and learning for recall, semi-supervised learning using high precision rules to generate initial corpus or “seeds”, etc..

 
Related posts：
Comparison of Pros and Cons of Two NLP Approaches 
Is Google ranking based on machine learning?
《立委随笔：语言自动分析的两个路子》
《立委随笔：机器学习和自然语言处理》

【置顶：立委科学网博客NLP博文一览（定期更新版）】