Thông tin tài liệu
INCORPORATING LINGUISTICALLY MOTIVATED
KNOWLEDGE SOURCES INTO DOCUMENT
CLASSIFICATION
GOH JIE MEIN
NATIONAL UNIVERSITY OF SINGAPORE
2004
INCORPORATING LINGUISTICALLY MOTIVATED
KNOWLEDGE SOURCES INTO DOCUMENT
CLASSIFICATION
GOH JIE MEIN
BSc(Hons 1, NUS)
A THESIS SUBMITTED
FOR THE DEGREE OF MASTER OF SCIENCE
DEPARTMENT OF INFORMATION SYSTEMS
NATIONAL UNIVERSITY OF SINGAPORE
2004
ACKNOWLEDGEMENTS
This thesis cannot be done without the constant guidance and assistance from many
whom I must acknowledge at this point.
Firstly, I am deeply grateful to my advisor, Associate Professor Danny Poo for his
constant guidance, encouragement and understanding. He is instrumental to the
development of this thesis and I sincerely thank him for providing valuable advice,
direction and insights for my research.
My deep appreciation also goes out to all my friends, peers and colleagues who have
helped my in one way or another:
I wish to thank Klarissa Chang, Koh Wei Chern, Cheo Puay Ling and Wong Foong Yin
for their listening ears and uplifting remarks.
Colleagues and friends such as Michelle Gwee, Wang Xinwei, Liu Xiao, Koh Chung
Haur, Li Yan, Li Huixian, Santosa Paulus, Wan Wen, Bryan Low, Chua Teng Chwan,
Tok Wee Hyong, Indriyati Atmosukarto, Colin Tan, Julian Lin, the pioneering batch of
Schemers, for keeping life pleasurable in the office.
I would also like to express my sincere thanks to A/P Chan Hock Chuan and A/P
Stanislaw Jarzabek for evaluating my research.
And to all professors, teaching staff, administrative staff, friends and students.
Last but not least, I would like to thank my family especially my parents, sisters and
Melvin Lye for their relentless moral support, motivation, advice, love and
understanding.
-i-
TABLE OF CONTENTS
TABLE OF CONTENTS
Acknowledgement
i
Contents
iv
List of Tables
v
List of Figures
vii
1. Introduction --------------------------------------------------------------------------
1
1.1 Background & Motivation
1
1.2 Aims and Objectives
2
1.3 Thesis Plan
4
2. Literature Review---------------------------------------------------------------------
6
2.1 Document Classification
8
2.2 Feature Selection Methods
9
2.3 Machine Learning Algorithms
9
2.3.1 Naïve Bayes
9
2.3.2 Support Vector Machines (SVM)
11
2.3.3 Alternating Decision Trees
13
2.3.4 C4.5
15
2.3.5 Ripper
16
2.3.6 Instance Based Learning – k-Nearest Neighbour
17
2.4. Natural Language Processing (NLP) in Document Classification
19
2.5. Conclusion
21
3. Linguistically Motivated Classification------------------------------------------
22
3.1 Considerations
22
3.2 Linguistically Motivated Knowledge Sources
25
3.2.1 Phrase
26
3.2.2 Word Sense ( Part of Speech Tags)
27
-i-
TABLE OF CONTENTS
3.2.3 Nouns
28
3.2.4 Verbs
28
3.2.5 Adjectives
29
3.2.6 Combination of Sources
29
3.3 Obtaining Linguistically Motivated Classifiers
4. Experiment-----------------------------------------------------------------------------
30
32
4.1 Evaluation Data Sets
32
4.1.1 Reuters-21578
32
4.1.2 WebKB
36
4.2 Dimensionality Reduction
37
4.2.1 Feature Selection
37
4.2.2 Document Frequency Thresholding
38
4.2.3 Stop Words Removal
40
4.3 Experiment Setup
40
4.3.1 Handling Multiple Categories Problems
40
4.3.2 Handling Multiple Categories Training Examples
41
4.4 Evaluation Measures
41
4.4.1 Loss Based Measures
41
4.4.2 Recall & Precision
43
4.4.3 Precision Recall Breakeven Point
43
4.4.4 Micro- and Macro- Averaging
44
4.5 Tools
5. Results & Evaluation -----------------------------------------------------------------
45
48
5.1 Results
48
5.2 Contribution of Different Linguistically Motivated Knowledge Sources to
48
Classification of Reuters-21578 Corpus
5.2.1 Words
49
5.2.2 Phrase
51
- ii -
TABLE OF CONTENTS
5.2.3 Word Sense (Part of Speech Tags)
51
5.2.4 Nouns
52
5.2.5 Verbs
53
5.2.6 Adjectives
53
5.2.7 Combination of Sources
54
5.2.8 Analysis of Reuters-21578Results
55
5.3 Contribution of Linguistically Motivated Knowledge Sources
to
61
Classification accuracy of WebKB Corpus
5.3.1 Words
61
5.3.2 Phrase
62
5.3.3 Word Sense (Part of Speech Tags)
63
5.3.4 Nouns
64
5.3.5 Verbs
64
5.3.6 Adjectives
65
5.3.7 Nouns & Words
66
5.3.8 Phrase & Words
67
5.3.9 Adjectives & Words
67
5.3.10 Analysis of WebKb Results
68
5.4 Summary of Results
6. Conclusion -----------------------------------------------------------------------------
71
72
6.1 Summary
72
6.2 Contributions
72
6.3 Limitations
75
6.3 Conclusion
75
- iii -
SUMMARY
This thesis describes an empirical study on the effects of linguistically motivated
knowledge sources with different learning algorithms. Using up to nine different
linguistically motivated knowledge sources and six different learning algorithms, we
examined the performance of classification accuracy by using different linguistically
motivated knowledge sources, with different learning algorithms on two benchmark data
corpora: Reuters-21578 and WebKB. The use of linguistically motivated knowledge
sources, nouns, outperformed the traditional bag-of-words classifiers for Reuters-21578.
The best results for this corpus were obtained with the use of nouns with support vector
machines. On the other hand, experiments with WebKB showed that classifiers built
using the novel combinations of linguistically motivated knowledge sources were as
competitive as those built using the conventional bag-of-words technique.
- iv -
LIST OF TABLES
LIST OF TABLES
Table 1
Summary of Related Studies
23
Table 2
Distribution of Categories in Reuters-21578
34
Table 3
Top 20 Categories in ModApte Split
35
Table 4
Breakdown of Documents in WebKB Categories
36
Table 5
Contingency Table
42
Table 6
Results using Words
49
Table 7
Results using Phrases
50
Table 8
Results using Tags
51
Table 9
Results using Nouns
52
Table 10
Results using Verbs
53
Table 11
Results using Adjectives
54
Table 12
Results using both Linguistically Knowledge Sources and Words
54
Table 13
Contribution of Knowledge Sources on Reuters-21578 (F1 measures)
55
Table 14
Contribution of Knowledge Sources on Reuters-21578 (Precision)
58
Table 15
Contribution of Knowledge Sources on Reuters-21578 (Recall)
59
Table 16
Results using Words
62
Table 17
Results using Phrases
63
Table 18
Results using Tags
63
Table 19
Results using Nouns
64
Table 20
Results using Verbs
65
Table 21
Results using Adjectives
66
-v-
LIST OF TABLES
Table 22
Results using Nouns & Words
66
Table 23
Results using Phrases & Words
67
Table 24
Results using Adjectives & Words
68
Table 25
Consolidated Results of WebKB
69
- vi -
LIST OF FIGURES
LIST OF FIGURES
Figure 1
Document Classification Process
7
Figure 2
Linear SVM
12
Figure 3
An Example of an ADTree
15
Figure 4
k-NN Algorithm
18
Figure 5
Extracting Linguistically Motivated Knowledge Sources
31
Figure 6
An Example of a Reuters-21578 Document
35
Figure 7
Design of System
46
Figure 8
Comparison of Different Linguistically Motivated Knowledge
56
Sources on Reuters-21578 (Micro F1 values)
Figure 9
Comparison of Different Linguistically Motivated Knowledge
57
Sources on Reuters-21578 (Macro F1 values)
Figure 10
Comparison of Different Linguistically Motivated Knowledge
59
Sources on Reuters-21578 (Precision)
Figure 11
Comparison of Different Linguistically Motivated Knowledge
60
Sources on Reuters-21578 (Recall)
Figure 12
Comparison of Different Linguistically Motivated Knowledge
70
Sources on WebKB (Micro F1 values)
Figure 13
Comparison of Different Linguistically Motivated Knowledge
70
Sources on WebKB (Macro F1 values)
- vii -
CHAPTER 1 INTRODUCTION
CHAPTER 1
INTRODUCTION
1.1 Background & Motivation
With the emerging importance of knowledge management, research in areas such as
document classification, information retrieval and information extraction each plays a
critical role in the success of knowledge management initiatives. Studies have shown that
the perceived output quality is an essential factor for successful implementation and
adoption of knowledge management technologies (Kankanhalli, et al., 2001). Large
document archives such as electronic knowledge repositories offer a huge wealth of
information whereby methods in the field of information retrieval and document
classification are used to derive knowledge.
Coupled with the accessibility to voluminous amount of information available in the
World Wide Web, this information explosion has brought about other problems. Users
are often overwhelmed by the deluge of information and suffer from a decreased ability
to assimilate information. Research has suggested that users feel bored or frustrated when
they receive too much information (Roussinov and Chen, 1999) which can lead to a state
where the individual is no longer able to effectively process the amount of information he
is exposed, giving rise to a lower decision quality in a given time set. This problem is
exacerbated with the proliferation of information in electronic repositories in
organizations and the World Wide Web (Farhoomand and Drury, 2002).
-1-
CHAPTER 1 INTRODUCTION
Document classification has been applied to the categorization of search results and has
been shown to alleviate the problem of information overload. The presentation of
documents in categories works better than a list of results because it enables users to
disambiguate the categories quickly and then focus on the relevant category (Dumais,
Cutrell & Chen, 2001). This is also useful for distinguishing documents containing words
with multiple meanings, or polysemy, a characteristic predominant in English words.
Experiments on supervised document classification techniques have predominantly used
the bag-of-words technique whereby words of the documents are used as features.
Alternate formulations of a meaning can also be introduced through linguistic variation
such as syntax which determines the association of words. Although there have been
some studies that have employed alternate features such as linguistic sources, these
studies have only employed a subset of linguistic sources and learning algorithms (Lewis,
1992; Arampatzis et al., 2000; Kongovi et al., 2002). Thus, this study extends previous
studies relating to document classification and aims to find ways to improve the
classification accuracy of documents.
1.2 Aims and Objectives
Differences in previous empirical studies could be introduced by the differences in
tagging tools used, different learning algorithms, parameters tuned for each of the
learning algorithms, feature selection methods employed and dataset involved (Yang,
-2-
CHAPTER 1 INTRODUCTION
1999). Thus, it is difficult to offer a sound conclusion based on previous works. Since
previous works documenting results based on linguistically motivated features with
learning algorithms produced inconsistent and sometimes conflicting results, we propose
to conduct a systematic study on multiple learning algorithms and linguistically
motivated knowledge sources as features. Some of these features are novel combinations
of linguistically motivated knowledge sources were not explored in previous studies.
Using a systematic and controlled study, we can resolve some of these ambiguities and
offer a sound conclusion. In our study, consistency in the dataset, learning algorithms,
tagging tools and features selections were maintained in the study so that we can have a
valid assessment of the effectiveness of linguistically motivated features.
The aim of this thesis is also to provide a solid foundation for research on feature
representations on text classification and study the factors affecting machine learning
algorithms used in document classification. One of these factors we want to look into is
the effect of feature engineering by utilizing linguistically motivated knowledge sources
as features in our study.
Thus, the objectives of this thesis are listed below:
1.
To examine the approach of using linguistically motivated knowledge
sources based on concepts derived from natural language processing as
features with popular learning algorithms, and systematically vary both
-3-
CHAPTER 1 INTRODUCTION
the learning algorithms and feature representations. We based our
evaluation on Reuters-21578 corpus and WebKB, benchmark corpus
that has been widely used in previous research.
2.
To examine the feasibility of applying novel combinations of
linguistically
motivated
knowledge
sources
and
explore
the
performance of these combinations as features on the accuracy of
document classification.
1.3 Thesis Plan
This thesis is composed of the following chapters:
Chapter 1 provides the background, motivation and objectives of this research.
Chapter 2 provides a literature review of document classification research. Here we
bring together literature from different fields, document classification, machine learning
techniques and natural language processing and give a detailed coverage of the
algorithms chosen for this study. In addition, this chapter also overview the rudimentary
knowledge required in later chapters.
Chapter 3 describes the types of linguistically motivated knowledge sources and the
novel combinations used in our study.
-4-
CHAPTER 1 INTRODUCTION
Chapter 4 provides a description of the experiment setup. It also gives a brief on the
performance measures used to evaluate the classifiers and tools employed to conduct the
study.
Chapter 5 provides an analysis of the results and suggests the implications for practice.
Chapter 6 concludes with the contributions, findings and the limitations of our study.
Suggestions on future research that can be an extension to this work are also provided.
-5-
CHAPTER 2 LITERATURE REVIEW
CHAPTER 2
LITERATURE REVIEW
Document classification has traditionally been carried out using the bag-of-words
paradigm. Research on natural language processing has shown fruitful results that can be
applied to document classification by introducing an avenue for improving accuracy by a
different set of features used. This chapter reviews related literature underpinning this
research. Section 2.1 gives an overview of document classification and the focus of
previous research. Section 2.2 overviews common features selection techniques used in
previous studies. Section 2.3 introduces the machine learning algorithms that will be
adopted in our study. Section 2.4 presents the concepts of natural language processing
employed to derive linguistically motivated knowledge sources.
2.1
Document Classification
Document classification, the focus of this work, refers to the task of assigning a
document to categories based on its underlying content. Although this task has been
carried out effectively using knowledge engineering approaches in the 80s, machine
learning approaches have superceded the use of knowledge engineering approaches for
document classification in recent years. While the knowledge engineering approaches
have produced effective classification accuracy, the machine learning approach offered
many advantages such as cost effectiveness, portability and competitive accuracy to that
-6-
CHAPTER 2 LITERATURE REVIEW
of human experts while producing considerable efficiency (Sebastiani, 2002). Thus,
machine learning supervised techniques are employed in this study.
A supervised approach involves three main phases: feature extraction, training, testing.
The entire process of document classification using machine learning methods is
illustrated in Figure 1.
Test Phase
Test corpus
Unlabeled Test Feature
Vector
Training
Phase
Training corpus
Feature
Selection
Labeled Training Feature
vectors
Classifier Model
Category
Figure 1: Document classification Process
There are two phases involved in learning based document classification: the training and
testing phase. In the training phase, pre-labeled documents are collected. This set of prelabeled documents is called the training corpus, training data or training set which is used
interchangeably throughout this thesis. Each document in the training corpus is
transformed into a feature vector. These feature vectors are trained with a learning
-7-
CHAPTER 2 LITERATURE REVIEW
classifier. Each classifier will then build a model based on the training set. The model
built based on the learning algorithm is then used in the testing phase to test a set of
unlabeled documents, called the test corpus, test data or test set, that are new to the
learning classifier, to be labeled.
The classification problem can be formally represented as follows:
Classification fc(d) -> {true, false} where d∈D
Given a set of training documents, D and a set of categories C, the classification problem
is defined as the function, f, to map documents in the test set, T, into a boolean value
where ‘true’ indicates that the document is categorized as C and ‘false’ indicates that the
document is not categorized under C, based on the characteristics of the training
documents, D where f (d)->c.
2.2
Feature Selection Methods
With a large number of documents and features, the document classification process
usually involves a feature selection step. This is done to reduce the feature dimension.
Feature selection methods extract only the important features derived from the original
set of features. Traditional features selection methods that have been commonly
employed in previous studies include document frequency (Dumais et al., 1998; Yang
and Pedersen, 1997), chi-square (Schutze et al., 1997; Yang and Pedersen, 1997),
information gain (Lewis and Ringuette, 1994; Yang and Pedersen, 1997), mutual
information (Dumais et al., 1998; Larkey and Croft, 1996) etc.
-8-
CHAPTER 2 LITERATURE REVIEW
After the feature reduction step, many supervised techniques can then be employed in
document classification. Subsequent section presents a review of the techniques that were
employed in this study.
2.3
Machine Learning Algorithms
This section reviews the state-of-the-art learning algorithms as text classifiers, to give a
background on the different methods and an analysis of the advantages and disadvantages
of each learning method that we have used in our empirical study. Past research in the
field of automatic document classification has focused on improving the document
classification techniques through various learning algorithms (Yang and Liu, 1999) such
as support vector machines (Joachims, 1998) and various feature selection methods
(Yang and Pedersen, 1997; Luigi et al., 2000). To make the study worthwhile, popular
learning algorithms that have reported significant improvements on the classification
accuracy of various learning algorithms in previous studies were used. These included a
wide variety of supervised-learning algorithms, including naïve bayes, support vector
machines, k-nearest-neighbours, C4.5, RIPPER, AdaBoost with decision stumps and
alternating decision trees.
2.3.1 Naïve Bayes (NB)
Bayes classification is a popular technique in recent years. The simplest Bayesian
classifier is the widely used Naïve Bayes classifier which assumes that features are
independent. Despite the inaccurate assumption of feature independence, Naïve Bayes is
-9-
CHAPTER 2 LITERATURE REVIEW
surprisingly successful in practice and has proven effective in text classification, medical
diagnosis, and computer performance management, among other applications.
Naïve Bayes classifier uses a probabilistic model of text to estimate the probability that a
document d is in class y, Pr(y|d). This model assumes conditional independence of
features, i.e. words are assumed to occur independently of the other words in the
document given its class. Despite this assumption, Naïve Bayes have performed well.
Bayes rule says that to achieve the highest classification accuracy, d should be assigned
to the class y ∈ {-1, +1} for which Pr(y|d) is the highest.
hBAYES (d ) = arg max y∈{−1, +1} Pr( y | d )
(1)
Pr(y|d) can be calculated by considering each document according to their length l.
Pr( y | d ) = ∑l =1 Pr( y | d , l ) • Pr(l | d )
∞
(2)
Pr(l|d) equals one for the length l’ of document d and is zero otherwise. In other words,
when we apply Bayes’ theorem to Pr(y|d,l) we can obtain the following equation:
Pr( y | d ) =
Pr(d | y, l ' ) • Pr( y | l ' )
(3)
∑ y '∈{−1,+1} Pr(d | y' , l ' ) • Pr( y'| l ' )
Pr(d|y,l’) is the probability of observing document d in class y given its length l’. Pr(y|l’)
is the prior probability that a document of length l’ is in class y. In the following we will
assume that the category of a document does not depend on its length so Pr(y|l’) = Pr(y).
An estimate of Pr(y) is as follows:
Pr' ( y ) =
∑
| y |
| y '|
y '∈ { − 1 , + 1 }
=
| y|
|D |
(4)
- 10 -
CHAPTER 2 LITERATURE REVIEW
|y| denotes the number of training documents in class y∈{-1,+1} and |D| is the total
number of documents.
Despite the unrealistic independence assumption, the Naïve Bayes classifier is
remarkably successful in practice. Researchers show that the Naïve Bayes classifier is
competitive with other learning algorithms such as decision tree and neural network
algorithms. Experimental results for Naïve Bayes classifiers can be found in several
studies (Lewis, 1992; Lewis and Ringuette, 1994; Lang, 1995; Pazzani 1996; Joachims,
1998; McCallum & Nigam 1998; Sahami, 1998). These studies have shown that Bayesian
classifiers can produce reasonable results with high precision and recall values. Hence,
we have chosen this learning algorithm in learning to classify text documents. The second
reason that this Bayesian method is important to our study of machine learning is that it
provides a useful perspective for understanding many learning algorithms that do not
explicitly manipulate probabilities.
2.3.2 Support Vector Machines (SVM)
Support vector machines were developed by Vapnik et al. (1995) based on structural risk
minimization principle from statistical learning theory. The idea of structural risk
minimization is to find a hypothesis h from a space H that guarantees the lowest
probability of error E(h) for a given training sample S consisting of n examples. Equation
(5) gives the upper bound that connects the true error of a hypothesis h with the error
E(h) of h on the training set and the complexity of h which reflects the well-known tradeoff between the complexity of the hypothesis space and the training error.
- 11 -
CHAPTER 2 LITERATURE REVIEW
n
d ln( ) − ln(η )
d
E (h) ≤ Etrain(h) +O(
)
n
(5)
A simple hypothesis space will most likely not contain good approximation functions and
will lead to a high training and true error. On the other hand a large hypothesis space will
lead to a small training error but the second term in the right-hand side of equation (5)
will be large. This reflects the fact that for a hypothesis space with high VC-dimension
the hypothesis with low training error may result in overfitting. Thus it is crucial to find
the right hypothesis space.
The simplest representation of a support vector machine, a linear SVM, is a hyperplane
that separates a set of positive examples from a set of negative examples with maximum
distance from the hyperplane to the nearest of the positive and negative examples. Figure
2 shows the graphical representation of a linear SVM.
+
+
-
-
+
+
+
+
+
-
Maximum
distance
Figure 2. Linear SVM
Joachims (1998) developed a model of learning text classifiers using support vector
machine and linked the statistical properties of text with the generalization performance
- 12 -
CHAPTER 2 LITERATURE REVIEW
of the learner. Unlike conventional generative models, SVM does not involve
unreasonable parametric or independence assumptions. The discriminative model focuses
on those properties of the text classification tasks that are sufficient for good
generalization performance, avoiding much of the complexity of natural language. This
makes SVM suitable for achieving good classification performance despite the high
dimensional feature spaces in text classification. High redundancy, high discriminative
power of term sets, and discriminative features in the high-frequency range are sufficient
conditions for good generalization. SVM is therefore chosen as one of the learning
algorithms in this study.
We used Platt’s (1999) sequential minimal optimization algorithm to process the linear
SVM more efficiently. This algorithm decomposes the large quadratic programming
problem into smaller sub-problems. Document classification using support vector
machine can be done either through a binary or multi-class classification but we have
adopted the binary approach which will be mentioned in a later chapter.
2.3.3 Alternating Decision Tree (ADTree)
Although a variety of decision tree learning methods have been developed with
somewhat differing capabilities and requirements, we have chosen one of the recent
method called the alternating decision trees (Freund & Mason, 1999). This is because this
method has been often applied to classification problems and applied to problems such as
learning to classify text or documents.
- 13 -
CHAPTER 2 LITERATURE REVIEW
Alternating decision tree learning algorithm is a new combination of decision trees with
boosting that generates classification rules that are small and often easy to interpret. A
general alternating tree defines a classification rule by defining a set of paths in the
alternating tree. As in standard decision trees, when a path reaches a decision node it
continues with the child that corresponds to the outcome of the decision associated with
that node. When a prediction node is reached, the path continues with all of the children
of the node. Path splits into a set of paths, where each path corresponds to one of the
children of the prediction node.
The difference between ADTree and conventional decision trees is that the classification
is based on traversing the path of the decision tree instead of the final leaf node of the
tree.
There are several key features of alternating decision trees. Firstly, compared to C5.0
with boosting, ADTree provides classifiers that are smaller and easier to interpret. In
addition, ADTree give a measure of confidence, called the classification margin, that can
be used to improve the accuracy of the cost of abstaining from predicting examples that
are hard to classify instead of only a class. However, the disadvantage of ADTree is its
susceptibility to overfitting in small data sets.
- 14 -
CHAPTER 2 LITERATURE REVIEW
Figure 3. An Example of an ADTree (Freund & Mason, 1999)
2.3.4 C4.5
A decision tree text classifier is a tree in which internal nodes are labeled by terms,
branches departing from them are labeled by tests on the weight that the term has in the
test document, and leaves are labeled by categories. In this classification scheme, a text
document d is categorized by recursively testing the weights that the terms labeling the
internal nodes have in vector d, until a leaf node is reached. The label of this node is then
assigned to d. Most of these classifiers use binary document representations represented
as a binary tree. There are a number of decision trees and among the most popular is C4.5
(Cohen and Hirsh, 1998). Thus we have chosen this learning method.
The most popular decision-tree algorithm that has shown good results on a variety of
problems is the C4.5 algorithm (Quinlan, 1993). Previous works based on this technique
are reported in Lewis and Ringuette, 1994, Moulinier et al., 1996, Apte and Damerau,
1994, Cohen, 1995, Cohen, 1996. The underlying approach to C4.5 is that it learns
- 15 -
CHAPTER 2 LITERATURE REVIEW
decision trees by constructing them top-down, from the root of the tree. Each instance
feature is then evaluated using a statistical test, like the information gain, to determine
how well it alone classifies the training examples. Information gain is otherwise known
as entropy in information theory. Entropy of a collection S is measured as follows:
Entropy ( S ) ≡ − p + log 2 p + − p − log 2 p − (6)
Where p+ is the proportion of positive instances in the collection S and p- is the
proportion of negative instances. The best feature is selected and employed as a root
node to the tree. For each possible value of this attribute, a descendant of the root node is
created, and the training examples are sorted to the appropriate descendant node. C4.5
forms a greedy search for a suitable decision tree in which no backtracking is allowed.
2.3.5 Ripper
This learning algorithm is a prepositional rule learner, RIPPER (Repeated Incremental
Pruning to Produce Error Reduction), proposed by Cohen (1995). The algorithm has a
few major phases that characterize it: grow, prune, optimization. RIPPER was developed
based on repeated application of Furnkranz and Widmer’s (1994) IREP algorithm
followed by two new global optimization procedures. Like other rule-based learners,
RIPPER grows rules in a greedy fashion guided by some information theoretic
procedures.
- 16 -
CHAPTER 2 LITERATURE REVIEW
Firstly, the rules are grown from a greedy process which adds conditions to the rule until
the rule is 100% accurate. The algorithm tries every possible value of each attribute and
selects the conditions with the highest information gain. The rules are incrementally
pruned and finally in the optimization stage, an initial rule set and pruned rule set are
obtained. One variant is generated from an empty rule while the other is generated by
greedily adding antecedents to the original rule. The smallest possible description length
for each variant and the original rule is computed and the variant with the minimal
description length is selected as the final representative of rules in the rule set. Rules that
would increase the description length of the rule set if they were included were deleted.
The resultant rules are then added to the rule set.
RIPPER has already been applied to a number of standard problems in text classification
with rather promising results (Cohen, 1995). Thus, it is chosen as one of the candidate
learning algorithms in our empirical study.
2.3.6 Instance Based Learning – k-Nearest Neighbour
The
basic
idea
behind
k-Nearest
Neighbors
(k-NN)
classifier
is
the assumption that examples located close to each other according to a user-defined
similarity metric are highly likely to belong to the same class. This algorithm is also
derived from Bayes’ rule. This technique has shown good performance on text
categorization in Yang & Liu (1997), Yang & Pederson (1999), Masand (1992).This
algorithm assumes that all instances correspond to points in the n-dimensional space. The
nearest neighbors of an instance are defined in terms of the standard Euclidean distance.
- 17 -
CHAPTER 2 LITERATURE REVIEW
An arbitrary instance x is described as a feature vector (a1(x), a2(x), … an(x)) where ai(x)
denotes the value of the ith attribute of the instance x. Then the distance between two
instances xi and xj is defined to be d(xi,xj) where,
d ( xi, xj ) =
n
∑ (ar ( x ) − ar ( x ))
r =1
i
2
j
The target function can either be discrete or real-valued. In our study, we will assume that
the target function is discrete valued thus we have a linear binary classifier for each
category. The pseudocode is as follows:
Train(training example x){
For each training example (x, f(x)) add example to list of training examples.
}
Classify(query instance xq){
Let xj… xk denote the k instances from training examples that are nearest to xq.
k
^
Return
f ( xq ) ←
arg max ∑ δ (v, f ( xi ))
v∈V
i =1
Where δ(a,b)=1 if a=b and where δ(a,b)=0 otherwise.
}
Figure 4: k-NN Algorithm (Mitchell, 1997)
The key advantage of instance-based learning is that instead of estimating the target
function once for the entire instance space, it can estimate it locally and differently for
each new instance to be classified. This method is a conceptually straightforward
approach to approximating real-valued or discrete-valued target functions. In general, one
disadvantage of instance-based approaches is that the cost of classifying new instances
- 18 -
CHAPTER 2 LITERATURE REVIEW
can be high due to the fact that nearly all computation takes place at classification time
rather than when the training examples are first encountered. A second disadvantage is
they consider all attributes of the instances when attempting to retrieve similar training
examples from memory. If the target concept depends on only a few of the many
available attributes, then the instances that are truly most “similar” may have a large
distance apart. However, as previous attempts to classify text with this approach has
shown to be effective (Yang, 1999), we have decided to include it inside our experiment.
2.4
Natural Language Processing (NLP) in Document
Classification
Most information retrieval (IR) systems are not linguistically motivated. Similarly, in
document classification research, most experiments are not linguistically motivated
(Cullingford, 1986). Closely related to the research on document classification is the
research on natural language processing and cognitive science. Traditionally, document
classification techniques are primarily directed at detecting performance accuracy and
hold little regard for linguistic phenomena. Much of the current document classification
systems are built upon techniques that represent text as a collection of terms such as
words. This has been done successfully using quantitative methods based on word or
character counts. However, it has been emphasized that vector space models cannot
capture critical semantic aspects of document content. In this case, the representation is
superficially related to content since language is more than simply a collection of words.
- 19 -
CHAPTER 2 LITERATURE REVIEW
Thus, natural language processing is a key technology for building information retrieval
systems of the future (Strzalkowski, 1999).
In order to study the effects of linguistically motivated knowledge sources with document
classification, it is imperative to learn about the grammar through natural language
processing so as to apply concepts in cognitive science on document classification
techniques. Natural language processing research attempts to enhance the ability of the
computer to analyze, understand and generate languages that are used. This is performed
by some type of computational or conceptual analysis to form meaningful structure or
semantics from a document. The inherently ambiguous nature of natural language makes
things even more difficult. A variety of research disciplines are involved in the successful
development of NLP systems. The mapping of words into meaningful representations is
driven by morphological, syntactic, semantic, and contextual cues available in words
(Cullingford, 1986). With the advancement of NLP techniques, we hope to incorporate
linguistic cues into document classification through NLP techniques. This can be done by
utilizing NLP techniques to extract the different representation of the documents and then
used in the classification process.
As defined by Medin (2000), other concepts identified are verb, count nouns, mass
nouns, isolated and interrelated concepts. We define such concepts as linguistically
motivated knowledge sources. They can be used to derive more complex linguistically
motivated features in the process of classification. It appears that the centrality of using
linguistic knowledge sources as features in the process of classification can serve as an
- 20 -
CHAPTER 2 LITERATURE REVIEW
important step for a good classification scheme. For example, besides individual words
and the relationships between words within a sentence, a document and the context of
what is already known of the world, helps to deliver the actual meaning of a text.
Research has focused on using nouns in the process of categorization in modeling the
process of categorization in the real world (Chen et al. 1992; Lewis, 1992; Arampatzis et
al, 2000; Basili, 2001). However, the significant differences of these results have led us
to examine these features with alternate representations.
2.5
Conclusion
Bag-of-words paradigm has appeared to be the feature used dominantly in supervised
classification studies. This could be due to the results of early attempts (Lewis, 1992)
which showed negative results. With the advent of NLP techniques, there seems a
propelling reason to examine the use of linguistically motivated knowledge sources.
Although there has been separate attempts made to study the effects of linguistically
motivated knowledge sources on supervised document classification techniques, it is
difficult to generalize a conclusion based on these separate attempts because of the
variations introduced across studies. In some cases, conflicting results were also reported.
Thus, there seems a need for us to fill the gap with a systematic study that covers an
extensive
variety
of
linguistically
motivated
knowledge
sources.
- 21 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
CHAPTER 3
LINGUISTICALLY MOTIVATED
CLASSIFICATION
Despite the existence of extensive research on document classification, the relationship
between different linguistic knowledge sources and classification model has not been
sufficiently or systematically explored. By bringing together the two streams of research
of document classification and natural language processing, we hope to shed light on the
effects of linguistically motivated knowledge sources with different learning algorithms.
Section 3.1 discusses the shortcomings of previous research. Section 3.2 explores the
linguistically motivated knowledge sources employed to resolve these issues. Finally
section 3.3 presents the technique to derive the features.
3.1
Considerations
Much research in the area of document classification has been focused mainly on
developing techniques or on improving the accuracy of such techniques. While the
underlying algorithm is an essential factor for classification accuracy, the way in which
texts are represented is also an important factor that should be examined. However,
attempts to produce text representations to improve effectiveness have shown
inconsistent results.
- 22 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
The classic work of Lewis (1992) has shown that there was low effectiveness of syntactic
phrase indexing in terms of its characteristics as a text representation but recent works by
Kongovi (2002), Basili (2001), has shown that there were improvements using the same
representation. Table 1 shows the conclusions made by some related works. For example,
noun phrase seems to behave differently with different learning algorithms. The results
differ due to the inconsistencies introduced in these studies through various datasets,
taggers, learning algorithms, parameters of the learning algorithms and feature selection
methods used.
Features
Noun Phrase
Algorithm/Method
Statistical
clustering algorithm
RIPPER
Corpus
Reuters-22173
Work
Lewis (1992)
Reuters-21578
Scott & Matwin
(1999)
Noun Phrase
Clustering
Reuters-21578
Kongovi (2002)
Noun Phrase
SOM
CANCERLIT
Tolle & Chen (2000)
Nouns
Rocchio
Reuters-21578
Proper Nouns
Rocchio
Reuters-21578
Tags
Rocchio
Reuters-21578
Basili, Moschitti &
Pazienza (2001)
Basili, Moschitti &
Pazienza (2001)
Basili, Moschitti &
Pazienza (2001)
Noun Phrase
Worse Performance than Words
Results
Better Performance than Words
Table 1: Summary of Previous Studies
To address the issues as discussed in the previous section and limitations of previous
work, this entails a systematic study on the effects of linguistically motivated knowledge
sources with various machine learning approaches for automatic document classification
is necessary. In contrast to previous work, this research conducts a comparative study and
analysis of learning methods among which, are some of the most effective and popular
techniques available, and report on the accuracies of linguistically motivated knowledge
- 23 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
sources and novel combinations of them using a systematic methodology to resolve any
of the issues that we have discussed in previous work.
Additionally, we try to see if we can break away from the traditional bag-of-words
paradigm. Bag-of-words basically refers to representing document using words, the
smallest meaningful unit of a document with little ambiguity. Word-based representations
have been the most common representation used in previous works related to document
classification. They are the basis for most work in text classification. The obvious
advantage of words is in its simplicity and straightforward processes to obtain the
representation. However the problem of using bag-of-words is that usually the logical
structure, layout and sequence of words are ignored.
A basic observation about using the bag of words representations for classification is that
a great deal of information from the original document associated with the logical
structure and sequence is discarded. The major limitation is the implicit assumption that
the order of words in a sentence is not relevant. In fact, paragraph, sentence and word
orderings are disrupted, and syntactic structures are ignored. However, this assumption
may always hold as words alone do not always represent true atomic units of meaning.
For example, the word “learning algorithm” could be interpreted in another manner when
broken up into two separate words, “learning” and “algorithm”.
Thus, we utilize
linguistically motivated knowledge sources as features, to see if we can resolve these
limitations associated with the bag-of-words paradigm. Novel combinations of
- 24 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
linguistically motivated knowledge sources are also proposed and presented in the next
section.
3.1.1 Linguistically Motivated Knowledge Sources
Machine learning methods require each example in the corpus to be described by a vector
of fixed dimensionality. Each component of the vector represents the value of one feature
of the example. As a linguistics knowledge source may provide the contextual cues about
a document that are useful as a feature representation for distinguishing the category of
the document, we are interested to study whether the choice of different feature
representations using different linguistic knowledge sources as the input vectors to the
learning algorithm have significant impact on document classification. We consider the
following linguistics knowledge sources in our research:
1. Word, this will be used as the baseline to do a comparative analysis with other
linguistically motivated knowledge sources;
2. Phrase;
3. Word sense or part of speech tagging;
4. Nouns;
5. Verbs;
6. Adjectives;
7. Combinations of sources with words.
- 25 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
The description of the above features and an analysis of the advantages and
disadvantages of feature representation are discussed.
3.1.2 Phrase
Phrases have been found to be useful indexing units in previous research. Kongovi,
Guzman & Dasigi’s (2002) has shown that phrases were salient features when used with
category profiles. We consider one class of phrases i.e. the syntactic phrases. Syntactic
phrases refer to any set of words that satisfy certain syntactic relations or constitute
specified syntactic structures or certain syntactic relations.
Phrase refers to the noun phrases that are identified by our parser. The data set is first
parsed into the appropriate format before being extracted and segmented. Noun Phrases is
defined as a sequence of words that terminates with a noun. More specifically, noun
phrases is defined as
NP = {A, N}*N , where NP stands for noun phrase, A for adjective and N for
nouns.
For example, the sentence, “The limping old man walks across the long bridge” the noun
phrases identified are “limping old man” and “long bridge”. In our work here, we do not
attempt to separate the noun phrases into its component noun phrases.
- 26 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
The advantage of phrase does not ignore the assumption that the ordering of words in not
relevant and the logical structure, layout and sequence of words are retained thus keeping
some information from the original document. On the other hand, the major limitation is
the greater degree of complexity when processing and extracting phrases as features.
Although phrase-based representation has been used in information retrieval, conclusions
derived from studies reporting the retrieval effectiveness of linguistic phrase-based
representations on retrieval have been inconsistent. Linguistic phrase identification has
been noted as improving retrieval effectiveness by Fagan (1987) but on the other hand,
Mitra et al. (1997) reported little benefits in using phrase-based representations. Smeaton
(1999) reported that the benefit of phrase-based representation varied with users. Lewis
(1992) undertook a major study of the use of noun phrases for statistical classification
and found that phrase representation did not produce any improvement on the Reuters22173 corpus.
As we are using a different corpus in our work, we decided to continue with the use of
phrase based representations in our experiment as it has not been studied before with
some of the learning algorithms that we have chosen.
3.2.3 Word Sense (Part of Speech Tagging)
Word sense refers to the incorporation of part of speech tags with the word so that the
exact word sense within a document is identified. The part of speech of a word provides a
- 27 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
syntactic source of the word, such as adjective, adverb, determiner, noun, verb,
preposition, pronoun and conjunction. As this feature incorporates both the tag and the
word, it will provide the word class or lexical tag for the classifier.
The intuition for using word sense is to capture additional information that will help to
distinguish homographs that can be differentiated based on the syntactical role of the
word. Homographs refer to words with more than one meaning. For example, the word
“patient” might have different meanings when utilized with different syntactical role such
as noun or verb. When used as a noun, a patient refers to an individual who is not feeling
well or sick but when used as an adjective, it could refer to the character of a person as
being tolerant.
3.2.4 Nouns
Gentner (1981) explored the differences between nouns and verbs and suggested that
nouns differ from verbs in the relational density of their representations. The semantic
components of noun meanings are more strongly interconnected than those of verbs and
other parts of speech. Hence, the meanings of nouns seem less mutable than the meanings
of verbs. Nouns have been used as a common candidate for distinguishing among
different concepts. Nouns are often called “substantive words” in the field of
Computational Linguistics and “content words” in Information Science.
3.2.5 Verbs
- 28 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
Verbs are associated with motions involving relations between objects (Kersten 1998).
From an information seeking perspective, verbs do not appear to contribute to the
classification accuracies. In order to validate this hypothesis, verbs are included as one of
the linguistically motivated knowledge sources that were examined in our study.
3.2.6 Adjectives
Bruce and Wiebe’s (1999) work has established a positive correlation with the presence
of adjectives with subjectivity. The presence of one or more adjectives is essential for
predicting that a sentence is subjective. Subjectivity tagging refers to distinguishing
sentences used to present opinions and evaluations from sentences used to objectively
present factual information. There are numerous applications for which subjectivity
tagging is relevant, including information retrieval and information extraction. This task
is essential to forums and news reporting. For a complete study on the use of
linguistically motivated knowledge sources, we have included adjectives as one source of
linguistic knowledge in our experiment.
3.2.7 Combination of Sources
Each linguistic knowledge source generates a feature vector from the context of the
document. However we also examine the combination of two linguistic knowledge
sources which is a novel technique. When sources are combined, the features generated
- 29 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
from each knowledge source are concatenated with each source contributing to half of the
total number of features and the dataset with all these features are generated. Here we
combine the words and the linguistically motivated knowledge sources, nouns, noun
phrase and adjectives as novel combinations to see if there are any improvements. Based
on the fact that the original features are retained, while some syntactical structure is
captured using this model, there appears to be an advantage in using a combination of
techniques.
3.3
Obtaining Linguistically Motivated Classifiers
Our technique has four steps (Figure 5). The input to the technique is a document, D.
Below is an outline of the generic process proposed and employed to use the
linguistically motivated knowledge sources as features:
1. Firstly, the document is broken up into sentences.
2. Morphological descriptions or tags are assigned to each term. This NLP
component does linguistic processing on the contents and attaches a tag to every
term.
3. Processed terms are parsed.
4. Linguistically motivated knowledge sources are then extracted based on the
tagging requirements as discussed earlier.
5. Features are combined at the binder phase if combinations of features are
required.
- 30 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
As a final step, the set of linguistically motivated knowledge sources obtained is used as
the input feature set for the training or testing phase of the documents.
Document
Sentence
Boundary
Detection
Words
Assignment of
Morphological
Descriptions
Tags
Parsing
Extractor
Phrase
Noun
Adjective
Verb
No
Combine
Tags?
Feature Type e.g.
Phrase
Yes
Binder
Combined Feature e.g. Phrase and Words
- 31 -
CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION
Figure 5: Extracting Linguistically Motivated Knowledge Sources
- 32 -
CHAPTER 4 EXPERIMENT
CHAPTER 4
EXPERIMENT
A controlled experimental study was conducted to validate the effectiveness of
linguistically motivated knowledge sources. This chapter describes the experimental
setup employed throughout the study. Section 4.1 describes the evaluation data sets used
in the study. Section 4.2 presents the preprocessing methods required in the study.
Section 4.3 provides more details on the handling of multiple categories. Section 4.4
presents the evaluation measures used. The last section, 4.5, describes the tools utilized in
the study.
4.1 Evaluation Data Sets
There are standard benchmark collections available for experimental purposes. We have
tested numerous linguistically motivated knowledge sources and their combinations that
were presented in Chapter 3 with two widely used data corpus: Reuters-21578 and
WebKB. These data corpus vary in many characteristics. This section articulates the
characteristics of all the data sets that were used in our experiment.
4.1.1 Reuters-21578
- 33 -
CHAPTER 4 EXPERIMENT
The dataset that we have chosen is the Reuters-21578 dataset. This is a widely used
collection. It accounts for most of the experimental work in classification (Sebastiani,
2002).
This
dataset
can
be
http://www.daviddlewis.com/resources/testcollections/reuters21578.
obtained
This
from
document
corpus was originally collected from the Reuters newswire stories manually indexed by
human indexers. The data was originally collected and labeled by Carnegie Group, Inc.
and Reuters, Ltd. in the course of developing the CONSTRUE text categorization
system.
The Reuters corpus consists of 21,578 documents classified into 135 categories. Of these
categories are 5 major categories and 118 subcategories. A total of more than 20,000
unique terms can be found in this corpus. The set of documents occur in SGML format
and majority of them contain categories labels which has been obtained through a manual
process. An example of a single document in the Reuters corpus is shown in Figure 6. In
our empirical study, we chose the “Modified Apte Split” (ModApte) version of Reuters21578 because of the abundance of recent work using this configuration. The ModApte
version place all documents written on or before 7 April 1987 into the training set and the
rest of the documents into the test set. Of these documents, 9603 training examples are
used as training documents and 3299 documents are used as testing examples, some of
which have no class labels attached. Only 90 classes for which at least one training
example and test example exists are included.
- 34 -
CHAPTER 4 EXPERIMENT
An analysis of the corpus reveals that the category distribution in the Reuters dataset is
highly skewed. Table 2 shows the overview of the category sets and the detailed
breakdown of the categories. The 674 categories are broken down into 5 groups:
1. Exchanges: Stock and commodity exchanges
2. Organizations: Economically important regulatory, financial, and political
organizations.
3. People: Important political and economic leaders.
4. Places: Countries of the world.
5. Topics: These are subject categories of economic interest.
Categorization
Subcategories
EXCHANGES
ORGS
PEOPLE
PLACES
TOPICS
39
56
267
175
135
Subcategories with >1
occurrences
32
32
114
147
120
Subcategories with > 20
categories
7
9
15
60
57
Table 2: Distribution of Categories in Reuters-21578
The training set has some categories with high numbers of documents whereas some with
extremely few documents. On average the documents belongs to 1.3 categories. We have
chosen a random subset of documents from the top 20 subcategories in our evaluation as
shown in Table 3.
Category
Train
Test
Acq
Bop
Corn
Cpi
Crude
1650
75
182
69
389
719
30
56
28
189
- 35 -
CHAPTER 4 EXPERIMENT
Dlr
Earn
Gnp
Grain
Interest
livestock
Money-fx
Money-supply
nat-gas
Oilseed
Ship
soybean
Sugar
Trade
Wheat
131
2877
101
433
347
75
538
140
75
124
197
78
126
369
212
44
1087
35
149
131
24
179
34
30
47
89
33
36
118
71
Table 3: Top 20 categories in ModApte Split
26-FEB-1987 15:01:01.79
cocoa
el-salvadorusauruguay
5; 5; 5;C T
22; 22; 1;f0704 31;reute
u f BC-BAHIA-COCOA-REVIEW
02-26 0105
2;
BAHIA COCOA REVIEW
SALVADOR, Feb 26 - Showers continued
throughout the week in
the Bahia cocoa zone, alleviating the drought since early
January and improving prospects for the coming temporao,
although normal humidity levels have not been restored,
Comissaria Smith said in its weekly review.
The dry period means the temporao will be late this year.
Arrivals for the week ended February 22 were 155,221 bags
of 60 kilos making a cumulative total for the season of 5.93
mln against 5.81 at the same stage last year. Again it seems
that cocoa delivered earlier on consignment was included in …
3;
Figure 6: An Example of a Reuters-21578 Document
- 36 -
CHAPTER 4 EXPERIMENT
4.1.2 WebKB
The other corpus that we used is the World Wide Knowledge Base dataset that was
collected by Craven et al. (1999). WebKB was collated by a crawler as part of an effort to
build a data corpus that mirror the World Wide Web.
The primary reason for choosing this corpus is that this corpus is of a different domain
and has dissimilar characteristics from the Reuters corpus. The WebKB corpus is of a
web-based domain. This corpus contains 8282 web pages from four academic domains
contributed by different authors with very different styles of writing and presenting
information on the web.
Divided into two different polychotomies, topic and web domain, we have decided to use
the first polychotomy in our experiments carried out by Nigam et al. (1998) and
Bekkerman et al. (2003) in which four categories were used: course, faculty, project and
student. This contributes to a total of 4199 documents. Our experiment also used these
four categories and used the pages from Cornell as the test pages. A detailed breakdown
of the data corpus is shown in Table 4.
Category
Course
Faculty
Project
Student
Number Proportion of
+ve e.g.(% )
930
22.1
1124
26.8
504
12.0
1641
39.1
Table 4: Breakdown of documents in WebKB Categories
- 37 -
CHAPTER 4 EXPERIMENT
4.2 Dimensionality Reduction
In a large document collection, the dimensionality of the feature space is very high
leading to more time and memory requirements in processing the feature vectors. This
calls for a feature selection phase that will address this issue by increasing data sparsity.
The first section gives an overview of feature selection methods used in document
classification studies followed by a detailed explanation of the techniques that we have
adopted for our study.
4.2.1 Feature Selection
Feature selection is typically performed by assigning a weight to each term and keeping a
certain number of terms with the highest scores, while discarding the rest of the terms.
Experiments then evaluate the classification performance on the resulting feature vectors
based on the features retained after the feature selection phase. Some learning algorithms
cannot scale to high dimensionality. For instance, LLSF algorithm cannot scale to high
dimensions. It has also been proven that some classifiers perform worse when using all
features e.g. K-NN, Rocchio, C4.5 and some classifiers like C4.5 are too inefficient to
use all features.
An advantage of dimensionality reduction is that it tends to reduce overfitting which
refers to the phenomenon by which a classifier is turned into the contingent
- 38 -
CHAPTER 4 EXPERIMENT
characteristics of the training data rather than just the constitutive characteristics of the
categories. Classifiers that overfit are extremely accurate on the training data such that
the performance on the test data is worse. Experiments have shown that in order to avoid
overfitting, some research suggested that 50-100 training examples per term may be
needed in document classification tasks (Fuhr & Buckley, 1991). Thus, it implies that
overfitting may be avoided by using a smaller amount of training examples. However, on
the other hand, removing features risk removing potential useful information on the
meaning of documents. Hence, the dimensionality reduction step must be done
appropriately.
Various dimensionality reduction methods have been proposed.
Methods such as
document frequency thresholding and empirical mutual information are used to select a
set of features to represent the documents. Other methods such as odds-ratio, chi-square
score, stopword removal and stemming are also used for feature selection. Document
frequency thresholding and stop word removal is employed in this study.
4.2.1.2 Document Frequency Thresholding
Of the many feature selection methods, document frequency is a simple and effective
feature selection method popular in conventional document classification literature.
Document frequency thresholding refers to the use of features that occur at least t times
in the training documents. In other words, it gives the weight to the feature by assigning a
value equal to that of its document frequency. Document frequency is the number of
- 39 -
CHAPTER 4 EXPERIMENT
documents in which the feature occurs.
This is one of the simplest but effective
techniques (Yang & Pederson, 1999) for vocabulary reduction. It scales easily to very
large corpus with a computational complexity approximately linear in the number of
training documents.
The document frequency for each feature in the training corpus is computed and removed
from the feature space where the features or terms whose document frequency was less
than some threshold, in the previous case. It is assumed that rare terms are not
informative in terms of global performance. This will greatly reduce the dimensionality
of the feature space. Although this is not always accurate, an improvement in
classification accuracy is possible if the removal of rare terms is actually removing noise
terms.
As the emphasis of our study is on the effects of different learning algorithms with
different linguistic knowledge sources, we will assume that the feature selection will not
be a major factor in our study as we will be using the same feature selection technique,
i.e. document frequency thresholding in all our experiments setup with different learning
algorithms and different linguistic knowledge sources. Furthermore, Yang & Pederson’s
(1999) work has consolidated the use of document frequency with Reuters corpus is not
just an ad hoc approach to improving classification accuracy but a reliable metric for
feature selection. They have suggested that document frequency should be a better
alternative when the computation of information gain or χ2 test proves to be too costly.
- 40 -
CHAPTER 4 EXPERIMENT
4.2.2 Stop Words Removal
Stop words refer to words that occur very frequently within one document such as “the”,
which does not add any discriminative value to the classifier. Thus, stop words
elimination is carried out to identify irrelevant features. When document frequency
thresholding removes particularly infrequent words, stop word elimination removes
mostly high frequency words from the attribute set. All words that occur in the list of stop
words will not be considered as features.
Feature selection based on rankings implies that the process is a greedy process that does
not account for dependencies between words. A list of stop words is provided in
Appendix A. We removed stop words for documents in Reuters-21578 but this was not
carried out for documents in WebKB as the number of features derived from the web
pages was much smaller than that of Reuters-21578.
4.3 Experiment Setup
4.3.1 Handling Multiple Categories Problems
For all learning algorithms, we used the default settings established in WEKA to train the
classifiers. We used binary classification for every learning algorithm. Each learning
classifier is created for each category. If there are more than two categories for a
- 41 -
CHAPTER 4 EXPERIMENT
document instance, an instance for each category will be treated as separate instances for
each category. Thus there are twice as many classifiers as there are number of categories.
4.3.2 Handling Multiple Categories Training Examples
As some documents in Reuters-21578 were labeled with more than 1 category, we
created one training instance for each category that a training example in the corpus has.
We created one positive training instance for each example labeled as belonging to that
category. Thus the number of training instances generated for the normal variant of
learning algorithms can be more than the number of training examples provided, but the
number of training instances generated for the 1-per-category variant is the same as the
number of training examples.
4.4 Evaluation Measures
4.4.1 Loss-based Measures
A commonly used performance measure in the machine learning community is the error
rate which is defined as the probability of the classification rule in predicting the wrong
class.
E(h) = Pr(h != Y|h)
- 42 -
CHAPTER 4 EXPERIMENT
Estimators for these measures can be defined using a contingency table of prediction so
on an independent test set. Each cell of the contingency table (see Table 5) represents one
of the four possible outcomes of prediction h(x) for an example:
f++, is the number of instances categorized under the category c, and the classifier predicts
it correctly as under the category c;
f+-, is the number of instances categorized not to belong to category c, but the classifier
predicts it incorrectly as belonging to category c;
f-+, is the number of instances categorized as belonging to category c, but the classifier
predicts it incorrectly as not belonging to category c;
f--, is the number of instances categorized not to belong to category c, and the classifier
predicts it correctly as not belonging to category c;
Predicted Category +1
Predicted Category -1
Actual Category y = +1
f++
f-+
Actual Category y=-1
f+f--
Table 5: Contingency Table
Formally, the conventional estimators, for these measures using the contingency table,
rooted in probabilistic theories, is
E ( h) =
f +− + f −+
f + + +f + − + f − + + f
−−
(7)
Sometimes cost matrix may be introduced where the utility of predicting a positive
example correctly is higher or lower than predicting a negative example and vice versa.
- 43 -
CHAPTER 4 EXPERIMENT
4.4.2 Recall and Precision
To avoid the problem of using the accuracy as a measure in text classification
experiments, we have considered two other performance measures, recall and precision.
Recall is the probability that a document with label y = 1 is classified correctly.
Re call (h) = Pr(h( x) = 1 | y = 1, h)
Re call (h)estimate =
f
f
++
++
+f
−+
(8)
(9)
Precision is the probability that a document classified as h(x) = 1 is classified correctly.
Pr ecision(h) = Pr( y = 1 | h( x) = 1, h) (10)
Pr ecision(h)estimate =
f
f
++
++
+f
+−
(11)
4.4.3 Precision and Recall Breakeven Point
While precision and recall accurately describe the classification performance, it is
difficult to compare different learning algorithms with two disparate scores. One popular
evaluation method for balancing precision and recall is the F-measure value proposed by
Rkjsbergen (1979). This measure combines precision and recall with a new parameter
specifying the importance of recall relative to precision. This measure is defined by
equation 12.
- 44 -
CHAPTER 4 EXPERIMENT
(1 + β 2 ) Pr ecision(h) Re call (h)
Fb(h) =
β 2 Pr ecision(h) + Re call (h)
(12)
If =1, this means that precision and recall are equally important and have the same
weight. The performance measure that we have employed is the F1 measure as given in
equation 13.
F 1(h) =
2 Pr ecision(h) Re call (h)
Pr ecision(h) + Re call (h)
(13)
4.4.4 Micro- and Macro- Averaging
F-measure measures the effectiveness of a classifier on a single class. However as most
text classification tasks contain many categories, each will have their own F-measure
value. Two other measures, macro-average value and micro-average values were used in
this study.
The macro-average values take the averages of all categories’ contingency table as
defined by:
F 1Macro =
1 m
∑ F1 (hi )
m i =1
(14)
- 45 -
CHAPTER 4 EXPERIMENT
On the other hand, the micro-average values makes use of all the contingency table
values for each category to obtain a new averaged contingency table by component-wise
addition to obtain averages for f++, f+-, f-+ and f-- . Finally, the micro F1 value is defined as:
F 1Micro =
2 f +Average
+
2 f +Average
+
Average
+ f +−
+ f −Average
+
(15)
4.5 Tools
Finally, to facilitate the study of the effects of linguistically motivated knowledge sources
with learning algorithms, a generic design of our system was conceptualized. The design
of our system is shown in Figure 7.
The system consists of four main modules:
1. The document management module;
2. The feature preprocessor module;
3. The learning classifier module;
4. The Graphical User Interface (GUI) module.
Tools that were readily available were tested and integrated into the system where
applicable such as the learning classifier module. Other modules had to be implemented.
- 46 -
CHAPTER 4 EXPERIMENT
Preprocess
document
Extract
Features
Document
Management
Module
Feature Engine
Module
Document
database
Learning
Classifier
Module
Model
Category
User Interface
Module
New
Document
Figure 7: Design of system
The details of each module follow:
The document management module contains functions to parse and process the
documents in the document repository. This module consists of a document handler and a
filter sub-component. The document handler parses the document based on the existing
format and the filter does additional pre-processing such as removing HTML tags for the
WebKB documents.
Having preprocessed the documents, the strings of text are passed to the feature engine
module where the documents are converted to the linguistic units or tokens that serve as
sources of linguistic knowledge to the learning classifier. The strings are tokenized and
an appropriate feature selection is performed on them. In our empirical study,
linguistically motivated tokens have been used. This entails the use of a Part of Speech
- 47 -
CHAPTER 4 EXPERIMENT
Tagger that tags the required system. In our experiments, we have employed SNOW
tagger (http://l2r.cs.uiuc.edu/~cogcomp/cc-software.html).
The classifier module is a module containing several learning algorithms. In our system,
WEKA 3-6 has been integrated with the system.
Finally, the GUI module contains codes to render the appropriate user interface for the
user to execute the classifier. Currently, the UI module consists of a raw text based
interface.
Based on the setup as described in this chapter, a series of experiments were conducted,
using 6 learning algorithms – Naïve Bayes (NB), Support Vector Machines (SVM), k-NN
Instance-Based Learner (k-NN), C4.5, RIPPER (JRip) and Alternating Decision Trees
(ADTree), with different linguistically motivated sources, including novel combinations
of these sources using the proposed approach that has been described in Chapter 3.
Results from the experiment with the use of this system will be discussed in details in the
next chapter.
- 48 -
CHAPTER 5 RESULTS & ANALYSIS
CHAPTER 5
RESULTS & ANALYSIS
In this chapter, results for our experiments are presented. The results of each type of
linguistically motivated knowledge source are then discussed in detail and we proceed to
give an overall assessment and analysis of the interaction effects of using different
linguistically motivated feature representations with different learning algorithms.
Section 5.1 presents an extensive set of experiment results based on the Reuters-21578
corpus. Comparisons of each of the linguistically motivated classifiers were then
described in the Section 5.1. In Section 5.2, the experiment results for the WebKB corpus
are then presented.
5.1
Results
Experiments were conducted on Reuters-21578 and WebKB corpora. This section
describes the series of experiments that have been carried out using different
linguistically motivated knowledge sources and learning algorithms mentioned in
Chapter 2 on these corpora.
5.2
Contribution of Different Linguistic Knowledge Sources to
Classification of Reuters-21578 Corpus
- 49 -
CHAPTER 5 RESULTS & ANALYSIS
5.2.1 Words
Table 6 shows the results of using word based representation. This set of results will be
used as a baseline for comparison with other linguistic knowledge sources. Since the
value of F1 occurs between 0 and 1, where the higher the value the better the classifier, it
was observed that relatively good performances have been achieved with the use of
words as representations for each of the learning algorithms tested. The results in Table 6
will be used as the baseline classifier for linguistically motivated classifiers.
Algorithm
Word
Micro F1
Macro F1
NB
0.860
0.860
SVM
0.884
0.882
k-NN
0.868
0.867
C4.5
0.870
0.866
Ripper
0.864
0.860
ADTree
0.887
0.885
Table 6: Results using Words
5.2.2 Phrase
Results of the use of noun phrase representation are being compared against word
representation in Table 7. Surprisingly, the majority of the results show that the word
based representations do better than using noun phrases. This result corresponds to the
work of Lewis (1992) and Scott (1999). The results indicate that some of the
informational content may have been lost.
- 50 -
CHAPTER 5 RESULTS & ANALYSIS
From our observations, we see that some of the noun phrases were not identified
correctly by SNOW which could have led to a decrease in the micro and macro values.
However, it is worth noting that the use of support vector machines has the best
performance with phrase based representations and showed minimal improvement to the
macro value.
The reason for the poor performance of the new linguistic classifiers built using phrases
as the linguistically motivated knowledge source could be due to:
i)
Sparse distribution of noun phrases.
ii)
Synonymous phrases which dilutes the contribution of each features which
essentially have the same meaning. For e.g., the occurrence of the noun phrase
“Standard oil company” and “oil company” becomes two separate features.
iii)
Noise introduced via the separation of synonymous phrases into multiple
phrases and the inaccurate identifications of noun phrases by the tagging tool.
Algorithm
Word
Micro F1
NB
SVM
k-NN
C4.5
Ripper
ADTree
Phrase
Macro F1
Micro F1
Change (%)
Macro F1
Change (%)
0.86
0.86
0.833
-3.140
0.83
-3.488
0.884
0.882
0.852
-3.620
0.884
0.227
0.868
0.867
0.835
-3.802
0.834
-3.806
0.87
0.866
0.842
-3.218
0.837
-3.349
0.864
0.86
0.828
-4.167
0.822
-4.419
0.887
0.885
0.848
-4.397
0.847
-4.294
Table 7: Results using Phrases
- 51 -
CHAPTER 5 RESULTS & ANALYSIS
5.2.3 Word Sense (Part of Speech Tags)
There has been a drop in the micro and macro averages of the word sense representation
which indicates that there is no clear advantage of using word sense as a representation
for document classification. Although using word senses has been reported to produce
better results in word sense disambiguation literature (Ng & Lee, 1996), it is not the case
for document classification. Adding part of speech tags will not add much improvement
because there are few terms that has very different meanings for different word senses.
The only learning algorithm that worked well with word senses, or part of speech tag,
was C4.5 trees. Our findings indicate that C4.5 actually does better for both micro and
macro F1 with the linguistically motivated feature, word senses, than with words.
Algorithm
NB
SVM
k-NN
C4.5
Ripper
ADTree
Word
Tags
Micro F1
Macro
F1
Micro F1
Change (%)
Macro
F1
Change (%)
0.86
0.86
0.815
-5.233
0.812
-5.581
0.884
0.882
0.864
-2.262
0.863
-2.154
0.868
0.867
0.843
-2.880
0.842
-2.884
0.87
0.866
0.879
1.034
0.88
1.617
0.864
0.86
0.856
-0.926
0.854
-0.698
0.887
0.885
0.881
-0.676
0.881
-0.453
Table 8: Results using Word Senses (Tags)
However, it is probable that word sense representation will be very effective especially in
data corpora containing terms that depict multiple meanings with different word senses.
For example, the word “sweet” has different word senses with different meanings. As a
- 52 -
CHAPTER 5 RESULTS & ANALYSIS
noun, the word has an approximate meaning of “candy”. As an adjective, it may mean
“lovable”. Since Reuters-21578 documents do not appear to exhibit this characteristic,
using this linguistically motivated knowledge source does not improve the baseline
classifier on most of the classifiers except for C4.5.
5.2.4 Nouns
Results with the linguistic classifiers built using bag-of-nouns actually produced superior
results with improvements ranging from 3.05- 4.76%. Except for Naïve Bayes, the results
indicate that using nouns are able to capture most of the informational content of the
documents. Thus, the results show that the use of noun representations shows signs of
improved micro and macro values. The reason for the improvement in the results could
be because the semantic components of noun meanings are more strongly interconnected
than those of verbs and word senses. Nouns appeared to have captured features with
salient concepts required for classification accuracy.
Algorithm
Word
Micro F1
NB
SVM
k-NN
C4.5
Ripper
ADTree
NN
Macro
F1
Micro F1
Change (%)
Macro F1
Change (%)
0.86
0.86
0.861
0.116
0.857
-0.349
0.884
0.882
0.924
4.525
0.924
4.762
0.868
0.867
0.901
3.802
0.899
3.691
0.87
0.866
0.908
4.368
0.906
4.619
0.864
0.86
0.891
3.125
0.888
3.256
0.887
0.885
0.913
2.931
0.912
3.051
Table 9: Results using Nouns
- 53 -
CHAPTER 5 RESULTS & ANALYSIS
5.2.5 Verbs
The results in table 10 show that there was a huge drop in the micro and macro F1 values.
Since verbs are more mutable than nouns, it follows that the micro and macro values
should fare worse than that of nouns as reflected in the results. Thus, verb is not a good
choice for feature representation for classification tasks. This may also indicate that the
removal of verbs from the documents may not affect the classification accuracy among
the learning algorithms adopted in this study.
Algorithm
NB
SVM
k-NN
C4.5
Ripper
ADTree
Word
Verb
Micro F1
Macro F1
Micro F1
Change (%)
Macro F1
Change (%)
0.86
0.86
0.636
-26.047
0.625
-27.326
0.884
0.882
0.637
-27.941
0.625
-29.138
0.868
0.867
0.636
-26.728
0.628
-27.566
0.87
0.866
0.619
-28.851
0.601
-30.60
0.864
0.86
0.594
-31.25
0.566
-34.186
0.887
0.885
0.628
-29.199
0.611
-30.960
Table 10: Results using Verbs.
5.2.6 Adjectives
From the results (see Table 11), there is a clear indication that there is no clear advantage
in using adjectives as feature representations due to the drop in performance compared to
using purely words. However, as compared to verbs, the performance is slightly better.
- 54 -
CHAPTER 5 RESULTS & ANALYSIS
Algorithm
NB
SVM
k-NN
C4.5
Ripper
ADTree
Word
Adjective
Micro F1
Macro F1
Micro F1
Change (%)
Macro F1
Change (%)
0.86
0.86
0.739
-14.070
0.73
-15.116
0.884
0.882
0.738
-16.516
0.732
-17.007
0.868
0.867
0.75
-13.594
0.747
-13.841
0.87
0.866
0.726
-16.552
0.717
-17.206
0.864
0.86
0.725
-16.088
0.713
-17.093
0.887
0.885
0.74
-16.573
0.733
-17.175
Table 11: Results using Adjectives.
5.2.7 Combination of Sources
The results using a combination of words and noun phrase representations led to
surprising results. This was due to the improvement shown across several learning
algorithms over word representations.
This shows that the use of noun phrase could have led to higher discriminating power
with the use of bag of words representation. The improvements were consistent across the
different learning algorithms.
Algorithm
NB
SVM
k-NN
C4.5
Ripper
ADTree
Word
Combine
Micro F1
Macro
F1
Micro F1
Change (%)
Macro
F1
Change (%)
0.86
0.86
0.864
0.465
0.861
0.116
0.884
0.882
0.892
0.905
0.892
1.134
0.868
0.867
0.88
1.382
0.879
1.384
0.87
0.866
0.878
0.920
0.877
1.270
0.864
0.86
0.864
0
0.862
0.233
0.887
0.885
0.892
0.564
0.891
0.678
Table 12: Results using both Linguistically Knowledge Source and Words.
- 55 -
CHAPTER 5 RESULTS & ANALYSIS
5.2.8 Analysis of Reuters-21578 Results
An analysis of the results was carried out from two perspectives, one from the features
and the other, the learning algorithm used. The former allows us to identify the features
that will improve the classification accuracy of the learning algorithm. The latter allows
us to select the appropriate type of learning algorithm that can be employed in other
applications based on the characteristics of the data, for example, if the data corpus
consists of many adjectives, we are able to tell which sort of learning algorithm is
suitable for classifying the documents.
Algorit
Word
Phrase
Combine
Tag
NN
Verb
Adjective
hm
Micro
Macro
Micro
Macr
Micro
Macr
Micro
Macr
Micro
Macr
Micro
Macr
Micro
Macr
F1
F1
F1
o F1
F1
o F1
F1
o F1
F1
o F1
F1
o F1
F1
o F1
NB
0.860
0.860
0.833
0.830
0.864
0.861
0.815
0.812
0.861
0.857
0.636
0.625
0.739
0.730
SVM
0.884
0.882
0.852
0.884
0.892
0.892
0.864
0.863
0.924
0.924
0.637
0.625
0.738
0.732
k-NN
0.868
0.867
0.835
0.834
0.880
0.879
0.843
0.842
0.901
0.899
0.636
0.628
0.750
0.747
C4.5
0.870
0.866
0.842
0.837
0.878
0.877
0.879
0.880
0.908
0.906
0.619
0.601
0.726
0.717
Ripper
0.864
0.860
0.828
0.822
0.864
0.862
0.856
0.854
0.891
0.888
0.594
0.566
0.725
0.713
ADTree
0.887
0.885
0.848
0.847
0.892
0.891
0.881
0.881
0.913
0.912
0.628
0.611
0.740
0.733
Table 13: Contribution of Knowledge Sources on Reuters-21578 data set (MicroAveraged F1, Macro-averaged F1)
Table 13, shows the micro-averaged and macro-averaged F1 measures for the different
knowledge sources and learning algorithms for the Reuters-2178 data set. The seven
columns in the table correspond to:
(i)
using only words;
(ii)
using only phrases;
- 56 -
CHAPTER 5 RESULTS & ANALYSIS
(iii)
using both words and phrases;
(iv)
using part of speech tag with words;
(v)
using only nouns;
(vi)
using only verbs and,
(vii)
using only adjectives.
Each of these knowledge sources were used with document frequency as the feature
selection technique.
Comparison of Different Linguistically Motivated Knowledge Sources
on Reuters-21578 (Micro F1 Values)
1
Micro F1
0.9
0.8
Words
0.7
Phrase
0.6
Combine
0.5
Tags
0.4
Nouns
0.3
Verbs
0.2
Adjectives
0.1
0
NB
SVM
k-NN
C4.5
Ripper
ADTree
Learning Algorithms
Figure 8: Comparison of Different Linguistically Motivated Knowledge Sources on
Reuters-21578 (Micro F1 values)
The best micro-averaged F1 value for Reuters-21578 is 92.4% (Figure 8), obtained by
using the linguistically motivated knowledge source, nouns, as features and SVM, as the
learning algorithm. The results indicate that SVM performs best with nouns, combined
words and phrases, words, phrases, relatively worse on adjectives and worst on verbs.
- 57 -
CHAPTER 5 RESULTS & ANALYSIS
The next best performing setup for micro-averaged F1 value gives values of 91.3% which
is obtained by ADTree with nouns.
Verb was consistently the worst performing
linguistically motivated knowledge source when employed as the sole feature
representation across all the learning algorithms. It is worthwhile to note that for each of
these learning algorithms tested, the best performing linguistically motivated knowledge
source was nouns, except for Naïve Bayes. Combining linguistically motivated
knowledge source generated the best performer for Naïve Bayes.
Comparison of Different Linguistically Motivated Knowledge Sources
on Reuters-21578 (Macro F1 values)
1
Macro F1
0.9
0.8
Words
0.7
Phrase
0.6
Combine
0.5
Tags
0.4
Nouns
0.3
Verbs
0.2
Adjectives
0.1
0
NB
SVM
k-NN
C4.5
Ripper
ADTree
Learning Algorithms
Figure 9: Comparison of Different Linguistically Motivated Knowledge Sources on
Reuters-21578 (Macro F1 values)
The best macro-averaged F1 value turned out to be the same as the results for microaveraged F1 values. SVM, together with nouns as a linguistically motivated knowledge
- 58 -
CHAPTER 5 RESULTS & ANALYSIS
source, outperformed all other classifiers. Similar to the micro-averaged F1 results, the
order of the performance of the linguistically motivated knowledge sources with SVM is:
nouns, combined words and nouns, words, phrases, tags, adjectives and verbs. Just as the
previous set of results, the next best performing classifier was obtained using AdTree
with nouns as the linguistically motivated knowledge source. The macro-averaged F1
value was 91.2%. Observation of the macro-averaged data also reveals the same trend in
the order of performance of each of these linguistically motivated features.
Algorithm
Word
Phrase
Combine
Tag
NN
Verb
Adjective
NB
0.898
0.872
0.884
0.845
0.891
0.687
0.750
SVM
0.902
0.871
0.886
0.867
0.930
0.677
0.734
k-NN
0.874
0.842
0.871
0.814
0.916
0.638
0.708
C4.5
0.898
0.873
0.861
0.889
0.920
0.678
0.724
Ripper
0.904
0.872
0.862
0.888
0.912
0.646
0.718
ADTree
0.902
0.871
0.880
0.884
0.918
0.670
0.730
Table 14: Contribution of knowledge sources on Reuters-21578 (Precision)
Table 14 shows the tabulation of precision values for each of the learning algorithms
using different feature representations as different knowledge sources. Figure 10 shows a
graphical representation of the results. Again, the results based on precision values
consolidates our previous observations that using nouns only gives the best results for
SVM, k-NN, C4.5, Ripper and AdTree. However, unlike the previous case whereby
Naïve Bayes works best with combined words and phrases as features. It is worth noting
that the precision is most of the time higher with words than the combined
- 59 -
CHAPTER 5 RESULTS & ANALYSIS
representations. The worst performing feature is obviously the verb as feature
representation which is consistent with all the learning algorithms.
Comparison of Different Linguistically Motivated Knowledge
Sources on Reuters-21578 (Precision)
Precision
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Word
Phrase
Combine
Tag
Nouns
Verbs
Adjectives
NB
SVM
k-NN
C4.5
Ripper
ADTree
Learning Algorithm
Figure 10: Comparison of Different Linguistically Motivated Knowledge Sources on
Reuters-21578 (Precision)
Algorithm
Word
Phrase
Combine
Tag
NN
Verb
Adjective
NB
0.830
0.806
0.852
0.793
0.836
0.600
0.746
SVM
0.874
0.847
0.906
0.871
0.923
0.620
0.766
k-NN
0.872
0.841
0.895
0.883
0.890
0.657
0.805
C4.5
0.851
0.827
0.904
0.883
0.903
0.589
0.755
Ripper
0.834
0.807
0.877
0.836
0.879
0.616
0.782
ADTree
0.881
0.841
0.911
0.888
0.912
0.614
0.774
Table 15: Contribution of knowledge sources on Reuters-21578 (Recall)
- 60 -
CHAPTER 5 RESULTS & ANALYSIS
From Table 15 we see that the combined feature representations actually outperform
nouns only in three of the learning algorithms. Figure 11 shows the graphical
representation of the results. From the results, it was observed that combined
representation works best with Naïve Bayes, k-NN and C4.5. On the other hand, noun
features outperforms other feature representations including combined representations
with SVM, Ripper and ADTree. The worst performing algorithm as reflected in the micro
and macro averaged values, is using verbs as the features. From our analysis of the
preliminary results, we feel that there is potential to further combine words, phrases and
nouns to produce even better results. These findings led us to further investigate the
contribution of novel combinations of linguistically motivated features in another
experiment that will be discussed in a later section.
Recall
Comparison of Different Linguistically Motivated
Knowledge Sources on Reuters-21578 (Recall)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Word
Phrase
Combine
Tags
Nouns
Verbs
Adjectives
NB
SVM
k-NN
C4.5
Ripper
ADTree
Learning Algorithm
Figure 11: Comparison of Different Linguistically Motivated Knowledge Sources on
Reuters-21578 (Recall)
- 61 -
CHAPTER 5 RESULTS & ANALYSIS
Based on the results, it appears that extracting nouns as features gives the best results for
all the learning algorithms that were tested. Adjectives and verbs were not so effective for
document classification with the learning algorithms. This could be because nouns extract
the informative terms in this data set as compared to other context source. The removal of
non-informative terms will increase the results due to the reduction of noise in the
category prediction process. The findings also led to an indication for the use of novel
combinations of linguistically motivated knowledge sources with bag-of-words which
will be carried out using the WebKB corpus.
5.3 Contribution of Linguistic Knowledge Sources to
Classification accuracy of WebKB Corpus
Following the interesting findings from our previous experiment, we now attempt to
reproduce the set of linguistically motivated features with a different data corpus. In
addition, we also attempt to use novel combinations of linguistically motivated features
with the bag-of-word representations to determine the effects of such novel combinations
on document classification. A similar experiment was conducted on the WebKB
collection.
5.3.1 Words
Table 16 shows the results of using word based representation. This will also be used as
the baseline classifier for the WebKB corpus. As seen in the Reuters-21578 corpus, good
- 62 -
CHAPTER 5 RESULTS & ANALYSIS
performances were also achieved with the use of words as representations for each of the
learning algorithms tested with WebKB.
Algorithm
Micro F1
Macro F1
Naïve Bayes
0.92
0.87
SVM
0.98
0.97
K-NN
0.73
0.65
AdaBoost
0.96
0.93
Adtree
1.00
0.99
Ripper
1.00
0.99
C4.5
0.77
0.99
Table 16: Results using Words
5.3.2 Phrase
From Table 17, the results of using phrase based features to build linguistically motivated
classifiers are shown. Similar to the Reuters corpus, the results showed no indication that
phrases help to improve the F1 measure. In fact, the results indicate a substantial drop in
both micro and macro values for each learning algorithm.
The reason for the drop in performance of the phrase based linguistically motivated
classifiers could be due to the sparse number of features especially noun phrases. The
content of HTML pages of WebKB corpus has much less features compared to that of
Reuters. Potentially, this also leads to a reduced effectiveness in the part of speech tagger
that was introduced in the preprocessing phase.
- 63 -
CHAPTER 5 RESULTS & ANALYSIS
Word
Algorithm
Naïve
Bayes
Phrase
Micro F1
Macro F1
Micro F1
Change
(%)
Macro F1
Change
(%)
0.92
0.87
0.73
-20.7
0.71
-18.4
SVM
0.98
0.97
0.72
-26.5
0.69
-28.9
K-NN
0.73
0.65
0.5
-31.5
0.4
-38.5
AdaBoost
0.96
0.93
0.92
-4.2
0.7
-24.7
Adtree
1
0.99
0.71
-29.0
0.67
-32.3
Ripper
1
0.99
0.8
-20.0
0.72
-27.3
0.771
0.994
0.75
-2.7
0.58
-41.6
C4.5
Table 17: Results using Phrases
5.3.3 Word Sense (Part of Speech Tags)
Using tagged terms as a linguistically motivated knowledge source in WebKB seemed to
show mixed results for different learning algorithms. As shown in Table 18, the results
for micro F1 values for SVM and C4.5 shows that the linguistically motivated classifiers
is as competitive as the bag-of-word classifier. However, such features showed a drop in
performance for other learning algorithms.
Word
Algorithm
Naïve
Bayes
TAG
Micro F1
Macro F1
Micro F1
Change
(%)
Macro F1
Change
(%)
0.92
0.87
0.75
-18.5
0.69
-20.7
SVM
0.98
0.97
0.98
0.0
0.72
-25.8
K-NN
0.73
0.65
0.57
-21.9
0.5
-23.1
AdaBoost
0.96
0.93
0.89
-7.3
0.68
-26.9
Adtree
1
0.99
0.56
-44.0
0.63
-36.4
Ripper
1
0.99
0.79
-21.0
0.72
-27.3
0.77
0.99
0.77
0.0
0.69
-30.3
C4.5
Table 18: Results using Word Senses (Tags)
- 64 -
CHAPTER 5 RESULTS & ANALYSIS
5.3.4 Nouns
The results for noun were surprising. This was because the results showed a drop in
performance for both micro and macro F1 values when noun based linguistically
motivated knowledge sources actually gave the best performance in Reuters-21578.
An analysis of the results and features shows that there were very few noun features in
WebKB that could give discriminating power to the learning algorithms. This finding
may also imply the possibility of the interaction effects of the characteristics of data with
the learning algorithms (Yang, 2001).
Word
Algorithm
Naïve
Bayes
Nouns
Micro F1
Macro F1
Micro F1
Change
(%)
Macro F1
Change
(%)
0.92
0.87
0.88
-4.3
0.78
-10.3
SVM
0.98
0.97
0.38
-61.2
0.76
-21.6
K-NN
0.73
0.65
0.4
-45.2
0.43
-33.8
AdaBoost
0.96
0.93
0.89
-7.3
0.77
-17.2
Adtree
1
0.99
0.76
-24.0
0.74
-25.3
Ripper
1
0.99
0.69
-31.0
0.66
-33.3
0.771
0.994
0.53
-31.3
0.54
-45.7
C4.5
Table 19: Results using Nouns
5.3.5 Verbs
Just as we had expected, the verb features had a huge performance drop when only verbs
were used. First, the micro and macro values, expected to fare worse than that of nouns
- 65 -
CHAPTER 5 RESULTS & ANALYSIS
was reflected in the results, because of the sparse distribution of verbs in the web pages.
Second, the verb features obtained from the feature engineering phase consisted of many
words that do not contribute to the semantic content of the web pages. For e.g., words
such as “do” and “go” do not confer distinguishing value to the linguistically motivated
classifiers because they could potentially occur in any of the four categories tested.
Word
Algorithm
Naïve
Bayes
Verb
Micro F1
Macro F1
Micro F1
Change
(%)
Macro F1
Change
(%)
0.92
0.87
0.48
-47.8
0.45
-48.3
SVM
0.98
0.97
0.49
-50.0
0.44
-54.6
K-NN
0.73
0.65
0.41
-43.8
0.32
-50.8
AdaBoost
0.96
0.93
0.56
-41.7
0.49
-47.3
1
0.99
0.43
-57.0
0.32
-67.7
1
0.99
0.41
-59.0
0.3
-69.7
0.771
0.994
0.51
-33.9
0.35
-64.8
Adtree
Ripper
C4.5
Table 20: Results using Verbs
5.3.6 Adjectives
Results for adjectives were also given in Table 21. The use of bag of adjectives also
caused a huge drop in performances. Unlike the results obtained in Reuters, as compared
to results obtained using verbs, the adjectives based linguistically motivated knowledge
sources fared worse than its peer. A close examination of the resulting set of knowledge
sources obtained reveals the distribution of adjectives in the WebKB corpus is sparser
compared to that of verbs. Thus, this suggests that the distribution was too sparse to
confer any distinguishing value to the classifiers.
- 66 -
CHAPTER 5 RESULTS & ANALYSIS
Word
Algorithm
Naïve
Bayes
Micro F1
Macro F1
Adjectives
Micro
Change
F1
(%)
Macro F1
Change
(%)
0.92
0.87
0.63
-31.5
0.48
-44.8
SVM
0.98
0.97
0.76
-22.4
0.35
-63.9
K-NN
0.73
0.65
0.38
-47.9
0.35
-46.2
AdaBoost
0.96
0.93
0.61
-36.5
0.37
-60.2
Adtree
1
0.99
0.29
-71.0
0.33
-66.7
Ripper
1
0.99
0.26
-74.0
0.28
-71.7
0.771
0.994
0.36
-53.3
0.48
-51.7
C4.5
Table 21: Results using Adjectives
5.3.7 Nouns & Words
Next we tried to combine both nouns and words and compare the results to that of using
bag of words. The resulting linguistically motivated classifiers were as competitive as the
baseline classifier and even performed better for some of the learning algorithms.
Competitive classification accuracies were found with AdTree, Ripper, and Adaboost.
Improvements were found with Naïve Bayes, SVM and C4.5.
Word
Algorithm
Naïve
Bayes
Micro F1
Macro F1
Nouns + Words
Micro
Change
F1
(%)
Macro F1
Change
(%)
0.92
0.87
0.93
1.1
0.9
3.4
SVM
0.98
0.97
0.99
1.0
0.98
1.0
K-NN
0.73
0.65
0.68
-6.8
0.59
-9.2
AdaBoost
0.96
0.93
0.95
-1.0
0.83
-10.8
1
0.99
1
0.0
0.99
0.0
Adtree
Ripper
C4.5
1
0.99
1
0.0
0.99
0.0
0.771
0.994
0.99
28.4
0.99
-0.4
Table 22: Results using Nouns & Words
- 67 -
CHAPTER 5 RESULTS & ANALYSIS
5.3.8 Phrase & Words
Linguistically motivated classifiers based on phrases and words were also tested. The
results are shown in Table 23. For micro values, improvements were shown in SVM and
C4.5. Naïve Bayes, AdaBoost, Adtree and Ripper with linguistically motivated
knowledge sources also showed competitive micro results. However, a slight decrease in
accuracy was observed with k-NN.
Word
Algorithm
Naïve
Bayes
Micro F1
Macro F1
Phrases + Words
Micro
Change
F1
(%)
Macro F1
Change
(%)
0.92
0.87
0.92
0.0
0.87
0.0
SVM
0.98
0.97
0.99
1.0
0.92
-5.2
K-NN
0.73
0.65
0.71
-2.7
0.51
-21.5
AdaBoost
0.96
0.93
0.96
0.0
0.82
-11.8
1
0.99
1
0.0
0.99
0.0
1
0.99
0.99
-1.0
0.99
0.0
0.771
0.994
0.98
27.1
0.84
-15.5
Adtree
Ripper
C4.5
Table 23: Results using Phrases & Words
5.3.9 Adjectives & Words
Lastly, the use of a combination of adjectives and bag of words were experimented and
the results are shown in Table 24. Micro F1 values were encouraging because
- 68 -
CHAPTER 5 RESULTS & ANALYSIS
improvements were shown for SVM, k-NN and C4.5. Competitive micro and macro
results were also shown for Naïve Bayes, Adtree and Adaboost.
Word
Algorithm
Naïve
Bayes
Micro F1
Macro F1
Adjectives + Words
Micro
Change
F1
(%)
0.0
0.88
Change
(%)
0.92
0.87
SVM
0.98
0.97
0.99
1.0
0.96
-1.0
K-NN
0.73
0.65
0.81
11.0
0.52
-20.0
AdaBoost
0.96
0.93
0.9
-6.2
0.81
-12.9
Adtree
1
0.99
1
0.0
0.99
0.0
Ripper
1
0.99
1
0.0
0.99
0.0
0.771
0.994
0.99
28.4
0.98
-1.4
C4.5
0.92
Macro F1
1.1
Table 24: Results using Adjectives & Words
5.3.10 Analysis of WebKB Results
From the results, it appears that bag of words does well with most of the learning
algorithms. However, it is noteworthy to highlight that the alternative combinations of
words and adjectives, nouns and phrases were as competitive as, and perform better than
the bag of words with some learning algorithms. However unlike the results obtained for
Reuters-21578, the results obtained for WebKB were less conclusive because of the
differing trends among the results. Best performing classifiers differs across different
learning algorithms but were mainly dominated by either the bag of words or the
combined linguistically motivated knowledge sources with words. In the case for SVM,
the combination of nouns and words performed better than the use of words alone.
Similar to the previous set of results, verbs and adjectives did not seem to add much
discriminative value to the classification.
- 69 -
CHAPTER 5 RESULTS & ANALYSIS
Observations based on the micro-averaged F1 values (Figure 12) shows that the best
performing classifier was found with AdTree and Ripper. AdTree works well with words,
combination of nouns and words, combination of phrase and words and combination of
adjectives and words. Ripper also works well with words, combination of nouns and
words and combination of adjectives and words. This suggests that combinations of
linguistically motivated classifiers can be as competitive as using words alone using a
web-based corpus. The best performing classifiers for the macro-averaged F1 values were
AdTree, Ripper and C4.5 (see Figure 13). Combinations of linguistically motivated
knowledge sources with words showed high macro-averaged scores with these learning
algorithms.
Algorithm
Word
Micro
F1
Phrase
Macro
F1
Tag
Micro
F1
Macro
F1
NN
Micro
F1
Macro
F1
Verb
Micro
F1
Macro
F1
Micro F1
Macro
F1
Naïve
Bayes
0.92
0.87
0.73
0.71
0.75
0.69
0.88
0.78
0.48
0.45
SVM
0.98
0.97
0.72
0.69
0.98
0.72
0.38
0.76
0.49
0.44
K-NN
0.73
0.65
0.5
0.4
0.57
0.5
0.4
0.43
0.41
0.32
AdaBoost
0.96
0.93
0.92
0.7
0.89
0.68
0.89
0.77
0.56
0.49
Adtree
1
0.99
0.71
0.67
0.56
0.63
0.76
0.74
0.43
0.32
Ripper
1
0.99
0.8
0.72
0.79
0.72
0.69
0.66
0.41
0.3
C4.5
0.77
0.99
0.75
0.58
0.77
0.69
0.53
0.54
0.51
0.35
Algorithm
Adjectives
Micro
F1
Naïve
Bayes
0.63
Macro
F1
0.48
Nouns & Words
Phrase & Words
Micro
F1
Micro
F1
Macro
F1
0.93
0.9
Macro
F1
0.92
0.87
Adjectives &
Words
Micro
Macro
F1
F1
0.92
0.88
SVM
0.76
0.35
0.99
0.98
0.99
0.92
0.99
0.96
K-NN
0.38
0.35
0.68
0.59
0.71
0.51
0.81
0.52
AdaBoost
0.61
0.37
0.95
0.83
0.96
0.82
0.9
0.81
Adtree
0.29
0.33
1
0.99
1
0.99
1
0.99
Ripper
0.26
0.28
1
0.99
0.99
0.99
1
0.99
C4.5
0.36
0.48
0.99
0.99
0.98
0.84
0.99
0.98
Table 25: Consolidated Results of WebKB
- 70 -
CHAPTER 5 RESULTS & ANALYSIS
Comparison of Different Linguistically Motivated Knowledge Sources on WebKB (Micro
F1 Value)
1.2
Word
1
Phrase
Tag
M icro F1
0.8
Nouns
Verbs
0.6
Adjectives
0.4
Nouns & Words
Phrase & Words
0.2
Adjectives & Words
0
Naïve
Bayes
SVM
K-NN
AdaBoost
Adtree
Ripper
C4.5
Learning Algorithms
Figure 12: Comparison of Different Linguistically Motivated Knowledge Sources on
WebKB (Micro F1 values)
Comparison of Different Linguistically Motivated Knowledge Sources on WebKB (Macro
F1 Value)
1.2
Word
1
Phrase
Tag
M acro F1
0.8
Nouns
Verbs
0.6
Adjectives
0.4
Nouns & Words
Phrase & Words
0.2
Adjectives & Words
0
Naïve
Bayes
SVM
K-NN
AdaBoost
Adtree
Ripper
C4.5
Learning Algorithms
Figure 13: Comparison of Different Linguistically Motivated Knowledge Sources on
WebKB (Macro F1 values)
- 71 -
CHAPTER 5 RESULTS & ANALYSIS
5.4 Summary of Results
The conclusions derived from the first part of our study suggested that document
classification techniques appear to work well with noun based linguistic source for
Reuters-21578. On the other hand, the later part of our study on WebKB, indicated that
linguistically motivated classification using combined representations were as
competitive as the baseline classifier using words.
Previous research has highlighted the interaction effects of data corpus with the feature
selection methods used (Yang, 1999). It appears that our results appear to suggest that the
use of linguistically motivated knowledge sources depends on the characteristics of the
data corpus used. Reuters-21578 is obtained from a financial domain and comprises of
financial news that contains a substantial number of terms. On the other hand, WebKB
was from a web based domain, comprising mainly of HTML pages. This suggests that
supervised document classification using nouns may be suitable for data corpus with a
large number of features. On the other hand, documents which have fewer features such
as HTML documents, could work well with a combination of words and linguistically
motivated knowledge sources.
Thus, based on our findings, we see a potential in employing the use of linguistically
motivated knowledge sources especially nouns and combinations of these sources with
words to improve classification accuracy.
- 72 -
CHAPTER 6 CONCLUSION
CHAPTER 6
CONCLUSION
In this final chapter, we conclude with the contribution of this project and some final
thoughts on this project.
6.1
Summary
This research builds on previous empirical studies on document classification. The use of
several linguistically motivated knowledge sources, including the use of novel
combinations of these sources as features, has been explored in this study. This study
extends previous research on the use of natural language processing in document
classification and covers several learning algorithms with different linguistically
motivated knowledge sources not explored in previous studies.
Linguistically motivated knowledge sources such as nouns have been found to have
significant interaction effects on the Reuters-21578 corpus. On the other hand, the use of
novel linguistically motivated knowledge sources using combinations of part of speech
with bag of words have also shown improvements in the performance measures on the
WebKB data corpus. This suggests that the use of linguistically motivated classifiers can
help to improve the performance of classification on these two corpora.
- 73 -
CHAPTER 6 CONCLUSION
6.2
Contributions
There are several aspects that this thesis contributes. Firstly, a thorough literature review
has been conducted on several streams of research, ranging from text classification,
human cognition, knowledge management and personalization. An integrative technique
that draws ideas from these fields has been proposed.
Previous works focused mainly on the improvement of techniques using the bag-ofwords technique but very few works examined the effects of the use of linguistically
motivated knowledge sources as feature representations and document classification
efficiency across several learning algorithms in a single experiment. In our experiment,
we have experimented with several combinations of linguistically motivated knowledge
sources with various learning algorithms not covered in previous studies.
This research adds a relatively new dimension to the current works of document
classification by shedding light on the use of natural language processing techniques to
employ linguistically motivated knowledge sources as features for document
classification. The potential of using such features for various popular learning
algorithms to improve document classification was also investigated. Even though these
research studies provide answers and interpretations of their results, several questions
were left open, such as does the different feature representations with different learning
algorithms play a role in the results of the document classification which we attempt to
answer in our research. In addition, we have also proposed novel combinations of
- 74 -
CHAPTER 6 CONCLUSION
linguistically motivated knowledge sources with words whose performance appeared to
be competitive.
With the increasing pressure from the exponential growth of information available,
research on document classification cannot be emphasized greater. The results of our
experiments can contribute to the studies which employ learning algorithms to categorize
information so as to facilitate people in finding and locate information. This will help to
resolve the information overload problem.
This thesis has examined and evaluated a series of classifiers, each of which is
incorporated
with
different
linguistically
motivated
knowledge
sources
and
representations. We find that certain linguistic knowledge sources such as nouns and
noun phrase combined with words can offer substantial improvements over classifiers
used with the bag-of-words technique.
In contrast with earlier works, this thesis presents a comparative empirical evaluation of
both learning algorithms with different linguistic knowledge sources. We evaluated 6
learning algorithms (Naïve Bayes, Support Vector Machine, k-Nearest Neighbours,
decision trees, RIPPER, ADTree), and 6 linguistic knowledge sources (word, phrase,
combined word and phrase, word sense, nouns, verbs and adjectives) and combinations
of some of these sources with words. We systematically evaluated the effectiveness of
each learning algorithm using these knowledge sources on the Reuters-21578 and
WebKb data corpora.
- 75 -
CHAPTER 6 CONCLUSION
It was found that learning algorithms can improve classification accuracies with an
appropriate use of knowledge sources. We have shown that the effectiveness of slightly
different linguistic knowledge sources can vary substantially with different classification
algorithm. From our experiments, we found that using support vector machine with nouns
as knowledge sources had the best accuracies for Reuters 21578. On the other hand, the
use of combinations of linguistically motivated knowledge sources with bag of words for
web based classification using learning algorithms such as AdTree, Ripper and C4.5
works just as well or even better than bag of words.
6.3
Limitations
This study employed several conventional yet popular learning algorithms. However,
with ongoing research in machine learning techniques, the use of six learning algorithms
in our study is a subset of the many learning algorithms available. Thus, it seems
inadequate to stop the empirical study at this point with the current number of learning
algorithms.
6.4
Conclusion
Document classification systems are an integral to the success of knowledge management
systems. There is increasing interest in applying supervised document classification
techniques in knowledge management systems. The findings of this study help to
enhance the understanding of the use of incorporating linguistically motivated knowledge
- 76 -
CHAPTER 6 CONCLUSION
sources into document classification. It is hoped that this research will prove to be
valuable as an extension to similar studies.
- 77 -
REFERENCES
REFERENCES
Arampatzis, A., van der Weide, Th.P., Koster, C.H.A., van Bommel, P. (2000). An
Evaluation of Linguistically-motivated Indexing Schemes. Proceedings of BCS-IRSG
2000 Colloquium on IR Research, 5th-7th April 2000, Sidney Sussex College,
Cambridge, England.
Arampatzis, A., van der Weide, Th.P., Koster, C.H.A., van Bommel, P. (2000).
Linguistically-motivated Information Retrieval, Encyclopedia of Library and Information
Science, Volume 69, pages 201-222, Marcel Dekker, Inc., New York, Basel, 2000.
Basili, R, Moschitti, A. and Pazienza, M.T. (2001). NLP-driven IR: Evaluating
Performances over a Text Classification task. In Proceedings of the 10th Intenational
Joint Conference of Artificial Intelligence (IJCAI-2001), August 4th, Seattle, Washington,
USA.
Basili, R, Moschitti, A. and Pazienza, M.T. (2002). Empirical investigation of fast text
classification over linguistic features. In Proceedings of the 15th European Conference on
Artificial Intelligence (ECAI-2002), July 21-26, 2002, Lyon, France.
Basu, Chumki, Hirsh, Haym, Cohen, William W. (1998). Recommendation as
Classification: Using Social and Content-Based Information in Recommendation. In
Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth
Innovative Applications of Artificial Intelligence Conference, AAAI 98, IAAI 98, July
26-30, 1998, Madison, Wisconsin, USA, pp. 714-720.
Bekkerman R., El-Yaniv, R., Tishby, N. and Winter, Y. (2003). Distributional Word
Clusters vs Words for Text Categorization. Journal of Machine Learning Research, 3, pp.
1183-1208.
Bruce, Rebecca F. & Wiebe, Janyce M. (1999). Recognizing subjectivity: a case study in
manual tagging. Natural Language Engineering, 5 (2).
Cardie C., Ng V., Pierce D., and Buckly C. (2000). Examining the Role of Statistical and
Linguistic Knowledge Sources in a General-Knowledge Question-Answering System.
Proceedings of the Sixth Applied Natural Language Processing Conference (ANLP2000), 180-187, ACL/Morgan Kaufmann.
Chen H. (1992). Knowledge-based document retrieval: framework and design. Journal of
Information Science, 18(1992) 293-314. Elsevier Science Publishers, B.V.
Cohen, William W. (1996). Learning Trees and Rules with Set-Valued Features. In
Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth
Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI 96, August
4-8, 1996, Portland, Oregon - Volume 1. AAAI Press / The MIT Press, 709-716.
- 78 -
REFERENCES
Cohen, William W. (1995). Fast Effective Rule Induction. In: Machine Learning, In
Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City,
California, USA, July 9-12, 1995. Armand Prieditis, Stuart J. Russell (Eds.). Morgan
Kaufmann, 115-123.
Cohen, William W. and Hirsh, H. (1998). Joins That Generalize: Text Classification
Using WHIRL. In Proceedings of the 4th International Conference on Knowledge
Discovery and Data Mining (KDD’98).
Craven, M., DiPasquo, D., Freitag, D., McCallum, A., Mitchell, T., Nigam, K., and
Slattery, S. (1999). Learning to Construct Knowledge Bases from the World Wide Web,
Artificial Intelligence, Elsevier.
Cullingford, Richard E. (1986). Natural Language Processing: A KnowledgeEngineering Approach. Rowman & Littlefield. New Jersey.
Dumais, S.T. (1994). Latent Semantic Indexing (LSI) and TREC-2. In: D. Harman (Ed.),
The Second Text Retrieval Conference (TREC2), National Institute of Standards and
Technology Special Publication, pp. 105-116.
Dumais, S. T., Platt, J., Heckerman D., and Sahami, M. (1998). Inductive learning
algorithms and representations for text categorization. In Proceedings of ACM-CIKM98,
Nov. 1998, pp. 148-155.
Dumais, S., Cutrell, E. and Chen, H. (2001). Optimizing Search by Showing Results in
Context. Proceedings of the SIG-CHI on Human Factors in Computing, March 31- April
4, 2001, Seattle, WA, USA, ACM, 2001.
Dumais, Susan T. and Chen, H. (2000). Hierarchical Classification of Web Content. In
Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval, July 24-28, 2000, Athens, Greece, pp. 256-263.
Fagan Joel L. (1987). Experiments in Automatic Phrase Indexing for Document
Retrieval: A Comparison of Syntactic and Non-Syntactic Methods. PhD thesis, Cornell
University, 1987.
Farhoomand, A. F. and Drury, D. H. (2002). Managerial Information Overload.
Communications of the ACM, 45, 127-131.
Freund Yoav, Schapire Robert E. (1996). Experiments with a New Boosting Algorithm.
In Proceedings of the Thirteenth International Conference (ICML '96), July 3-6, Bari,
Italy, pp.148-156.
Freund Yoav, Mason Llew. (1999). The Alternating Decision Tree Learning Algorithm.
In Proceedings of the Sixteenth International Conference on Machine Learning (ICML
1999), June 27 - 30, Bled, Slovenia, pp. 124-133.
- 79 -
REFERENCES
Fuhr, N., and Buckley, C. (1991). A Probabilistic Learning Approach for Document
Indexing, ACM Transactions on Information Systems, 9(3), pp. 223-248.
Furnkranz, J., Mitchell, T., Riloff, E. (1998). A Case Study in Using Linguistic Phrases
for Text Categorization on the WWW, In Proceedings of the 1st AAAI Workshop on
Learning for Text Categorization, pp. 5-12, Madison, US.
Furnkranz, J., and Widmer, G. (1994). Incremental Reduced Error Pruning. In
Proceedings of the 11th International Conference on Machine Learning (ML-94), pp. 7077, New Brunswick, JN, Morgan Kaufmann.
Gentner, Dedre (1981). Some interesting differences between nouns and verbs. Cognition
and Brain Theory 4. 161-178.
Joachims, T. (1997). A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for
Text Categorization. In Proceedings of the 14th International Conference on Machine
Learning (ICML' 97).
Joachims, T. (1998). Text Categorization with Support Vector Machines: Learning with
Many Relevant Features. Proceedings of Machine Learning: ECML-98, 10th European
Conference on Machine Learning, April 21-23, Chemnitz, Germany, Springer, pp. 137142.
Joachims, T. (2002). Learning to Classify Text Using Support Vector Machines. Kluwer
Academic Publishers, Boston Hardbound.
Kankanhalli A, Tan B C Y and Wei K K. (2001). Seeking Knowledge in Electronic
Knowledge Repositories: An Exploratory Study, Proceedings of the Twenty-Second
International Conference on Information Systems, New York, Association for Computing
Machinery, pp. 123-133.
Kongovi Madhusudhan, Guzman Juan Carlow, and Dasigi Venu. (2002). Text
Categorization: An Experiment Using Phrases. In Advances in Information Retrieval,
Proceedings of the 24th BCS-IRSG European Colloquium on IR Research Glasgow, UK,
March 25-27, 2002. ECIR, 2002, Springer Verlag Berlin Heidelberg, Lecture Notes in
Computer Science 2291, Springer, pp. 213-228.
Moulinier, I., Raskinis, G., and Ganascia, J.G. (1996). Text Categorization: A Symbolic
Approach. In Proceedings of the 5th Annual Symposium on Document Analysis and
Information Retrieval (SDAIR'96), pp. 87-99.
Lang K. (1995). NewsWeeder: Learning to filter netnews, Machine Learning,
Proceedings of the Twelfth International Conference on Machine Learning, July 9-12,
Tahoe City, California, USA, Morgan Kaufmann, pp.331-339.
- 80 -
REFERENCES
Larkey, L.S. and Croft, W.B. (1996). Combining Classifiers in Text Categorization. In
Proceedings of the 19th Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR'98), pp. 179-187.
Lewis, D. (1992). Representation and Learning in Information Retrieval. PhD Thesis ,
University of Massachusetts, Amherst.
Lewis, D. D. and Ringuette, M. (1994). A Comparison of Two Learning Algorithms for
Text Categorization. Third Annual Symposuim on Document Analysis and Information
Retrieval. pp 81-93.
Luigi, G., Fabrizio, S. and Maria, S. (2000). Feature Selection and Negative Evidence in
Automated Text Categorization. Proceedings of the ACM KDD-00 Workshop on Text
Mining, Boston, US.
Masand Brij M., Linoff Gordon, Waltz David L. (1992). Classifying News Stories using
Memory Based Reasoning. In Proceedings of the 15th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval, Copenhagen,
Denmark, June 21-24, 1992. ACM, pp. 59-65.
McCallum, A., and Nigam, K. (1998). A Comparison of Event Models for Naïve Bayes
Text Classification. AAAI-98, Workshop on Learning for Text Categorization, Technical
Report, WS-98-05, AAAI Press.
Mitchell, T.M. (1997). Machine Learning. McGraw Hill, New York, NY.
Medin, D. L., Lynch, E. B. and Solomon, K. O. (2000). Are There Kinds of Concepts?
Annual Review Psychology, 51, 121-147.
Mitra M., Buckley C., Singhal A., and Cardie C. (1997). An analysis of statistical and
syntactic phrases. Proceedings of RIAO '97, Montreal, Canada, June 25-27, 1997.
Mladenic, D. (1998). Feature Subset Selection in Text Learning. In Proceedings of the
10th European Conference on Machine Learning (ECML'98), pp. 95-100.
Ng, H. T. and Lee, H. B. (1996). Integrating Multiple Knowledge Sources to
Disambiguate Word Sense: An Exemplar-Based Approach. Proceedings of the 34th
Annual Meeting of the Association for Computational Linguistics, June 24-27, 1996,
University of California, Santa Cruz, California, USA, Proceedings. Morgan Kaufmann
Publishers / ACL 1996, pp. 40-47.
Nigam, K, McCallum, A., Thrun, S. and Mitchell, T. (1998). Learning to Classify Text
from Labeled and Unlabeled Documents. In Proceedings of the Fifteenth National
Conference on Artificial Intelligence (AAAI-98), pp. 792-799.
- 81 -
REFERENCES
Pedro, Domingos, Michael, J. Pazzani. (1996). Simple Bayesian Classifiers Do Not
Assume Independence. In Proceedings of the Thirteenth National Conference on
Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence
Conference, AAAI 96, IAAI 96, August 4-8, 1996, Portland, Oregon - Volume 2. AAAI
Press / The MIT Press, 1996.
Platt, J. (1999). Using Sparseness and Analytic QP to Speed Training of Support Vector
Machines, Advances in Neural Information Processing Systems, 11, M. S. Kearns, S. A.
Solla, D. A. Cohn, eds., MIT Press.
Platt, J. (1998). Sequential Minimal Optimization: A Fast Algorithm for Training Support
Vector Machines, Microsoft Research Technical Report MSR-TR-98-14.
Quinlan J. Ross. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.
Reuters-21578 data set [online] http://www.research.att.com/~lewis/reuters21578.html
Richardson R. and Smeaton A.F. (1995). Using WordNet in a knowledge-based approach
to infornation retrieval. [online]
Rogati , M. and Yang, Y. (2002). High-performing feature selection for text
classification, In Proceedings of the 10th Conference for Information and Knowledge
Management (CIKM-2002).
Roussinov, D.G., Chen, H.C. (1999). Document clustering for electronic meetings: an
experimental comparison of two techniques, Decision Support Systems, 27 (1-2): 67-79.
Sahami, M. (1998). Using Machine Learning to Improve Information Access. PhD
Thesis, Stanford University, Computer Science Department. STAN-CS-TR-98-1615.
Schapire, R.E., Singer, Y. and Singhal, A. (1998). Boosting and Rocchio Applied to Text
Filtering. In Proceedings of the 21st Annual International Conference on Research and
Development in Information Retrieval (SIGIR'98), pp. 215-223.
Schutze, H. and Silverstein H. (1997). Projections for efficient document clustering. In
Proceedings of the 20th Annual International Conference on Research and Development
in Information Retrieval (SIGIR'97), pp. 74–81, Philadelphia, PA, July 1997.
Scott S. and Matwin S. (1998). Text Classification Using WordNet Hypernyms. In
Proceedings of Coling-ACL'98 Workshop on the Usage of WordNet in Natural Language
Processing Systems, Montreal, Canada.
Scott S. and Matwin S. (1999). Feature Engineering for Text Classification. In
Proceedings of the Sixteenth International Conference on Machine Learning (ICML'99),
Bled, Slovenia, June 27-30, Morgan Kaufmann.
- 82 -
REFERENCES
Sebastiani Fabrizio, (2002). Machine Learning in Automated Text Categorization. ACM
Computing Surveys, Vol 34, No. 1, March 2002, pp 1-47.
Siegel E.V. and McKeown K.R. (2001). Learning Methods to Combine Linguistic
Indicators: Improving Aspectual Classification and Revealing Linguistic Insights.
Association for Computational Linguistics.
SNOW URL:http://l2r.cs.uiuc.edu/~cogcomp/cc-software.html
Smeaton, A. (1999). Using NLP or NLP resources for Information Retrieval Tasks. In
Natural Language Information Retrieval, Kluwer, Boston, MA.
Strzalkowski, T., Carballo, J.P., Karlgren, J., Hulth, A., Tapanainen, P., Lahtinen, T.
(1999). Natural Language Information Retrieval: TREC-8 Report. In Proceedings of the
Eighth Text REtreival Conference (TREC-8), Gaithersburg, Maryland, November 17-19,
NIST.
Tolle K.M. and Chen H. (2000). Comparing Noun Phrasing Techniques for Use with
Medical Digital Library Tools. Journal of the American Society for Information Science,
51(4):352-370, 2000.
Vapnik, V. (1995). The Nature of Statistical Learning Theory, Springer-Verlag.
WEKA URL: http://www.cs.waikato.ac.nz/ml/weka/
Yang, Y. and Pedersen, J. O. (1997). A Comparative Study on Feature Selection in Text
Categorization, In Proceedings of the Fourteenth International Conference on Machine
Learning (ICML 1997), Nashville, Tennessee, USA, July 8-12, 1997. Morgan Kaufmann
1997, pp. 412-420.
Yang, Y. and Liu, X. (1999). A Re-examination of Text Categorization Methods. In
Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR'99), August 15-19, 1999, Berkeley, USA.
ACM, pp. 42-49.
Yang, Y. (2001). A Study on Thresholding Strategies for Text Categorization. In
Proceedings of the 24th Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR,01), pp. 137-145.
Zelikovitz S., and Hirsh, H. (2000). Improving Short Text Classification Using Unlabeled
Background Knowledge to Assess Document Similarity. In Proceedings of the 17th
International Conference on Machine Learning, pp. 1183-1190.
Zelikovitz S., and Hirsh, H. (2001). Using LSI for Text Classification in the Presence of
Background Text. In Proceedings of the 10th Conference for Information and Knowledge
Management, 2001.
- 83 -
[...]... a systematic study that covers an extensive variety of linguistically motivated knowledge sources - 21 - CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION Despite the existence of extensive research on document classification, the relationship between different linguistic knowledge sources and classification model has not been sufficiently or systematically... broken up into two separate words, “learning” and “algorithm” Thus, we utilize linguistically motivated knowledge sources as features, to see if we can resolve these limitations associated with the bag-of-words paradigm Novel combinations of - 24 - CHAPTER 3 LINGUISTICALLY MOTIVATED CLASSIFICATION linguistically motivated knowledge sources are also proposed and presented in the next section 3.1.1 Linguistically. .. Processing (NLP) in Document Classification Most information retrieval (IR) systems are not linguistically motivated Similarly, in document classification research, most experiments are not linguistically motivated (Cullingford, 1986) Closely related to the research on document classification is the research on natural language processing and cognitive science Traditionally, document classification techniques... supervised classification studies This could be due to the results of early attempts (Lewis, 1992) which showed negative results With the advent of NLP techniques, there seems a propelling reason to examine the use of linguistically motivated knowledge sources Although there has been separate attempts made to study the effects of linguistically motivated knowledge sources on supervised document classification. .. employed to derive linguistically motivated knowledge sources 2.1 Document Classification Document classification, the focus of this work, refers to the task of assigning a document to categories based on its underlying content Although this task has been carried out effectively using knowledge engineering approaches in the 80s, machine learning approaches have superceded the use of knowledge engineering... Since previous works documenting results based on linguistically motivated features with learning algorithms produced inconsistent and sometimes conflicting results, we propose to conduct a systematic study on multiple learning algorithms and linguistically motivated knowledge sources as features Some of these features are novel combinations of linguistically motivated knowledge sources were not explored... explored By bringing together the two streams of research of document classification and natural language processing, we hope to shed light on the effects of linguistically motivated knowledge sources with different learning algorithms Section 3.1 discusses the shortcomings of previous research Section 3.2 explores the linguistically motivated knowledge sources employed to resolve these issues Finally section... effectiveness of linguistically motivated features The aim of this thesis is also to provide a solid foundation for research on feature representations on text classification and study the factors affecting machine learning algorithms used in document classification One of these factors we want to look into is the effect of feature engineering by utilizing linguistically motivated knowledge sources as features... feature representations using different linguistic knowledge sources as the input vectors to the learning algorithm have significant impact on document classification We consider the following linguistics knowledge sources in our research: 1 Word, this will be used as the baseline to do a comparative analysis with other linguistically motivated knowledge sources; 2 Phrase; 3 Word sense or part of speech... linguistic cues into document classification through NLP techniques This can be done by utilizing NLP techniques to extract the different representation of the documents and then used in the classification process As defined by Medin (2000), other concepts identified are verb, count nouns, mass nouns, isolated and interrelated concepts We define such concepts as linguistically motivated knowledge sources They ... CHAPTER LINGUISTICALLY MOTIVATED CLASSIFICATION linguistically motivated knowledge sources are also proposed and presented in the next section 3.1.1 Linguistically Motivated Knowledge Sources. .. covers an extensive variety of linguistically motivated knowledge sources - 21 - CHAPTER LINGUISTICALLY MOTIVATED CLASSIFICATION CHAPTER LINGUISTICALLY MOTIVATED CLASSIFICATION Despite the existence... use of linguistically motivated knowledge sources Although there has been separate attempts made to study the effects of linguistically motivated knowledge sources on supervised document classification
Ngày đăng: 09/10/2015, 11:24
Xem thêm: Incorporating linguistically motivated knowledge sources into document classification
