Experimental Analysis of Neural Network Based Feature Extractors for Cursive Handwriting Recognition Ling Gang, Brijesh Verma and Siddhi Kulkarni School of Information Technology, Griffith University-Gold Coast Campus PMB 50, GCMC, Qld 9726, Australia E-mail: B.Verma@mailbox.gu.edu.au, S.Kulkarni@mailbox.gu.edu.au Web: http://intsun.int.gu.edu.au ABSTRACT Artificial neural networks have been widely used in many real world applications including classification of cursive handwritten segmented characters. However, the feature extraction ability of MLP based neural networks has not been investigated properly. In this paper, a new MLP based approach such as an auto-associator for feature extraction from segmented handwritten characters is proposed. The performance of Auto-Associator (AA), Multilayer Perceptron (MLP) and Multi-MLP as a feature extractor have been investigated and compared. The results and detailed analysis of our investigation are presented in the paper. 1. INTRODUCTION 1.1 Motivations and aims of the research There are a number of classification techniques widely used by researchers in many real world applications. However, there are a very few researchers who have tried MLP based neural networks as a feature extractor. The need for research to further improve and embellish current character recognition techniques has been widely recognized. It is also recognised that the types of feature extractors used contribute to some of the errors caused. Therefore a need can be seen to find a new feature extractor and investigate the NN-based feature extraction techniques, to show which are indeed the best and most efficient techniques to use. 1.2 Background There are only a few empirical comparative studies of NN-based feature extraction paradigms have been made. The paradigms in Mao and Jain [1] are compared only for exploratory data projection and two-dimensional classification and in Lerner et al. [2] only for one database. In the research carried out by Boaz Lerner, Hugo Guterman, Mayer Aladjem [3]; the complex architectures of more than two layers were not considered as candidates for the classifier and the number of output units was three, which is quite small. The comparative studies of different MLP-based feature extraction have not been considered yet. So it is necessary to do more over this issue, the primary aim of this research is to investigate the feature extraction ability of Auto-Associator, MLP and Multi-MLP to determine which one is more suitable and reliable to be used in real-world handwriting character recognition systems. The origins of character recognition [4-6] can be found as early as 1870. It first appeared as an aid to the visually handicapped, and the first successful attempt was made by the Russian scientist Tyurin in 1900 [7]. From then on, many papers about neural networks [8-15] and applications have been presented and widely used in pattern recognition areas, the modern version of character recognition appeared in the middle of the 1940s with the development of digital computers. Thenceforth it was realized as a data processing approach with application to the business world. The principal of motivation for the development of character recognition is the need to cope with enormous quantities of paper such as bank checks, commercial forms, government records, credit card imprints and mail sorting generated by the expanding technological society. Presently, the methodologies in character recognition have advanced from the earlier use of primitive techniques for the recognition of machine printed numerals and a limited number of English letters to the application of sophisticated techniques for the recognition of a wide variety of complex handwriting characters, symbols and word/script. 1.3 Organization of the paper This paper consists of five sections. Section 1 presents the motivations and background. Section 2 details research methodology, proposing and describing the methods that were employed in this research. Section 3 details experimental results, listing the results obtained during the experiments. Section 4 provides a discussion, an analysis of the experimental results and compares the three different techniques that have been investigated. Section 5 provides conclusions that have been drawn from this research. 2. PROPOSED RESEARCH METHODOLOGY The Figure 1 below outlines the proposed research methodology and it is described in the sections below. 0-7803-7278-6/02/$10.00 ©2002 IEEE Figure 1. Block diagram of research methodology 2.1 Character acquisition and preprocessing Before experiments could be carried out, there was a need to process the original images. The techniques employed to prepare input files for various techniques are discussed in the following sections. 2.1.1 Character database acquisition The training and test characters/words used in this research came from the following directories on CEDAR CD-ROM (Benchmark Database): TRAIN/BINANUMS/BD/* TEST/BINANUMS/BD/* TRAIN/CITIES/BD/* TEST/CITIES/BD/* All the images were black & white lower case characters and stored in PBM format. All useless white spaces around the images have been removed. 2.1.2 Character resizing Resizing was the first technique used to process the images. The resizing process partially employed an existing program written by R. Crane [16] using the C programming language, which was modified by us, and all the images were resized to 30 rows by 40 columns. 2.1.3 Chain code feature extraction For all the training and test characters, the character images were first boundaried, all the pixels of each image were changed to the background colour, except the outmost ones of the image. The images were then processed using chain code technique with 8 directions. After chain coding, each image was divided into some small sub-images whose size was 10 rows by 10 columns. The numbers of each direction in a single sub- window were added, and these numbers were recorded to be used later. After all the images were chain coded, all the numbers were divided by the biggest one among them to create the inputs whose maximum value was 1 and minimum value was 0, each character had 12 * 8 =96 inputs. 3. EXPERIMENTAL METHOD There were a total of three character feature extraction/recognition techniques investigated in this research: AA, MLP and Multi-MLP. The BP algorithm was employed as a common training algorithm. The networks used were feed-forward neural networks. There was only one hidden layer for all kinds of networks in this research. The number of neurons in the input layer of all these extractor was governed by the size of the sub- windows of the training characters. Training character matrices had 30 rows by 40 columns, and the sub- window had the size 10 rows by 10 columns. So each image had 12 sub-windows, since each sub-window contributed 8 elements, the number of units in the input layer was 96. 3.1 Auto-associator (AA) feature extractor An AA, as its name implies, is a network that learns an input-output mapping such that the output is the same as the input, in other words, the target data set is identical to the input data set. Hence, an AA has configuration of d: m: d with d units in both the input and output layers and m<d in the hidden layer. The dimensionality of the input and output is obviously the same and the network is trained, using the error back propagation algorithm, to generate output vectors o as close as possible to the input vectors x by again minimizing the mean squared error over all patterns in the training set: ∑∑ = = −= p p k k p k p k xo E 1 1 2)(2/1 where o p k represents the kth output of the nth input vector x n =( x n 1 …. x n k , … x n d ) and N is the number of training patterns. The key aspect of the auto associative MLP is that the number of hidden units at the center of the network is usually chosen to be much smaller than the input/output dimensionality. As a result of this bottleneck, the hidden units extract a low-dimensional representation of the input data and such a network can therefore be used for feature extraction. 0-7803-7278-6/02/$10.00 ©2002 IEEE To be consistent with all the neural networks being compared in this research, the training file was set to the same format, they all consisted of two parts. The first part was obtained by chain coding as described previously. For the AA, the second part was exactly the same as the first part. 3.2 MLP The values of the MLP’s learning rate and momentum were both set to 0.1 as those of the AA. The number of inputs and outputs were set to 96 and 24, respectively. The number of hidden units of MLP was set to 26. The input vector of the MLP was obtained by employing the chain code feature extractor as described previously, the second part was the output vector. This part indicates to the network which class the current character belongs to. 3.3 Multi-MLP For the Multi-MLP feature extractor, the situation was different. The possible outputs governed the number of neural networks instead of the number of output units. Each neural network had only 2 units in the output layer, in other words, each neural network was dedicated to recognize one letter. The hidden units of these networks were all set to 26. The input vector to the Multi-MLP was the same as for the AA and MLP, but the desired output of the Multi- MLP only had two classes. 3.4 Criteria for training termination When using the back propagation algorithm, the usual criterion for termination is the reduction of the Root Mean Squared (RMS) error to an acceptable level. There is no general level for the RMS error, however, the smaller, the better. It was found in experiments, the convergence of the RMS error was very slow, therefore, another criterion for termination that was considered, was to set the network to stop training after a certain number of iterations. 4. EXPERIMENTAL RESULTS The proposed approaches were all implemented and run on SP2 supercomputer and on the NIAS (UNIX) machine at Griffith University. SP2 is an IBM-based product that consists of eight RS/6000 390 machines and 14 RS/6000 590 machines connected by a high speed switch. The operation system is UNIX. The programming language used for implementation was C. This section shows the results obtained by using the three neural network based character feature extraction techniques. In this research, the training data set consists of 16 lowercase letters: a, b, c, d, e, h, i, l, m, n, o, r, s, t, u, x. After training, the three techniques were assessed by their classification rates for both training data and test data. There were many experiments conducted, all of them were very time consuming. Sometime it took more than several days. Only the most relevant results shall be shown in this and following sections. 4.1 Preliminary results Before training these networks with 16 letters, some preliminary experiments were conducted. In these experiments, only the AA and the MLP were trained and, instead of 16 letters, only four letters were chosen: a, c, d, e. All training characters were hand printed characters. There were 96 training pairs. Both the MLP and the AA were trained with 2000 iterations. The number of test characters was 181. The results are displayed in Table 1 below. There were two AAs were trained, one AA had 26 hidden units and the other AA had 96 hidden units. As can be seen from Table 1, the classification rates of the training data were 100%. The classification rates for the test data were quite high as well. All of them were between 82% and 87%. Of course, the classification would be lower than these figures if the number of classes increased, but these figures were very promising for doing more research. Subsequent experiments used more classes and larger datasets, more and more images were segmented from the handwritten words, which were very hard to recognize, so that these ANNs could be trained with a more diverse and challenging training set. TABLE 1. PRELIMINARY RESULTS (#inputs: 96) Classification Rate (%) MLP /AA No. of hidden units No. of outp uts RMS Training set Test set MLP 26 4 0.01252 100 82.3 AA 26 96 0.03259 100 86.7 AA 96 96 0.02989 100 83.9 4.2 Classification rates for the MLP After doing some preliminary experiments, the number of letters in the training and test database were increased from 4 to 16. The numbers of characters in training and test database were increased as well. The MLP based network is a very popular character recognition technique and has been widely used in many fields. So the major comparison of work was conducted between the MLP and the AA. The results in Table 1 (rows 1, 2) were obtained by training the MLP with 352 hand printed characters. The test data set presented to the MLP was 280 hand printed characters. The MLP was trained 0-7803-7278-6/02/$10.00 ©2002 IEEE with 2000 and 5000 iterations, respectively. As can be seen from Table 2, the classification rate for the training set was 99.7% for two MLPs, near 100%. Test sets of two MLPs obtained high figures as well: 78.2%. TABLE 2. CLASSIFICATION RATES USING MLP (#inputs: 96, #outputs: 24, #hidden units: 26) Classification Rate (%) Training pairs No. of iterations RMS Training set Test Set 352 2000 0.0141 99.7 78.2 352 5000 0.0118 99.7 78.2 506 5000 0.0184 94.4 52.8 656 5000 0.0604 91.5 55.1 951 5000 0.0678 94.2 60 The results in Table 2 (rows 3,4,5) were obtained by adding more and more cursive characters to the previous training data set. At the beginning, there were no cursive characters, there were only 352 hand-printed characters in the training data set. The number of cursive characters was increased from 0 to 144, 304 and 599. Accordingly, the number of training characters increased from 352 to 506, 656, 951. TABLE 3. CLASSIFICATION RATES FOR TWO MLPS Classification Rate (%) Trai ning pairs No. of Iteratio ns RMS Training set Test set Test (top5) 656 5,000 0.0604 92.3 60.2 79.8 614 5,000 0.0505 93.9 60.8 86.1 The above table (Table 3) contains the classification rates for two MLPs. The result in first row was trained by all 16 letters including l, but tested without l. The second one was trained and tested without l: 4.3 Classification rates for the AA The comparison of performance between the AA and the MLP was the major aim of this research. The results in Table 4 (rows 1,2,3,4) were obtained by training the AA with 352 characters. The test data set presented to the AA was 280 characters. To achieve better results, two AAs with a different number of hidden units were trained. One AA had 26 hidden units, so the structure of its classifier was 26 inputs, 26 hidden units and 24 output units. The second AA had 96 hidden units, accordingly, the structure of its classifier was 96 input units, 26 hidden units and 24 output units. The two kinds of AA were trained for 2000 and 5000 iterations respectively. TABLE 4. CLASSIFICATION RATES FOR AA (#inputs: 96, #outputs: 96) Classification Rate(%) No. of training chars No. of hidden units No. of Itera tions RMS Training set Test set 352 26 2000 0.0240 97.4 75 352 26 5000 0.0317 98.9 76.1 352 96 2000 0.0240 99.4 76.8 352 96 5000 0.0180 99.7 78.2 506 96 5000 0.0482 95.1 53.3 656 96 5000 0.0592 94.4 56.7 951 96 5000 0.0650 94.3 61.9 As can be seen from the above table, the AA with 96 hidden units outperforms the AA with 26 hidden units. Therefore the subsequent experiments were only conducted using an AA with 96 hidden units. The results listed in Table 4 (last 3 rows) were obtained by adding the previous training and test data sets, which contained only hand printed characters, to some cursive characters from data set B as used in the case with the MLP. The number of iterations for all experiments was 5000. The number of hidden units of AA was 96. The classifier had 26 hidden units. The training pairs were increased from 352 to 506, 656 and 951. The test set had 1,056 training pairs. As can be seen from the above table, the classification rates increased when the number of training pairs increased. Of course, since the number of training iterations for each AA was 5000, the classification rate for the training set decreased when the number of training pairs increased. As the RMS error increased, the classification rates of the training set decreased, in this case, from 95.1% to 94.3%. Compared with its MLP counterpart, some letters obtained higher classification rates. For example, letter l obtained a 17.9% increase in classification rate, whereas letter a decreased by 11.9%. No changes to the classification rate of letter x. In total 8 letters of the AA obtained higher classification rates than the MLP, 7 letters obtained lower classification rates, they were letters a, b, d, e, h, i, m, respectively. 4.4 Classification rates for the Multi-MLP Multi-MLP was the one that had 16 neural networks each of which had only two classes. The training files were different to each of these networks. Each network was 0-7803-7278-6/02/$10.00 ©2002 IEEE designed to respond to a particular letter. Each network had only two classes. The first one was trained to respond to a given letter. The other one was trained to respond to the rest of the letters. Since it was very time consuming, only one experiment was conducted so far: the number of training pairs for each network was 951, the same as for the MLP and the AA. Some of its testing results are listed in Table 5. TABLE 5. CLASSIFICATION RATES OF MULTI-MLP (#inputs: 96, #outputs: 2, #hidden units: 26) Classification Rate (%) Multi -MLP RMS Training set Test set Test (top5) A 0.072687 93.5 56.3 81.0 B 0.065042 92.9 50.0 75.0 C 0.056367 96.7 65.4 94.3 D 0.046068 96.7 81.3 100 E 0.072770 90.3 57.6 79.7 H 0.117057 80.6 52.8 88.9 I 0.08604 90.3 57.9 86.3 L 0.12983 74.2 31.9 76.8 M 0.032646 98.4 72.7 95.5 N 0.102719 83.9 46.3 73.1 O 0.010263 83.9 67.0 91.3 R 0.072746 93.5 56.0 93.3 S 0.079608 90.3 56.9 81.5 T 0.046014 96.8 75.7 98.6 U 0.056317 95.2 64.7 100 X 0.056279 85.2 66.7 83.3 These results were obtained by training the multi-MLP with the same training characters as those used to train MLP and AA and test characters were the same as those of MLP and AA as well. But the number of classes of each network was only 2 rather than 24. The number of iterations for all networks was 5000. As can be seen, the RMS for different MLP were varied dramatically, and the classification rates of these MLP were also different, the highest one was 81.3%, the lowest one was 31.9%. 5. DISCUSSION 5.1 Classification rate As can be seen from the tables in previous section, among three feature extraction techniques, the AA provided the top results for character recognition: 61.9% for the test set and 94.3% for the training set, The second best one was the MLP, followed by Multi-MLP, whose classification rate for the training set was 94.1%, and the test set was 57.1%. From the experiments we can observe that increasing the number of training pairs for the AA or the MLP is a good way to increase the classification rates, but the characters added were all cursive characters. Does that increase the classification rates of the hand printed character section which has 280 out of a total of 1056 characters in test dataset? The classification rates for only the 280 hand printed characters rather than the whole test set were calculated. From above, we can observe that when the number of cursive characters in the training dataset of the MLP increased from 0 to 599, the classification rates for printed characters only increased by 0.7%. Meanwhile, the classification rates for cursive characters dramatically increased from 30.4% to 53.2%. As can be seen, when the number of cursive characters in the training dataset of the AA increased from 0 to 599, the classification rates for printed characters only increased by 1.9%. Even so, it is better than the MLP ‘s 0.7% increase. Meanwhile, the classification rates for cursive characters dramatically increased from 26.2% to 55.9%, the percentage increase was 29.7%, while the MLP counterpart was 22.8%. 5.2 General problems with classification rates It was found that when the number of training pairs was increased to 951 and the number of iterations was set to 5,000, the best classification rate for handwritten characters using the three feature extraction techniques was less than 62%. Increasing the number of iterations did little to improve the classification rates. It was deduced that three main factors influenced the classification rates obtained by these techniques: Small training dataset, Difficulty of training and testing data, Resizing problems, Similarity of characters. 5.2.1 Small training data set As can be seen from the previous experiments, the classification rate increased when the number of training pairs increased. For example, when the number of training pairs increased by 150 from 506 to 656. The classification rate increased by 3.4 percent. When 295 more training pairs were added to obtain a total of 951, the classification rate increased by 5.2 percent accordingly. A 60 percent classification rate is quite high considering that a maximum of 951 training pairs was used. Due to time constraints, more training pairs were not employed. 5.2.2 Difficulty of training and testing data The other factor that influenced the recognition rate was the nature of the handwritten data. As the characters sampled were real world characters, it was found that the actual writing styles of two different people could be extremely diverse. Misclassification could easily occur by both human and automated systems. The diversity of characters was not only shown between people, but also between samples of the same person. This increased the difficulty of training and test data dramatically. The following characters (Figure 2) are some samples for lowercase b: 0-7803-7278-6/02/$10.00 ©2002 IEEE Figure 2. Examples of lowercase b The other important reason was that some training and test samples were segmented from handwritten words, sometimes they became very hard to recognize (Figure 3) even for a human. (a) (b) (c) Figure 3. Examples of segmented characters. (a) Lowercase r, (b) Lowercase s, (c) Lowercase x. 5.2.3 Resizing problems Because the sizes of training and test images were different from each other, all the characters were resized to the same size before being chain coded in order to attain more comparable features. However, one of the major disadvantages of resizing is that it can cause some of the character’s characteristics to be lost, and that may be critical for extracting features. 5.2.4 Similarity of characters After analyzing the tables describing target and actual outputs of each class, we found that some classes had very high classification rates, whereas some others got as low as 40%. After further analysis, it was found that some letters were easily recognized as some other particular letters. For example, the letter l was easily recognized as letter e or letter i. It has been found that about 20 percent of some letters were recognized as the other letters, such as 19.6 percent of l’s were recognized as e’s. 6. CONCLUSIONS We have investigated neural network based 3 feature extraction techniques. The Auto-associator feature extractor proposed by us, achieved the highest recognition rates. The highest rate for difficult handwritten characters from CEDAR benchmark database were approximately 61.9%. The classification rate was quite high considering it only used 951 training pairs and 5,000 iterations. The classification rates of the MLP were lower than those of the AA. The best classification result for handwritten characters was 60.0% (1.9% less than the AA). The recognition rates and overall performance of the Multi-MLP were the lowest of all three techniques tested. The highest classification rate it provided was 57.1% (4.8% lower than the AA and 2.9% lower than the MLP), and this method took the longest training time. References [1] J. Mao, A. K. Jain, Artificial neural networks for feature extraction and multivariate data projection. IEEE Trans. Neural Networks 6, 296 –317, 1995. [2] B. Lerner, Toward a completely automatic neural network based human chromosome analysis. IEEE Trans, Syst. Man Cyber. 28, Part B, Special issue on artificial neural networks, 544-552, 1998. [3] B. Lerner, H. Guterman, Mayer Aladjem, A comparative study of neural network based feature extraction paradigms. Pattern Recognition. Vol 20, 1999. [4] M. E. Stevens, Introduction to the special issue on optical character recognition (OCR), Pattern Recognition 2, 147 – 150, 1970. [5] J. Rabinow, Whither OCR and whence Datamation. 38 – 42, July 1969. [6] P. L. Andersson, Optical character recognition – a survey. Datamation, 43 – 48, July 1969. [7] J. Mantas, An overview of character recognition methodologies, Pattern Recognition 19, 425 –430 1986. [8] R. Davis and J. L. Yall, Recognition of handwritten characters – a review, Image vision Comput. 4, 208 –218. [9] Y. Cheng and C. H. Leung, Chain – code transform for chinese character recognition, IEEE 1985, Proc. Int. Conf. Cyb. Soc. Tucson, AZ, U.S.A. pp. 42 – 45, 1985. [10] H. I. Avi-Itzhak, T. A. Diep, and H. Garland, “High accuracy optical character recognition using neural networks with centroid dithering”, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 17, pp. 218-224, 1995. [11] S-W. Lee, “Off-Line Recognition of totally unconstrained handwritten numerals using multilayer cluster neural network”, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 18, pp. 648-652, 1996. [12] S-B. Cho, “Neural-network classifiers for recognizing totally unconstrained handwritten numerals”, IEEE Trans. on Neural Networks, Vol. 8, pp. 43-53, 1997. [13] N.W. Strathy, C.Y. Suen, and A. Krzyzak, “Segmentation of handwritten digits using contour features”, ICDAR ‘93, pp. 577- 580, 1993. [14] B. A. Yanikoglu, and P. A. Sandon, “Off-line cursive handwriting recognition using style parameters”, Tech. Report PCS-TR93-192, Dartmouth College, NH., 1993. [15] J-H. Chiang, “A hybrid neural model in handwritten word recognition”, Neural Networks, Vol. 11, pp. 337-346, 1998. [16] R. Crane, A simplified approach to image processing: classical and modern techniques in C. Prentice Hall. 1996. 0-7803-7278-6/02/$10.00 ©2002 IEEE . Experimental Analysis of Neural Network Based Feature Extractors for Cursive Handwriting Recognition Ling Gang, Brijesh Verma and Siddhi Kulkarni School of Information Technology,. algorithm. The networks used were feed-forward neural networks. There was only one hidden layer for all kinds of networks in this research. The number of neurons in the input layer of all these. Multi-MLP For the Multi-MLP feature extractor, the situation was different. The possible outputs governed the number of neural networks instead of the number of output units. Each neural network