Extracting Information from Text and Images for Location Proteomics

Extracting Information from Text and Images for Location Proteomics Zhenzhen Kou Center for Automated  Learning & Discovery Carnegie Mellon University Pittsburgh, PA woomy@cs.cmu.edu William W. Cohen Center for Automated  Learning & Discovery Carnegie Mellon University Pittsburgh, PA wcohen@cs.cmu.edu ABSTRACT There is extensive interest in automating the collection, organization and summarization of biological data Data in the form of figures and accompanying captions in literature present special challenges for such efforts Building on our previously developed tools for analyzing fluorescence microscope images depicting protein subcellular location information, we have implemented a system called SLIF, which combines image analysis methods with text mining methods to extract information about protein subcellular localization from the text and images found in online journals Our current system can generate assertions such as “Figure N depicts a localization of type L for protein P in cell type C” Keywords Information extraction, Bioinformatics, text mining, image mining INTRODUCTION The vast size of the biological literature and the knowledge contained therein makes it essential to organize and summarize pertinent scientific results This leads to the creation of curated databases, like the Entrez databases, SwissProt, and YPD However, curated databases are expensive to create and maintain Moreover, they not typically permit extensive links to specific supporting data, not estimate confidence of assertions, not allow for divergence of opinion, and not readily permit updating or reinterpretation of previously entered information Information extraction (IE) methods can be used to at least partially overcome these limitations by creating selfpopulating knowledge bases that can automatically extract and store assertions from biomedical text [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] However, most existing IE systems are limited to extracting information only from text, not from image data In this paper we describe techniques for extracting information about protein subcellular locations from both text and images These techniques build on previous work [12] in using image processing methods to analyze fluorescence microscope images and extract a quantitative description of the localization patterns of the tagged proteins This work was later extended to process images harvested from on-line publications [13] Here we will describe a further extension to this system, which extracts detailed textual annotations of the images (and associated proteins) by analyzing the Robert F. Murphy Center for Automated  Learning & Discovery and Dept. of Biological Sciences Carnegie Mellon University Pittsburgh, PA murphy@cmu.edu accompanying captions The system is called SLIF (for Subcellular Location Image Finder), and our long-term goal is to develop a large library of annotated and analyzed fluorescence microscope images, in order to support datamining More generally, there are many reasons for wishing to investigate extraction from the text and images in figures Figures occupy large amounts of valuable page space, and are likely to be seen disproportionately by casual readers Thus figure and caption pairs often concisely summarize a paper’s most important results as perceived by the author In the following sections, we will first describe briefly how SLIF works We will then describe in detail a recent extensions to the system, specifically, our approach to associating information derived from analyzing caption text with information derived by analyzing image data THE SLIF SYSTEM 2.1 Overview SLIF applies both image analysis and text interpretation to the figure and caption pairs harvested from on-line journals, so as to extract assertions such as “Figure N depicts a localization of type L for protein P in cell type C” The protein localization pattern L is obtained by analyzing the figure, and the protein name and cell type are obtained by analysis of the caption Figure illustrates some of the key technical issues The figure encloses a prototypical figure harvested from a biomedical publication, and the associated caption text Note that the text “Fig Kinase…experiments” is the associated caption from the journal article, and that the figure contains several panels (independently meaningful subfigures) Since most figures in biomedical publications contain several panels, associating caption information with individual panels is a non-trivial problem SLIF performs several distinct tasks The first task is to extract image/caption pairs from each figure in some set of on-line journal articles The next task is figure analysis: to identify panels that contain fluorescence microscope images, and compute numerical features that adequately capture information about subcellular location The third task is caption analysis: to extract protein names and cell types from This figure is reproduced from the article “Ras Regulates the Polarity of the Yeast Actin Cytoskeleton through the Stress Response Pathway”, by Jackson Ho and Anthony Bretscher, Molecular Biology of the Cell Vol 12, pp 1541– 1555, June 2001 captions The fourth step is mapping the information extracted from the caption to the right panels of the figure Fig Kinase inactive Plk inhibits Golgi fragmentation by mitotic cytosol (A) NRK cells were grown on coverslips and treated with 2mMthymidine for to 14 h Cells were subsequently permeabilized with digitonin, washed with 1M KCl-containing buffer, and incubated with either mgyml interphase cytosol (IE), 7mgyml mitotic extract (ME), or mitotic extract to which 20 mgyml kinase inactive Plk (ME + Plk-KD) was added After a 60-min incubation at 32C, cells were fixed and stained with anti-mannosidase II antibody to visualize the Golgi apparatus by fluorescence microscopy (B) Percentage of cells with fragmented Golgi after incubation with mitotic extract (ME) in the absence or the presence of kinase inactive Plk (ME + Plk-KD) The histogram represents the average of four independent experiments Figure 1: A figure caption pair reproduced from the biomedical literature The original SLIF system used a web robot to automatically retrieve PDF versions of online journal articles from PubMed Central that matched a particular query Figures and accompanying captions were extracted and paired together using a modified version of PDF2HTML, a public domain tool The figure-extraction step achieved a precision (number of correct figure-caption pairs returned divided by the number of figure-caption pairs returned) of 98% and a recall (number of correct pairs returned divided by the number of actual pairs) of 77% The new version of SLIF includes web robots to extract papers from sources such as BioMedCentral, and we also have obtained an extensive collection of articles directly from the publisher These sources are in XML format, so figure/caption pairs can be extracted without errors 2.2 Figure processing 2.2.1 Decomposing figures into panels For figures containing multiple panels, the individual panels must be recovered from the figure In the current system, figures are decomposed into panels by recursively subdividing the figure by looking for horizontal and vertical white-space partitions The current system achieves a precision of 73% and a recall of more than 60% on these steps 2.2.2 Identifying fluorescence microscope images Once panels have been identified, it is necessary to determine what sort of image the panel contains, so that appropriate image processing steps can be performed In the current system, panels are classified as to whether they are fluorescence microscope images using the grey-scale histogram as features The k-nearest neighbor classifier used for this task achieves a precision of 97% and a recall of 92% 2.2.3 Image preprocessing To compute subcellular location features (SLFs) , the analysis techniques we have developed require images containing a single cell with a known resolution To apply these techniques to images from on-line articles, some preprocessing is required Annotation detection and removal Many microscope images (micrograph) contain annotations such as labels, arrows and indicators of scale, within the image itself These must be detected, analyzed, and then removed from the image Annotation detection relies on finding areas that are bright and have sharp edges Annotation removal consists of filling the annotation area with background pixel values On a test set of 100 fluorescence microscope panels, this step achieves precision of 83% and recall of 82% Multi-cell image segmentation Many (if not most) published fluorescence microscope images contain more than one cell and our methods for classifying subcellular location patterns require images of a single cell Each micrograph must be segmented into individual cells using the “seeded watershed” algorithm The seeded watershed segmentation works well for some location classes (e.g., tubulin, with 52% precision and 41% recall) but is not expected to work well for others (e.g., Golgi, with 62% precision but only 32% recall) Improving this step is a subject of current research Determining the scale of each micrograph Automated analysis of fluorescence microscope images requires knowing the scale of an image since some of our previously developed subcellular location features (SLF) are directly depend on the scale of the images Imaging processing techniques are used to locate the scale bar associated with a panel The size of the scale bar is extracted from the accompanying caption Scale bar extraction is currently done with a precision of 76% and a recall of 50% Improving this step is a subject of current research 2.2.4 Subcellular location feature computation Finally, SLFs are produced that summarize the localization pattern of each cell We will not discuss these features at length, except to note that we have explored a number of feature types, and have developed what we believe to be quantitative, numeric measurements that provide a great deal of information about subcellular localization For example, when applied to the particular problem of classifying proteins from SLFs, we have achieved over 92% accuracy using these features to classify single cells in a tenway classification task [12,13] Our previous work [12] also demonstrated the feasibility of determining the subcellular location patterns via SLFs of individual cells in on-line journals, despite the challenge of differences in the magnification and pixel resolution, differences in sample preparation, cell type and microscopy method, and image alterations introduced during publication 2.3 Caption processing 2.3.1 Entity name extraction Caption interpretation aims to identify the name and cell type of the visualized protein in each microscope image These extraction tasks have been heavily studied in the literature; however, there are still few publicly-available extraction systems Rather than expend substantial resources to developing our own extractors, for the current version of SLF we hand-coded some relatively simple extraction methods for this task Protein names tend to be either single words with upper case letters, numerical figures, and nonalphabetical letters such as Nef, p53, or compound words with upper case letters, numerical letters, and nonalphabetical letters such as Interleukin (IL-1)-responsive kinase, or single lowercase words ending in –in or –ase such as “actin”, “tubulin”, “insulin” Similar rules are used to identify cell type The protein-name extractor obtains a precision of 63% and a recall of 95%, and the cell-type extractor obtains a precision of 85% and recall of 92% 2.3.2 Entity to panel alignment To integrate the features obtained via figure processing and entity names extracted from caption, entity to panel alignment must be done The goal here is to determine, for each entity extracted from the caption, to which panel that entity is associated The linkage between the images which are the figure panels and the text of captions is usually based on textual labels which appear as annotations to the images, and which are also interspersed with the caption text So, entity to panel alignment is based on extracting the labels from panels, and extracting the corresponding image pointers from captions Image pointers are strings in the caption that refer to places in the accompanying images, for example, “A”, and “B”, in Figure In the remainder of the paper, we will discuss in detail the methods used to find the panel-label annotations which appear in images, and the methods used to match these annotations to image pointers In the remainder of this section, we will briefly review how image pointers are found and associated with extracted entities [14] In analyzing caption text, we decided to break down the task of entity to panel alignment into several subtasks The first step is image-pointer extraction After image pointers are extracted, they are classified according to their linguistic function Bullet-style image pointers function as compressed versions of bulleted lists, for example, the strings “(A)” and “(B)” in figure NP-style image pointers are used as proper names in grammatical text, for example, the string “(A)” in the text: “Following a procedure similar to that used in (A), …” Citation-style image pointers are interspersed with grammatical caption text, in the same manner that bibliography citations are interspersed with ordinary text The remaining image pointers in Figure are citation-style We combined the steps of extraction and classification, as follows Most image pointers are parenthesize, and relatively short We thus hand-coded an extractor that finds all parenthesized expressions that are (a) less than 40 characters long and (b) not contain a nested parenthesized expression, and also extracts all whitespace-surrounded expressions of the form “x”, “X”, “x-y” or “X-Y” that are preceded by one of the words “in”, “from”, or “panel” This extractor has high recall (98%) but only moderate precision (74.5%) on the task of finding image pointers Using a classifier trained using machine learning, we then classify extracted image pointers as bullet-style, citationstyle, NP-style, or “other” Image pointers classified as “other” are discarded, which compensates for the relatively low precision of the extractor This classifier has an overall accuracy of 87.8% Performance is extremely good (recall of 98% and precision of 94.6%) on bullet-style labels, which are the ones most likely to severely impact performance Most errors are made by incorrectly rejecting citation-style image pointers [14] After image-pointer classification, the scope of each image pointer is determined The scope of an image pointer specifies, indirectly, what text should be associated with that image pointer The scope of a NP-style image pointer is the set of words that (grammatically) modify the proper noun it serves as The scope of a bullet-style image pointer is all the text between it and the next bullet-style image pointer The scope of a citation-style image pointer is some sequence of tokens around the image pointer, usually corresponding to a nearby noun phrase currently approximated with heuristic hand-coded methods Figure shows the overall structure of SLIF Tasks described in light grey characters represent future or ongoing work Not illustrated are supporting tools for browsing and querying the extracted information, which are also under development; however, flexible tools for accessing extracted information are also extremely important, since often only part of the information present in a figure is extracted Figure Diagram of SLIF (A) original image Data set 427 (B) binary edge map (C) regions rich in edge Figure 2: Process of text detection Total ‘text’ region 467 Real text region 380 Table 4: Text detection result PANEL LABEL EXTRACTION AND PANEL-TEXT MATCHING Figure Panels with internal labels Precision 81.3% Recall 89.0% Extracting panel labels and mapping information derived from captions to panels are crucial steps in SLIF since it serves at the bridge between image analysis and caption interpretation Since most panels contain internal labels (such as “A”~”F” in figure 3), we focused our initial work on extracting these internal labels Automatic detection and recognition of panel labels is a challenging problem because the label is usually a single character embedded in the panel, and the background might be complex However, current OCR (optical character recognition) technology is largely restricted to finding text printed against clean backgrounds, and cannot handle text printed against shaded or textured background, or embedded in images directly[15, 16] Our current system applies a fourstage strategy to the label contained within the panel itself The step is text detection, where a segmentation scheme is used to focus attention on regions where a panel label may occur The next step is image enhancement, where the text region is enhanced by increasing the resolution of characters, and converting the gray-scale image to a binary image The next step is OCR, where the enhanced text image is passed through an OCR engine for recognition The final step is approximate string matching, where the OCR results of all the panels in one figure are matched against the list of panel labels obtained by interpreting the caption associated with this figure Missing (or incorrect) labels produced by OCR can be corrected in the string-matching step This final step also serves as the way of mapping between labels recognized from the image and the labels (image pointers) obtained by caption interpretation, so as to combine the information extracted from the figure and the caption Below we will describe results of our four-stage strategy for panel label extraction and panel-text matching These experiments are based on a dataset of 427 hand-labeled panels from 95 randomly-chosen Pubmed Central papers As a baseline, we note that simply running the OCR software we are using (GOCR [17]) directly on panels produces only yielded correct 15 labels This emphasizes the point that the current OCR software is not designed to recognize text embedded in images Text detection Because characters usually form regions of high contrast against the background, a typical text region can be characterized as a rectangular region with a high density of sharp edges Therefore our text detection method relies on finding areas that have sharp edges We used the Roberts method [18] for edge detection Applying edge detection to the original panel (image A) resulted in a binary image B Image B contains the edges of the labels as well as some noise We noticed that the noise usually consisted of short line segments while label edges were represented by longer continuous regions or short nearly-connnected segments; for example the edges for the letter “a”, might be disjoint We therefore used a two-stage process to reduce noise We first closed the binary image using a 3x3 pixel structural element to connect the disjoint sections making up the edges of the labels Then we removed any objects of size 25 pixels or less to delete any remaining noise This results in a binary image C in which connected regions have a high density of sharp edges As an example, figure shows the process of text detection through A to C The text region appeared as a connected component in C We then bounded the connected components in C with their maximum and minimum coordinates in the x and y directions to get candidates for text regions Several constraints were then applied to filter out candidates that are not text regions  Panel labels are usually a single letter, so the ratio between the height and the width of a text region should be in a certain range We only kept regions for which this ratio was greater or equal to 0.3, i.e we discarded horizontally strip-shaped region  Panel label is usually located in one corner of the panel, so the distance between the boundary of a text region and the panel boundary should be small We only kept regions for which this horizontalvertical distance was less than 1/10 of the widthheight of the panel  Panel labels are usually small in size compared with the panel, so the area of the text region should be in a certain range Experimentally, we noticed the height/width of the text region was between 1/20 and 1/4 of the height/width of the panel, so we only kept regions with areas in the range of [1/202a, 1/42a] compared to the area a of the panel The experimental results are shown in Table A total of 380 of the 467 candidate text regions were correctly detected While 81.3% precision appears low, most of the regions incorrectly considered to be text regions don’t contain characters at all They therefore will not yield any characters during OCR and won’t affect the final set of panel labels Intensity normalization Because GOCR assumes the character to be black font printed in white background, we must determine whether the text appears as a black font on a white background or a white character on a black background before applying OCR In panel images, the intensity value of text may be lower or higher than that of the background For the later, the intensity of text images should be inversed before running GOCR We call this procedure intensity normalization Our normalization method is as follows First, we choose the top 20% pixels and the bottom 15% pixels of a text region, where pixels are mainly background pixels, and calculate the mean value m1 of these pixels Then we choose the middle area from 0.3h to 0.65h (h is the height of text region) and calculate the mean value m2 If m1 < m2, we consider the intensity of text is lower than that of the background; otherwise the intensity of text is considered to be higher than that of the background Image enhancement Applying GOCR to normalized text regions obtained a precision 71.3% of and a recall of 63.5% We hypothesized that most errors are because GOCR is designed for recognizing high resolution text printed against clean backgrounds In order to increase the recognition rate, we introduced an interpolation method and a binarizing algorithm to increase the image quality Sub-pixel Interpolation One crucial condition for GOCR success rate is the resolution of input image GOCR prefers fonts of 20~60 pixels However, the area of the text region might be less than 20 x 20 pixels GOCR usually fails on such low resolution images To obtain higher resolution images, we expanded regions smaller than 20 x 20 by applying bicubic interpolation[19] Bicubic interpolation estimates the grey value at a pixel in the destination image by an average of 16 pixels surrounding the closest corresponding pixel in the source image Figure (a) original text region (b) binary image (a) (b) (c) (d) # panels shows one example where GOCR failed when given the original text region (a), while GOCR successfully recognized background character the binary image (b) obtained by dynamic thresholding (described in the following paragraph) A number of binarization algorithms have been proposed We chose Niblack’s method [21], which performed well in a recent survey [22] Niblack’s algorithm calculates the threshold dynamically by gliding a rectangular window across the image Another effective method for binarization is based on Gaussian mixture models [23,24], in which a histogram of gray value frequencies is modeled as mixture of Gaussians, (e) and one peak is used to define a thresholding scheme This is Figure Binarization based on Gaussian mixture illustrated in Figure 7(e) It is assumed that the range of models Here (a) is the original text region; (b) and (c) are gray-levels corresponding to “character intensity” thresholding results by assuming mixtures of and corresponds to one Gaussian, and the background intensities Gaussians, respectively; (d) is the segmentation result by corresponds to the remaining Gaussians dynamic thresholding; and (e) is the smoothed histogram of The number of Gaussian mixtures is crucial in the text region appropriately modeling the background In Figure 7, (a) is the original text image, and (b) and (c) are segmentation results Binarization Complex backgrounds pose another difficulty created by assuming two and three Gaussian mixture models for OCR GOCR can accept a grey image as input and it does respectively The character is confused with the background the binarization to separate text from the background by when using only two Gaussians, but can be extracted when global thresholding[17,20] Unfortunately, global using three Gaussians Hence we choose three as the number thresholding is usually not possible for complicated images of Gaussians in the mixture The underlying reason for three Consequently, GOCR works poorly in these cases Figure being a good choice for the number of Gaussians # Labels extracted with Labels extracted without image Labels extracted with image Labels extracted after correctly detected running GOCR on panels Recall enhancement # Precision enhancement Recall # Precision modification # Precision Recall # 15 3.9% 3.5% 271 71.3% 63.5% 302 79.1% 70.7% 316 Table 5: Labels correctly extracted, with/without image enhancement, before/after modification Precision Recall 83.2% 74.0% text 427 region 380 mixtures could be that the character is more uniform in grey value than the background The parameters of the Gaussian mixture models can be estimated via the EM algorithm [24] Each of the two binarization algorithms introduced in this section has advantages The modified Niblack’s algorithm is faster, but sometimes it is too sensitive to the local noise; this is illustrated in Figure 7, where the dynamic thresholding (d) didn’t work as well as the Gaussian mixture model (c) The Gaussian mixture model is good at grey value distribution modeling, but the estimation of the parameters is more expensive Therefore we used the following strategy: first we apply the modified Niblack’s algorithm and run GOCR If and only if no characters are recognized, we apply the Gaussian mixture model algorithm and run GOCR again With image enhancement, we obtained a precision 79.1% of and a recall of 70.7% (Table 5) Figure 8: A case where current algorithm failed (a) original anel, (b) detected text region Modification based on grids and string match Even with image enhancement, we might still fail to extract labels from some panels Part of the reason is that our current binarizing algorithm is not robust enough For example, the current binarization process couldn’t recognize the character in Figure correctly To succeed on cases in which OCR fails, we can turn to “context” information, i.e., labels extracted from other panels which are in the same figure as the failed one If labels of these “sibling” panels are extracted correctly, we can use the context to guess what label the panel holds To find out the missing/incorrect label, we must figure out all the possible labels, and the pattern for how these labels are assigned to the panels We can usually determine the range of possible labels from caption analysis Since the caption analysis is based only on the text, the list of labels generated from caption processing is fairly reliable In general the arrangement of labels might be complex: labels may appear outside panel, or several panels may share one label However, in the majority of cases, panels are grouped into grids, each panel has its own label, and the labels are assigned to panels either in column-major or rowmajor order The six panels shown in Figure are typical of this case For this case, we analyze the locations of the panels in the figure and reconstruct this grid, i.e., the number of total columns and rows, and also determine the row and column position of each panel Given the list of all panel labels, extracted from caption analysis, the grid, and the distribution of panels, we compute the label sequence assigned to panels in column-major order and row-major order, resulting in two strings SC and SR Taking Figure as an example, suppose that panel C’s label is mis-recognized as “G”, and that no label is found for panel E In this case the string SC will be “ADB GF” and SR will be “ABGDF” Then we computed the similarities between the string of labels S resulting from caption analysis, and the strings SC and SR resulting from OCR and grid analysis For instance, if caption analysis produces the string “ABCDEF”, we would compare this string to “ADB GF” and “ABGDF” Here we used Needleman-Wunsch edit distance (using substitution costs reflecting likely OCR errors, and implemented with a package described elsewhere [25]) to compute the similarity between two strings The edit-distance alignment for the string with the smaller distance to the OCR result is then used to correct the OCR result For Figure 3, using our strategy, we can infer that the labels for panel 1~6 should be ABCDEF Table shows the contribution of the modification process Note that this procedure also produces a the mapping between labels extracted from images of panels and those generated by caption interpretation CONCLUSION Mining the biological literature is crucial to organize and summarize scientific results Most existing IE systems for the biological domain are limited to extracting information from text However, figure-caption pairs in scientific publications are extremely dense in information We have set as our long-term goal building an accurate automated toolset, SLIF, to extract information about protein subcellular localization from the text and images found in online journals In this paper, we gave a review of SLIF, and presented new results for the task of integrating information obtained from textual analysis and image analysis Specifically, we have augmented previously developed tools to find fluorescence microscope images depicting protein subcellular location patterns, by adding caption processing to extract image pointers and entity names from the text Additional image processing and OCR techniques were used to extract panel labels Finally image pointer to panel label alignment via approximate string matching and geometric analysis of a figure was used to integrate the two sets of labels The final current system can generate assertions such as “Figure N depicts a localization of type L for protein P in cell type C” We believe that SLIF demonstrates that IE from Both biological text and image is feasible REFERENCES [1] Blaschke, C., Andrade, M A., Ouzounis, C., and Valencia, A.: Automatic extraction of biological information from scientific text: Protein-protein interactions In Proceedings of the 1999 International Conference on Intelligent Systems for Molecular Biology (ISMB-99) 1999, pp 60–67 [2] Sekimizu, T., Park, H., and Tsujii, J Identifying the interaction between genes and gene products based on frequently seen verbs in Medline abstracts In Genome Informatics, pp 62–71 Universal Academy Press, Inc,1998 [3] Pustejovsky, J., Casta˜no, J., Zhang, J., Kotecki, M., and Cochran, B.: Robust relational parsing over biomedical literature: Extracting inhibit relations In Proceedings of 2002 the Pacific Symposium on Biocomputing (PSB-2002) 2002, pp 362–373 [4] Thomas, J., Milward, D., Ouzounis, C., Pulman, S., and Carroll, M.: Automatic extraction of protein interactions from scientific abstracts In Proceedings of 2000 the Pacific Symposium on Biocomputing (PSB-2000).2000, pp 538–549 [5] Stephens, M., Palakal, M., Mukhopadhyay, S., Raje, R., and Mostafa, J.: Detecting gene relations from medline abstracts In Pacific Symposium on Biocomputing 2001, pp 483–496 [6] Humphreys, K., Demetriou, G., and Gaizauskas, R.: Two applications of information extraction to biological science journal articles: Enzyme interactions and protein structures In Proceedings of 2000 the Pacific Symposium on Biocomputing (PSB-2000) 2000, pp 502–513 [7] Fukuda, K., Tsunoda, T., Tamura, A., and Takagi, T.: Toward information extraction: Identifying protein names from biological papers In Proceedings of 1998 the Pacific Symposium on Biocomputing (PSB-1998)1998, pp 707–718 [8] Rindflesch, T., Tanabe, L., Weinstein, J N., and Hunter, L.: Edgar: Extraction of drugs, genes and relations from the biomedical literature In Proceedings of 2000 the Pacific Symposium on Biocomputing (PSB-2000).2000, pp 514–525 [9] Bunescu, R., Ge, R., Mooney, R J., Marcotte, E., and Ramani, A K Extracting gene and protein names from biomedical abstracts Unpublished Technical Note, Available from http://www.cs.utexas.edu/users/ml/publication/ie.html, 2002 [10] Craven, M and Kumlien, J.: Constructing biological knowledge bases by extracting information from text sources In Proceedings of the 7th International Conference on Intelligent Systems for Molecular Biology (ISMB-99) AAAI Press, 1999, pp 77–86 [11] Stapley, B., Kelley, L., and Sternberg, M.: Predicting the sub-cellular location of proteins from text using support vector machines In Proceedings of the 2002 Pacific Symposium on Biocomputing 2002, pp 374–385 [12] R.F Murphy, M Velliste and G Porreca.: Robust Classification of Subcellular Location Patterns in Fluorescence Microscope Images In Proceedings of 2002 IEEE Intl Workshop Neural Networks Signal Processing (NNSP 12), pp 67-76 [13] R F Murphy, M Velliste, J Yao, and G Porreca.: Searching Online Journals for Fluorescence Microscope Images Depicting Protein Subcellular Location Patterns In Proceedings of IEEE Int Symp Bio-Informat Biomed Eng (BIBE 2001) 2, pp.119-128 [14] W Cohen, R Wang, R Murphy.: Understanding Captions in Biomedical Publications, in KDD-2003 (to appear) [15] Wu V., Manmatha R., Riseman E M : TextFinder: An Automatic System to Detect and Recognize Text In Images IEEE Transaction on Pattern Analysis and Machine Intelligence, November 1999,21(11), pp 1224-1229 [16] Mori S., Suen C Y., and Yamamoto K Historical review of OCR research and Development Proceedings of The IEEE, 80(7), July 1992, pp1029-1058 [17] GOCR0.37, http://jocr.sourceforge.net/ [18] Shapiro L and Stockman G Computer Vision Prentice Hall, 2001, p156-176 [19] Bicubic Interpolation for Image Scaling, http://astronomy.swin.edu.au/~pbourke/colour/bicubic/ [20] Otsu, N A thresholding selection method from gray-level histogram IEEE Transactions on Systems, Man, and Cybernetics, 1979,9(1): pp62-66 [21] W Niblack An Introduction to Digital Image Processing, pp115–116 Englewood Cliffs, N.J.: Prentice Hall, 1986 [22] Christian Wolf , Jean-Michel Jolion and Francoise Chassaing Text Localization, Enhancement and Binarization in Multimedia Documents In Proceedings of the International Conference on Pattern Recognition (ICPR) 2002, volume 4, pp 1037-1040 [23] J Yang, J Gao, Y Zang, X Chen, and A Waibel An Automatic Sign Recognition and Translation System In Proceedings of the Workshop on Perceptive User Interfaces (PUI), 2001 [24] T Mitchell Machine learning McGraw Hill, 1997 pp191-196 [25] W W Cohen and P Ravikumar and S E Fienberg, A Comparison of String Distance Metrics for NameMatching Tasks In Proceedings of the IJCAI-2003 Workshop on Information Integration on the Web(IIWeb-03), To applear  ... IE from Both biological text and image is feasible REFERENCES [1] Blaschke, C., Andrade, M A., Ouzounis, C., and Valencia, A.: Automatic extraction of biological information from scientific text: ... to extract information about protein subcellular localization from the text and images found in online journals In this paper, we gave a review of SLIF, and presented new results for the task... existing IE systems for the biological domain are limited to extracting information from text However, figure-caption pairs in scientific publications are extremely dense in information We have