(LUẬN VĂN THẠC SĨ) Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies

Extraction des informations sur le Web

The growing number of web pages and the information they contain makes it increasingly challenging to search for and extract relevant data Consequently, web information extraction has become a significant focus of research in this complex landscape.

Information extraction (IE) is influenced by the nature of web pages, which can be categorized into two types: the hidden web and the explicit web On hidden web pages, information is only displayed after users complete certain forms, while explicit web pages present information immediately upon entry Consequently, research on IE varies according to these two web types.

The focus of our analysis is on two aspects of information extraction (IE) from the web First, we explore automatic generation of data to populate forms for retrieving essential information from hidden web content Second, we examine explicit information extraction, emphasizing how to identify regions on a webpage that contain necessary data and the methods used to extract relevant information from those areas.

In general, information on the web is presented explicitly in HTML pages, which have a flexible structure that allows content to be positioned anywhere on the page The indexing of web pages involves two major steps.

Premièrement, la détermination des régions contenant des informations nécessaires

Deuxièmement, l’extraction des informations pertinentes à partir des régions déterminées

Les sous-sections suivantes vont présenter ces deux étapes

Comme les informations apparaissent n’importe ó dans la page Web Le problème qui se pose est : comment peut-on déterminer si une région contient des informations

Internship Report: This document focuses on the collection of web information related to specific domains, utilizing agents and necessary ontologies Various studies address this issue, and three main approaches can be identified within this research.

Approche basée sur le traitement du langage naturel (TLN)

The semantic marker-based approach facilitates the creation of personalized markers to enhance content tagging on a webpage This method is increasingly being adopted in the field of Semantic Web.

A common structured model approach organizes information from various sources into a unified framework, regardless of the original source This method enhances the efficiency and ease of extracting insights from multiple data sources Currently, this third approach is increasingly being adopted in various research efforts.

Information is organized in various ways, and the goal is to restructure these formats into a common framework that is independent of the information source The challenge lies in how to create this unified structure from different web pages Three main approaches can be identified for this task.

L’utilisation des techniques XML L’utilisation d’arbre des marqueurs L’utilisation d’ontologie du Web Les sous-sections suivantes vont développer plus ces trois approches a Utilisation de XML

The structure of HTML markers in a web page resembles that of an XML document, as the goal is to project an HTML page onto an XML format Initially, an HTML page is converted to XML by completing any incomplete markers Subsequently, the areas containing information are identified using various techniques, such as domain ontologies or transformations through XSLT (Extensible Stylesheet Language Transformations).

This approach offers the significant advantage of being simple and understandable, as XML is widely used and its techniques, such as XSLT, are well-known However, it also has a major drawback: XML transformation is only effective when the HTML page structure remains stable Currently, many web pages frequently change their structure, rendering this approach less applicable Additionally, the use of a markup tree can be considered.

This approach is based on the structure of HTML markers in a web page, resembling a tree where the nodes correspond to pairs of HTML markers Initially, it involves creating a Document Object Model (DOM) tree for each HTML page by completing markers that have only a single tag, such as

and .

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies

Remove unnecessary markers while retaining the content within them Each remaining pair of markers will form a node in the DOM tree, with a marker contained directly within a parent marker becoming a direct child of the corresponding parent node Subsequently, noise (unnecessary nodes) can be eliminated from the tree using techniques from the Style Tree Next, regions containing information are identified using various methods, such as assessing the edit distance between character strings and calculating the similarity between character strings These identified regions are then designated for the data extraction phase.

Creating a DOM tree involves clear and concrete steps, but it is most effective for pages that present information in a tabular format (rows and columns) or in a consistent manner across multiple pages, such as those showcasing products like books, cars, or clothing This approach becomes less effective for websites with pages that contain varying forms.

This approach is grounded in the structure of HTML markers within a web page An HTML page is composed of a set of HTML markers, which include defined elements with a limited number of types These markers can be represented by a collection of classes and instances, forming what is known as a web ontology.

The process of determining regions involves projecting an HTML page onto a web ontology, where each class or instance corresponds to an HTML marker Subsequently, the regions containing information are identified based on the projection results; for instance, studies like [7] and [10] utilize rules constructed in JessTab [9] to define these regions.

Utilisation de la syntaxe des marqueurs

This method utilizes the same technique as the one used for determining regions, focusing on various aspects of the HTML page structure It employs XML and XSLT transformation, measures the edit distance between character strings, and calculates the similarity between character sequences.

This method utilizes tools like RoadRunner, which automatically extracts data from web pages by analyzing HTML coding grammar It processes two pages simultaneously, calculating the similarity and dissimilarity between them based on the matching of character strings.

B.Lui et al ([17], [44]) a proposé une méthode extrayant des lignes de données en se basant sur l’arbre DOM avec deux hypothèses : (i) une région ne contient qu’un ou plusieurs tableaux ayant des lignes et des colonnes de données ; (ii) les données dans le même tableau ont la même structure La phase d’extraction ne traite que les sous arbres dont la racine est le nœud composant tous les marqueurs d’une région : on calcul la similarité entre les fils d’un nœud en utilisant la distance éditée S’ils sont semblables, ils seront considérés comme les lignes d’un tableau de données Dans le cas contraire, ils seront considérés comme des tableaux indépendants et chaque nœud et ses fils seront traités comme une région indépendante

Based on the assumption that data only appears in tables with rows and columns, K Lerman et al proposed a data extraction algorithm that generates templates for HTML pages through three key steps: (i) searching for a template for the page; (ii) identifying data columns with the assumption that data within the same column shares the same type; and (iii) identifying rows within the columns using labels This algorithm eliminates the need to separate the processes of determining regions and extracting data.

X Meng et al [24] a proposé une méthode qui s’appelle SG-WRAM (schema-guided approach to wrapper maintenance) Elle maintient les wrappers en se basant sur les caractéristiques syntaxiques selon quatre étapes [24] : (i) les caractéristiques sont obtenues à partir de schéma définie par l’utilisateur, des règles d’extraction précédentes ou des rộsultats extrait ; (ii) reconnaợtre des donnộes en basant ces caractộristiques ; (iii) regrouper des données à la schéma, chaque groupe peut être une instance de la schéma ; (iv) les instances représentatives sont sélectionnées pour diminuer les règles d’extraction dans la phase suivante

Pour conclure, les méthodes basées sur la syntaxe des marqueurs se caractérisent ainsi :

Elles réalisent seulement un traitement syntaxique Le traitement est basé sur la grammaire du codage HTML, sur la syntaxe des marqueurs HTML ou sur la

Internship Report: The project involved gathering information from the web related to specific domains using agents and ontologies within the structure of HTML pages As a result, these methods are relatively straightforward, with clear algorithms guiding the process.

The methods rely on several assumptions: either the HTML pages of the same site maintain a stable structure, or the data regions are organized in one or more tables, or the data within rows share the same structure, or the data within the same column is of the same type Each method is only effective on a limited number of web pages.

Extracted data is often noisy and typically consists of information found within a table cell or between complete HTML tags These methods can only retrieve a portion of the data, specifically relating to a particular domain within a table cell.

Utilisation d’ontologie de domaine

Récemment, les ontologies sont de plus en plus utilisées dans le contexte du Web Sémantique Svab et al [37] distingue trois types d’ontologie pour l’EI :

L’ontologie terminologique : elle représente les termes dans les langages humains Elle est applicable dans l’annotation de texte

L’ontologie de représentation : elle représente les objets Web dans les pages Web Elle est applicable dans l’analyse de discours linguistique

L’ontologie de domaine : elle représente les classes, les instances, les slots… des objets dans un domaine spécifique

Figure 1 représente une partie de l’ontologie de domaine scientifique Les rectangles représentent les classes, instances (objets) Les arcs représentent les relations entre eux

Figure 1 : Une partie de l’ontologie de domaine d’événement scientifique (source [7])

This document focuses solely on domain ontology as it pertains to the issue of information extraction (IE) Some of the discussed tools leverage domain ontology for data extraction from the web.

Snoussi et al define ontology as a method for decomposing the world into objects and describing them They utilize SOX (Schema for Oriented-Object XML) to define and model an ontology To facilitate web data extraction, they developed a tool called WeDaX (Web Data eXtraction), which employs domain ontology and follows four key steps: (i) downloading HTML pages and converting them into XML documents; (ii) constructing a data model using SOX and an ontology; (iii) mapping the XML documents to the ontology's classes, instances, and slots; and (iv) allowing users to select the specific data elements they need from the page.

McDowell et al proposed OntoSyphon, a tool designed to extract web data using an ontology This tool takes an ontology and HTML pages as input and outputs extracted data that corresponds to the classes, instances, and slots defined in the ontology The extraction process involves five key steps: (i) identifying a class from the ontology; (ii) generating possible lexical-syntactic phrases for the class instances through heuristic processing; (iii) searching the page for occurrences of these phrases and extracting them as candidate instances; (iv) repeating steps 2 and 3 until the search set is empty; and (v) evaluating the candidate instances to select the correct instances for the class.

Espinasse et al proposed AGATHE, an architecture designed for web information extraction using agents and ontologies Similar to MasterWeb, AGATHE operates on two main principles: (i) extracting data functionality based on web ontology, and (ii) semantically extracting data from web pages using domain ontology and inference rules in Jess A detailed presentation of this architecture will be provided in the third section of the document.

Banko et al introduced TextRunner, a tool designed to extract data with an indefinite structure, focusing on open information Buitelaar et al developed SOBA, a subcomponent of the SmartWeb system Additionally, Liddle et al presented a tool that enables users to graphically create and test web information extraction.

[31] a proposé un outil qui permet de générer automatiquement les annotations entité- només avec les classes et les instances correspondantes

L’approche d’utilisation d’ontologie en EI se caractérise ainsi :

Semantic extraction involves retrieving information based on the ontology of each domain, ensuring that the data is more relevant This method allows for the extraction of specific text segments from HTML pages, enhancing the quality and precision of the information gathered.

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies approches syntaxiques n’extraient qu’une celle d’un tableau dans les pages HTML

The domain ontology is declarative, allowing for openness and flexibility This means that users can easily add, edit, or remove classes, instances, and slots within the ontology.

Independence has two key aspects Firstly, ontology is independent of the web page structure and the presentation of information on the page Therefore, the approaches to this trend remain unaffected by changes in HTML structure Secondly, ontology is also independent of the application, allowing for easy expansion of the system from one domain to another by simply replacing the corresponding domain ontology.

Par contre, la tendance d’utilisation de l’ontologie domaine a des difficultés dans la phase d’inférence sur l’ontologie, par ex WeDaX [34] utilise SOX, AGATHE [7] et MasterWeb

[10] utilisent les règles d’inférence en Jess [9].

Extraction des informations par agent

An agent is defined as an autonomous entity operating within a specific environment, capable of living independently and flexibly by perceiving and interacting with its surroundings to achieve its own goals The four primary characteristics of an agent include autonomy, reactivity, proactivity, and sociability.

A multi-agent system is a network of entities that interact to achieve their individual objectives, thereby contributing to the overall goal of the system This approach proves to be effective in various applications across different contexts.

Le système a besoin de communications complexes Une communication est considérée comme complexe si : elle a au moins deux participants et, échange des informations entres eux se passe en plusieurs fois

Le système doit fonctionner dans les cas ó on ne peut pas prévoir ou spécifier tout son comportement en des cas-en-cas (case-by-case)

Le système concerne la négociation, la collaboration ou la compétition entre plusieurs entités différentes

Le systốme doit fonctionner de faỗon autonome

Le système est prévu qu'il y ait des changements dans le système ou que le but du système soit changé dynamiquement

Le système multiagent convient naturellement au problème de l’EI car, en effet, l’EI a des

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies caractéristiques correspondant aux contextes ci-dessus ([4], [27]):

Les sources des informations sont distribuées : elles appartiennent à plusieurs systèmes qui se situent aux serveurs différents sur le réseau Internet

Les sources des informations sont hétérogènes : elles sont réalisées par personnes différentes, en temps différents et en utilisant des mécanismes et des plates-formes différents

Les sources des informations changent dynamiquement selon le besoin de leurs propriétés

According to [4], an artificial intelligence (AI) problem is no longer just a straightforward query on a database; rather, it is a complex issue that can be broken down into several interdependent sub-problems Furthermore, the solution requires integration from multiple subsystems, highlighting the need for complex communication and autonomy within the system This scenario exemplifies an effective context for multi-agent systems.

Numerous studies have identified multi-agent systems as an effective solution for the issue of Information Extraction (IE) I Bouslimi et al introduced the Cooperative Information Gathering System (CIGS) organizational model, which is developed using the Gaia methodology, a framework focused on agent-oriented system analysis and design.

R Okada et al [27] a proposé le modèle CAS (Coopreative Agents Society) pour l’extraction d’informations à partir d’Internet F Freitas et al [10] a proposé MasterWeb, un système d’extraction des informations en basant sur l’ontologie domaine B Espinasse et al [7] a proposé AGATHE (Agent information GATHEring), une architecture permettant d’extraire des informations sur le Web en basant sur des agents et des ontologies L Sofiane et al [36] a proposé MathNet, un système d’extraction des informations sur le Web en basant sur les agents mobiles T Xiaohui et al [43] a proposé une méthode d’expansion des requêtes basées sur des sujets hiérarchies, c’est une solution pour le problème d’EI S Cazalens et al [5] a proposé BONOM, c’est un système multiagent pour la recherche des informations sur le Web

En basant sur la nature du système utilisé, on peut regrouper les recherches en trois groupes principaux :

Using a single agent, such as MasterWeb, limits the benefits of a multi-agent system This approach restricts the potential for task sharing through agent interaction and diminishes the advantages of parallel processing inherent in multi-agent systems.

The use of multi-agent systems, such as CIGS, CAS, and AGATHE, is becoming increasingly popular due to their ability to leverage the full benefits of this technology However, challenges remain, particularly concerning analysis and evaluation within these systems.

Internship Report: Gathering information from the web related to specific domains using agents and ontologies The design of multi-agent systems currently lacks a standardized method and effective tools for analysis and development.

The use of mobile agents, such as MathNet, offers the potential for effective adaptation to the distributed, heterogeneous, and dynamic environment of the Internet However, challenges related to resource access rights and network security must be carefully managed These security issues are complex and contribute to the limited popularity of mobile multi-agent systems.

In summary, the multi-agent system presents a suitable solution for web information extraction, demonstrating effectiveness in a distributed, dynamic, and heterogeneous environment that aligns with the nature of the Internet.

Conclusion

Today, the challenge of extracting information from the Web is becoming increasingly difficult and complex due to the distributed, dynamic, and heterogeneous nature of the Internet Numerous studies are currently focused on this issue, proposing various methods for effective web information extraction.

En effet, chaque méthode a l’air efficace dans un domaine spécifique et sur des pages particulières

This chapter presents the state of the art in the field of Information Extraction (IE), which encompasses two types: IE from the Hidden Web and IE from the Explicit Web Typically, IE from the Explicit Web involves two key stages: region extraction and data extraction from those regions Region extraction can be approached through three methods: natural language processing, user-defined semantic markers, and HTML page structure analysis The HTML structure-based approach has three trends: utilizing XML and XSLT techniques, employing the Document Object Model (DOM), and leveraging Web ontologies The second stage of IE, data extraction from regions, also exhibits three trends: applying syntactic techniques, using semantic techniques grounded in domain ontologies, or employing agents.

Le chapitre suivant va présenter MatserWeb et AGATHE Ce sont deux architectures pour le problème d’EI Elles composent les deux approches qui utilisent des ontologies et des agents

MasterWeb

MasterWeb is a multi-agent system designed for data extraction from the Internet, utilizing ontologies for enhanced information retrieval This system is part of a Brazilian-German project focused on a semantic approach to data retrieval.

This section introduces MasterWeb, a multi-agent system designed for the EI It consists of two approaches that utilize ontologies and agents Section 1 provides a general overview of MasterWeb, while Section 2 focuses on the extraction agent, which is the core component of MasterWeb For more details about MasterWeb, please refer to [10].

L’architecture de MasterWeb a trois parties :

Chargement des pages : chercher et télécharger les pages Web à partir de plusieurs sources différentes Le composant principal de cette partie est le méta moteur

Information extraction involves gathering relevant data from pages received by the meta-search engine, which is the core function of MasterWeb The primary component of this process is the extraction agent.

Fourniture des services : fournir des informations extraites aux utilisateurs ou aux autres agents Le composant principal de cette partie est le médiateur

Le fonctionnement du système est illustré dans la figure 2 :

A meta search engine scours the internet for web pages by utilizing existing search engines like Altavista or Excite It then downloads these pages and forwards them to extraction agents for further processing.

Each extraction agent specializes in retrieving information from a specific domain They receive web pages through a search engine and extract relevant data based on domain and web ontologies The extracted information is then stored in a database.

From the user's perspective, they can request the information they need The mediator receives this request and subsequently queries the database to provide an appropriate response.

Figure 2 : Architecture générale de MasterWeb (source [10])

The extraction agent is responsible for retrieving information from a specific domain, known as a cluster It receives web pages through a meta engine and extracts relevant information based on domain ontologies and web ontologies The extracted data is then stored in a database.

Figure 3 : Agent d’extraction de MasterWeb (source [10])

L’agent d’extraction fonctionne selon quatre étapes (figure 3 [10]) :

Validation involves checking whether a web page (linked to a URL) is validated A page is deemed validated if it meets certain criteria: it is accessible, it is in HTML format, and it has not been previously processed.

Preprocessing involves extracting key elements from each validated page, including content, page titles, links, and email addresses, among other components This data is then transmitted to the inference engine of the agent.

Reconnaissance involves functionally classifying a webpage, where the extraction agent determines whether the page is a list, message, frame, or HTML page Only HTML pages are processed; if a page contains information relevant to its cluster, it moves on to the extraction phase Conversely, if the page includes information pertaining to other clusters, it is recommended to agents in those respective clusters.

Extraction : extraire des informations concernées en utilisant des ontologies de domaine.

AGATHE

This section introduces AGATHE, an architecture designed to address the issue of EI It encompasses two approaches utilizing ontologies and agents Section 1 will outline the objectives, architecture, and functioning of AGATHE, while Section 2 will provide an in-depth look at the extraction subsystem, which is the core of the AGATHE architecture For more detailed information on AGATHE, please refer to [7].

The AGATHE system is a generic software architecture designed for the development of information collection systems on the web, focusing on one or more specific domains It implements a cooperative information gathering approach using agents and ontologies, and takes into account research contexts by considering clusters of web pages related to specific areas.

Figure 4 : L’architecture générale de AGATHE (source [7])

Le système AGATHE contient trois sous-systèmes principaux [7] (figure 4) :

The Search Subsystem (SSR) is responsible for querying external search engines on the web, such as Google, to retrieve web pages that will be processed by the Extraction Subsystem.

The Extraction Subsystem (ESS) consists of various Extraction Clusters (EC), each tailored to process web pages within a specific domain, such as academic research or tourism.

The Usage Subsystem (USS) is responsible for storing data extracted from processed web pages by the Extraction Subsystem, and it offers a query interface for users, which may include both humans and other software agents.

The operation of AGATHE is depicted in Figure 4, highlighting the interaction arrows numbered among its various subsystems Arrow 1 indicates that an extraction cluster from the SSE requests a search for specific pages from the SSR Arrows 2 and 3 show that the SSR functions as a meta-search robot, querying existing search engines like Google and Altavista for web pages Subsequently, arrow 4 illustrates that these pages are sent to the SSE, specifically to the extraction agent that made the request If necessary, arrow 5 indicates that recommendations are sent from the relevant cluster to other extraction clusters, suggesting pages that may be of interest Finally, arrow 6 demonstrates that the extracted information is forwarded to the SSU for storage in a specific database, which will be accessible for user queries as shown in arrow 7.

Ce document se concentre seulement sur le sous-système d’extraction, les autres sont décrits détaillé dans [7]

Typically, the SSE first generates web page requests sent to the SSR, and its primary responsibility is to process the results of these requests from the SSR This process involves several key tasks, including validation, functional classification, and information extraction.

Figure 5 : L’architecture interne de SSE (source [7])

As illustrated in Figure 5, each extraction cluster operates as a multi-agent system that classifies web pages and extracts information from them These extraction clusters consist of various types of agents, including extractor agents, preparatory agents, a supervisory agent, a recommendation agent, and a storage agent.

Each cluster has one or more preparer agents that receive web pages from SSR These agents are created by the supervisor agent of the same cluster and can be deleted by the supervisor when they are no longer needed.

Ces agents réalisent un premier traitement, le traitement fonctionnel, des pages Web en la séquence des tâches suivantes :

Validation : vérifier si les pages Web sont valides C'est-à-dire si elles sont au format HTML, elles sont accessibles, et si elles n’ont pas déjà été stockées dans

Internship Report: Collection of web information related to specific domains using agents and ontologies for database management Pages that do not meet these criteria will not be processed further.

Preprocessing involves identifying the content, titles, links, and available emails on a webpage using techniques from Information Retrieval (IR) and potentially natural language processing methods.

Classification fonctionnelle : Cette tâche, à base de connaissances, utilise le moteur d’inférence Jess et exploite une ontologie définissant des concepts liés au Web (l’ontologie Web)

Figure 6 illustrates the sequence of tasks performed by the preparation agent The preparation agent receives raw HTML pages from the supervising agent, initiating the validation process If the page is valid, the agent proceeds with pre-processing; otherwise, the invalid page is sent to the storage agent to prevent future processing Once pre-processing is completed, functional classification takes place, producing pages ready for extraction and classification, which are then forwarded to the extraction agents If the page is deemed invalid, it is directed to the storage agent.

Each cluster consists of multiple extraction agents tasked with semantically classifying the received pages and extracting the information they contain Each extraction agent is linked to a specific concept within the domain ontology related to the extraction cluster in question.

La figure 7 illustre l’enchaợnement de tõches rộalisộes par l’agent extracteur L’agent extracteur reỗoit les pages prộparộes de l’agent prộparateur (1) et effectue la

The internship report discusses the process of collecting web information related to specific domains using agents and semantic classification ontologies Classified page addresses and their respective classes are sent to both the storage agent and the recommendation agent Subsequently, the extraction agent performs information extraction from these pages, with the extracted data also forwarded to the storage and recommendation agents Additionally, the extraction agent can independently query the web by sending a request to the SSR supervisor agent.

The recommending agent, as illustrated in Figure 8, receives pages processed by extractor agents and suggests some of these pages to other extractor agents within the same cluster or to recommending agents from different clusters In the latter scenario, the recommending agent is also aware of certain ontologies associated with clusters other than its own.

Conclusion

Généralement, AGATHE respecte le principe du problème d’EI sur le Web explicite Il traite les pages Web selon deux étapes :

Functional treatment involves assessing whether a webpage contains essential data by utilizing Web ontology and inference rules To ascertain if the data pertains to a specific domain, it also employs domain ontology during this evaluation process.

Traitement sémantique : classifier les données dan la page sont les instances de quelle classe dans l’ontologie domaine en utilisant l’ontologie domaine et des règles d’inférence

AGATHE is distinguished by its semantic information extraction capabilities The use of ontology offers a declarative advantage, allowing for extraction that is independent of the page structure, unaffected by changes in how information is presented, and versatile across different domains of information.

D’ailleurs, en basant sur le système multiagent, cette architecture est efficace dans un environnement distribué, dynamique et hétérogène correspondant de la nature de Internet

Ce chapitre a présenté deux systèmes d’EI qui composent les deux approches qui utilisent des ontologies et des agents :

MasterWeb, un système existant pour l’EI avec une architecture dont la partie d’extraction a un seul agent

AGATHE, qui s’appuise sur MasterWeb, c’est une architecture proposée pour le problème d’EI avec une architecture distribuée selon un système multi agents

Both systems aim for the same goal of artificial intelligence (AI), employing similar techniques such as ontology-based AI and agents The next chapter will introduce the MasterWeb projection into the new AGATHE architecture.

Spécification du problème

MasterWeb's extraction component relies on a single type of agent, known as the extraction agent, which sequentially handles the entire extraction process within a cluster, from validation to web page extraction As a result, MasterWeb is unable to leverage the benefits of a true multi-agent system, such as distributed and parallel processing.

AGATHE is introduced as an innovative architecture featuring a multi-agent extraction system, where multiple agents collaborate to handle different segments of the extraction process Each agent is responsible for a specific task within the extraction chain, enabling them to process web pages simultaneously This parallel processing capability enhances the efficiency of data extraction.

The main issue is how to align the work of the extraction agent at MasterWeb with the tasks performed by agents in the Extraction Sub-System (SSE) of AGATHE.

Diviser et modifier le code, la base de règles, les ontologies de agent d’extraction de MasterWeb aux ceux des agents dans le SSE de AGATHE

Définir et implémenter les protocoles, les messages pour la communication entre les agents dans le SSE de AGATHE

Reconstruire la base de données de MasterWeb pour s’adapter au nouveau système AGATHE

Les sections suivantes vont prộsenter en dộtail la faỗon dont on rộsout le problốme

La figure 9 illustre la spécification générale du SSE/ les rectangles sont agents, les arcs sont protocoles entre les agents Le SSE contient cinq types d’agent :

Agent superviseur : Il supervise l’activitộ des agents prộparateurs Il reỗoit les

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies résultats des requêtes envoyées par le SSR

Agent préparateur : Il vérifie si la page est sous forme html ; identifie le contenu, le titre, les liens dans la page reỗue Il classifie fonctionnellement la page

Agent extracteur : Il classifie sémantiquement la page et puis, extraie des informations dans son propre domaine sur la page prétraitée

Agent stockeur : Il sauvegarde les résultats classifiés et ceux extraits dans sa base des données

Agent recommandeur : Il reỗoit des pages classifiộes de l’agent extracteur et les envoie aux agents recommandeur des autres clusters

Ces agents sont décrits en détail dans la section 3.1.3

Figure 9 : Les agents et les protocoles dans SSE

Entre les agents du SSE, il y a les protocoles suivants (le protocole est nommé par le nom d’agent transmetteur plus le nom d’agent rộcepteur plus ô protocole ằ):

Super_Prepa_Protocole : Il a lieu entre l’agent superviseur et l’agent préparateur dốs que l’agent superviseur reỗoit une page de agent recherche

The Prepa_Stock_Protocole occurs between the preparatory agent and the storage agent when, after preprocessing, there are invalidated pages, or after classification, there are pages categorized as Message or List.

Prepa_Extrac_Protocole : Il a lieu entre l’agent préparateur et un agent extracteur

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies si, après avoir classifié, il y a des pages sous forme html

Extrac_Stock_Protocole : Il a lieu entre un agent extracteur et l’agent stockeur dès que l’agent extracteur termine sa classification ou son extraction

Extrac_Recom_Protocole : Il a lieu entre un agent extracteur et l’agent recommandeur si, après avoir classifié, il y a des pages à recommander

Recom_Recom_Protocole : Il a lieu entre l’agent recommandeur d’un cluster et celui d’autre cluster dốs que le transmetteur reỗoit une page recommandộe d’un agent extracteur du même cluster

Recom_Extrac_Protocole : Il a lieu entre l’agent recommandeur et un agent extracteur dốs que l’agent recommandeur reỗoit une page recommandộe de l’autre agent recommandeur de l’autre cluster

Ces protocoles vont être décrit en détaillé dans la section 3.1.4

This section will provide a detailed overview of the internal behavior of each agent within the SSE, focusing specifically on the first three agents: the Preparer, Extractor, and Stocker The Preparer agent plays a crucial role in the overall system.

L’agent préparateur a trois tâches principales :

Vộrifier si la page reỗue est validộe Pré-traiter la page si elle est validée

Classifier fonctionnellement la page pour détecter si la page est de type html, ou message ou liste

The preparatory agent operates through a series of defined steps Initially, it enters a waiting state to receive pages from the supervisory agent, remaining inactive to optimize system resources Upon receiving a page, it sends a confirmation of receipt and transitions to the validation state, where it checks if the page is validated If validated, it moves to the preprocessing state; if not, it sends the invalid page to the storage agent and returns to an active waiting state for confirmation of receipt During preprocessing, the agent identifies the content, title, and links on the page before advancing to the classification state, where it determines if the page is in HTML format If it is, the HTML page is sent to the extractor agent, followed by a return to active waiting for confirmation If not, the page is sent to the storage agent, and the agent again enters the active waiting state, where it remains until confirmation of receipt is received, as the waiting period is relatively short.

L’agent extracteur s’occupe de deux tâches principales :

Classifier sémantiquement la page html pour détecter si son contenu appartient à quel domaine

Extraire des informations sur la page si elle appartient à son propre domaine

The extractor agent operates in several key stages Initially, it enters a waiting state to receive HTML pages from the preparer agent, remaining inactive to conserve system resources Upon receiving a page, it sends a confirmation and moves to the classification state, where it semantically categorizes the HTML content to determine if it falls within its domain If the content is relevant, the agent transitions to the extraction state; if not, it forwards the page, along with relevant rules and classes, to both the recommender and storage agents before returning to an active waiting state for confirmation During the extraction phase, the agent retrieves information from the page and, if it pertains to its domain, sends it to the storage agent Otherwise, it relays the extracted information, along with pertinent rules and classes, to the recommender and storage agents Finally, it enters an active waiting state, where it awaits confirmation of receipt, as the waiting period is relatively short.

L’agent stockeur s’occupe de sauvegarder les résultats qui viennent de l’agent préparateur ou l’agent extracteur :

Une page invalidée venant de l’agent préparateur Une page de type message ou liste venant de l’agent préparateur Des informations extraites par l’agent extracteur

The storage agent operates through several key stages: Initially, it enters an idle state to await HTML pages sent by the preparation agent, optimizing system resources during this inactive waiting period Upon receiving an invalidated page, a message page, or a list page from the preparation agent, it sends a confirmation of receipt before transitioning to the invalidated storage state In this state, the agent stores the invalidated page in the database and returns to the idle state When it receives extracted information or recommended information from the extraction agent, it again confirms receipt and moves to the results storage state, where it saves the extracted or recommended information to the database before returning to the idle state.

Cette section va présenter en détaillé les protocoles dans les trois agents : Préparateur, Extracteur et Stockeur a Protocole Prepa_Extrac_Protocole

Il a lieu entre l’agent préparateur et un agent extracteur si, après avoir classifié, il y a des pages sous forme html

Figure 13 : Protocole Prepa_Extrac_Protocole

L’agent préparateur : transmetteur L’agent extracteur : récepteur Les informations échangées :

De l’agent préparateur à l’agent extracteur : une page html validée (VALID_PAGE)

De l’agent extracteur à l’agent préparateur : une confirmation (CONFIRMATION) b Protocole Prepa_Stock_Protocole

Il a lieu entre l’agent préparateur et l’agent stockeur si, après avoir classifié, il y a des pages invalidées après avoir prétraité ou des pages de type Message, Liste

Figure 14 : Protocole Prepa_Stock_Protocole

L’agent préparateur : transmetteur L’agent stockeur : récepteur Les informations échangées :

From the preparatory agent to the storage agent, there are three types of messages: an invalid HTML page (INVALID_PAGE), a message-type page (MESSAGE), and a list-type page (LIST).

De l’agent stockeur à l’agent préparateur : une confirmation (CONFIRMATION) c Protocole Extrac_Stock_Protocole

Il a lieu entre un agent extracteur et l’agent stockeur dès que l’agent extracteur termine sa classification ou son extraction

L’agent extracteur : transmetteur L’agent stockeur : récepteur

Figure 15 : Protocole Extrac_Stock_Protocole

From the extractor agent to the storage agent, there are three types of messages: extracted information (INFORMATION), a recommended page along with rules and classes (RECOMMENDED_PAGE), and recommended information accompanied by rules and classes (RECOMMENDED_INFOR).

De l’agent stockeur à l’agent préparateur : une confirmation (CONFIRMATION).

Conception

JADE [14] est un outil pour développer le système multiagent Il contient :

Un environnement sur lequel les agents de JADE vivrent activement sur un hôte particulier

Une bibliothèque des classes qu’on peut utiliser pour développer de nouveaux agents

Une chaợne des outils graphiques qui permettent de gộrer les agents en temps marché

JADE a des termes importants (figure 16):

A container serves as a JADE environment hosted on a machine, where agents reside Within this setup, there is a Main Container that oversees the management of other containers in the system.

Platform : Un ensemble des containers dans un même système Dans un Platform, il n’y a qu’un seul Main Container

AMS : Agent Management System C’est un agent au niveau de système qui fournit le service nommé

DF : Directory Facilitator C’est un autre agent au niveau de système qui aide aux agents à chercher leurs partenaires

Les agents : ce sont des agents qui sont développés par les développeurs Ils sont placés dans les Containers dans le système

Figure 16 : Platform et Container de JADE (source [14])

Généralement, un agent dans JADE a des états [14] (figure 17) :

Initial : l’objet d’agent est né, mais il n’est pas encore inscrit à l’agent AMS Il n’a pas encore le nom et l’adresse Il ne peut pas encore communiquer avec les autres

Actif : l’objet d’agent est activé Il est inscrit à l’agent AMS Il a son propre nom et son adresse

Suspende : l’objet d’agent s’est arrêté Ses comportements ne sont plus réalisés

Attend : l’objet d’agent est bloqué pour attendre quelques choses Le thread Java est en dorme

Supprime : l’objet d’agent est vraiment supprimé Il n’existe plus dans le système

Transfert : un agent mobile se déplace à la nouvelle place

Figure 17 : La vie d’un agent de JADE (source [14])

De côté de codage, un agent contient des classes concernées :

Une classe agent qui inhérente la classe Agent de JADE

Un ensemble des classes qui héritent des classes Behavior qui implémentent les comportements des agents b Jess : Java Expert System Shell

Jess, developed by Sandia National Laboratories in the late 1990s, is a rule-based programming language embedded in Java, designed for applications requiring rule inference.

Jess contient des termes importants :

Rule : une règle qui est sous forme ( si alors ) Il représente une règle d’inférence qui est grillée par le moteur d’inférence de Jess

Template : une définition d’un fait par un ensemble des slots (attributs) dans la base de faits Elle est correspondante aux classes de type bean de Java

Instance : une instance concrète d’une template Elle est correspondante aux objets de Java

Slot : une représentation d’un attribut d’une template (ou instance) Elle est correspondante aux attributs d’un objet de Java

Jess can be embedded in Java, allowing a Jess program to access Java variables and objects Conversely, a Java program can also interact with Jess facts, including templates, instances, and instance slots.

Le langage de Jess est de type déclaratif Le code de Jess peut être exécuté sans recompilation

Le moteur d’infộrence de Jess marche en se basant sur le principe ô pattern matching ằ

Un programme Jess peut être exécuté dans un programme Java selon un script suivant :

Créer un objet de type Rete Charger la base des faits et la base des règles Transformer les variables initiales

Appeler le commande Rete.run() Récupérer les variables nécessaires de la sorties

Il y a dix classes principales (figure 18):

Les classes pour prétraiter fonctionnement la page Html : PaginaWWW, TextHtml, PageHtml, Termo

Les classes sont utilisées dans des règles : PageProcessor, PageProce

Figure18 : Les classes de MasterWeb

Une classes pour charger des pages : CorpusLoading

Une classe pour lancer les agents dans JADE : LaunchedAgent

Les classes de agent CFP : AgentCFP, PageProcessor_Behaviour, PageAnalyze_Behaviour

Pour projeter de MasterWeb à AGATHE, on ne modifie que les classes de agent CFP : AgentCFP, PageProcessor_Behaviour, PageAnalyze_Behaviour Le reste est gardé le même dans Masterweb

On divise le travail de agent CFP aux ceux des cinq agents dont les trois agents principaux (figure 19): agent Préparateur : Preparateur, PrepaProcessor_Behaviour, PrepaAnalyze_Behaviour

Figure19: Les classes de AGATHE

This report focuses on web information collection in specialized domains using agents and ontologies The Extractor agent is responsible for data extraction and processing, utilizing ExtraProcessor_Behaviour and ExtraAnalyze_Behaviour Meanwhile, the Stocker agent manages data storage and analysis through StockProcessor_Behaviour and StockAnalyze_Behaviour.

Ces agents sont implémentés en détaillé dans les sous sections suivantes, les prototypes sont présentés en annexe C a Préparateur

Agent Préparateur contient trois classes (figure 20) : Préparateur, PrepaProcessor_Behaviour et PrepaAnalyze_Behaviour

The Preparateur class, which inherits from the Agent class in JADE, includes several key methods: the setup() method, which overrides the Agent class's setup method to create an instance of the PrepaProcessor_Behaviour; the openDB() method, responsible for opening the database and invoked by the setup method; and the openRete() method, which initializes the rule execution engine in Java, also called by the setup method.

The PrepaProcessor_Behaviour class, which inherits from the CycleBehaviour class of JADE, features a single method: action() This method awaits pages sent by the Supervisor agent and creates an instance of the PrepaAnalyze_Behaviour class upon receiving a page.

La classe PrepaAnalyze_Behaviour qui hérite de la classe OneShotBehaviour de JADE, a trois méthodes : action() : Elle traite la page reỗue en appelant la mộthode processInfo

Préparateur : Agent setup() openDB() openRete() PrepaProcessor : CyclicB action()

PrepaAnalyze : OneShotB action() verifyDB() processInfo()

Figure20: Les classes de Préparateur

The internship report focuses on gathering information from the web related to specific domains using agents and ontologies The processInfor() method is utilized to analyze the functionality of a webpage, while the verifyDB() function checks whether the page has been processed Additionally, an extractor is employed to facilitate data retrieval.

Agent Extracteur contient trois classes (figure 21) : Extracteur, ExtraProcessor_Behaviour et ExtraAnalyze_Behaviour

The Extractor class, which inherits from the JADE Agent class, includes the setup() method, a redefined function from the Agent class in JADE that creates an instance of the ExtraProcessor_Behaviour Additionally, it features the openRete() method, responsible for launching a rule engine in Java, which is invoked by the setup method.

The ExtraProcessor_Behaviour class, which inherits from the CycleBehaviour class in JADE, contains a single method: action() This method waits for pages sent by the Preparateur agent and creates an instance of the ExtraAnalyze_Behaviour class as soon as a page is received.

The ExtraAnalyze_Behaviour class, which inherits from the OneShotBehaviour class in JADE, includes four key methods The action() method processes the received page by invoking the processInfo method The processInfo method is responsible for semantically analyzing a page, while the treatLinksToSend method checks if the page is recommended.

Extracteur : Agent setup() openRete() ExtraProcessor : CyclicB action()

ExtraAnalyze : OneShotB action() treatLinksToSend() processInfo() extractRelevInform()

Figure21: Les classes de Extracteur

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies extractRelevInform() : extrait des infromations sur la page c Stockeur

Agent Stockeur contient trois classes (figure 22) : Stockeur, StockProcessor_Behaviour et StockAnalyze_Behaviour

The Stockeur class, which inherits from the JADE Agent class, includes the setup() method, an overridden function that creates an instance of the ExtraProcessor_Behaviour class Additionally, it features the openDB() method, responsible for opening the database, which is invoked by the setup method.

The StockProcessor_Behaviour class, which inherits from the CycleBehaviour class in JADE, contains a single method: action() This method waits for results sent by the Preparer agent or Extractor agents Upon receiving a page, it creates an instance of the StockAnalyze_Behaviour class.

The StockAnalyze_Behaviour class, which inherits from the OneShotBehaviour class in JADE, includes several key methods The action() method processes the received page by invoking one of the following functions: saveURLwError() for retaining error pages, saveURLbadLink() for storing irrelevant pages, saveStatistics() for saving statistical information, and saveNotRec() for archiving pages with processing errors.

Stockeur : Agent setup() openDB() StockProcessor : CyclicB action()

StockAnalyze : OneShotB action() saveURLwError() saveURL() saveURLbadLink() saveNotRec() saveStatistics()

Figure22: Les classes de Stockeur

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies saveURL() : sauvegarde les informations extraites

The exchanged messages are created according to the FIPA standard, incorporating essential attributes such as performative, conversationID, and content For instance, a message from the Preparer agent to the Extractor agent, indicating that a page has been validated, includes the following attributes: performative: INFORM, conversationID: VALID_PAGE, and content: an instance of the ValidPage class (detailed in Annex C).

En se basant sur la base de données de MasterWeb, on garde les tables : cfp_error_notrec, cfp_preference, cfp_profile, cfp_queries, cfp_statistics, cfp_engines

Figure 23 : Les relations dans la BD

We are updating the three tables cfp_rec, cfp_errors, and cfp_notrec by replacing the attribute 'url' with 'url_id' This change is being implemented to facilitate the addition of a new table dedicated to managing URLs.

On ajoute les nouvelles tables : la table cfp_pages pour garder les composants découpérés de la page La table cfp_infos pour garder les informations extraites La table

Mise en œuvre

Le domaine de teste : événement scientifique (dont un cluster)

Le nombre de pages testées : 310

La plateforme de teste : JADE 3.4, Eclipse 3.1, Java SDK 1.5.1 L’environnement distribué : réseau local (100Mb/s),

L’ontologie utilisé : CFP (Call for papers), il est manipulé sous Protégé

Le moteur d’inférence des règles : Jess

L’ensemble des agents : un Préparateur, un Extracteur, un Stockeur (du cluster événement scientifique) Chaque agent est lancé sur une machine différente

Le rộsultat reỗu par la phase de classification fonctionnelle :

Les résultats de AGATHE et de MasterWeb sont identiques car tous les deux systèmes prennent le même processus d’extraction, la même base de faits, la même base de

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies règles, la même base de teste

In theory, AGATHE can process multiple agents simultaneously, while MasterWeb relies on a single extraction agent to handle the entire extraction process Consequently, we anticipate that AGATHE will operate more quickly than MasterWeb.

The processing speed is not precisely defined, as the time taken differs significantly when multiple agents share a centralized server compared to when each agent operates on a distributed server Additionally, conflicts may arise between agents utilizing the inference engine if they are located on the same machine Currently, it remains challenging to determine processing time in a centralized setup.

Les informations extraites sont tenues par les instances de la classe Slot-Found qui a les slots suivants (figure 24) :

Figure 24 : Les slots de la classe Slot-Found

L’utilisateur s’intéresse aux slots principaux :

Slots-of-Web-Page: représente le slot de la classe Web-Page dans lequel l’instance a été trouvé

Values: reprộsente le contenu qui apparaợt dans la page

Start-Position: le point départ de la valeur

Slot-in-Process: représente le slot de la classe auquel l’instance corresponds

Here are some instances identified on a page using the scientific event domain ontology For example, instance 17, as shown in figure 26, contains the value "papers," which is categorized under the "Event" class.

The publication appears in the 8008th character slot of the Lowercase-Contents section within the Web-Page class The first line indicates that the page is classified as belonging to the Conference class, as shown in Figure 26.

Figure 25 : Une page à classifier et à extraire

… Extracteur : ========================= instance :17 :Slots-of-Web-Page: Slot(Lowercase-Contents)

:Start-Position: 8008 :Slot-in-Process: Slot(Event-Publication)

… Extracteur : ========================= instance :23 :Slots-of-Web-Page: Slot(Lowercase-Contents)

:Start-Position: 4932 :Slot-in-Process: Slot(Deadline)

Extracteur : ========================= instance :30 :Slots-of-Web-Page: Slot(Lowercase-Contents)

:Start-Position: 764 :Slot-in-Process: Slot(Program-Committee)

Figure 26 : Le résultat de classification et d’extraction

The initial version of the AGATHE system is restricted to a single domain for extraction, utilizing a topology consisting of a Preparer, an Extractor, and a Storage unit Future developments aim to expand the system's capabilities.

Valider le système avec plusieurs extracteurs, plusieurs préparateurs dans un même cluster

Réaliser la partie d’extraction avec tous les cinq types d’agent dans le SSE

Réaliser le système avec tous les trois sous systèmes : SSR, SSE et SSU

Appliquer le système à plusieurs domaines différents (plusieurs clusters différents)

En particulier, résoudre le problème de conflit quand plusieurs agents utilsant le moteur d’inférence sont miss sur la même machine.

Conclusion

This chapter introduced the initial version of the MasterWeb system projection within the AGATHE architecture It is restricted to the extraction phase focused solely on the scientific event domain, specifically the "Call for Papers" agent, which is central to information systems (IS).

Déterminé et analysé le problème de projeter de l’agent d’extraction de MasterWeb au sous système d’extraction de AGATHE

Réalisé le système, en particulier le sous système d’extraction de AGATHE dont les trois agents Préparateur, Extracteur et Stockeur sont priorités, sur la platforme JADE

The current system is limited to three agents within the SSE and has yet to be validated In the future, it has the potential to expand to multiple agents capable of parallel processing within the same cluster, encompassing all agents in the SSE, all three subsystems of AGATHE, and multiple clusters corresponding to various domains.

The challenge of Information Extraction (IE) is increasingly complex in the context of the Internet, where information proliferates rapidly and sources are diverse These sources are generated by various individuals for different purposes and under varying structures To address this issue, several approaches have been proposed Typically, explicit IE on the web involves two main steps: region extraction and data extraction from those regions Region extraction can be approached through natural language processing, user-defined semantic markers, or HTML page structure The HTML structure approach has three trends: utilizing XML and XSLT techniques, leveraging the Document Object Model (DOM), and employing Web ontologies The second step, data extraction from regions, also has three trends: applying syntactic techniques, using semantic techniques based on domain ontologies, or utilizing agents Systems like MasterWeb and AGATHE exemplify tools that extract information from specific web domains using agents and ontologies.

MasterWeb is an extraction information system (EI) that relies on a single agent to manage all stages of the extraction process In contrast, AGATHE proposes a multi-agent architecture where each agent handles a specific task within the extraction chain The main challenge lies in adapting the AGATHE architecture to the MasterWeb system, specifically by redistributing the extraction responsibilities from the MasterWeb agent to the various agents within AGATHE's extraction subsystem.

This document outlines the implementation of AGATHE using the MasterWeb system on the JADE platform It details the identification of each agent's role within AGATHE's SSE in relation to MasterWeb, the division and modification of the extraction agent's coding in MasterWeb to align with the agents in AGATHE's SSE, the establishment of concrete protocols among the agents in SSE, and the adaptation of the database to fit the new architecture.

The current system is limited to three SSE agents and has not yet been validated In the future, it will undergo validation and can be expanded to accommodate multiple agents working in parallel within the same cluster This expansion will include all agents in the SSE, all three subsystems of AGATHE, and multiple clusters corresponding to various domains.

[1] J Ambite and C Knoblock Agents for information gathering In Software Agents Bradshaw, J (ed.), MIT Press, Pittsburgh, PA, USA, 1997

[2] P Buitelaar, P Cimiano and S Racioppa Ontology-based Information Extraction with SOBA In: Proc of LREC06, Genoa, Italy, May 2006

[3] M Banko, M.J Cafarella, S Soderland, M Broadhead and O Etzioni Open Information Extraction from the Web Proc of the International Joint Conference on Artificial Intelligence (IJCAI), 2007

[4] I Bouslimi, K Ghedira and C Hanachi An agent-based organizational model for cooperative information gathering In : Signal-Image Technology & Internet Based Systems (SITIS 2006), Hammamet, Tunisia, 17/12/06-21/12/06, IEEE, p 414-424, décembre 2006

BONOM is a multi-agent system designed for knowledge-driven information retrieval on the Internet This innovative approach was presented at the Francophone Days of Knowledge Engineering (IC'2000) held in Toulouse from May 10-12, 2000, highlighting its significance in enhancing information search capabilities.

[6] V Crescenzi, G Mecca and P Merialdo Roadrunner: Towards automatic data extraction from large Web sites Technical Report n RT-DIA-64-2001, D.I.A., Universit a di Roma Tre, 2001

[7] B Espinasse, F Freitas and S Fournier AGATHE: an Agent and Ontology based System for Restricted-Domain Information Gathering on the Web IEEE-RCIS, Research Challenges in Information Sciences, 23-26 April, Ouarzazate, Marocco, 2007

[8] L Eikvil Information extraction from the world wide Web: a survey Technical Report 945, Norwegian Computing Center, 1999

[9] H Eriksson Using JessTab to Integrate Protégé and Jess IEEE Intelligent Systems, 18(2):43-50, 2003

[10] F Freitas and G Bittencourt An Ontology-Based Architecture for Cooperative Information Agents IJCAI, 2003

[11] N Georgiev, J.M Labat, J.L Minel and L Nicolas Extraction de données à partir de pages HTML par création semi-automatique de règles XSLT Congrès IC 2003, p117-

[12] H Guo and A Stent Taxonomy Based Data Extraction from Multi-item Web Pages Workshop on Web Content Mining with Human Language Technologies at the ISWC'06,

[13] D Hérin, B Espinasse, E Andonoff and C Hanachi Des systèmes d’information coopératifs aux agents informationnels In Ingénierie des systèmes d’information, C.Cauvet, C.Rosenthal-Sabroux, Hermès 2001

[14] JADE, JADE tutorial http://jade.tilab.com

[15] JESS, http://hezberg.casandia.gov/jess

[16] S.W Liddle, D.W Embley, D.T Scott and S.H Yau Extracting Data Behind Web Forms

Proceedings of the Workshop on Conceptual Modeling Approaches for e-Business,

[17] B Lui, R Grossman and Y Zhai Mining Data Records in Web Pages Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD-2003), Washington, DC, USA, August 24 - 27, 2003

[18] S.W Liddle, K.A Hewett and D.W Embley An Integrated Ontology Development

Environment for Data Extraction Proceedings of Information Systems Technology and its Applications, International Conference ISTA'2003, Kharkiv, Ukraine, June 19-

[19] K Lerman, C Knoblock and S Minton Automatic data extraction from lists and tables in

Web sources In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-

[20] J.P Lage, A.S da Silva, P.B Golgher and A.H.F Laender Automatic generation of agents for collecting hidden Web pages for data extraction Data Knowl Eng 49(2): 177-196 (2004)

In their 2001 paper presented at the International Workshop on Data Semantics in Web Information Systems in Yokohama, S.W Liddle, S.H Yau, and D.W Embley explore the automatic extraction of data from the Hidden Web This research addresses the challenges of accessing and utilizing data that is not readily available through conventional search engines.

[22] L McDowell and M Cafarella Ontology-driven Information Extraction with OntoSyphon Fifth International Semantic Web Conference (ISWC 2006), November 2006

In their 2003 paper presented at the International Conference on Parallel and Distributed Processing Techniques and Applications in Las Vegas, L Ma, N Goharian, and A Chowdhury discuss innovative methods for automatic data extraction from web pages generated by templates Their research highlights the significance of improving data retrieval processes in the context of web technologies.

[24] X Meng, D Hu and C Li Schema-guided wrapper maintenance for Web-data extraction Fifth ACM CIKM International Workshop on Web Information and Data Management (WIDM 2003), New Orleans, Louisiana, USA, November 7-8, 2003

[25] J Myllymaki Effective Web data extraction with standard XML technologies In Proc of the Tenth International World Wide Web Conference, WWW10, pp 689 696, Hong Kong, China, May 2001

[26] S.A O’Malley and S.A DeLoach Determining When to Use an Agent_Oriented Software Engineering Paradigm Proceedings of the Second International Workshop On Agent- Oriented Software Engineering(AOSE 2001), Montreal, Canada, May 29th 2001

[27] R Okada, E-S Lee and N.Shiratori Multimedia Information Gathering based on a Society of Cooperative Agents International Conference on Multimedia and Networking (MmNe'96), pp.86-95, Oct, 1995

[28] H Ouahid and A Karmouch An XML-Based Web Mining Agent Proceeding of MATA’99, Ahmed KARMOUCH and Roger IMPEY eds., World Scientific, Ottawa, 1999

[29] B Omelayenko Learning of ontologies for the Web: the analysis of existent approaches

In Proceedings of the International Workshop on Web Dynamics, 2001

[30] H.D Pfeiffer and R.T Hartley Visual CP Representation of Knowledge In Supplimental

Proceedings of the 8th International Conference On Conceptual Structures, Darmstadt, Germany, 2000

[31] B Popov, A Kiryakov, D Ognyanoff, D Manov, Al Kirilov and M Goranov Towards semantic Web information extraction In proceedings of ISWC, Sundial Resort, Florida, USA, October, 2003

[32] M Saywell Negotiation Agents: An Overview In: Multimedia Systems Coursework, Dept of Electronics and Computer Science, UK, 2000

[33] H Snoussi, L Magnin and J-Y Nie Heterogeneous Web data extraction using ontology The AI-2002 Workshop on Business Agents and the Semantic Web, Calgary, may 2002

In their paper, "Toward an Ontology-based Web Data Extraction," authors H Snoussi, L Magnin, and J-Y Nie discuss innovative methods for extracting web data using ontological frameworks This research was presented at the AI-2002 Workshop on Business Agents and the Semantic Web (BASeWEB) during the AI-2002 Conference held in Calgary, Alberta, Canada, on May 26, 2002.

[35] M Sintek, M Junker, L.V Elst and A Abecker Using Information Extraction Rules for Extending Domain Ontologies Position Statement for the IJCAI-2001 Workshop on Ontology Learning, 2001

[36] L Sofiane and N.H Perrreira Mobile Agents for Information Extraction in the MathNet System In: International Conference Frontieres in Education, Boston, 2002

[37] O Svab, V Svatek and M Labsky Types and Roles of Ontologies in Web Information Extraction In: ECML/PKDD04 Workshop on Knowledge Discovery and Ontologies, Pisa,

[38] K.P Sycara Multiagent Systems AI Magazine, vol 10, n°2, pages 79-93, 1998

F Villa presented a case study on declarative modeling aimed at achieving architecture independence and enhancing data/model integration This research was showcased at the 3rd Biennial meeting of the International Environmental Modelling and Software Society (iEMSs 2006) held at The Wyndham Hotel in Burlington, Vermont, USA, on July 9.

[40] A Wessman, S.W Liddle and D.W Embley A Generalized Framework for an Ontology-

Based Data-Extraction System Information Systems Technology and its Applications, 4th International Conference ISTA'2005, Palmerston North, New Zealand, 23-25 May,

[41] M Wooldridge and N.R Jennings Inteliigent Agents: Theory and Practice The

[42] L Xu, D.W Embley and Y Ding Combining Declarative and Procedural Knowledge to

Automate and Represent Ontology Mapping SemWAT 2006 - 1st International Workshop on Semantic Web Applications: Theory and Practice, April 2006, pp 395-

In their 2005 paper presented at the 7th International Conference on Information Integration and Web-Based Applications & Services (iiWAS2005) in Kuala Lumpur, Malaysia, T Xiaohui, K John, and L Yuefeng explore information fusion through a subject-based information gathering method tailored for intelligent multi-agent models The research, detailed on pages 861-869, emphasizes the significance of integrating diverse information sources to enhance the functionality and efficiency of multi-agent systems.

[44] L Yi, B Lui and X Li Eliminating Noisy Information in Web Pages for Data Mining

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD-2003), Washington, DC, USA, August 24 - 27, 2003

Les messages

Départ de l’agent Préparateur à l’agent Stockeur performative : INFORM conversationID : INVALID_PAGE contenu : une instance de la classe InvalidPage

Départ de l’agent Préparateur à l’agent Stockeur performative : INFORM conversationID : BAD_PAGE contenu : une instance de la classe BadPage

Départ de l’agent Préparateur à l’agent Stockeur performative : INFORM conversationID : REJECTED_PAGE contenu : une instance de la classe RecommandedPage

Départ de l’agent Préparateur à l’agent Extracteur performative : INFORM conversationID : VALID_PAGE contenu : une instance de la classe ValidPage

Départ de l’agent Préparateur à l’agent Stockeur performative : INFORM conversationID : STATISTIC contenu : une instance de la classe Statistic

Départ de l’agent Extracteur à l’agent Stockeur performative : INFORM conversationID : INFORMATION contenu : une instance de la classe Information Message 7 : des informations d’une page Html

Départ de l’agent Préparateur à l’agent Stockeur performative : INFORM conversationID : RECOGNIZED_PAGE contenu : une instance de la classe Page

Message 8 : une confirmation performative : CONFIRM

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies conversationID : CONFIRMATION contenu : nulle // facultatif

Les tables de la BD

Cette table contient des attributs : url_id : number // l’identification de URL url_name : varchar // le nom complet de URL b cfp_pages

The table includes several attributes: `url_id`, which references `cfp_urls.url_id`; `host`, representing the host address; `port`, indicating the port number; and `protocol`, specifying the protocol used Additionally, it contains `content_type` for the type of content, `page_date` for the creation date, and `length` for the page body size Other attributes include `title` for the page title, `description` for the page description, and `meta_keyword` for the list of keywords The table also features `summary` for content summaries, `content` for the page content, and `content_without_tag` for the content stripped of markers Furthermore, it includes `content_without_line` for content without line breaks, `email` for emails found on the page, `term_in_sequence` for terms appearing in sequence, `link` for links present on the page, and `term` for terms that appear throughout the page.

This table includes several attributes: id, which is a number representing the object's identification; url_id, a number that references cfp_urls.url_id; start_position, a number indicating the starting point of the object; and content, which is a varchar that holds the object's content.

This table includes several attributes: agent_id, a number representing the identification of the agent; cluster_id, a number that references cfp_clusters.cluster_id; agent_type, a number indicating the type of agent; and name, a varchar field for the agent's full name.

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies adresse : varchar // l’adresse de l’agent e cfp_classes

Cette table contient des attributs : classe_id : number // l’identification de la classe reconnue name : varchar // le nom de la classe e cfp_clusters

Cette table contient des attributs : cluster_id : number // l’identification du cluster name : varchar // le nom du cluster f cfp_cluster_class

Cette table contient des attributs : cluster_id : number // référence vers le cfp_clusters.cluster_id class_id : number // référence vers le cfp_class.class_id

Prototypes

Préparateur public class Preparateur extends Agent{ protected void setup();

Connection openDB(); void openRete(String fileName); // le nom du fichier contenant des règles } public class PrepaProcessor_Behaviour extends CyclicBehaviour{ public void action();

The `PrepaAnalyze_Behaviour` class extends `OneShotBehaviour` and includes a method called `action()` It processes information through the `processInfo` method, which takes parameters such as the source number, source identification, page URL, and page parameters Additionally, it features a `verifyDB` method to validate the database.

String url, // l’adresse de la page URLConnection ulc); // la connexion ver la base des données }

Extracteur public class Extracteur extends Agent{ protected void setup();

Rapport du stage : Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies void openRete(String fileName); // le nom du fichier contenant des règles

} public class ExtraProcessor_Behaviour extends CyclicBehaviour{ public void action();

The ÈxtraAnalyze_Behaviour` class extends ÒneShotBehaviour` and includes the àction` method It processes information through the `processInfo` method, which takes parameters such as the source number, source ID, URL, and page parameters Additionally, the `treatLinksToSend` method handles the classification of links by utilizing the source ID, semantic classification status, and main fact information.

Fact mainFact) ; // les informations classifiées }

Stockeur public class Stockeur extends Agent{ protected void setup();

} public class StockProcessor_Behaviour extends CyclicBehaviour{ public void action();

} public class StockAnalyze_Behaviour extends OneShotBehaviour{ public void action(); public void saveURLwError(

URLConnection ulc, // la connexion ver la BD String url, // l’adresse de la page Int codErr); // le type d’erreur public void saveURLbadLink(

The code snippet defines a method to save statistics, which includes parameters for the source number, source identification, result type, and the cause of the result Additionally, it includes a string URL for the page address and an integer for the error type This method is essential for tracking and managing data related to various sources and their outcomes.

URLConnection ulc, // la connexion ver la BD String url, // l’adresse de la page Int codErr); // le type d’erreur public long saveURL(

URLConnection ulc, // la connexion ver la BD String url, // l’adresse de la page Vector infRelev); // les informations extraites }

C.2 Prototype de contenu des message

Cette section va présenter les classes qui encapsulent les informations échangées entre les agents

HttpURLConnection ulc; // la connexion vers la page

String nameUrl ; // l’adresse url de la page int constat ; // le type d’erreur

String nameUrl, // l’adresse url de la page int constat // le type d’erreur

String nameUrl, // l’adresse url de la page int rule // la règle utilisée ou recommandée

Statistic { int source; // la source de la page int idSource; // l’identification de la source int status; // l’état de la page int constat; // le type de résultat

String url; // l’url de la page

String nameUrl; // le nom complet de la page int idSource; // l’identification de la source

Page pageContent; // le contenu de la page

Tiêu đề	Collecte d’information sur le Web relatif à des domaines restreints, à base d’agents et d’ontologies
Tác giả	Nguyen Manh Hung
Người hướng dẫn	Bernard Espinasse, Sébastien Fournier, Norbert Giambiasi
Trường học	Institut de la Francophonie pour l’Informatique
Thể loại	rapport de stage
Năm xuất bản	2007
Thành phố	Marseille

Định dạng
Số trang	54
Dung lượng	1,07 MB