Information retrieval and web search = truy xuất thông tin và truy cập web

BO GIAO Dl}C vA \ DAo T~O TRUONG D~I HQC NGO~I NGU - TIN HQC TP.HCM KHOA CONG NGH:E:THONG TIN -cgm~ - KHOA LuAN TOT NGHIEP • • INFORMATION RETRIEVAL AND WEB SEARCH GIANG VrEN HU6NG DAN: PGS.TS LE HoAI BAc SINH VIEN THVC HI:E:N: • NGUYEN vO LONG MSSV: 08DH11113 NGUYEN HUU ,LV MSSV: 08DH11240 /' ',/"'"_ :"4"-_ - :,':' , "::~ ~~ 1: F i"" " ~ It ~- • ": i-J;~ ~;{""f ",-', i ~'1".~ \''C ~'L' • , , "' •, - "TP.H~\" 1\ :- ••• ~ ~ Ii ••••••• -/" t A, ,'' ,./ " ",,-\ N' - - 08/2012 M a £)Au 1~ONG QUAN Chuang NHUNG KHAI NI$M Cd BAN VB THU THAI> THONG TIN 1.1 Tu kh6a troy vdn (Keyword queries) 1.2 Troy vdn logic (Boolean queries) 1.3 M~nh d€ troy vdn (Phrase queries) 1.4 Troy vdn gfin (Proximity queries) 1.5 Toim tai li~u (Full document require) 1.6 Cau h6i ngon ngu tlJ nhien (Natural language question) Chuang cAc MO HlNH TRICH XUAT DO' LI$U 2.1 Mo hinh Boolean 2.2 Mo hinh khong gian vector 2.3 Mo hinh ngon ngu th6ng ke 12 Chuang MUC £) MQt spammer eling eo thS su d\lng eae do~n rna dS ~n mQt s6 cac y@ut6 hinh anh tren trang, vi d\l, b~ng cach thi@tl~p thuQc tinh hiSn thi HTML la false Che d~y: Spam may chu web dS tni vS mQt tai li~u HTML dS nguoi su dvng va cac tai li~u khac thu th~p thong tin web B~g cach nay, spammer co thS trinh bay eho nguoi su d\lng web vai nQi dung va gui mQt trang thu rae d@ncac cong C\ltim ki@ml~p chi m\le Spam may chu eo thS xae dinh eac trinh thu th~p theo mQt hai cach: WEB GOl THV RAe 73 No tri danh sach cac dia chi IP cua cong C\ltim kiSm va xac dinh trinh thu th~p thong tin b~ng cach kSt hQ'pdia chi IP No xac dinh cac trinh duy~t web d\la tren truo-ng d~i di~n cac yeu c~u thong bao HTTP Vi d\l, ten d~i di~n Clla yeu c~u thong bao HTTP duai day duQ'c su d\lllg boi trinh duy~t Microsoft Internet Explorer 6: GET /pub/WWW/TheProject.html HTTP/l.l Host: www.w3.org User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Nhfrng ten d~i di~n khong chm1n, vi v~y dS tm cac yeu c~u tmg d\lllg bao g6m truemg tin nh~n tuang tmg Tuy nhien, cac trinh thu th~p cong C\ltim ki~m thucmg xac dinh minh b~ng ten rieng bi~t tll' cac trinh duy~t web binh thucmg dS cho phep nQi dung t6t va t6i uu hoa hqp phap Vi d\l, mQt s6 trang web ph\lc V\l cong C\l tim kiSm mQt phien ban cua cac trang cua hQ 1£1 chuySn huang lien kSt va quang cao S\f chuy~n huo-ng: Nguai gui thu rac cling co thS fin mQt trang spam b~ng cach t\l dQng chuySn huang trinh duy~t dSn mQt URL khac sau trang web duQ'ctai Nhu v~y, cac trang spam cho cac cong C\l tim kiSm dS l~p danh m\lC (nguai su d\lng se khong bao gia nhin thciy), va cac trang m\lc tieu duQ'ctrinh bay cho nguai su d\lng web thong qua S\l chuySn huang MQt cach dS d~t duQ'cchuySn huang 1£1 su d\lng "lam mai" meta-tag, va thiSt l~p thai gian lam mai cho tai muc b~ng O.MQt cach khac 1£1 su d\lng cac kich ban 10.4 Chang thu rae MQt s6 ho~t dQng gui thu rac, gi6ng nhu chuySn huang b~ng cach su d\lllg meta-tags, rc1tdS dS phat hi~n Tuy nhien, chuySn huang b~ng cach su d\lng kich ban rcit kho dS xac dinh boi vi trinh thu th~p cong C\l tim kiSm khong th\lc hi~n kich ban DS ngan ch~n che d~y, cong C\l tim kiSm thu th~p thong tin co thS xac dinh chinh no nhu la mQt trinh duy~t Web thuemg xuyen WEB GUr THV RAe 74 Sir d\ffig cac di~u khoan cua van ban neo cua cac lien k@ttra d@nmQt trang chi s6 trang co thS ch6ng l~i nQi dung thu rac d@nmuc dQ nao d6 b61 vi van ban neo tit cac trang khac la dang tin C?y han Phuang phap ban dfru duQ'c ling hQ d~t cho cac trang chi m\lc rna khong duQ'cl~y boi trinh thu th?p cong C\l tim ki@m Bay gia, mQt ky thu?t chung duQ'c su d\lng boi cac cong C\l tim ki@mnhu chung ta c6 thS nhin th~y chuang 8, vi d\l nhu cong C\l tim ki@mcho cac di~u khoan van ban neo co trQng lugng cao han Trong th\fc t@,cac tit ngu gfrn mQt phfrn cua van ban neo cung cung c~p S\f dang gia bien t?P t6t v~ trang m\lc tieu Cac thu?t toan x@plo~i trang co thS dS ch6ng l~i nQi dung thu rac d@n mQt muc dQ tri nao d6 boi vi n6 duQ'c d\fa tren cac lien k@ttro d@ncac trang m\lc tieu, va nhfrng trang d6 d@n cac trang m\lc tieu cfrn phai co uy tin cung nhu diSm x@p lo~i trang cao han Tuy nhien, no khong d6i v6i lien k@tgiri thu rac d\fa tren cac phuang phap thao lu?n tren Thay vi ch6ng tUng lo~i ca nhan cua thu rac, rnQt phuang phap (gQi la TrustRank) duQ'c d~ xu~t dS ch6ng l~i t~t ca cac lo~i phuang phap giri thu rac cling mQt hk No co lQ'ith@cua S\f co l?p gfrn dung cua cac trang co ti@ngtam va khong thu rac va cac trang thu rac thuang lien k@tv6i nhi~u trang co uy tin (trong mQt S\f n6 l\fc dS cai thi~n diSm s6 trung tam cua hQ cac phuang phap phan tich lien k@t duQ'c su d\lng dS cac trang rieng bi~t co uy tin va b~t ki cac hinh thuc nao cua thu rac rna khong giai quy@tm6i ki thu?t gui thu rac mQt cach ca nhan Ch6ng thu rac cung co thS duQ'cxem nhu la rnQtv~n d~ phan lo~i, vi d\l nhu, d\f doan xem li~u mQt trang co thS la mQt trang thu rac hay khong Nguai ta co thS su d\lng b~t ky thu?t toan giarn sat nao dS t~o mQt chuang trinh phan lo~i thu rac V~n d~ quan trQng la tinh nang thi@t k@duQ'csu d\ffig vi~c hQc Sau day la rnQt s6 tinh nang vi d\l duQ'c su d\ffig dS phat hi~n nQi dung thu rac S6 tit trang: MQt trang thu rac co xu huang chua nhi~u tit han rnQt trang khong phai trang thu rac dS bao g6rn rnQt s6 lUQ'ngIan cac tit ph6 bi@n WEB Gill THU RAe 75 Chi@udai tu trung binh: chi@u dai tu trung binh cho van xuoi tiSng Anh la khoang chu cai Chi@udai tu trung binh cua n9i dung t6ng hgp thuemg khac S6 lugng tu tieu d@trang: boi vi c6ng C\l tim kiSm thong thuemg cung celp tr " "" { • • I • ; • ., I" , f ~ ".' \." ~: r, ",,~;~:' I,e'l I _'~' • the