In this thesis, Chapter 1 will introduce an overview of the research topic, present the reason for the birth of the PGN-LM model and introduce the problem of automatic heading.. Chapter
Word Embedding
Mô hình Skip-Gram
9ӟLP{KuQK6NLS-*UDPWDVӁVӱGөQJWӯPөFWLrXOjPÿҫXYjRYjÿҫXUDPRQJÿӧL OjWӯQJӳFҧQKÿӇKXҩQOX\ӋQPҥQJQHXUDO1KѭYұ\PӛLPүXVDPSOHKXҩQOX\ӋQVӁ OjPӝWFһSWӯPөFWLrXWӯQJӳFҧQKYjWұSGӳOLӋXKXҩQOX\ӋQFӫDWDVӁEDRJӗPWҩW FҧFiFFһSWӯWURQJNKRQJӳOLӋX
+uQKWKӇKLӋQYtGөVѫÿӗKXҩQOX\ӋQFӫDP{KuQK6NLS-*UDPYӟLNtFKWKѭӟFFӱDVәN /~FQj\WӯPөFWLrXVӁOjtea YjFiFWӯQJӳFҧQKVӁOjbest và you.
Mô hình CBOW
0{KuQK&%2:WKuQJѭӧFOҥLYӟL6NLp-*UDPĈҫXUDPRQJÿӧLO~FQj\OjFiFWӯQJӳ FҧQKéWѭӣQJFKtQKFӫDP{KuQK&%2:OjGӵÿRiQWӯPөFWLrXGӵDYjRFiFWӯQJӳ FҧQK[XQJTXDQKQyWURQJPӝWSKҥPYLQKҩWÿӏQK&KRWӯPөFWLrX w t WҥLYӏWUt t
WURQJFkXYăQEҧQNKLÿyÿҫXYjROjFiFWӯQJӳFҧQK(w tm , ,w w t 1, t 1 1 , ,w t m ) xung TXDQKWӯw t WURQJSKҥPYL m
+uQKWKӇKLӋQYtGөVѫÿӗKXҩQOX\ӋQFӫDP{KuQK&%2:YӟLNtFKWKѭӟFFӱDVә
N /~FQj\FiFWӯQJӳFҧQKVӁOjThe, best, you, can YjWӯPөFWLrXVӁOjtea
7URQJOXұQYăQQj\FK~QJW{LVӱGөQJNӻWKXұW:RUG9HFÿӇSKөFYөYLӋF
HPEHGGLQJWUѭӟFNKLÿѭDYjRKXҩQOX\ӋQFKRP{KuQKFKtQKYLӋFQj\VӁJL~SELӇXGLӉQFiFWӯKӧSOêPDQJQKLӅXQJӳQJKƭDKѫQ
Mô hình Sequence-to-6HTXHQFHYjFѫFKӃ$WWHQWLRQ
1KѭÿmWUuQKEj\ӣPөFFiFF{QJWUuQKQJKLrQFӭXOLrQTXDQSKѭѫQJSKiSWyPWҳW WUӯXWѭӧQJabstraction methodWKӇKLӋQWtQKYѭӧWWUӝLYӟLNKҧQăQJWҥRUDWӯPӟL GӵDWUrQQӝLGXQJYjSKRQJFiFKFӫDYăQEҧQJӕFÿLӅXQj\JL~SFiFFkXWyPWҳWWӵ QKLrQPҥFKOҥFJLӕQJYӟLFiFKKRҥWÿӝQJFӫDFRQQJѭӡLKѫQ7URQJSKѭѫQJSKiS Qj\WKuP{KuQK6HT6HTOjPӝWWURQJQKӳQJ P{KuQKPҥQJ1ѫ-URQÿѭӧFVӱGөQJ QKLӅXQKҩWWURQJGӏFKPi\(Machine translation)ÿѭӧFӭQJGөQJUӝQJUmLWURQJFiF P{KuQKWyPWҳWYăQEҧQWKHRKѭӟQJWUӯXWѭӧQJ
0{KuQK6HT6HTEDRJӗPEӝPmKyDEncoder YjEӝJLҧLPmDecoderYӟLFiFQ~W PҥQJ/6707URQJEjLWRiQQj\FK~QJW{LOӵDFKӑQNLӃQWU~F/670YuQyUҩWWKtFK KӧSÿӇ[ӱOêGӳOLӋXGҥQJFKXӛLYjFyNKҧQăQJQҳPEҳWWK{QJWLQFӫDFiFFkXGjL KLӋXTXҧKѫQVRFiFNLӃQWU~FPҥQJ511WUX\ӅQWKӕQJ%ӝ(QFRGHUJӗPQKLӅXOӟS /670FyQKLӋPYөOҫQOѭӧWÿӑF FiFWK{QJWLQWӯFKXӛLYăQEҧQÿҫXYjRYjWҥRUD FKXӛLWUҥQJWKiLҭQFXӕLFQJÿyQJJyLWRjQEӝWK{QJWLQFӫDFKXӛLÿҫXYjRWKjQK PӝWYHFWRUGX\QKҩW7LӃSWKHREӝ'HFRGHUFNJQJOjPӝWOӟS/670FyQKLӋPYөÿӑF WUҥQJWKiLҭQFXӕLFQJÿmÿѭӧFPmKRiEӣLWҫQJ(QFRGHUWUѭӟFÿyYjVLQKUDWӯQJ WKjQKSKҫQFӫDFkXWyPWҳW
Hình 7.L͇QWU~FP{KuQK6HT6HTWUX\͉QWK͙QJ
+uQKELӇXGLӉQNLӃQWU~FP{KuQK6HT6HTWUX\ӅQWKӕQJ%ӝ(QFRGHUFyQKLӋPYө OҫQOѭӧWÿӑFFiFWK{QJWLQWӯFKXӛLYăQEҧQÿҫXYjRYjWҥRUDFKXӛLWUҥQJWKiLҭQOj h1, h2, h3, , h9, h106DXÿy%ӝ'HFRGHUFyQKLӋPYөÿӑFWUҥQJWKiLҭQFXӕLFQJOj h10 ÿmÿѭӧFPmKRiEӣLWҫQJ(QFRGHUWUѭӟFÿyOҫQOѭӧWVLQKUDWӯQJWKjQKSKҫQFӫD
&iFNӃWTXҧNKӣLÿҫXNKiWӕWYӟLP{KuQK6HT6HTQj\WX\QKLrQÿm[XҩWKLӋQPӝW VӕKҥQFKӃ7URQJP{KuQKQj\EӝPmKyDEҳWEXӝFSKҧLFKX\ӇQÿәLWRjQEӝFKXӛL ÿҫXYjRWKjQKPӝWYHFWRUWUҥQJWKiLҭQGX\QKҩWVDXÿyEӝJLҧLPmGӵÿRiQFKXӛLÿҫX UDGӵDWUrQYHFWRUWUҥQJWKiLQj\9uYұ\P{KuQKQj\FKӍKRҥWÿӝQJWӕWÿӕLYӟLFiF FKXӛLQJҳQYuEӝJLҧLPmÿDQJ[HP[pWWRjQEӝFKXӛLÿҫXYjRWURQJYHFWRUGX\ QKҩWGRÿyUҩWNKyÿӇEӝPmKyDJKLQKӟFiFFKXӛLGjLÿӕLYӟLPӝWYHFWRUFyÿӝGjL FӕÿӏQKQKѭYұ\ ĈӇNKҳFSKөFKҥQFKӃQj\FѫFKӃ$WWHQWLRQÿѭӧFÿѭDUDYjWtFKKӧSYjRP{KuQKÿӇ JLҧLTX\ӃWEjLWRiQéWѭӣQJFӫDFѫFKӃQj\OjGӵÿRiQPӝWWӯEҵQJFiFKFKӍ[Hm xét PӝWYjLSKҫQFөWKӇFӫDFKXӛLWKD\YuWRjQEӝFKXӛL
7URQJFѫFKӃ$WWHQWLRQSKkQSKӕLFK~êAttention DistributionÿѭӧFWtQKWRiQGѭӟL GҥQJSKkQSKӕL[iFVXҩWGӵDWUrQFiFWӯWURQJYăQEҧQQJXӗQJL~SEӝJLҧLPmTX\ӃW ÿӏQKWKjQKSKҫQQjRVӁQrQWұSWUXQJYjRNKLQyWҥRUDWӯWLӃSWKHR[14]3KkQSKӕL chú ý a t ÿѭӧFWtQKWRiQFKRWӯQJEѭӟFJLҧLPmWKӭt YӟLF{QJWKӭFQKѭVDX tanh( ) t T i h i s t attn e v W h W s b (1) max( ) t t a soft e (2)
7URQJF{QJWKӭFWUrQv W W b, h , s , attn OjFiFWKDPVӕFҫQKӑFӢPӛLEѭӟFJLҧLPmKӋ
VӕFK~êa i t VӁÿѭӧFWtQKWRiQ7UӑQJVӕFK~êVӁÿҥLGLӋQFKRPӭFÿӝFK~êFҫQGjQK FKRPӝWWӯQKҩWÿӏQKÿӇWҥRUDPӝWWӯÿҫXUDWURQJEѭӟFJLҧLPm [15]
3KkQSKӕLFK~êÿѭӧFVӱGөQJÿӇWtQKWәQJWUӑQJVӕFӫDFiFWUҥQJWKiLҭQFӫDEӝPm KyDÿѭӧFJӑLOjYHFWѫQJӳFҧQKContext vector) h t * YHFWRUQj\ELӇXWKӏQKӳQJJuÿm ÿӑFÿѭӧFWӯYăQEҧQQJXӗQFKREѭӟFQj\YjFyWKӇÿѭӧFWtQKQKѭVDX
7URQJOXұQYăQQj\FK~QJW{LVӱGөQJP{KuQK6HT6HTYӟLFѫFKӃ$WWHQWLRQÿӇWҥRWLrXÿӅWӯSKҫQIXOO-WH[WP{KuQKQj\ÿѭӧFNêKLӋXOjP{KuQKBase-Seq2Seq3KѭѫQJSKiSQj\VӁÿѭӧFFK~QJW{LVӱGөQJÿӇVRViQKYӟLP{KuQKPGN-LM mà chúng tôi ÿmÿӅ[XҩW
Hình 8.L͇QWU~FP{KuQK6HT6HTN͇WKͫSYͣLF˯FK͇$WWHQWLRQ
+uQKWKӇKLӋQNLӃQWU~FFӫDP{KuQK6HT6HTYjFѫFKӃ$WWHQWLRQ7URQJYtGөQj\WҥLWKӡLÿLӇPGӵÿRiQWӯWLӃSWKHRVDXFkXYummy and hard toYHFWRUQJӳFҧQKÿѭӧFWәQJKӧSEҵQJFiFKWtQKWUXQJEuQKFiFWUӑQJVӕFӫDh1, h2, h3, , h9, h10 YjWҥRWKjQKSKkQSKӕL$WWHQWLRQ7DFyWKӇWKҩ\O~FQj\WUӑQJVӕ$WWHQWLRQFӫDWӯ find UҩWFDR9uYұ\EӝJLҧLPmÿmWҥRUDFkXWyPWҳWKRjQFKӍQKOjYummy and hard to find.
Mô hình Pointer Generator Network
Mô hình Pointer Generator Network
Sau EѭӟFWLӅQ[ӱOêGӳOLӋXEDQÿҫXFKRWRjQEӝYăQEҧQYjWLrXÿӅQKӳQJGӳOLӋX Qj\VӁÿѭӧFKXҩQOX\ӋQYӟLP{KuQKPointer Generator Network+uQKWKӇKLӋQ NLӃQWU~FFӫDP{KuQKQj\1KѭÿmWUuQKEj\WURQJSKҫQF{QJWUuQKOLrQTXDQP{ KuQKQj\VӁJL~SJLҧLTX\ӃWKDLYҩQÿӅFKtQKVRYӟLP{KuQKBase-Seq2Seq thông WKѭӡQJ x 9ҩQÿӅÿҫXWLrQOjJL~SWҥRUDFiFWӯWURQJWLrXÿӅWӵÿӝQJFyêQJKƭD liên TXDQYӟLYăQEҧQÿҫXYjRKѫQӢEѭӟFWҥRFkXWLrXÿӅEҵQJEӝJLҧLPmP{ KuQKQj\VӁFyNKҧQăQJTX\ӃWÿӏQKKRһFVӁWҥRWӯPӟLKRһFVӁVDRFKpSPӝW WӯWURQJYăQEҧQQJXӗQĈLӅXQj\VӁJL~SFkXWLrXÿӅWӵÿӝQJFyêQJKƭDOLrQ TXDQYӟLYăQEҧQJӕFKѫQ VRYӟLP{KuQKWUX\ӅQWKӕQJ x 9ҩQÿӅWKӭKDLOjJLҧLTX\ӃWYҩQÿӅJһSSKҧLFiFWӯOҥWURQJYăQEҧQÿҫXYjR OjFiFWӯFKѭDEDRJLӡ[XҩWKLӋQWURQJ9RFDEXODU\KD\FzQJӑLOjOut-of-
Vocabulary - OOV) Mô hình Base-Seq2Seq NKLJһSFiFWӯOҥQj\FKӍELӃW
6DXNKLTXiWUuQKKXҩQOX\ӋQKRjQWҩWFK~QJW{LVӱGөQJEӝJLҧLPmÿmÿѭӧFKXҩQ OX\ӋQÿӇWҥRWLrXÿӅWӵÿӝQJWӯYăQEҧQÿҫXYjR
1KѭÿmWUuQKEj\VѫOѭӧFWURQJSKҫQNLӃQWKӭFQӅQWҧQJLanguage Model OjPӝW WKXұWWRiQÿѭӧFKXҩQOX\ӋQQKҵPQҳPEҳWFiFÿһFÿLӇPYjSKkQEӕFKXӛLWӯWURQJ QJ{QQJӳWӵQKLrQYjFKRSKpSÿѭDUDGӵÿRiQWӯWLӃSWKHR[XҩWKLӋQFQJYӟL[iF VXҩWFӫDQyGӵDWUrQFKXӛLWӯKLӋQWҥL[16]3KѭѫQJSKiSWLӃSFұQGӵDWUrQ1JUDPOj NӻWKXұWSKәELӃQQKҩWFӫDLanguage Model0{KuQKGӵDWUrQJLҧÿӏQKFӫD0DUNRY UҵQJ[iFVXҩW[XҩWKLӋQFӫDPӝWWӯFөWKӇWURQJFKXӛLFKӍSKөWKXӝFYjRVӵ[XҩWKLӋQ FӫDn ± 1 WӯWUѭӟFÿy [17][18]
0{KuQKQj\ÿѭӧF[k\GӵQJÿӇJLҧLTX\ӃWYҩQÿӅOһSOҥLFiFWӯWURQJWLrXÿӅÿѭӧF WҥRWӵÿӝQJWӯP{KuQKPointer Generator NetworkFK~QJW{LVӱGөQJLanguage
Model PӭFWӯÿӇVӱDOӛLFKRFiFWLrXÿӅÿѭӧFWҥRUDWӵÿӝQJ%rQFҥQKÿyFK~QJ
W{LFNJQJVӱGөQJLanguage Model ÿӇVӱDOӛLFKRFiFWLrXÿӅÿѭӧFGӵÿRiQEӣLP{ hình Base-Seq2Seq, FK~QJW{LJӑLQyOjP{KuQKNӃWKӧSJLӳDBase-Seq2Seq và
Language Model là Smooth-Seq2SeqPӝWNLӃQWU~FӣJLӳDÿӇVRViQKVӵFҧLWKLӋQ
FӫDYLӋFiSGөQJLanguage Model VRYӟLP{KuQK6HT6HTWUX\ӅQWKӕQJ
Hình 133K˱˯QJSKiSKX̭QOX\ QP{KuQKQJ{QQJͷ
7URQJEjLOXұQYăQQj\FK~QJW{LÿmKXҩQOX\ӋQLanguage Model WӯWұS&RUSXVEә sung, Language Model FӫDFK~QJW{LVӱGөQJPҥQJ/670ÿӇKXҩQOX\ӋQTXiWUuQK KXҩQOX\ӋQÿѭӧFP{WҧQKѭKuQK
6DXNKLKXҩQOX\ӋQLanguage Model Qj\ÿѭӧFVӱGөQJÿӇVӱDOӛLFKRWLrXÿӅWӵ ÿӝQJ&{QJWKӭFVӱDOӛLÿѭӧFWKӇKLӋQWKHRF{QJWKӭFQKѭVDX
3KѭѫQJSKiSVӱDOӛLQJӳSKiSÿѭӧFFK~QJW{LWKӇKLӋQQKѭKuQKĈҫXWLrQFK~QJW{LVӁWUX\[XҩWGDQKViFKFKӍPөFFiFWӯEӏOһSOҥLWURQJWLrXÿӅYӯDWҥRUDVDXÿy[yDOҫQOѭӧWWӯQJWӯEӏOһSOҥLQj\YjÿiQKJLiÿLӇPOҥLFKRFkXWLrXÿӅWUѭӟFYjVDXNKLORҥLEӓWӯOһSOҥLGӵDWUrQF{QJWKӭFFXӕLFQJJLӳOҥLWLrXÿӅFyÿLӇPFDRKѫQ
Hình 146͵GͭQJP{KuQKQJ{QQJͷÿ͋V͵DO͟LQJͷSKiS
+uQKWKӇKLӋQPӝWYtGөFӫDYLӋFVӱGөQJP{KuQKQJ{QQJӳÿӇVӱDOӛLQJӳSKiS WURQJP{KuQKFӫDFK~QJW{L9ӟLWLrXÿӅÿҫXYjROj³RQHRIWKHWKHEHVWJUHHQJUHHQ
WHDVLKDYHHYHUKDG´ ÿҫXWLrQFK~QJW{LWUtFK[XҩWUDFiFFKӍPөFWӯEӏOһSOҥLWURQJ
FkXVDXÿyWtQKÿLӇPFKRFkXWUѭӟFYjVDXNKLORҥLEӓWӯQJWӯQj\FiFKWtQKÿLӇP FKRPӝWFkXÿѭӧFWKӇKLӋQWKHRF{QJWKӭF6DXQKLӅXEѭӟFORҥLEӓYjWtQKÿLӇP chRFkXFkXWLrXÿӅ³RQHRIWKHEHVWJUHHQWHDVLKDYHHYHUKDG´ OjNӃWTXҧVPRRWK FӫDFK~QJW{LNӃWTXҧOjFK~QJW{LÿmFyWKӇORҥLEӓÿѭӧFFiFWӯthe, green không KӧSOêWURQJWLrXÿӅEDQÿҫX
Hình 159tGͭV͵GͭQJP{KuQKQJ{QQJͷV͵DO͟LQJͷSKiS
&KѭѫQJ HIӊN THӴ&9ơĈẩ1+*,ẩ
Mô hình PGN-LM FӫDFK~QJW{LVӱGөQJWұSGӳOLӋXKXҩQOX\ӋQ
7ұSGӳOLӋXÿҫXWLrQOjYăQEҧQÿҫXYjRFҫQÿѭӧFWҥRWLrXÿӅ&K~QJW{LOҩ\WұSGӳ OLӋXÿҫXYjRFKRP{KuQKFӫDPuQKWӯ.DJJOH https://www.kaggle.com/snap/amazon-fine-food-reviews%ӝGӳOLӋXQj\OjFiFÿiQK
JLiFӫDFiFORҥLWKӵFSKҭPWӯ$PD]RQ0ӛLÿiQKJLiEDRJӗPQӝLGXQJWyPWҳWYj WLrXÿӅ'ӳOLӋXNpRGjLWURQJNKRҧQJWKӡLJLDQKѫQQăPEDRJӗPWҩWFҧ ÿiQKJLiWtQKÿӃQWKiQJQăP0{KuQKPGN-LM VӁÿѭӧFKXҩQ OX\ӋQÿӇQҳPEҳWYăQSKRQJFӫDYăQEҧQJӕFYjÿӝGjLFӫDWLrXÿӅWӯWұSGӳ OLӋXÿҫX YjRÿӇWҥRUDWLrXÿӅWӵÿӝQJ
Hình 167̵SGͷOL X$PD]RQ)LQH)RRG
+uQKWKӇKLӋQWұSGӳOLӋX$PD]RQ)LQH)RRGPjFK~QJW{LVӱGөQJFKRYLӋFKXҩQOX\ӋQYjWҥRWLrXÿӅWӵÿӝQJFKRP{KuQKPGN-LM7URQJWұSGDWDVHWQjy có FiFFӝWQKѭProductId, UserId, Title, Text, WX\QKLrQYӟLPөFWLrXWҥRWLrXÿӅFK~QJW{LFKӍVӱGөQJFӝWOjText và Title
7ұSGӳOLӋXWKӭKDLOjWұS&RUSXVEәVXQJÿѭӧFJӑLOjextra corpusGQJÿӇKXҩQ OX\ӋQLanguage Model7ұSGӳOLӋXQj\FҫQUҩWOӟQYjWәQJTXiWYuYұ\FK~QJW{L TX\ӃWÿӏQKVӱGөQJFiFEjLÿiQKJLiFӫD$PD]RQ https://snap.stanford.edu/data/web-Amazon.html.KRWjLOLӋXQj\EDRJӗPFiFEjL ÿiQKJLiWӯ$PD]RQ'ӳOLӋXNpRGjLWURQJNKRҧQJWKӡLJLDQQăPEDRJӗP WULӋXEjLÿiQKJLiWtQKÿӃQWKiQJQăP&iFEjLÿiQKJLiEDRJӗPWK{QJWLQ VҧQSKҭPYjQJѭӡLGQJ[ӃSKҥQJYjEjLYLӃWÿiQKJLi
Hình 177̵SGͷOL XE͝VXQJ
+uQKWKӇKLӋQWұSGӳOLӋXEәVXQJPjFK~QJW{LVӱGөQJFKRYLӋFKXҩQOX\ӋQ
Language Model7ұSGDWDVHWEәVXQJQj\FyFiFFӝWQKѭreviewerID, revieweName, Text, Title, reviewTime, 7X\QKLrQFK~QJW{LFKӍVӱGөQJFӝWText
GjQKFKRYLӋFKXҩQOX\ӋQLanguage Model/êGROjYuFӝWQj\PDQJQKLӅXêQJKƭDQKҩWWәQJTXiWQKҩWUҩWWKtFKKӧSFKRYLӋFKXҩQOX\ӋQLanguage Model
7URQJSKҫQQj\FK~QJW{LVӁWUuQKEj\YtGөFөWKӇYӅYLӋFWҥRUDFkXWLrXÿӅWӵ ÿӝQJWӯQӝLGXQJFӫDQyYӟLP{KuQKBase-Seq2Seq và PGN-LM Hình 18 cho WKҩ\PӝWYtGөWҥRFkXWLrXÿӅWӯSKҫQQӝLGXQJFӫDQyYӟLP{KuQKBase-Seq2Seq ÿmÿѭӧFÿӅFұSWURQJSKҫQ.LӃQWKӭFQӅQWҧQJ9ӟLP{KuQK6HT6HTNӃWKӧSFQJ FѫFKӃ$WWHQWLRQWӯYăQEҧQJӕF³@>@ score
Covered BLUE Total (16) ĈӕLYӟLEjLWRiQÿһWWLrXÿӅWӯQJӳ[XҩWKLӋQWURQJWLrXÿӅNK{QJQKҩWWKLӃWSKҧLOjFiFWӯOӵDFKӑQWӯYăQEҧQJӕFPLӉQVDRQӝLGXQJFӫDQyWKӇKLӋQÿѭӧFêFKtQKFӫDYăQEҧQJӕF9uYұ\SKѭѫQJSKiS%/8(YjFiFSKѭѫQJSKiSÿRÿӝFKtQK[iFWUX\ӅQWKӕQJÿmOӝQKLӅXNKX\ӃWÿLӇPNK{QJWKӇKLӋQÿѭӧFWtQKKӧSOêYjWKӵFWLӉQFӫDP{KuQK9uYұ\FK~QJW{LTX\ӃWÿӏQKQKӡWKrPWuQKQJX\ӋQYLrQWLӃQKjQK ÿiQKJLiNӃWTXҧÿҫXUDFӫDP{KuQKEҵQJVӭFQJѭӡL
ĈiQKJiá
ĈiQKJLiWӵÿӝQJ
7URQJSKѭѫQJSKiSÿiQKJLiWӵÿӝQJFK~QJW{LWtQKÿLӇP%/(8JLӳDYăQEҧQJӕFYjWLrXÿӅWӵÿӝQJÿѭӧFWҥRUDWӯP{KuQKÿѭӧFÿӅ[XҩW0{KuQKPGN-LM FӫDFK~QJW{LÿѭӧFKXҩQOX\ӋQQKҵPQҳPEҳWQӝLGXQJFӫDYăQEҧQJӕFYjWҥRUD PӝWWLrXÿӅPӟLWKӇKLӋQêWѭӣQJFKtQKFӫDYăQEҧQJӕFYӟLFiFWӯQJӳNK{QJFKӍSKөWKXӝFKRjQWRjQYjRWӯQJӳFӫDYăQEҧQJӕF'RÿyÿLӇP%/(8WKѭӡQJWKҩSYjNK{QJFKRWKӇKLӋQÿѭӧFÿӝFKtQK[iFFӫDP{KuQK
Hình 20:Ví GͭFiFKWtQKÿL͋P%/(8FKRP͡WFkXWLrXÿ͉
7KHRQKѭOêWKX\ӃWWUѭӟFÿy%/(8ÿѭӧFWtQKGӵDWUrQVӕOѭӧQJQ-JUDPVJLӕQJ QKDXJLӳDFkXGӏFKFӫDP{KuQKWLrXÿӅYӟLFiFFkXWKDPFKLӃXWѭѫQJӭQJÿRҥQ YăQJӕF7URQJWKӱQJKLӋPFӫDFK~QJW{LFK~QJW{LVӱGөQJ%/(8-JUDPÿӇÿR ÿҥF
+uQKELӇXGLӉQFiFKWtQKÿLӇP%/(8PjFK~QJW{LVӱGөQJ3KѭѫQJSKiSFK~QJ W{LWtQKÿLӇPOj-JUDPWURQJYtGөQj\WKuÿLӇP%/(8JLӳDYăQEҧQJӕFOjThis fruit is very healthy YjWLrXÿӅWӵÿӝQJThis fruit is so healthy VӁOj 4/5 = 0.8
Hình 21&iFKWtQKÿL͋P%/(8FKRP͟LP{KuQK
+uQKWKӇKLӋQF{QJWKӭFWtQKÿLӇP%/(8FӫDFiFP{KuQKEҵQJFiFKWtQKWәQJ ÿLӇPFӫDWҩWFҧFiFFkXWLrXÿӅFKLDFKRWәQJVӕFkXWLrXÿӅ
Base-Seq2Seq Smooth-Seq2Seq PGN-LM
%̫QJ2%̫QJN͇WTX̫ÿiQKJLi- ÿL͋P%/(8
&XӕLFQJEҧQJELӇXGLӉQFiFÿLӇP%/(8ÿѭӧFWtQKFKRP{KuQKPjFK~QJW{L ÿmÿӅ[XҩWYjP{KuQKFyÿLӇPFDRQKҩWFNJQJFKӍÿҥWĈyFNJQJOjOêGRWҥL VDRFK~QJWDFҫQWKrPSKѭѫQJSKiSÿiQKJLiYӟLFRQQJѭӡL0һFGYұ\YӟLÿLӇP BLEU, mô hình PGN-LM YүQFKRNӃWTXҧFDRKѫQVRYӟLP{KuQKBase-Seq2Seq và mô hình Smooth-Seq2Seq.
ĈiQKJLiYӟLFRQQJѭӡL
1KѭÿmWUuQKEj\ӣSKҫQSKѭѫQJSKiSÿiQKJLiWUѭӟFÿyErQFҥQKSKѭѫQJSKiS ÿiQKJLiWӵÿӝQJFK~QJW{LFNJQJVӱGөQJSKѭѫQJSKiSÿiQKJLiEҵQJFRQQJѭӡLÿӇ ÿROѭӡQJWtQKKӧSOêYjNKҧWKLFӫDFiFP{KuQK
&K~QJW{LÿmWLӃQKjQKÿiQKJLiWKӫF{QJYӟLVӵJL~SÿӥFӫDWuQKQJX\ӋQYLrQÿm WӕWQJKLӋS&K~QJW{LVӱGөQJWLrXFKtÿӇÿiQKJLiWLrXÿӅWӵÿӝQJĈҫXWLrQWtQK KӧSOêÿѭӧFWtQKEҵQJWӹOӋWLrXÿӅÿ~QJQJӳSKiSYjFyêQJKƭDKӧSOêVRYӟLQӝL
GXQJÿRҥQYăQEҧQJӕF7KӭKDLWtQKNKҧWKLÿѭӧFWtQKEҵQJWӹOӋWLrXÿӅWӵÿӝQJ ÿѭӧFWҥRUDҩQWѭӧQJKѫQKӧSOêYjWUX\ӅQWҧLÿѭӧFQKLӅXQӝLGXQJKѫQVRYӟLWLrX ÿӅJӕF ĈӇWLӃQKjQKÿiQKJLiFK~QJW{L FKӑQQJүXQKLrQYăQEҧQWӯEӝGӳOLӋXÿҫXYjR YjQKӡVӵJL~SÿӥFӫDFiFWuQKQJX\ӋQYLrQÿiQKJLiPӝWFiFKFKӫTXDQ0ӛLYtGө EDRJӗPPӝWÿRҥQYăQEҧQJӕFWLrXÿӅJӕFYjWLrXÿӅWӵÿӝQJÿѭӧFWҥREDRJӗP WLrXÿӅWӯP{KuQKBase-Seq2SeqWLrXÿӅWӯ mô hình Smooth-Seq2Seq YjFXӕLFQJ OjWLrXÿӅWӯP{KuQKPGN-LM&iFWuQKQJX\ӋQYLrQÿmFKӑQWLrXÿӅWӵÿӝQJWӕW
QKҩWFKRPӛLYăQEҧQJӕFWKHRFiFWLrXFKtWUrQWURQJFyWKӇFyQKLӅXOӵDFKӑQ
Hình 229tGͭP̳XÿiQKJLiFͯDFiFWuQKQJX\ QYLrQ
+uQKWKӇKLӋQYtGөPүXÿiQKJLiFӫDFiFWuQKQJX\ӋQYLrQ&iFGӳOLӋXEDRJӗP YăQEҧQJӕFWLrXÿӅJӕFYjWLrXÿӅWӵÿӝQJÿѭӧFWҥRUDEӣLP{KuQKPjFK~QJ W{LÿӅ[XҩW&iFWuQKQJX\ӋQYLrQVӁFKӑQFiFWLrXÿӅKӧSOêKRһFWӕWKѫQKD\EҵQJ VRYӟLWLrXÿӅJӕFFKRWӯQJFһS&XӕLFQJFK~QJW{LVӁWәQJKӧSFiFNӃWTXҧQj\OҥL ÿӇFyÿѭӧFNӃWTXҧFXӕLFQJ
6DXNKLFyNӃWTXҧÿiQKJLiÿҫ\ÿӫFӫDKѫQWLrXÿӅWӵÿӝQJWӯWuQKQJX\ӋQ YLrQW{LWLӃQKjQKÿRÿҥFFKӍVӕOjÿLӇPKӧSlý và ÿLӇPNKҧWKL ÿӇÿiQKJLiFiF mô hình
Hình 23&iFKWtQKÿL͋PKͫSOêFͯDP{KuQK
+uQKWKӇKLӋQF{QJWKӭFWtQKÿLӇPKӧSOêFӫDP{KuQKOjWӹOӋFiFWLrXÿӅWӵ ÿӝQJÿѭӧFVLQKUDFyQӝLGXQJWKӇKLӋQÿ~QJYӟLQӝLGXQJFӫDYăQEҧQJӕF
Hình 24&iFKWtQKÿL͋PNK̫WKLFͯDP{KuQK
+uQKWKӇKLӋQF{QJWKӭFWtQKÿLӇPNKҧWKLFӫDP{KuQKOjWӹOӋFiFFkXWLrXÿӅWӵ ÿӝQJÿѭӧFWҥRUDNK{QJFKӍÿ~QJYӅQӝLGXQJPjFzQÿҧPEҧRQJӳSKiSҩQWѭӧQJ KѫQVRYӟLWLrXÿӅJӕFYjWKD\WKӃÿѭӧFWLrXÿӅJӕF
Base-Seq2Seq Smooth-Seq2Seq PGN-LM
%̫QJ3%̫QJN͇WTX̫ÿiQKJLiWtQKKͫSOê
Base-Seq2Seq Smooth-Seq2Seq PGN-LM
%̫QJ4%̫QJN͇WTX̫ÿiQKJLiWtQKNK̫WKL
%ҧQJYjFKRWKҩ\NӃWTXҧFXӕLFQJÿѭӧFWәQJKӧSWӯÿiQKJLiFӫDWuQK
QJX\ӋQYLrQ.ӃWTXҧFKRWKҩ\FҧWLrXÿӅWӵÿӝQJÿӅXWѭѫQJÿӕLKӧSOêYjNKҧWKLWKӇKLӋQÿѭӧFWK{QJÿLӋSFӫDYăQEҧQJӕF.ӃWTXҧFKRWKҩ\P{KuQKPGN-LM KRҥW ÿӝQJWӕWKѫQVRYӟLP{KuQKBase-Seq2Seq và Smooth-Seq2SeqYӟLWtQKKӧSOêYjWtQKNKҧWKLOj99,49 %
7URQJQJKLrQFӭXÿӅWjLQj\W{LÿmKLӋQWKӵFÿѭӧFYLӋF[k\GӵQJP{KuQKPGN-LM FyNKҧQăQJWӵÿӝQJÿһWWLrXÿӅFKRÿRҥQYăQWK{QJTXDYLӋFiSGөQJFiFNӻWKXұW KӑFVkX'HHS/HDUQLQJKLӋQÿҥLNӃWKӧSYӟLP{KuQKQJ{QQJӳ0{KuQKÿmWKӇKLӋQ WtQKYѭӧWWUӝLYӟLNKҧQăQJWҥRUDQKӳQJWLrXÿӅҩQWѭӧQJFKRÿӝFJLҧ%rQFҥQKNKҧ QăQJWҥRFiFWLrXÿӅYӟLFiFWӯNK{QJ[XҩWKLӋQWURQJÿRҥQYăQJӕFNKҧQăQJ[ӱOê YҩQÿӅ2XW-Of-9RFDEXODU\VRYӟLFiFP{KuQKWUX\ӅQWKӕQJFNJQJJL~SWiFJLҧFyWKӇ WҥRUDQKӳQJWLrXÿӅY{FQJSKRQJSK~YjÿDGҥQJYӟLQKӳQJWӯQJӳFKѭDWӯQJ[XҩW KLӋQWUѭӟFÿk\FiFWӯQj\FyWKӇOjFiFGDQKWӯULrQJÿѭӧFÿӝFJLҧÿѭDYjRÿӇWăQJ WtQKҩQWѭӧQJ
6.2 Hѭӟng mӣ rӝng cӫDÿӅ tài
1JRjLQKӳQJF{QJYLӋFÿmÿѭӧFKLӋQWKӵFWURQJP{KuQKÿӅ[XҩWW{LWLQUҵQJQKӳQJ KѭӟQJPӣUӝQJVDXVӁÿHPOҥLQKӳQJNӃWTXҧÿҫ\KӭDKҽQFKRFiFQJKLrQFӭXWURQJ WѭѫQJODL x 0ӣUӝQJYӅWtQKӭQJGөQJWKuErQFҥQKWұSGӳOLӋX$PD]RQ)LQH)RRGYӅ WKӵFSKҭPQj\FK~QJWDFyWKӇPӣUӝQJWUrQQKLӅXWұSGӳOLӋXNKiFSKRQJ SK~YjÿDGҥQJKѫQ x &yWKӇiSGөQJP{KuQKYӟLQKLӅXQJ{QQJӳNKiFQKDXKѫQ
7URQJVXӕWTXiWUuQKWKӵFKLӋQOXұQYăQW{LÿmKӑFWKrPÿѭӧFUҩWQKLӅXEjLKӑFFy WKӇNӇÿӃQQKѭNӻQăQJ[k\GӵQJPӝWKӋWKӕQJӭQJGөQJNӻWKXұWKӑFVkXFҫQ QKӳQJTX\WUuQKQKѭWKӃQjRYLӋFWLӃSFұQQKӳQJNLӃQWKӭFPӟLQKѭWKӃQjRFNJQJ QKѭQKӳQJNKyNKăQNKLÿѭDPӝWVҧQSKҭPUDVӱGөQJWKӵFWӃ0һFGFyQKӳQJ NKyNKăQQKӳQJNLӃQWKӭFPjW{LÿmKӑFÿѭӧFVӁOjKjQKWUDQJTXêJLiWUrQFRQ ÿѭӡQJSKiWWULӇQVӵQJKLӋSӫDW{L
7{L[LQFҧPѫQTXê7Kҫ\&{WURQJ.KRD.KRDKӑFPi\WtQK- 7UѭӡQJÿҥLKӑF%ách KRD7KjQK3Kӕ+ӗ&Kt0LQKÿһFELӋWOjWKҫ\3*6764XҧQ7KjQK7KѫQJѭӡLÿm GүQGҳWW{LÿLTXDJLDLÿRҥQOXұQYăQQj\
[1] 6-RQHV³$XWRPDWLFVXPPDUL]LQJWKHVWDWH RIWKHDUW´Information Processing and Management, Elsevier, vol 43, no 6, Nov 2007
[2] N P C Khatri, and G 6LQJK ³$EVWUDFWLYH DQG H[WUDFWLYH WH[W summarization using document context vector and recurrent neural QHWZRUNV´Computation and Language (cs.CL), arXiv:1807.08000, Jul
Computation and Language (cs.CL), arXiv:1512.01712, Dec 2015
[4] & KDWUL * 6LQJK DQG 1 3DULNK ³$EVWUDFWLYH DQG H[WUDFWLYH WH[W summarization using document context vector and recurrent neural QHWZRUNV´Computation and Language (cs.CL), arXiv:1807.08000, Jul
[5] +3(GPXQGVRQ³1HZPHWKRGVLQDXWRPDWLFH[WUDFWLQJ´Journal of the
ACM, vol 16, iss 2, pp 264-285, Apr 1969
[6] +3(GPXQGVRQ³/H[UDQN*UDSK-based lexical centrality as salience in WH[W VXPPDUL]DWLRQ´ Computation and Language (cs.CL), arXiv:1109.2128, Sep 2011
[7] 9 *XSWD DQG * /HKDO ³$ VXUYH\ RI WH[W VXPPDUL]DWLRQ H[WUDFWLYH WHFKQLTXHV´Journal of Emerging Technologies in Web Intelligence 2, 3,
[8] C D M Bigail See, and P -/LX³*HWWRWKHSRLQW6XPPDUL]DWLRQZLWK pointer-JHQHUDWRU QHWZRUNV´ Computation and Language (cs.CL), arXiv:1704.04368, Apr 2017
[9] Z Liu, Angela Ng, S Lee, Ai Ti Aw, and N ) &KHQ ³7RSLF-aware pointer-JHQHUDWRU QHWZRUNV IRU VXPPDUL]LQJ VSRNHQ FRQYHUVDWLRQV´
Computation and Language (cs.CL), arXiv:1910.01335, Oct 2019
[10] Y Tay, S Wang, L A Tuan, J Fu, M C Phan, X Yuan, J Rao, Siu C Hui, and A =KDQJ ³6LPSOH DQG Hffective curriculum pointer-generator QHWZRUNVIRUUHDGLQJFRPSUHKHQVLRQRYHUORQJQDUUDWLYHV´Computation and Language (cs.CL), arXiv:1905.10847, May 2019
[11] W Li, R Peng, Y Wang, and Z