Nghiên cứu xây dựng công cụ hỗ trợ sửa lỗi viết tiếng anh cho báo cáo khoa học lĩnh vực công nghệ thông tin

ỦY BAN NHÂN DÂN TP.HCM ĐẠI HỌC QUỐC GIA TP.HCM SỞ KHOA HỌC CÔNG NGHỆ TRƯỜNG ĐH CÔNG NGHỆ THÔNG TIN BÁO CÁO NGHIỆM THU ĐỀ TÀI NGHIÊN CỨU XÂY DỰNG CÔNG CỤ HỖ TRỢ SỬA LỖI VIẾT TIẾNG ANH CHO BÁO CÁO KHOA HỌC LĨNH VỰC CÔNG NGHỆ THÔNG TIN Chủ nhiệm đề tài: TS NGUYỄN LƯU THÙY NGÂN Tp Hồ Chí Minh, Tháng 8/2017 BÁO CÁO NGHIỆM THU Tên đề tài: Nghiên cứu xây dựng công cụ hỗ trợ sửa lỗi viết tiếng Anh cho báo cáo khoa học lĩnh vực Công nghệ Thông tin Chủ nhiệm đề tài: TS Nguyễn Lưu Thùy Ngân Cơ quan chủ trì: Trường Đại học Công Nghệ Thông Tin, ĐH Quốc gia TP.HCM Thời gian thực đề tài: 24 tháng Kinh phí duyệt: 500 triệu đồng Kinh phí cấp: 450 triệu đồng, theo thông báo số 109/TB-SKHCN ngày 29/6/2015 thông báo số 174/TB-SKHCN ngày 16/8/2016 Mục tiêu: (Theo đề cương duyệt) Đề tài hướng đến việc xây dựng cơng cụ hỗ trợ người nước ngồi viết tiếng Anh, đặc biệt viết báo khoa học tiếng Anh Công cụ cho phép sử dụng rộng rãi cho mục đích học thuật, nghiên cứu Đối tượng sử dụng công cụ hướng đến học sinh, sinh viên, nhà nghiên cứu Việt Nam mà có người nước ngồi nước khơng nói tiếng Anh Cơng cụ sử dụng cho lớp học kỹ viết báo cáo khoa học (academic writing) Đề tài xây dựng tảng cho việc nghiên cứu chuyển giao công nghệ nhiều lĩnh vực: Xử lý ngôn ngữ tự nhiên, Trí tuệ nhân tạo, Dạy/học tiếng Anh ứng dụng công nghệ thông tin Công cụ hỗ trợ viết tiếng Anh xây dựng theo đề tài có khả tích hợp nhiều chức hỗ trợ cho việc viết tiếng Anh nói riêng việc học tiếng Anh nói chung Ứng dụng sử dụng nhiều kỹ thuật liên quan đến nhiều vấn đề nghiên cứu từ đơn giản đến phức tạp lĩnh vực nghiên cứu nói Nội dung thực (Theo đề cương duyệt) : Xem Bảng Bảng 1: Nội dung (theo đề cương duyệt) Các nội dung chủ yếu Nội dung 1: Xây dựng ngữ liệu chuẩn phục vụ nghiên cứu Nội dung 2: Nghiên cứu kỹ thuật phân tích lỗi tự động Nội dung 3: Nghiên cứu kỹ thuật tự động phát lỗi văn phong Nội dung 4: Nghiên cứu cải tiến số kỹ thuật bắt lỗi tả, ngữ pháp Nội dung 5: Xây dựng công cụ hỗ trợ sửa lỗi viết Tiếng Anh Nội dung 6: Viết báo cáo tổng kết Kết cần đạt Ngữ liệu xây dựng từ 40 báo khoa học, gồm khoảng 7000 câu tiếng Anh viết người không xứ, liên kết từ gán nhãn phân loại lỗi Chất lượng ngữ liệu thể qua độ quán interannotator agreement đạt 70% Tài liệu hướng dẫn gán nhãn phân loại lỗi Cơng cụ phân tích lỗi tự động Input: văn bản, văn gốc văn sau sửa lỗi Output: thống kê số lỗi sai loại Kết thử nghiệm đánh giá ngữ liệu chuẩn xây dựng mục Chất lượng tự động liên kết từ phân loại liên kết từ (ngữ) đạt 50% F-score Báo cáo kỹ thuật, phát triển thành báo khoa học Công cụ phát lỗi văn phong tự động Input: Văn gốc Output: Những câu văn cần phải viết lại có lỗi văn phong Kết thử nghiệm đánh giá ngữ liệu chuẩn xây dựng mục Độ phủ (recall) kỹ thuật tự động phát lỗi đạt 30% Báo cáo kỹ thuật, phát triển thành báo khoa học Có thể đăng ký thành giải pháp hữu ích Cơng cụ sửa lỗi tự động Input: Văn gốc Output: Những lỗi ngữ pháp văn đề xuất cách sửa Kết thử nghiệm đánh giá ngữ liệu chuẩn xây dựng mục ngữ liệu chuẩn khác Chất lượng kỹ thuật sửa lỗi ngữ pháp (của một vài loại lỗi) đạt F-score cao 40% Báo cáo kỹ thuật, phát triển thành báo khoa học Cơng cụ tích hợp số chức thực đề tài có khả tích hợp chức khác Báo cáo kỹ thuật Có thể đăng ký thành giải pháp hữu ích Báo cáo việc thực tất nội dung Sản phẩm cần đạt đề tài: Bảng 2: Sản phẩm cần đạt đề tài Tên sản phẩm 01 ngữ liệu chuẩn phục vụ nghiên cứu 01 báo khoa học đăng tạp chí chuyên ngành 01 báo khoa học đăng kỷ yếu hội nghị chuyên ngành 01 giải pháp hữu ích Đào tạo 01 thạc sĩ (luận văn tốt nghiệp) 01 công cụ hỗ trợ sửa lỗi văn khoa học tiếng Anh cho lĩnh vực công nghệ thông tin i Kết thực Hồn thành Hồn thành (01 báo tạp chí quốc tế Data and Knowledge Engineering - DKE 2017, SCI-indexed) Hoàn thành vượt tiêu (02 báo hội nghị quốc tế KSE 2015, 2016, 01 báo hội nghị nước FAIR 2017) Hoàn thành (Quyết định chấp nhận đơn hợp lệ ngày 26/5/2017) Hoàn thành vượt tiêu (02 thạc sĩ, 01 cử nhân bảo vệ thành cơng) Hồn thành Mục lục Mục lục i Danh mục hình vi Danh mục bảng viii Danh sách thuật ngữ ix TÓM TẮT NỘI DUNG NGHIÊN CỨU 1 TỔNG QUAN 1.1 Giới thiệu 1.2 Mục tiêu đề tài 1.2.1 Mục tiêu tổng quát 1.2.2 Mục tiêu cụ thể NỘI DUNG, PHƯƠNG PHÁP VÀ KẾT QUẢ THỬ NGHIỆM 2.1 Nội dung 1: Xây dựng ngữ liệu chuẩn phục vụ nghiên cứu 2.1.1 Tổng quan xây dựng ngữ liệu chuẩn phục vụ nghiên cứu hỗ trợ viết tiếng Anh 2.1.2 Mô tả nội dung 2.1.3 Mơ hình xây dựng ngữ liệu chuẩn sở 10 2.1.4 Mơ hình xây dựng ngữ liệu chuẩn mở rộng 12 2.1.5 Thu thập xử lý liệu 16 2.1.6 Kết gán nhãn 17 ii 2.2 2.3 2.1.7 Phân tích đánh giá ngữ liệu chuẩn 18 2.1.8 Kết luận 23 Nội dung 2: Kỹ thuật phân tích lỗi tự động 23 2.2.1 Tổng quan phân tích lỗi tự động 23 2.2.2 Mơ hình phân tích lỗi tự động dựa kỹ thuật dịch máy 28 2.2.3 Kết quả, phân tích đánh giá 31 Nội dung 3: Kỹ thuật tự động phát lỗi văn phong 40 2.3.1 Tổng quan kỹ thuật phát lỗi văn phong 40 2.3.2 Mơ hình phát lỗi văn phong dựa luật 41 2.3.3 Thử nghiệm đánh giá 44 2.3.4 Mơ hình phát lỗi văn phong dựa thống kê - nhằm phát phần không quan trọng câu 46 2.3.5 Phân tích đánh giá Mơ hình phát phát phần không quan trọng câu 49 2.4 Nội dung 4: Kỹ thuật tự động phát sửa lỗi tả, ngữ pháp 55 2.4.1 Mơ hình sửa lỗi ngữ pháp cho loại lỗi danh từ số ít, số nhiều 55 2.4.2 Thử nghiệm, đánh giá mơ hình sửa lỗi danh từ số ít, số nhiều 56 2.5 2.4.3 Mơ hình sửa lỗi tả 58 2.4.4 Tra cứu cụm từ viết báo cáo khoa học 60 Nội dung 5: Xây dựng phần mềm trực tuyến hỗ trợ viết sửa văn khoa học tiếng Anh 66 2.5.1 Mô tả yêu cầu - Thiết kế chức (use-case) 66 2.5.2 Thiết kế giao diện 67 2.5.3 Cài đặt phần mềm 68 KẾT QUẢ VÀ THẢO LUẬN 75 3.1 Tóm tắt kết chung 75 3.2 Nhận xét thảo luận 76 iii KẾT LUẬN VÀ ĐỀ NGHỊ 78 4.1 Kết luận 78 4.2 Đề nghị 80 PHỤ LỤC - HƯỚNG DẪN GÁN NHÃN 85 PHỤ LỤC - HƯỚNG DẪN SỬ DỤNG CÔNG CỤ GÁN NHÃN 89 PHỤ LỤC - THUYẾT MINH ĐỀ TÀI ĐƯỢC DUYỆT 90 PHỤ LỤC - HỢP ĐỒNG NGHIÊN CỨU 91 PHỤ LỤC - MINH CHỨNG SẢN PHẨM BÀI BÁO 92 PHỤ LỤC - MINH CHỨNG ĐÀO TẠO 128 PHỤ LỤC - QUYẾT TOÁN KINH PHÍ 129 TÀI LIỆU THAM KHẢO 129 iv Danh sách hình 2.1 Quá trình xây dựng ngữ liệu 2.2 Bộ nhãn phân loại lỗi sở 10 2.3 Hai cặp câu gán nhãn sử dụng Mơ hình sở, câu câu gốc, câu câu chỉnh sửa 11 2.4 Bộ nhãn phân loại lỗi mở rộng 13 2.5 Ví dụ liên kết từ Meteor 26 2.6 Cấu trúc tổng quan hệ thống 28 2.7 Ví dụ nhóm lỗi (1): Các từ có liên quan khơng liên kết với 34 2.8 Ví dụ nhóm lỗi (2): Các từ không liên quan liên kết với 34 2.9 Liên kết từ NUCLE - Ví dụ 38 2.10 Liên kết từ NUCLE - Ví dụ 39 2.11 Ví dụ lỗi văn phong (trước sửa: 1b, sau sửa: 1c) 40 2.12 Kiến trúc hệ thống 46 2.13 Cấu trúc tập tin câu ví dụ sau chuẩn hóa 62 2.14 Mơ hình rút trích cụm từ 62 2.15 Phương pháp tìm kiếm câu ví dụ 65 2.16 Thiết kế giao diện công cụ 68 2.17 Thiết kế giao diện chức năng: Sửa lỗi văn phong 69 2.18 Thiết kế giao diện chức năng: Sửa lỗi tả/ngữ pháp 69 2.19 Thiết kế giao diện chức năng: Phân tích lỗi 70 2.20 Thiết kế giao diện chức năng: Tra cứu cụm từ 70 v 2.21 Màn hình chức phát lỗi văn phong 71 2.22 Màn hình chức sửa lỗi tả ngữ pháp 72 2.23 Màn hình chức phân tích lỗi 73 2.24 Màn hình chức tra cứu cụm từ 74 4.1 Tagset used in the SWA corpus 81 vi Danh sách bảng Nội dung (theo đề cương duyệt) 2 Sản phẩm cần đạt đề tài i 2.1 Thống kê số lượng gióng hàng từ lỗi gán nhãn 18 2.2 Bảng ánh xạ nhãn nhãn KJ, NUCLE SWA, dùng để so sánh ngữ liệu 19 2.3 Thống kê lỗi ngữ pháp (Grammar), so sánh với KJ NUCLE α = TOTALSWA / TOTALKJ , β = TOTALSWA / TOTALNUCLE 20 2.4 Những loại lỗi loại Paraphrase cho thấy nhiều kiểu lỗi sai khác người viết 21 2.5 Các đặc trưng cho phân lớp liên kết từ 30 2.6 Kết liên kết từ 32 2.7 Kết liên kết từ phân loại theo nhãn 33 2.8 Phân tích lỗi liên kết từ 34 2.9 Kết phân lớp (5-fold cross validation) 35 2.10 Ánh xạ nhãn SWA METEOR 37 2.11 Kết chạy thử nghiệm METEOR ngữ liệu SWA 37 2.12 Kết phân lớp nhãn METEOR 38 2.13 Thống kê ngữ liệu 39 2.14 Kết phân lớp SWA, NUCLE FCE 39 2.15 Lựa chọn đặc trưng cho mơ hình huấn luyện CRF 49 2.16 Tỉ lệ nén (theo %) hệ thống nén câu sử dụng liệu tiếng Việt 53 2.17 Kết hệ thống nén câu sử dụng liệu tiếng Việt 53 vii Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, 2013, pp 590–600 URL http://aclweb.org/anthology/D/D13/D13-1056.pdf 425 [13] B MacCartney, M Galley, C D Manning, A phrase-based alignment model for natural language inference, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, Association for Computational Linguistics, Stroudsburg, PA, USA, 2008, pp 802–811 URL http://dl.acm.org/citation.cfm?id=1613715.1613817 430 [14] T.-G Noh, S Pad´o, V Shwartz, I Dagan, V Nastase, K Eichler, L Kotlerman, M Adler, Multi-level alignments as an extensible representation basis for textual entailment algorithms, Lexical and Computational Semantics (* SEM 2015) (2015) 193 [15] S Stymne, Blast: A tool for error analysis of machine translation output, in: 435 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Systems Demonstrations, HLT ’11, Association for Computational Linguistics, Stroudsburg, PA, USA, 2011, pp 56– 61 URL http://dl.acm.org/citation.cfm?id=2002440.2002450 440 [16] B Xu, G Gao, S R Fussell, D Cosley, Improving machine translation by showing two outputs, in: Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems, CHI ’14, ACM, New York, NY, USA, 2014, pp 3743–3746 doi:10.1145/2556288.2557171 URL http://doi.acm.org/10.1145/2556288.2557171 445 [17] M Popovi´c, H Ney, Towards automatic error analysis of machine translation output, Comput Linguist 37 (4) (2011) 657–688 doi:10.1162/COLI_a_ 00072 URL http://dx.doi.org/10.1162/COLI_a_00072 23 [18] D Vilar, J Xu, L D’haro, H Ney, Error analysis of statistical machine transla450 tion output, in: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC-2006), European Language Resources Association (ELRA), Genoa, Italy, 2006, aCL Anthology Identifier: L06-1244 URL http://www.lrec-conf.org/proceedings/lrec2006/pdf/ 413_pdf.pdf 455 [19] F Fancellu, B Webber, Translating negation: A manual error analysis, ExProM 2015 (2015) [20] M Denkowski, A Lavie, Meteor universal: Language specific translation evaluation for any target language, in: Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014 460 [21] M Denkowski, A Lavie, Meteor-next and the meteor paraphrase tables: Improved evaluation support for five target languages, in: Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, WMT ’10, Association for Computational Linguistics, Stroudsburg, PA, USA, 2010, pp 339–342 465 URL http://dl.acm.org/citation.cfm?id=1868850.1868901 [22] F J Och, H Ney, A systematic comparison of various statistical alignment models, Comput Linguist 29 (1) (2003) 19–51 doi:10.1162/ 089120103321337421 URL http://dx.doi.org/10.1162/089120103321337421 470 [23] P F Brown, V J D Pietra, S A D Pietra, R L Mercer, The mathematics of statistical machine translation: parameter estimation, Comput Linguist 19 (2) (1993) 263–311 URL http://dl.acm.org/citation.cfm?id=972470.972474 [24] S Vogel, H Ney, C Tillmann, Hmm-based word alignment in statistical transla- 475 tion, in: Proceedings of the 16th conference on Computational linguistics - Volume 2, COLING ’96, Association for Computational Linguistics, Stroudsburg, 24 PA, USA, 1996, pp 836–841 doi:10.3115/993268.993313 URL http://dx.doi.org/10.3115/993268.993313 [25] D Klein, C D Manning, Accurate unlexicalized parsing, in: Proceedings of 480 the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, Association for Computational Linguistics, Stroudsburg, PA, USA, 2003, pp 423–430 doi:10.3115/1075096.1075150 URL http://dx.doi.org/10.3115/1075096.1075150 [26] M J Sabet, H Faili, G Haffari, Improving word alignment of rare words with 485 word embeddings., in: COLING, 2016, pp 3209–3215 25 PHỤ LỤC - MINH CHỨNG ĐÀO TẠO 128 PHỤ LỤC - QUYẾT TỐN KINH PHÍ 129 Tài liệu tham khảo Tiếng Anh [2] Afantenos, Stergos et al “An empirical resource for discovering cognitive principles of discourse organisation: the ANNODIS corpus” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-29517408-7-7 [3] Allan, Keith “Review of "Toward a cognitive semantics: volume I: concept structuring systems, volume II: typology and process in concept structuring" by Leonard Talmy The MIT Press 2000.” In: Comput Linguist 27.2 (June 2001), pp 309–315 issn: 0891-2017 url: http://dl acm.org/citation.cfm?id=972667.972680 [4] Androutsopoulos, Ion and Malakasiotis, Prodromos “A survey of paraphrasing and textual entailment methods” In: J Artif Int Res 38.1 (May 2010), pp 135–187 issn: 1076-9757 url: http://dl.acm.org/citation.cfm?id=1892211.1892215 [5] Bouamor, Houda, Max, Aurélien, and Vilnat, Anne “Monolingual alignment by edit rate computation on sentential paraphrase pairs” In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume Stroudsburg, PA, USA: Association for Computational Linguistics, 2011, 395–400 [6] Boyd, Adriane “Pronunciation modeling in spelling correction for writers of English as a foreign language” In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Student Research Workshop and Doctoral Consortium SRWS ’09 Boulder, Colorado: Association for Computational Linguistics, 2009, pp 31–36 url: http://dl.acm.org/citation.cfm? id=1620932.1620938 [7] Brill, Eric and Moore, Robert C “An improved error model for noisy channel spelling correction” In: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics ACL ’00 Hong Kong: Association for Computational Linguistics, 2000, pp 286–293 doi: 10.3115/ 1075218.1075255 url: http://dx.doi.org/10.3115/1075218.1075255 [8] Brockett, Chris, Dolan, William B., and Gamon, Michael “Correcting ESL errors using phrasal SMT techniques” In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics ACL-44 Sydney, Australia: Association for Computational Linguistics, 2006, pp 249–256 doi: 10.3115/ 1220175.1220207 url: http://dx.doi.org/10.3115/1220175.1220207 130 [9] Brown, Peter F et al “The mathematics of statistical machine translation: parameter estimation” In: Comput Linguist 19.2 (1993), pp 263–311 issn: 0891-2017 [10] Burstein, Jill, Chodorow, Martin, and Leacock, Claudia “Automated essay evaluation: the criterion online writing service” In: AI Mag 25.3 (2004), pp 27–36 issn: 0738-4602 url: http: //dl.acm.org/citation.cfm?id=1045744.1045749 [11] C.-C., Shei and H., Pain “An ESL Writer’s Collocational Aid” In: Computer Assisted Language Learning 13.2 (2000), pp 167–182 [12] Callison-Burch, Chris, Cohn, Trevor, and Lapata, Mirella Annotation guidelines for paraphrase alignment Dec 2006 [13] Chang, Chih-Chung and Lin, Chih-Jen “LIBSVM: A library for support vector machines” In: ACM Transactions on Intelligent Systems and Technology (3 2011), 27:1–27:27 [14] Chang, Jason S and Chang, Yu-chia “Computer Assisted Language Learning Based on Corpora and Natural Language Processing: The Experience of Project CANDLE” In: IWLeL 2004: An Interactive Workshop on Language e-Learning 2004, pp 15–23 url: http://citeseerx.ist psu.edu/viewdoc/summary?doi=10.1.1.100.5318 [15] Chodorow, Martin and Leacock, Claudia “An unsupervised method for detecting grammatical errors” In: Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference NAACL 2000 Seattle, Washington: Association for Computational Linguistics, 2000, pp 140–147 url: http://dl.acm.org/citation.cfm?id=974305.974324 [16] Clarke, James and Lapata, Mirella “Global Inference for Sentence Compression An Integer Linear Programming Approach” In: Journal of Artificial Intelligence Research 31 (2008), pp 399– 429 url: http://jamesclarke.net/media/papers/clarke-lapata-jair2008.pdf [17] Cohn, Trevor, Callison-Burch, Chris, and Lapata, Mirella “Constructing corpora for the development and evaluation of paraphrase systems” In: Comput Linguist 34.4 (Dec 2008), pp 597– 614 issn: 0891-2017 doi: 10.1162/coli.08-003-R1-07-044 url: http://dx.doi.org/10 1162/coli.08-003-R1-07-044 [18] Crammer, Koby, Kulesza, Alex, and Dredze, Mark “Adaptive Regularization of Weight Vectors” In: Advances in Neural Information Processing Systems 22 Ed by Bengio, Y et al Curran Associates, Inc., 2009, pp 414–422 url: http://papers.nips.cc/paper/3848- adaptiveregularization-of-weight-vectors.pdf [19] Dahlmeier, Daniel and Ng, Hwee Tou “Correcting semantic collocation errors with L1-induced paraphrases” In: Proceedings of the Conference on Empirical Methods in Natural Language Processing EMNLP ’11 Edinburgh, United Kingdom: Association for Computational Linguistics, 2011, pp 107–117 isbn: 978-1-937284-11-4 url: http://dl.acm.org/citation.cfm?id= 2145432.2145445 [20] Dahlmeier, Daniel and Ng, Hwee Tou “Grammatical error correction with alternating structure optimization” In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume HLT ’11 Portland, Oregon: Association for Computational Linguistics, 2011, pp 915–923 isbn: 978-1-932432-87-9 url: http://dl acm.org/citation.cfm?id=2002472.2002588 131 [21] Dale, Robert and Kilgarriff, Adam “Helping Our Own: Text Massaging for Computational Linguistics as a New Shared Task” In: Proceedings of the International Natural Language Generation Conference 2011 Nancy, France, 2011 [22] Dale, Robert and Kilgarriff, Adam “Helping Our Own: The HOO 2011 Pilot Shared Task” In: Proceedings of the 13th European Workshop on Natural Language Generation Dublin, Ireland, 2010 [23] Dale, Robert and Narroway, George “A Framework for Evaluating Text Correction” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-2-9517408-7-7 [24] De Felice, Rachele and Pulman, Stephen G “A classifier-based approach to preposition and determiner error correction in L2 English” In: Proceedings of the 22nd International Conference on Computational Linguistics - Volume COLING ’08 Manchester, United Kingdom: Association for Computational Linguistics, 2008, pp 169–176 isbn: 978-1-905593-44-6 url: http://dl.acm.org/citation.cfm?id=1599081.1599103 [25] Denis, Alexandre et al “Representation of linguistic and domain knowledge for second language learning in virtual worlds” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-2-95174087-7 [26] Diaz-Negrillo, A and Dominguez-Fernandez, J “Error tagging systems for learner corpora” In: 19 (2006), pp 83–102 [27] Dickinson, Markus and Ledbetter, Scott “Annotating Errors in a Hungarian Learner Corpus” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-2-9517408-7-7 [28] Dinh, Dien and Vu, Thuy “A maximum entropy approach for Vietnamese word segmentation” In: 4th International Confernce on Computer Sciences: Research, Innovation and Vision for the Future, February 12-16, 2006, Ho Chi Minh City, Vietnam 2006, pp 248–253 url: http: //dx.doi.org/10.1109/RIVF.2006.1696447 [29] Duong, Mai, Nghiem, Minh-Quoc, and Nguyen, Ngan Luu-Thuy “Exploring Alignment-Classification Methods in the Context of Professional Writing Assistance” In: Data Knowledge Engineering (2017), pp – issn: 0169-023X doi: https://doi.org/10.1016/j.datak.2017.08.005 url: http://www.sciencedirect.com/science/article/pii/S0169023X1730037X [30] Elson, David K and McKeown, Kathleen R “Tense and aspect assignment in narrative discourse” In: Proceedings of the 6th International Natural Language Generation Conference INLG ’10 Trim, Co Meath, Ireland: Association for Computational Linguistics, 2010, pp 47–56 url: http://dl.acm.org/citation.cfm?id=1873738.1873747 [31] Fan, Rong-En et al “LIBLINEAR: A Library for Large Linear Classification” In: Journal of Machine Learning Research (2008), pp 1871–1874 132 [32] Futagi, Yoko “The effects of learner errors on the development of a collocation detection tool” In: Proceedings of the fourth workshop on Analytics for noisy unstructured text data AND ’10 Toronto, ON, Canada: ACM, 2010, pp 27–34 isbn: 978-1-4503-0376-7 doi: 10.1145/1871840 1871846 url: http://doi.acm.org/10.1145/1871840.1871846 [33] Gamon, Michael “High-order sequence modeling for language learner error detection” In: Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications IUNLPBEA ’11 Portland, Oregon: Association for Computational Linguistics, 2011, pp 180– 189 isbn: 9781937284039 url: http://dl.acm.org/citation.cfm?id=2043132.2043154 [34] Gamon, Michael “Using mostly native data to correct errors in learners’ writing: a meta-classifier approach” In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics HLT ’10 Los Angeles, California: Association for Computational Linguistics, 2010, pp 163–171 isbn: 1-932432-65-5 url: http: //dl.acm.org/citation.cfm?id=1857999.1858018 [35] Gamon, Michael et al “Using contextual speller techniques and language modeling for ESL error correction” In: In Proceedings of IJCNLP 2008 [36] Gamon, Micheal et al “Using statistical techniques and web search to correct ESL errors” In: 26.3 (2009) [37] Germann, Ulrich “Yawat: yet another word alignment tool” In: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session HLT-Demonstrations ’08 Columbus, Ohio: Association for Computational Linguistics, 2008, pp 20–23 url: http://dl.acm.org/citation.cfm?id=1564144.1564150 [38] Ghosh, Sucheta et al “Improving the Recall of a Discourse Parser by Constraint-based Postprocessing” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-2-9517408-7-7 [39] Golding, Andrew “A bayesian hybrid for context sensitive spelling correction” In: Proceedings of the 3rd Wordshop on Very Large Corpora Cambridge, USA, 1995, pp 39–53 [40] Golding, Andrew and Roth, Dan “Applying Winnow to context-sensitive spelling correction” In: Proceedings of the Int Conference on Machine Learning 1996, pp 182–190 [41] Goodman, Michael Wayne and Bond, Francis “Using generation for grammar analysis and error detection” In: Proceedings of the ACL-IJCNLP 2009 Conference Short Papers ACLShort ’09 Suntec, Singapore: Association for Computational Linguistics, 2009, pp 109–112 url: http: //dl.acm.org/citation.cfm?id=1667583.1667618 [42] Granger, Sylviane The computer learner corpus: A versatile new source of data for SLA research S Granger (ed), 1998, pp 3–18 [43] Ha, Nguyen Thi Thu and Dung, Vu Thi Ngoc “Improve Bayesian Network to Generating Vietnamese Sentence Reduction” In: IERI Procedia 10.0 (2014) International Conference on Future Information Engineering (FIE 2014), pp 190 –195 issn: 2212-6678 url: http://www sciencedirect.com/science/article/pii/S2212667814001245 [44] Han, Na-Rae, Chodorow, Martin, and Leacock, Claudia “Detecting errors in English article usage by non-native speakers” In: Nat Lang Eng 12.2 (June 2006), pp 115–129 issn: 1351-3249 doi: 10.1017/S1351324906004190 url: http://dx.doi.org/10.1017/S1351324906004190 133 [45] Han, Na-Rae et al “Using an Error-Annotated Learner Corpus to Develop an ESL/EFL Error Correction System” In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10) Ed by Chair), Nicoletta Calzolari (Conference et al Valletta, Malta: European Language Resources Association (ELRA), 2010 isbn: 2-9517408-6-7 [46] Hana, Jirka et al “Building a learner corpus” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-2-9517408-7-7 [47] Howarth, P “The phraseology of learners’ academic writing” In: Phraseology Theory Analysis and Applications Ed by Cowie, Anthony PEditor Howarth 1994 Oxford University Press, 1998, pp 161–186 [48] Ibrahim, Ali, Katz, Boris, and Lin, Jimmy “Extracting structural paraphrases from aligned monolingual corpora” In: Proceedings of the second international workshop on Paraphrasing Volume 16 PARAPHRASE ’03 Sapporo, Japan: Association for Computational Linguistics, 2003, pp 57–64 doi: 10 3115 / 1118984 1118992 url: http : / / dx doi org / 10 3115 / 1118984.1118992 [49] Islam, Aminul and Inkpen, Diana “Real-word spelling correction using Google web 1Tn-gram data set” In: Proceedings of the 18th ACM conference on Information and knowledge management CIKM ’09 Hong Kong, China: ACM, 2009, pp 1689–1692 isbn: 978-1-60558-512-3 doi: 10.1145/1645953.1646205 url: http://doi.acm.org/10.1145/1645953.1646205 [50] Izumi, Emi, Uchimoto, Kiyotaka, and Isahara, Hitoshi “SST speech corpus of Japanese learners’ English and automatic detection of learners’ errors” In: 28 (2004), pp 31–48 [51] Kernighan, Mark D., Church, Kenneth W., and Gale, William A “A Spelling Correction Program Based on a Noisy Channel Model” In: Proceedings of the 13th Conference on Computational Linguistics - Volume COLING ’90 Helsinki, Finland: Association for Computational Linguistics, 1990, pp 205–210 doi: 10.3115/997939.997975 url: http://dx.doi.org/10 3115/997939.997975 [52] Koehn, Philipp, Och, Franz Josef, and Marcu, Daniel “Statistical phrase-based translation” In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume NAACL ’03 Edmonton, Canada: Association for Computational Linguistics, 2003, pp 48–54 doi: 10.3115/1073445 1073462 url: http://dx.doi.org/10.3115/1073445.1073462 [53] Kroll, Barbara and Reid, Joy “Guidelines for designing writing prompts: Clarifications, caveats, and cautions” In: Journal of Second Language Writing 3.3 (1994), pp 231 –255 issn: 1060-3743 doi: 10.1016/1060- 3743(94)90018- url: http://www.sciencedirect.com/science/ article/pii/1060374394900183 [54] Kukich, Karen “Techniques for automatically correcting words in text” In: ACM Comput Surv 24.4 (Dec 1992), pp 377–439 issn: 0360-0300 doi: 10 1145 / 146370 146380 url: http://doi.acm.org/10.1145/146370.146380 [55] Lafferty, John D., McCallum, Andrew, and Pereira, Fernando C N “Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data” In: Proceedings of the Eighteenth International Conference on Machine Learning ICML ’01 2001, pp 282–289 isbn: 155860-778-1 url: http://dl.acm.org/citation.cfm?id=645530.655813 134 [56] Langacker, Ronald W “An Introduction to Cognitive Grammar” In: Cognitive Science 10.1 (1986), pp 1–40 issn: 1551-6709 doi: 10.1207/s15516709cog1001_1 url: http://dx.doi org/10.1207/s15516709cog1001_1 [57] Leacock, Claudia, Gamon, Michael, and Brockett, Chris “User input and interactions on Microsoft Research ESL Assistant” In: Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications EdAppsNLP ’09 Boulder, Colorado: Association for Computational Linguistics, 2009, pp 73–81 isbn: 978-1-932432-37-4 url: http://dl.acm org/citation.cfm?id=1609843.1609854 [58] Leacock, Claudia et al Automated Grammatical Error Detection for Language Learners Morgan and Claypool Publishers, 2010 isbn: 1608454703, 9781608454709 [59] Liang, Percy, Taskar, Ben, and Klein, Dan “Alignment by agreement” In: Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics HLT-NAACL ’06 New York, New York: Association for Computational Linguistics, 2006, pp 104–111 doi: 10.3115/1220835.1220849 url: http://dx.doi.org/10.3115/1220835.1220849 [60] Lin, Chin-Yew and Och, Franz Josef “Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics” In: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain 2004, pp 605–612 url: http://acl.ldc.upenn.edu/acl2004/main/pdf/215_pdf_2col.pdf [61] Liu, Anne Li-E, Wible, David, and Tsao, Nai-Lung “Automated suggestions for miscollocations” In: Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications EdAppsNLP ’09 Boulder, Colorado: Association for Computational Linguistics, 2009, pp 47–50 isbn: 978-1-932432-37-4 url: http : / / dl acm org / citation cfm ? id = 1609843.1609850 [62] Liu, Jyi-Shane, Hung, Pei-Chun, and Lee, Ching-Ying “A Language Information Retrieval Approach to Writing Assistance” In: International Journal of Computational Linguistics & Chinese Language Processing, Special Issue on Selected Papers from ROCLING XIX 13.3 (2008) [63] Mani, Inderjeet, Pustejovsky, James, and Sundheim, Beth “Introduction to the special issue on temporal information processing” In: 3.1 (Mar 2004), pp 1–10 issn: 1530-0226 doi: 10.1145/ 1017068.1017069 url: http://doi.acm.org/10.1145/1017068.1017069 [64] Mays, Eric, Damerau, Fred J., and Mercer, Robert L “Context based spelling correction” In: Information Processing Management 27.5 (1991), pp 517 –522 issn: 0306-4573 doi: http: //dx.doi.org/10.1016/0306- 4573(91)90066- U url: http://www.sciencedirect.com/ science/article/pii/030645739190066U [65] Milton, John and Cheng, Vivying S Y “A toolkit to assist L2 learners become independent writers” In: Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics and Writing: Writing Processes and Authoring Aids CL&W ’10 Los Angeles, California: Association for Computational Linguistics, 2010, pp 33–41 url: http://dl.acm.org/citation.cfm?id= 1860657.1860662 135 [66] Moulin, Bernard and Dumas, Stéphanie “The Temporal Structure of a Discourse and Verb Tense Determination” In: Proceedings of the Second International Conference on Conceptual Structures: Current Practices ICCS ’94 London, UK, UK: Springer-Verlag, 1994, pp 45–68 isbn: 3-540-58328-9 url: http://dl.acm.org/citation.cfm?id=645487.757709 [67] Nagata, Ryo and Nakatani, Kazuhide “Evaluating performance of grammatical error detection to maximize learning effect” In: Proceedings of the 23rd International Conference on Computational Linguistics: Posters COLING ’10 Beijing, China: Association for Computational Linguistics, 2010, pp 894–900 url: http://dl.acm.org/citation.cfm?id=1944566.1944669 [68] Nagata, Ryo, Whittaker, Edward, and Sheinman, Vera “Creating a manually error-tagged and shallow-parsed learner corpus” In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume HLT ’11 Portland, Oregon: Association for Computational Linguistics, 2011, pp 1210–1219 isbn: 978-1-93243287-9 url: http://dl.acm.org/citation.cfm?id=2002472.2002625 [69] Ng, Hwee Tou et al “The CoNLL-2013 Shared Task on Grammatical Error Correction” In: Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task Sofia, Bulgaria: Association for Computational Linguistics, 2013, pp 1–12 url: http: //www.aclweb.org/anthology/W13-3601 [70] Nghiem, Minh, Dinh, Dien, and Nguyen, Mai “Improving Vietnamese POS tagging by integrating a rich feature set and Support Vector Machines” In: 2008 IEEE International Conference on Research, Innovation and Vision for the Future in Computing & Communication Technologies, RIVF 2008, Ho Chi Minh City, Vietnam, 13-17 July 2008 2008, pp 128–133 url: http: //dx.doi.org/10.1109/RIVF.2008.4586344 [71] Nguyen, Dat Quoc et al “From Treebank Conversion to Automatic Dependency Parsing for Vietnamese” In: Proceedings of 19th International Conference on Application of Natural Language to Information Systems, NLDB’14 2014, pp 196–207 [72] Och, Franz Josef and Ney, Hermann “A systematic comparison of various statistical alignment models” In: Computational linguistics 29.1 (2003), pp 19–51 [73] Pan, Feng, Mulkar-Mehta, Rutu, and Hobbs, Jerry R “Annotating and learning event durations in text” In: Comput Linguist 37.4 (2011), pp 727–752 issn: 0891-2017 doi: 10.1162/COLI_ a_00075 url: http://dx.doi.org/10.1162/COLI_a_00075 [74] Park, Y Albert and Levy, Roger “Automated whole sentence grammar correction using a noisy channel model” In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume HLT ’11 Portland, Oregon: Association for Computational Linguistics, 2011, pp 934–944 isbn: 978-1-932432-87-9 url: http://dl acm.org/citation.cfm?id=2002472.2002590 [75] Pendar, Nick and Cotos, Elena “Automatic identification of discourse moves in scientific article introductions” In: Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications EANL ’08 Columbus, Ohio: Association for Computational Linguistics, 2008, pp 62–70 isbn: 978-1-932432-08-4 url: http://dl.acm.org/citation.cfm?id= 1631836.1631844 [76] Pravec, Norma A “Survey of learner corpora” In: ICAME Journal 26 (2002), pp 81–114 136 [77] Qian, Xian and Liu, Yang “Fast Joint Compression and Summarization via Graph Cuts” In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL 2013, pp 1492–1502 url: http: //aclweb.org/anthology/D/D13/D13-1156.pdf [78] Qian, Xian and Liu, Yang “Polynomial Time Joint Structural Inference for Sentence Compression” In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers 2014, pp 327–332 url: http://aclweb.org/anthology/P/P14/P14-2054.pdf [79] Rozovskaya, Alla and Roth, Dan “Annotating ESL errors: challenges and rewards” In: Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications IUNLPBEA ’10 Los Angeles, California: Association for Computational Linguistics, 2010, pp 28–36 url: http://dl.acm.org/citation.cfm?id=1866795.1866799 [80] Rozovskaya, Alla and Roth, Dan “Generating confusion sets for context-sensitive error correction” In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing EMNLP ’10 Cambridge, Massachusetts: Association for Computational Linguistics, 2010, pp 961–970 url: http://dl.acm.org/citation.cfm?id=1870658.1870752 [81] Rozovskaya, Alla and Roth, Dan “Training paradigms for correcting errors in grammar and usage” In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics HLT ’10 Los Angeles, California: Association for Computational Linguistics, 2010, pp 154–162 isbn: 1-932432-65-5 url: http: //dl.acm.org/citation.cfm?id=1857999.1858017 [82] Saint-Dizier, Patrick “DISLOG: A logic-based language for processing discourse structures” In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12) Ed by Chair), Nicoletta Calzolari (Conference et al Istanbul, Turkey: European Language Resources Association (ELRA), 2012 isbn: 978-2-9517408-7-7 [83] Stein, Benno, Potthast, Martin, and Trenkmann, Martin “Retrieving customary web language to assist writers” In: Proceedings of the 32nd European conference on Advances in Information Retrieval ECIR’2010 Milton Keynes, UK: Springer-Verlag, 2010, pp 631–635 isbn: 3-64212274-4, 978-3-642-12274-3 doi: 10.1007/978- 3- 642- 12275- 0_64 url: http://dx.doi org/10.1007/978-3-642-12275-0_64 [84] Stymne, Sara “Blast: A Tool for Error Analysis of Machine Translation Output” In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Systems Demonstrations HLT ’11 Portland, Oregon: Association for Computational Linguistics, 2011, pp 56–61 isbn: 978-1-932432-90-9 url: http : / / dl acm org / citation.cfm?id=2002440.2002450 [85] Sun, Guihua et al “Detecting Erroneous Sentences using Automatically Mined Sequential Patterns” In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) Prague, Czech Republic: Association for Computational Linguistics, 2007, pp 81–88 [86] Tetreault, Joel, Filatova, Elena, and Chodorow, Martin “Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk” In: Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications 2010 137 [87] Tetreault, Joel, Foster, Jennifer, and Chodorow, Martin “Using Parse Features for Preposition Selection and Error Detection” In: Proceedings of the ACL 2010 Conference Short Papers 2010 [88] Tetreault, Joel R and Chodorow, Martin “The ups and downs of preposition error detection in ESL writing” In: Proceedings of the 22nd International Conference on Computational Linguistics - Volume COLING ’08 Manchester, United Kingdom: Association for Computational Linguistics, 2008, pp 865–872 isbn: 978-1-905593-44-6 url: http://dl.acm.org/citation cfm?id=1599081.1599190 [89] Uzawa, Kozue “Second language learners’ processes of L1 writing, L2 writing, and translation from L1 into L2” In: Journal of Second Language Writing 5.3 (1996), pp 271 –294 issn: 10603743 doi: 10 1016 / S1060 - 3743(96 ) 90005 - url: http : / / www sciencedirect com / science/article/pii/S1060374396900053 [90] Wagner, Joachim, Foster, Jennifer, and Genabith, Josef van “A Comparative Evaluation of Deep and Shallow Approaches to the Automatic Detection of Common Grammatical Errors” In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) Prague, Czech Republic: Association for Computational Linguistics, 2007, pp 112–121 url: http://www.aclweb.org/ anthology/D/D07/D07-1012 [91] Wang, Dong, Qian, Xian, and Liu, Yang “A Two-step Approach to Sentence Compression of Spoken Utterances” In: The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 2: Short Papers 2012, pp 166–170 url: http://www.aclweb.org/anthology/P12-2033 [92] Whitelaw, Casey et al “Using the web for language independent spellchecking and autocorrection” In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume - Volume EMNLP ’09 Singapore: Association for Computational Linguistics, 2009, pp 890–899 isbn: 978-1-932432-62-6 url: http://dl.acm.org/citation.cfm? id=1699571.1699629 [93] Williams, Joseph M and Colomb, Gregory G Style: Lessons in clarity and grace Boston, MA: Longman, 2010 [94] Wu, Jian-Cheng et al “Automatic collocation suggestion in academic writing” In: Proceedings of the ACL 2010 Conference Short Papers ACLShort ’10 Uppsala, Sweden: Association for Computational Linguistics, 2010, pp 115–119 url: http://dl.acm.org/citation.cfm?id= 1858842.1858863 [95] Wu, Jian-Cheng et al “Helping our own: NTHU NLPLAB system description” In: Proceedings of the Seventh Workshop on Building Educational Applications Using NLP Montreal, Canada: Association for Computational Linguistics, 2012, pp 295–301 url: http : / / dl acm org / citation.cfm?id=2390384.2390421 [96] Xing, Junwen et al “UM-Checker: A Hybrid System for English Grammatical Error Correction” In: Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task Sofia, Bulgaria: Association for Computational Linguistics, 2013, pp 34–42 url: http://www.aclweb.org/anthology/W13-3605 [97] Xue, Huichao and Hwa, Rebecca “Improved Correction Detection in Revised ESL Sentences.” In: ACL (2) 2014, pp 599–604 138 [98] Yasuda, Sachiko “Genre-based tasks in foreign language writing: Developing writers?? genre awareness, linguistic knowledge, and writing competence” In: Journal of Second Language Writing 20.2 (2011), pp 111 –133 issn: 1060-3743 doi: 10.1016/j.jslw.2011.03.001 url: http: //www.sciencedirect.com/science/article/pii/S1060374311000191 [99] Yoshimoto, Ippei et al “NAIST at 2013 CoNLL Grammatical Error Correction Shared Task” In: Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task Sofia, Bulgaria: Association for Computational Linguistics, 2013, pp 26–33 url: http://www.aclweb.org/anthology/W13-3604 139