Smart agent for card games using reinforcement learning

VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE AND ENGINEERING ——————– * ——————— THESIS Smart agent for card-games using reinforcement learning COUNCIL : COMPUTER SCIENCE SUPERVISOR : PhD Nguyen Duc Dung REVIEWER : MSc Vuong Ba Thinh —o0o— Students: Vo Khac Thanh Cap Dang Xuan Kiet 1713164 1711852 HO CHI MINH CITY, October, 2021 ฀ Declaration We hereby declare that unless a clear reference is made to the work of others, the substance of this dissertation is original and has not been submitted in whole or in part for consideration at this or any other university for any other degree or qualification This study is our work and contains nothing that, except as stated in the text and acknowledgments, it the product of work done in collaboration with others If this statement is not true, our team will accept all responsibility to the Dean of the School and the School Administration Acknowledgments We would want to express our heartfelt and sincere appreciation to Dr.Nguyen Duc Dung For roughly one year, we have faced many challenges Many tasks that we thought they could be finished in hours turned out to be weeks, months We saw things did not go as we expected, some dragged us down However, we were eventually clear-headed and found solutions for every problems in the thesis, thanks to your advice and guidance We will be eternally thankful and obliged to you for your direction, wisdom, passion, and encouragement in assisting us in studying and implementing this project We would like to thank all of our faculty lecturers, without whom we would not have had the essential knowledge to complete our research Later, armed with this university’s invaluable experience, we shall fearlessly face our future and be exceptional Four years have passed since the first day we started our first lecture in the university We had new friends, who were not only classmate, but also great teachers, teammates and assistants It would have taken longer than four years for us to complete all classes if it had not been for you Furthermore, this four-year period would never be fulfilled without happy, sad, cheerful and emotional memories that we had together We will always respect and appreciate those as the best days that we have ever had Despite our efforts, we are aware that this project is currently insufficient and will inevitably contain flaws We’d love to hear from professors on how we might improve even more Finally, we wish you health, wealth, and success in your endeavors Students Abstract Artificial intelligence and games have long been in a relationship Gamers demand sophisticated AI, while games provide hard and diverse challenges whose solutions often apply to real-life problems In the past, Chess and Go are played by computers at a superhuman level Recently, in Defend of the Ancients (DotA 2), computers defeated the best professional player, yet there were some restrictions in the match Collectible Card Games (CCG), a branch of the strategy genre, challenges minds with their complex environments These games are ideal for researching artificial intelligence as they are harder to play compared to traditional card games and the responding time requirement is relaxed In this dissertation, we aim to create a CCG game and training smart agents who are able to take efficient actions and play at an accomplished level Table of contents Introduction 1.1 Overview 1.1.1 Video games & Card-games 1.1.2 Artificial Intelligence 1.2 Objectives 1.3 Development process 1 3 Background 2.1 Game Engines 2.1.1 Definition 2.1.2 Runtime Engine Architecture 2.1.2.1 Third-Party SDKs and Middleware 2.1.2.2 Resource Manager 2.1.2.3 Rendering Engine 2.1.2.4 Collision and Physics 2.1.2.5 Human Interface Devices - HID 2.1.2.6 Gameplay Foundation Systems 2.2 Reinforcement Learning 2.2.1 Markov Decision Process 2.2.2 Q-learning 2.2.3 Fictitious Self-play 2.3 Supervised Learning 2.3.1 Multilayer perceptron 2.3.2 Recurrent neural network 5 8 9 10 11 12 12 13 13 13 16 18 18 18 19 19 19 21 21 22 22 22 23 23 Related Works 3.1 Game Engines 3.1.1 Unity 3.1.1.1 Pros 3.1.1.2 Cons 3.1.2 Cocos Creator 3.1.2.1 Pros 3.1.2.2 Cons 3.1.3 Summary 3.2 Games 3.2.1 Yu-Gi-Oh! 3.2.1.1 Pros 3.2.1.2 Cons i 3.2.2 3.3 3.4 Hearthstone 3.2.2.1 Pros 3.2.2.2 Cons 3.2.2.3 Summary Reinforcement learning approaches 3.3.1 Deep Q-network 3.3.1.1 Overview 3.3.1.2 Pros 3.3.1.3 Cons 3.3.2 Double Deep Q-Network 3.3.2.1 Overview 3.3.2.2 Pros 3.3.2.3 Cons 3.3.3 Neural Fictitious Self-play 3.3.3.1 Overview 3.3.3.2 Pros 3.3.3.3 Cons 3.3.4 Remarks Summary Quest of the Divinity 4.1 System Architecture 4.1.1 Application Programming Interface 4.1.2 Communication Server 4.2 Quest of the Divinity 4.2.1 Game board 4.2.2 Ruleset 4.2.3 Game Controls 4.2.4 How To Play 4.2.5 Card Design 4.2.6 Architecture Design 4.2.6.1 Client 4.2.6.2 API Director 4.2.6.3 Log Writer 4.2.6.4 Mouse 4.2.6.5 Timer 4.2.6.6 Game Manager 4.2.6.7 Player 4.2.6.8 Hand 4.2.6.9 Deck 4.2.6.10 Boardside 4.2.6.11 Card 4.2.6.12 Visualizer 4.3 Game agent 4.3.1 Feature Extraction 4.3.2 Reward functions 4.3.3 NFSP agent with double DQN 4.3.3.1 Imitation Agent 23 24 25 25 25 25 25 26 26 27 27 27 27 29 29 30 30 31 31 32 32 32 32 33 33 35 35 36 36 38 38 38 40 40 40 40 40 41 41 41 41 42 42 42 43 43 43 4.3.3.2 4.3.3.3 Double DQN Agent NFSP Agent Experiments & Results 5.1 Experiment 5.1.1 Experimental setup 5.1.2 Data description 5.1.2.1 Data overview 5.1.2.2 Data problem 5.1.2.3 Solution and result 5.2 Results 5.2.1 Imitation Agents 5.2.2 Evaluation with collected data 5.2.3 Evaluation with game 5.2.4 Evaluation with human 44 44 47 47 47 48 48 48 49 50 50 51 52 54 Conclusion 6.1 What went well 6.2 What was not great 6.3 What could be different 55 55 55 55 A In-game Images 59 B Card Details & Graphics References B.1 Card Details B.1.1 Minions B.1.1.1 None B.1.1.2 Sage B.1.1.3 Warrior B.1.1.4 Abyss B.1.1.5 Nature B.1.1.6 Dragon B.1.2 Spell Cards B.1.3 Sacrifice Cards B.2 Graphics References B.2.1 Asset Packs B.2.2 Minions B.2.2.1 None B.2.2.2 Sage B.2.2.3 Warrior B.2.2.4 Abyss B.2.2.5 Nature B.2.2.6 Dragon B.2.2.7 Summon Minions B.2.3 Spells B.2.4 Sacrifices B.2.5 Albums B.2.6 Others 64 64 64 64 65 65 66 66 67 67 67 68 68 68 68 68 69 69 69 70 70 70 70 71 71 IN-GAME IMAGES Figure A.2: Simple particle effect on activating a minion card Figure A.3: Apprentice Mage with initial chapion effect tuitively, the minion component will request the visualizer to move to the target and move back, right after the translation call, the component will deal damage to its target, and its target, on the same time, deals damage back This may arise a problem where the minion dies right after the deal damage call, which makes it explode at a position very close to its original position, without moving at all Thus, the minion component only calls the visualizer to move to its target position, when the position translation is complete, it then deals damage while the visualizer moves back to the original position (Figure A.5) 60 IN-GAME IMAGES Figure A.4: Minion outline types, green for minions which are available to attack, yellow for enemy guards, and blinking red for chosen minion to perform an attack The cursor will temporarily change its texture to a sword Figure A.5: A minion deals damage to its target, with some debris splash On minion’s death, it makes a simple exploding effect (Figure A.6) On activating sacrifice or spell card, it stays at the dropped position, and wait for player to choose target (Figure A.7) For non-target spell cards, they will automatically disappear after a short delay Some minions have their own special effect on board, for example: Lord of the Apocalypse 61 IN-GAME IMAGES Figure A.6: Minion’s simple exploding effect Figure A.7: A spell card stays at its dropped position and wait for target and its two Infernal creatures present on the board with dark red rifts on their backs (Figure A.8) Dragons have their own attack visuals, instead of directly translate their position to their target’s position (translation is proceeded in 2D plane), they fly up and then translate their position to their target’s position This creates the feel that they attack their prey from above Especially, The One From Above does not take reflect damage from its target, it flies up and throws an exploding fireball to its target (Figure A.9) 62 IN-GAME IMAGES Figure A.8: Lord of the Apocalypse and Infernal creatures special effects Figure A.9: The One From Above with special attack 63 Appendix B Card Details & Graphics References B.1 Card Details As introduced in Chapter 4, cards are the center of the game They are divided into three main sections: Minions, Spells and Sacrifices, each card contributes differently to player’s playstyle and strategies Below is the detail of each card, their functions and information B.1.1 Minions B.1.1.1 None There are cards which not belong to any class: Sailor, Wandering Shaman, Outcast Sage, The Butcher, Forsaken Witch and Mazelina the Jinxed Figure B.1: Card in None class 64 CARD DETAILS & GRAPHICS REFERENCES B.1.1.2 Sage Sage cards are decorated with a blue outline on their portrait Their functions are built around protection and survivability Figure B.2: Cards in Sage class B.1.1.3 Warrior Warrior cards are outlined by red Their functions focus on damage Figure B.3: Cards in Warrior class 65 CARD DETAILS & GRAPHICS REFERENCES B.1.1.4 Abyss There is no color that best suit for a deadly class than the death itself, so that the Abyss card skin is replaced with a deadly, ruined looking skin Figure B.4: Cards in Abyss class B.1.1.5 Nature Nature is associated with green outline and plays supporting role in the game Figure B.5: Cards in Nature class 66 CARD DETAILS & GRAPHICS REFERENCES B.1.1.6 Dragon Dragons, with minions, decorate themselves with a golden patterned card skin Dragons can be classified as the strongest at their specific cost, if they are played together, they will be unstoppable Figure B.6: Cards in Dragon class B.1.2 Spell Cards Figure B.7: The Spell card subset B.1.3 Sacrifice Cards Sacrifice cards are described in detail at section 4.2.5, they boost the team’s strength remarkably, yet they could the the double-edged sword if the players play them recklessly 67 CARD DETAILS & GRAPHICS REFERENCES Figure B.8: The Sacrifice card subset B.2 Graphics References In Quest of The Divinity, we have used one Card Game GUI Pack, which is available on Unity Asset Store, and a lot of pictures from sources including Pinterest, Imgur, Artstation and more Some card portraits are taken from the same album, so instead of repeatedly show an url across all involved images, we put the album’s name on those images Kindly refer section B.2.5 for detailed images B.2.1 Asset Packs • PONETI, Card Game GUI Unity Asset Store https://assetstore.unity.com/packages/2d/gui/card-game-gui-130628 B.2.2 Minions B.2.2.1 None • Sailor: Pinterest https://www.pinterest.com/pin/540361655298022013/ • Wandering Shaman: Alexei Konev https://www.artstation.com/artwork/K1WNR • Outcast Sage: Donutello, Wizard/Sorcerer D&D Character Dump • The Butcher: Donutello, Wizard/Sorcerer D&D Character Dump • Forsaken Witch: Donutello, Wizard/Sorcerer D&D Character Dump • Mazelina the Jinxed: Donutello, Wizard/Sorcerer D&D Character Dump B.2.2.2 Sage • Oathkeeper: Pinterest https://www.pinterest.com/pin/740701469944563786/ • Apprentice Mage: Magali Villeneuve https://www.inprnt.com/gallery/magvilleneuve/teyo-the-shieldmage/ • The Alchemist: Imgur.com, TES:L card art • The Oracle: Donutello, Wizard/Sorcerer D&D Character Dump 68 CARD DETAILS & GRAPHICS REFERENCES • The Priest: Fabio Stibiel https://www.pinterest.com/pin/532480355937305686/ • The Rune Controller: Donutello, Wizard/Sorcerer D&D Character Dump • The Enlighten: Anna LAKISOVA https://www.artstation.com/artwork/OBmJw • Aaron the Wisekeeper: Livia Prima https://www.artstation.com/artwork/Xy33n B.2.2.3 Warrior • Windfield Soldier: Macky O https://www.pinterest.com/pin/700239442036172893/ • Barbarian: Imgur.com, TES:L card art • Charming Swordsman: Billy Christian https://www.artstation.com/artwork/zaR1w • Royal Knight: Character Portraits https://characterportraits.tumblr.com/post/175930595839 • Vicious Traitor: Imgur, Fantasy Character Art for your DND Campaigns https://imgur.com/gallery/uKuYK • Cryptic Legion Spearmaster: Mocah Wallpeppers https://mocah.org/4560126-warrior-drawing-fantasy-art-armor.html • Windfield Commander: Imgur https://imgur.com/kkTOOm6 • Lionroar General: Ye Fan https://www.artstation.com/artwork/5DeQ1 B.2.2.4 Abyss • The Bloodlust: Imgur.com, TES:L card art • Betrayed Shapeshifter: Donutello, Wizard/Sorcerer D&D Character Dump • Uncaged Hellhound: Alex Orgaz https://www.pinterest.com/pin/490399846922368874/ • Withered Mandarin: Donutello, Wizard/Sorcerer D&D Character Dump • Emperor of the Frozenheart: Jon Pintar https://www.artstation.com/artwork/KQBmr • Uprising Warmaster: mattforsyth https://www.deviantart.com/mattforsyth/art/Warlord-639494101 • Incarnation of Death: Mega Rusa Things https://www.pinterest.com/pin/753156737658342895/ • Lord of The Apocalypse: Pinterest https://www.pinterest.com/pin/AWWzkTcNIqv4RsIpoO2sJb72s8fQvMj92iFCEILPGCWYTO5b-uqcxo/ B.2.2.5 Nature • Dragonfly Archer: PigeonHero, Creatures [multifarious] 69 CARD DETAILS & GRAPHICS REFERENCES • Greenwood Guardian: PigeonHero, Creatures [multifarious] • Skull Hunter: Imgur.com https://imgur.com/vcWZOjQ • The Jungle Guidance: u/Thanmarkou https://www.reddit.com/r/Fantasy/comments/4q8n0e/forest_princess/ • Viper Sisters: Sombre Choniqueur https://www.pinterest.com/pin/437623288789145842/ • Gloryhorn: Sickbrush https://www.deviantart.com/sickbrush/art/Elara-Deer-guardian-392888237 • Serperus: mobius-9 https://www.deviantart.com/mobius-9/art/Green-dragon-568861529 B.2.2.6 Dragon • Ender Flame: Zudarts Lee https://www.pinterest.com/pin/585960601492999759/ • Eternal Frost: DragonFan https://art.alphacoders.com/arts/view/43615 • The One Above All: Dennis Mooney https://www.pinterest.com/pin/556968678912253095/ B.2.2.7 Summon Minions • Piggy: Bo˙zenka Chadzy´nska https://www.artstation.com/artwork/yLdlQ • Raven: Odin’s Grey Hawk https://www.pinterest.fr/pin/521150988137046500/ • Withered Soldier: Donutello, Wizard/Sorcerer D&D Character Dump • Infernal Creature: Stefan Koidl https://www.artstation.com/artwork/o5KRW B.2.3 Spells • • • • • • • • Icetomb: Imgur.com, TES:L card art Arcane Ritual: Imgur.com, TES:L card art Dancing Flame: Imgur.com, TES:L card art Reckless Gamble: myD&D https://mydnd.tumblr.com/post/37499403441/dragon-magazine-issue-417 Falling Meteor: Imgur.com, TES:L card art Gift of the Nature: Imgur.com, TES:L card art Electrocuted: Imgur.com, TES:L card art Divine Wrath: Imgur.com, TES:L card art B.2.4 Sacrifices • Ceremony of The Doom: Flint Chen https://www.artstation.com/artwork/L2eOKw 70 CARD DETAILS & GRAPHICS REFERENCES • Unbound Darkness: Julie Dillion https://loschaos.com/darkness-fantasy-art-by-julie-dillon/ • Righteous Immolation: Andeavenor https://art.alphacoders.com/arts/view/39342 • The Final Duel: DevianArt https://www.pinterest.com/pin/559501953684386036/ B.2.5 Albums • Imgur.com, TES:L card art https://imgur.com/a/uSrDd • Donutello, Wizard/Sorcerer D&D Character Dump https://imgur.com/gallery/6GEvd43 • PigeonHero, Creatures [multifarious] https://imgur.com/gallery/XPKbDI0 B.2.6 Others • Player Avatar: https://www.pinterest.pt/pin/508906826634239751/ • Enemy Avatar, evil king by yefumm https://www.deviantart.com/yefumm/art/evil-king-318297999 71 Appendix C API: Card IDs Card ID 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Name Sailor Wandering Shaman Outcast Sage The Butcher Forsaken Witch Mazelina the Jinxed Oathkeeper Apprentice Mage The Alchemist The Oracle The Priest The Rune Controller The Enlighten Aaron The Wisekeeper Windfield Soldier Barbarian Charming Swordman Royal Knight Vicious Traitor Cryptic Legion Spearmaster Windfield Commander Lionroar General The Bloodlust Betrayed Shapeshifter Uncaged Hellhound Withered Mandarin Emperor of the Frozenheart Uprising Warmaster Card ID 29 30 51 52 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 53 54 55 56 Name Incarnation of Death Lord of The Apocalypse Withered Soldier Infernal Creature Dragonfly Archer Greenwood Guardian Skull Hunter The Jungle Guidance Viper Sisters Gloryhorn Serperus Ender Flame Eternal Frost The One Above All Icetomb Arcane Ritual Dancing Flame Reckless Gamble Falling Meteor Gift of The Nature Electrocuted Divine Wrath Piggy Raven Ceremony of the Doom Unbound Darkness Righteous Immolation The Final Duel Table C.1: Card ID list 72 Appendix D API: Action IDs Action ID 10 11 12 13 14 15 16 999 Description Choose 1st player’s card on board Choose 2nd player’s card on board Choose 3rd player’s card on board Choose 4th player’s card on board Choose 5th player’s card on board Choose 6th player’s card on board Choose 7th player’s card on board Play 1st card on hand Play 2nd card on hand Play 3rd card on hand Play 4th card on hand Play 5th card on hand Play 6th card on hand Play 7th card on hand Play 8th card on hand Play 9th card on hand End turn Required Associated Action ID Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory Optional Optional Optional Optional Optional Optional Optional Optional Optional None Table D.1: Action ID list 73 API: ACTION IDS Associated Action ID 11 → 17 21 → 27 31 → 37 41 → 47 51 → 57 61 → 67 71 → 77 81 82 83 84 85 86 87 Description Target 1st opponent’s card on board Target 2nd opponent’s card on board Target 3rd opponent’s card on board Target 4th opponent’s card on board Target 5th opponent’s card on board Target 6th opponent’s card on board Target 7th opponent’s card on board Target opponent’s avatar Target 1st player’s card and 1st - 7th nent’s card on board Target 2nd player’s card and 1st - 7th nent’s card on board Target 3rd player’s card and 1st - 7th nent’s card on board Target 4th player’s card and 1st - 7th nent’s card on board Target 5th player’s card and 1st - 7th nent’s card on board Target 6th player’s card and 1st - 7th nent’s card on board Target 7th player’s card and 1st - 7th nent’s card on board Target 1st player’s card on board Target 2nd player’s card on board Target 3rd player’s card on board Target 4th player’s card on board Target 5th player’s card on board Target 6th player’s card on board Target 7th player’s card on board Note oppooppooppooppooppooppooppo- Only Duel Only Duel Only Duel Only Duel Only Duel Only Duel Only Duel used for The Final used for The Final used for The Final used for The Final used for The Final used for The Final used for The Final Table D.2: Associate Action ID list 74 ... methods applying reinforcement learning for MDP, such as Q -learning 2.2.2 Q -learning Q -learning is one of the core methods applying reinforcement learning for MDP The goal of Q -learning is as same... aim for a modern card game with smart agents based on modern machine learning approaches To achieve this goal, we are going to perform the following tasks: • Investigate card games to prepare for. .. designing an alternative card game with smart agents based on machine learning models • Build the proposed Card game in order to assist the smart agent • Build Intelligent Agents using modern methods