Khóa luận tốt nghiệp: Applying neural network models to classification of skin diseases and building applications for diagnosing skin diseases

Skin cancer is divided into many different types include:melanoma, Basal cell carcinoma, keratoacanthoma,...Based on related research, our research is to classify skin cancers using neur

NATIONAL UNIVERSITY HOCHIMINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY

ADVANCED PROGRAM IN INFORMATION SYSTEMS

LUONG THI THUY TRANG - 19521037

TRAN QUANG MINH - 19521856

BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS

THESIS ADVISOR

NGUYEN DINH THUAN

ASSESSMENT COMMITTEE

The Assessment Committee is established under the Decision

TỪ by Rector of the University of Information Technology.

ID - Chairman

)M - Secretary

4 - Member

carrying out our graduation project with better results.

Next, our group would like to thank Mr Nguyen Minh Nhut and the FITSU

research team for giving us the opportunity to participate and learn from research as

well as exchange useful knowledge with everyone in the group Receiving a lot of

help from friends in the group helped us gain a lot of new useful knowledge to equip

ourselves in particular and carry out our graduation project in general.

To this day, we do not forget to thank the teachers of the information systemsdepartment and the subject teachers who dedicatedly taught and shared usefulknowledge with us What we learn from our teachers is the foundation that helps us

successfully complete this graduation thesis.

And we would like to thank the reviewers and the council members for reading and

commenting on our graduation thesis so that we can improve the errors, thereby

improving the thesis this topic.

TABLE OF CONTENTS

œ4LL]k›

THANK YOU 2 — i

TABLE OF CONTENTS o5 5= << << HH TH TH 000 4 60 ii LIST OF FIGU RÌEVS 5 <5 5< 2 HH HH HH Hư 0900850 Y LIST OF TAABLEẺS 5 << 5< << SH Họ TH TH TH 00000000850 00 vii ABSTTIRRA CC T o 5-5 << HH HH HH 000006000 viii Chapter 1 Introduction - <5 < 5< 5< S5 2< 895 895 5938998998998998998958958958558588589559 1 1.1 Introduction topic dc s6 s6 9É 9.98 9 99.99 99.0.0009 00 0096999600650 1 1.2 Related WOIK doc 359559556 5969195 Error! Bookmark not defined 1.3 Callenge€ so s55 s5 9 Hi 00.000 000000001 00 2 1.4 Object and Target lÑ @S€ATCH s << << << + 5599964 095980598 55006580656 2 Chapter 2 Overview Algorithm IRes€arrCH 5-5555 < S4 S944 5935959595 4.9596 3 2.1 Efficient Neti nh TƯ .THT cu Ác HH HY HH hgg00000000000600003006 3 2.2 Performance of EfficientNet o- 5< 5G 5< 1 0060906096 98 50 08356 6 2.3 MESNGL.-T,, Ti 7 di HIŸHHAHhHYHgHY10300886866806806806000000806 6 2.4 Operating Principle Of MIFSÌNGI œs- << 5< s9 9 9 ni 09.00096896 50 7 Chapter 3 Methods Of Algorithm Implemenfafionn <=<<=<< << «<< << 11 3.1 DaASGÍ co cọ HC In 00000006 0096000408896 11 3.2 Data ÏTPAITI o G5 6É 9 999 99 99.99.990.900 09.00.0091 908095048966040 996 13 3.3 Processing Ï)4fa c << 9 9 Họ HT 0 00.00 000909908090 15 3.4 MFSNet ImpleimenfafIOI d-o- s5 << 9999.99.99.95 999900009 04 09ø 16 3.5 EfficlentNet ImpleimernfafIOI s- << << <5 5.5.9 99099 99 48 4 9ø 18 3.6 Evaluation And COTICÏUSIOH d- << <6 %8 9 984 9.984 999589699558969955896996 22 EU VY.vor an 22

EM» s20 1 22

E8 €Ề 22

3.6.4 ưa vo 23

Chapter 4 Analysis and Design The Application -.<-<<=<<<<<<<sesse 24

4.1 System Architecture cccscccsccsssssscsssssccsssssscscsssscscsssssscssssssccsssssscsssssssssssesseees 24

4.2 USCCASC rrscccrscscccsscscecsscsscccsccsessscsssscecsscsssssscssossncssessnessesssessssossncssessnesssssnessones 24

4.2.1 Usecase diapram - Ăn HT HT HH HH nưệp 24

4.2.2 List (on 3 25

4.2.3 ListOfUS€-CaSG Ặ Ăn HH TH TH HH HH nhe, 25

4.2.4 Usecase specification and activity diàram - ¿+ scssss+sxsxsxs+ 26

4.2.5 Sequence Dlagiraim - -c- + vn HT HH ưệp 39

4.3 Entiry Relationship ÏD1aBTAIH o-so <5 << 939356565 69859599889696 48

4.3.1 Entity Relationship Model ccceceseeseseeceeeeseeeeeeeseeseeeeeeeeeeneeeensens 48

4.3.2 Design Data nan ae 49

4.3.3 Entity Relationship model expÏaInation - - +++s<++x++sss2 49

4.4 System Structure ccceecccsssssessesseeseeseensceseceeceeeeeenseeseeeeesseeeseeseeeeeeaeeeeeeeeneeeaes 53

4.4.1 Back-end S€TV©T - St HH TH TH TH TH TH TH TT HH Hit 534.4.2 Mobile appÏICafIOT ĩ2 212211121121191 9111 11111 11 111 1 11 vn rt 55

4.5 System Work ion e 58

4.5.1 LOGIN USCL eee eee ecececeeescceseeseeeseeseeseeeseesecesecseeeseesceesesseeeseeseeeeeneeeeeaes 58

4.5.2 Register pationt 0.cccccccccccssessesscsseseeseeseeseesesecececseesecseeseeseeseeseeaeees 59 4.5.3 Doctor create schedule -: - + t1 12 391 2 ng gưkt 62

4.5.4 Doctor view scheduÏes - - c + x22 + 21 HH H1 1111111111111 re 644.5.5 Doctor ¿2á ồ.ƯỐỒỐỒ.ễễo.-'.Ä£ 66

4.5.6 Doctor check-out appOInI€HIE - c3 332 33+ E+EE+EsrEerrersrrrrexee 67 4.5.7 Doctor prescribe mediCIn€ - - 3 3+ + *2*+tE+EEEEEEEEEErrrrrrrrrrrrrree 70

4.5.8 Patient view and update prOẨiÏG - - - 5 5+ S+x + + xsEsrrsrrsrrerrkrrrrree 714.5.9 Patient book appO1InfI€TIĂ - - (2 + 3221311331811 rrkree 744.5.10 Patient view appointment and medicine - - 5 s+ss+ss+ss+ex+xs+ 754.5.11 Patient scan skin for diagnosis c5 + St +seserrrrrrrrrrrrrrrrrres 78

Chapter 5 Application Deployment scssssssssscssesscsssssscssesscessesseeees 80

ru USF oe 80

5.2 Register US€T Án HT TT TT TT Hàn TH ng HH 81

5.3 Doctor create schedule - - 2 E111 11231 11 5311113111113 1 E183 key 81 5.4 Doctor view schedùS - - 22c 32012313 1121111111111 111111 1 811 11 g1 HH re 82

5.5 vao 825.6 Doctor check-out appOITN€TIE - - - c3 22 1323833135111 E11 1E Erkrree 83

5.7 Doctor prescribe Imme€dICIn€ - c2 22322331133 151 1511115111111 rrke 84 5.8 Patient view and update profile 00 ccececeseeseeseeseeeeeeeseeeeeseeseeseeeeeeeeeeseeneeatens 85

5.9 Patient book appointment eeeeceecesceseeseeseeseeeeeeeaeeseeseeseeseeeeeaeeeeereeeateas 865.10 Patient view appointment and medicine -¿- -5-++++s+s+sx+sc+exserserses 87

5.11 Patient scan skin for diaQmosis 2.0.2.0 cccceceesescececeseseeeesceeeseeeeseeeeseeesaeeeeeeeeeeeees 88

Chapter 6 Conclusion and Future Work sscsccssscsssessesseessscssesseeees 90

Su 0) 0001 90 6.2 Future Work 7.58 TS BS S5 ĐT g4 4 90

REFERENCES 0 0G G Ọ cỌ cọ g0 000000000096 0090808 92

LIST OF FIGURES

os Lie

Figure 2.1.1 Width Scaling in EfficientNet illustration - s55 5< c+xsecsxsee 4Figure 2.1.2 Depth Scaling in EfficientNet 1lÏusfrafion - 55-5 S-+s+csccsecseres 4

Figure 2.1.3 Resolution Scaling in EfficienNet illustralon - 55555 <+s<++ 5

Figure 2.2.1 Performance EffficientnET [7] - +: +: +22 ++s£+£+EseEexereeserrereeres 6Figure 2.4.1 MFSNet i]ÏusfratiOT -.- - th ghe 7

Figure 3 1.1 Data QV€TVICW ST S111 11111111111 11 T1 H101 1111 11 11 g1 rà Hy 12

Figure 3 2.1 Data Augmentation Process Illustration «++s++<++sx++2 14Figure 3 3.1 Processing ÏDafa c c1 11 1211 11111118111 111111 11g n1 H1 Hi nh riệt 15Figure 3.4.1 MFSNet Implementation Illustrafion - « Figure 3.5.1 EfficientNet Implementat1oT -. 2c 32332 **+*£++vE+e+eeEeeeeereeresse 18

Figure 4.2.1 System Arch1f€CẦUT€ - - 2c 2212212112591 191 1811515511111 11111 11 g1 ray 24

Figure 4.2.1.1 Usecase Diagram for PaftI€nit - ¿22333 *+£++exexessereereeres 24

Figure 4.2.1.2 Usecase Diagram for ÏDOCfOF -¿- ¿22t **3 S2 £+E+eE+eeeeeereeresrss 25

Figure 4.2.4.1 Book An Appointment Activity Diagram +55 ssx+£+x+x 27

Figure 4.2.4.2 View And Update Profile Activity Diapram - - 5+5 +++++s 29

Figure 4.2.4.3 View Appointment And Medicine Activity Diagram 30Figure 4.2.4.4 Scan Skin Activity DIaBraim - cty 32Figure 4.2.4.5 Scan QR Code to Check Activity Diagram -.- +5 s+ss+ss+x+s 34Figure 4.2.4.6 Create Schedule Activity DiagTam - :- cSccs+xsexseserssrrrrrrres 35Figure 4.2.4 7 Prescribe Medication Activity Diagram 55c se s+s sex 37

Figure 4.2.4 8 View Profile for Doctor Diagram ACIVIẨV cccccccccscsereres 38

Figure 4.2.4.9 View List Schedule Activity IDiapram - + s+sssssersseesee 39

Figure 4.2.5.1 Book An AppoInt€ñf - ¿5s c3 3 23333931 EEeEEereexeeresrrrrrsres 40

Figure 4.2.5.2 View and Update Profile Patient - 5s sssssvrssseereseesee 41Figure 4.2.5.3 View Appointment And Medicine - - ¿5+ + s+s<xss+eserserseres 42

Figure 4.2.5.4 vi s nh 43 Figure 4.2.5.5 nv00) 1n 4A 44 Figure 4.2.5.6 Create Schedulπ c1 S11 1112112121111 1111111111111 11211 1 re 45

Figure 4.2.5.7 Prescribe MedICafIOT - + t1 9v 191 111g vn ng ngư 46Figure 4.2.5.8 View Profile for [DOCfOT + S12 9 vn ngư 47

Figure 4.2.5.9 View List ScheduÏe - - + tk ng ng HH ng HH re 48

Figure 4.3.1 Entity Relationship Model - s- cty 49

Figure 5.1.1 Back-end server architecture running on Google Cloud Platform

ENVITONMENE occ cece ee ccceecccccesscceececsscsscceesnseeccesussscceesasseecesssssececssasecesessssseeeessaeess 53Figure 5.1.2 Cross-platform mobile application architecture developed with

Flutter framework 0.0 a 55

Figure 5.2.1 Login User FÏOWH4T - (c2 3c 3233111931211 EEkrrrrke 59Figure 5.2.2 Register Patient FÏOWCHF ch ng ng Hư 61

Figure 5.2.3 Doctor Create Schedule Flowchart - - St sxs++eseserserrerrres 63 Figure 5.2.4 Doctor View Schedules Flowchart - ¿c5 Sc + ssxseseeserserseres 66 Figure 5.2.5 Doctor View Profile Flowchart - ¿- + St *+‡++eseeereereerssres 67

Figure 5.2.6 Doctor Check Out Patient Appointmenf s55 s++ss+++sz+s+2 69Figure 5.2.7 Doctor Add Prescribe MedIcIne ¿- 555 +2 *++‡+s£++vEeeeeeexseresss 71Figure 5.2 8 Patient View And Update Profile FlowchaTf -‹ - 5s <+s+s<++ 73Figure 5.2.9 Patient Book An Appointment Flowchart 55+ ++ss+ss+sx+sx+s 75

Figure 5.2 10 Patient View Appointment And Medicine Prescription Flowchart

I 03010020023 re 77

Figure 5.2.11 Patient Scan Skin For Diagnosis Flowchart - -5- «+ <+ec++ 79

Figure 5.3.1 Login User UL ccccccccssesseseesessenseeeseeseeseeseeseeesseesesaecsessesseeseeseeeseeeeees 80 Figure 5.3.2 Register Patien UI ccccccccscescssesseseeseeeeseeseeseeecseeecsecseeseeseeseeseseeeeesees 81

Figure 5.3.3 Doctor Create Schedule UI ccc ccccccsssseeseesseeeeseeeseeseeeseeeceseeeeeseeas 82Figure 5.3.4 Doctor View Schedules ] - - 6 SH HH gi 82Figure 5.3.5 Doctor View Profile Í, - «kg ng ghe 83Figure 5.3.6 Doctor Check Out AppoInmefI - - +: +c + s+xsxexeeserrerrsres 84

Figure 5.3 7 Doctor Prescribe Medicine - ¿22c 3t 331 1 EEErrrererrrrrrsree 85 Figure 5.3.8 Patient View And Update Profile - ¿- + St s+xsxexsersereeres 86

Figure 5.3.9 Patient Book AppoInf€nI - 5 + * + + E+E£eEeeeeereeseerreerree 87Figure 5.3.10 Patient View Appointment And Medicine Prescription 88Figure 5.3.11 Patient Scan Skin for Diagnosis ccccccesceceeceeceeceeeeeeeseeseeseeseeeeeeenees 89

LIST OF TABLES

os Lx

IE10GSE 5086.60.9111 25 Table 4.2.3.1 List of UseCase 1 26

Table 4.2.4.1 Book An AppOInIN€TIE - - 2 2232322331131 191 1911111111111 xe 27 Table 4.2.4.2 View And Update PrOfIÏe 25c S 223 SE +EESE+EESSEEErrrkrrrrrrrrkrrrrek 28 Table 4.2.4.3 View Appointment And Medicine - ¿+ tt EsEsEEvskersertekrrerrree 30 Table 4.2.4.4 Scan Skin 8 31 Table 4.2.4.5 Scan QR Code to CecKK - 5c 3231121191191 1511511111111 xe 33 Table 4.2.4.6 Create Schedule 1 35Table 4.2.4.7 Prescribe Medication cccesesseesesseseteeeeseseseseeeseseseneeeeeaeseeeeesseseneeserseseeeeeeees 36Table 4.2.4.8 View Profile For Doctor n1 38 Table 4.2.4.9 View List Schedule ccccccceececcssessecesceseseeeeseeeeseesceeeecseeecaeeseeeeateeeaceeeateesaeeeeaes 39

Table S5 INP, vip bhoaaaaaaaadầadi 50 Table 4.3.3.2 Schedule Entity 17177 50 Table 4.3.3.3 Appointment Entity - c2 2232232323 E532E 1211121111111 rkek 51Table 4.3.3.4 Pationt Emtity c ccccccccccccescscessescsesseseeccseeseseesesesseseeseseeseseesesecseeecseeecseeecseneeseneeaes 51Table 4.3.3.5 Medicien EnIẨV - c2 2c 2122325321131 151151151111 111 1111111111111 1 E1 T1 tre 52 Table 4.3.3.6 ResultDetail EETILIẨy - - c6 tt 121 1 119121111 HT 52

I0 SS52808092.0:00.015 ẻ 59 Table 5.2.2 Register Daf{I€TI( - + vn HT HT HH HT HH TT ngư 60Table 5.2.3 Doctor Create SchedUIÏC - - ¿5 tt St S912 2E 11121111 1011111 ve, 63Table 5.2.4 Doctor 4i 65 Table 5.2.5 Doctor 420i) 11211 67 Table 5.2.6 Doctor Check Out AppoIntN€TI - - - + t3 #EESEEEEkekEerskskekrkrrrree 68Table 5.2 7 Doctor Prescribe Medicine - - S3 129k EVESE SE re, 70Table 5.2.8 Patient View And Update ProfiÌe - - 5+ + S+*+t SE +EEseEEsrrrrrrrrrrrrrrrrree 72 Table 5.2.9 Patient Book Appointment - - (S331 VESEEvSkEgggnggggtgrrưy 74 Table 5.2 10 Patient View Appointment And Medieine - 55: 5+++s++x+sx+xsexsxseesrs 76

Table 5.2.11 Patient Scan Skin For Diagnosis - c6 3E E*EEEEeEerererrrsrreeee 78

Currently, the world in general and Vietnam in particular are facing climate change

We are already suffering the consequences of climate change on our health, and the

skin is the part directly affected by this change, thereby causing diseases including cancer skin and its early diagnosis is pivotal for the complete elimination of malignant

tumors from the body Skin cancer is divided into many different types include:melanoma, Basal cell carcinoma, keratoacanthoma, Based on related research, our

research is to classify skin cancers using neural network algorithms to support

differential diagnosis these types of skin cancers We use a newly designed neuralarchitecture based on an underlying, scaling network called EfficientNet[1], whichsupports image classification with better accuracy and improvement This study also

implements a disease segmentation procedure to determine disease identification and

localization, monitor morphological changes, and extract discriminating features forfurther diagnosis using the proposed framework, called MFSNet (Multi-FocusSegmentation Network) [2], which uses feature maps of different scales to calculate

the final segmentation mask However, the drawback of data imbalance makes the

models not perform well in their analysis, so we have researched some more methods

to handle data imbalance [3] Finally, we built a mobile application to support scanning

skin images to diagnose skin diseases and added features to support users in making

appointments with doctors and viewing a list of drug recommendations from doctors.doctor Frameworks used in the application include Flutter, building servers usingPython Django, and some other related technologies

The report include 7 chapter with contents on below:

- Chapter 1: Overview Present the research motivation, introduce the topic,

challenges and final goals

- Chapter 2: Overview Algorithm Research Presents an overview of skin cancer,

methods of classifying the disease, important factors in classifying and describing the technologies applied.

Chapter 3: Methods Of Implementation Presents data processing methods,details of neural network algorithms, and ways to apply them to research.

Chapter 4: Analysis and Design System Application Present the system

architecture and system analysis of the mobile application in the study

Chapter 5: Systen Implementation Present the method of integrating the modelinto the application and analyze the results after implementation

Chapter 6: Conclusion and Future Work Research summary, lessons and

limitations, future development direction

LIST MODIFY REPORT

No Description Modify Page

1 Change title of chapter 1 to 1

Introduction title.

2 Change title name of part 1.2 to 3

Literature Review and move to

chapter 2.

3 Change title name of chapter 3 to 11

Methods of Algorithm Implementation.

4 Merge chapter 4 and chapter 5 24

5 Change part 5.2 to Main Flow of 80

Application

Move part 3 of chapter 5 to new chapter and change to name

Deployment Application

Chapter 1 Introduction

1.1 Introduction topic

According to the Global Cancer Statistic 2018 report, 300,000 new instances of

melanoma were reported worldwide in 2018 [4] Skin cancer is uncommon in Asian nations, such as Vietnam, but more prevalent among Caucasians residing in the USA

and Australia [4] Because melanin is present in darker skin, those with darker

complexion among Asians and Africans are less likely to get skin cancer One way

to think about melanin is as a shield against the effects of UV A and UV B Furthermore, several studies have shown a rising trend in the prevalence of skin

cancer in European nations [5], but rates in Asia have generally remained steady In

Vietnam, based on the estimate from the International Agency for Research on Cancer, in 2018, there were 157 new cases of melanoma, and the mortality rate associated with this type of cancer was quite low compared with that associated with

the other types of cancer [6] Nonetheless, in Vietnam, the prevalence and death rate

from malignant skin cancer have been consistent in recent times Skin cancer is less

common in Vietnam compared to that in other countries But one of the few cancersthat can be prevented is skin cancer, and UV exposure is a modifiable risk factor.Furthermore, as a result of the effects of global climate change, tropical nations like

Vietnam are now subject to higher levels of UV radiation on top of their already protracted and intense sun exposure As a result, it is critical to increase public

knowledge about skin cancer and the preventative measures that may be taken due

to Vietnam's extensive sun exposure In other nations, skin cancer awareness and

understanding were well-established Studies on skin cancer awareness and behaviors, however, are scarce in Vietnam The fact that skin cancer is not common

in Vietnam makes people indifferent to it and there are no measures to promptly

diagnose the disease.

Therefore, in this study, we conduct research on neural network algorithms based

on existing technologies to classify skin cancers using Efficient network models and

skin segmentation to determine the exact location of skin cancer by MFSNet Alongwith that is building a mobile application to provide timely diagnosis and support to

users about skin wounds, scheduling an examination with a doctor, and seeing medication recommendations, if any.

1.2 Challenges

- Limited by lack of data: With the goal of adding skin data of Southeast Asian

or Asian people to expand and develop more suitable research in Vietnam, however

this data set is still rare and limited, currently there are only data sets of ISIC 2018

but does not clearly distinguish between skin colors.

- Unbalance of dataset: The project conducts research on a number of data sets

HAMIOK, Fitzpatrick17k, these data sets are divided into many different types ofskin cancer, however there is a data imbalance between types of skin cancer in these

dataset.

- Assessing the accuracy of the model is limited due to factors such as dataimbalance and unsatisfactory output, leading to the evaluation model not being

optimal the accuracy of the model.

1.3 Object and Target Research

- Object: Dataset Research HAMI0K, Fitzpatrick17k.

+ Process data and build models to train the dataset

+ Install and experiment with EfficientNet, MFSNet model algorithms

+ Analyze and evaluate the results obtained

+ Built an application with features to diagnose skin diseases, schedule

examinations and view medications

Chapter 2 Overview Algorithm Research

2.1 Literature Review

Mingxing Tan et al.[7], with research EfficientNet: Rethinking Model Scaling

for Convolutional Neural Networks to study and rethink the process of scaling up ConvNets Unlike conventional practice, which arbitrary scales these factors, this

method uniformly scales network width, depth, and resolution with a set of fixed

scaling coefficients It achieve much better accuracy and efficiency than previous ConvNets In particular, their EfficientNet-B7 achieves state-of-the-art 84.3% top-1

accuracy.

Doyoon Kim et al [8], with research Deep Transfer Learning for AutomatedDiagnosis of Skin Lesions from Photographs study on ImageNet with finetuning

on melanoma detection They compared EfficientNet, MnasNet, MobileNet, and

found the mobile network EfficientNet and achieves the best mean performancewith an area under the receiver operating characteristic curve of 0.931+0.005

Hritam Basak et al [2], with research MFSNet: A Multi Focus Segmentation

Network for Skin Lesion Segmentation develops an Artificial Intelligence (AI)framework for supervised skin lesion segmentation employing the deep learningapproach, the proposed framework is MFSNet, uses differently scaled feature mapsfor computing the final segmentation mask, the images are preprocessed to removeunwanted artifacts and noises

2.2 EfficientNet

EfficientNet is a convolutional neural network built upon a concept called

"compound scaling.” This concept addresses the longstanding trade-off between model size, accuracy, and computational efficiency The idea behind compound scaling is to scale three essential dimensions of a neural network: width, depth, and resolution.

+ Width scaling: means more feature maps (filters) in the convolutional

Figure 2.2.1 Width Scaling in EfficientNet illustration

+ Depth scaling: means more convolutional layers

Depth scaling

layer_i

higher resolution

deeper

Figure 2.2.2 Depth Scaling in EfficientNet illustration

+ Resolution scaling: means that it processes input images with larger width

and depth (spatial resolutions).

Resolution scaling

higher resolution

Figure 2.2.3 Resolution Scaling in EfficienNet illustraion

This scaling method has generally contributed to improved accuracies on the

majority of benchmarking datasets However, the traditional methods of scaling models are highly arbitrary Certain models have a depth scale, while others have a width scale To get better outcomes, some models simply take in pictures with a

higher quality Randomly scaling models is a labor-intensive process that needs

manual tweaking and several man-hours, and it frequently produces little or no

performance gain In a far more moral approach, the EfficientNet authors suggestedscaling up CNN models to achieve more accuracy and efficiency [10]

EfficientNet is divided into many models with different structures EffectiveNet

BO through B7: Every version correlates to distinct width, depth, and resolution measurements Compared to earlier iterations, later models are usually bigger,

include more information, and can learn more complicated features Minor instances

such as B0 or B1 are typically lightweight and smaller in size, making them suitable

for deployment on devices with limited resources Meanwhile, larger instances such

as B5, B6, and B7 provide more complex feature learning but require morecomputational resources

% fh 3 eResNet-152 Top! Acc #Params

° ‘DenseNet-201 RewNer152 (He etal, 0016) | 778% — 60M

5 ep TEEN 2 EfficientNet-B1 79.1% — 78M

Selif s ResNeXt-I01 (Xie etal 2017)| 80.9% 84M

764 4 EfficientNet-B3 81.6% 12M

" fi ResNet-50 SENet (Hu et al, 2018) 32.7% 146M

1 a NASNet-A (Zoph et al., 2018) 82.7% 89M

Number of Parameters (Millions)

Figure 2.3.1 Performance EffficientNet [7]

According research [7], the author gave result of performance that the biggest

EfficientNet model EfficientNet B7 obtained state-of-the-art performance on theImageNet, EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy onImageNet, while being 8.4x smaller and 6.1x faster on inference than the bestexisting ConvNet, EfficientNet-B1 is 7.6x smaller and 5.7x faster than ResNet-152.EfficientNetBO having very less number of parameters and still having betteraccuracy than ResNet 50 which has significant amount of parameters [11]

Extracting and classifying specific features from medical images also facilitates theapplication of machine learning techniques to automate the analysis process and

empower medical professionals with additional support tools important in their daily work [2 ].

In order to address the issue of skin lesion segmentation in photos, we have

focused our study work on analyzing a particular segmentation model known as

MFSNet (Multi-Scale Feature Fusion Network) in medical image MFSNet uses feature maps at different scales to calculate skin lesion characteristics at various

levels of detail The process of integrating feature maps with different scales helpsthe model take advantage of information from low-level to high-level features,thereby enhancing the ability to recognize and segment more accurately [2]

MFSNet use the convolutional neural network (CNN) model Res2Net, which

was recently developed, as its (backbone) Using the deep features acquired byRes2Net, MFSNet creates a global map for the segmentation mask using a ParallelPartial Decoder (PPD) module

2.5 Operating Principle of MFSNet

A neural network's first three layers are frequently used to extract low-levelfeatures, or features with a high resolution but limited spatial information Neural

networks frequently concentrate on recognizing fundamental elements in the image, such as edges, corners, or minute details, while extracting these characteristics,

giving little attention to the entire spatial structure of the picture Although a lot ofdetail is frequently recovered at this point, nothing of the image's extensive spatialorganization is shown

Two- and three-level features, containing important edge information, are used

in the BA (Boundary Attention) module to improve boundary representation Theyare then used for two different purposes First, they are fed into the next two layers

of the neural network, whose output is fed into the PPD (Predictive Path Decoder) module to generate a global segmentation map This map is used as a global map to

determine the general location for segmentation Second, they are fed into RA(Regional Attention) branches along with the segmentation map, to be used as global

instructions for the entire learning process of the neural network.

In diagnosing medical pathology, medical professionals often perform an initialestimate of skin melanoma before looking in more detail at the cellular level to

determine the exact location and assignment suitable label However, during the learning process, the neural network cannot easily adjust the rest of the information

to recognize important regions without specific instructions, leading to less accuratesegmentation Most existing methods mainly focus on using image classification

networks that are lightly tuned to discriminative regions in the image This causes bias when predicting dense regions MFSNet proposes a two-stage segmentation

method using the RA parallel structure to mitigate this problem and extend the

diagnostic method in practice.

Edge information can guide the feature extraction process for segmentation by providing useful supervision with refined boundary constraints MFSNet used a BA

module combined with parallel RA branches to extract accurate boundary

information This makes it easier for the neural network to focus on more detailed

boundary recognition, thereby improving the feature extraction process for imagesegmentation.

In the field of image processing, the Partial Parallel Decoder (PPD) module is a component of neural network architecture that is frequently utilized in image

segmentation models Usually, the goal of this module's design is to provide preciseand thorough segmentation maps in order to solve the picture segmentation problem

In particular, the PPD module frequently reproduces certain characteristics andinformation by utilizing data from earlier layers of the neural network, whichenhances picture segmentation skills PPD often use a variety of methods, includingskip connections, upsampling, and combining data from earlier layers, to produce

intricate and superior segmentation maps With the help of this module, the neural

network may recover significant characteristics from earlier layers and utilize them

to produce segmentation maps that are more precise and comprehensive

Workflow of MF SNet propsed:

To extract features, the input picture is processed through several convolution

layers Since they retain sufficient information, only the features from the secondand third convolution layers are thought to be helpful for edge guiding throughout

the learning process As a result, the BA module uses them to explicitly learn boundary information The BA module simultaneously receives input from the

global segmentation map (output from PPD) and shallow features from theconvolution layers By performing a series of distance transformations and other

mathematical operations, an improved boundary map is created, which is then used

by the RA modules The upsized segmentation map from the following layer, the

BA module, and the features from the associated convolution layer are fed into the

RA module The RA module learns features to generate segmentation masks

including the foreground and background layers using two distinct input branches

As a result, in places where two adjacent layers share high-level semantic

information, the RA module creates a mask for each layer to improve the response

of the other layer Ultimately, the segmentation output from the RA branch isgenerated by combining the predictions of these two branches.

Chapter 3 Methods Of Algorithm Implementation

3.1 Dataset

Based on further observation and analysis of the datasets we studied, we collect

and process two main datasets: ISIC2018 and Fitzpatrickl7k The dataset wecollected ISIC2018 dataset [28], which provides a large and diverse dataset, with 7

different disease labels assigned to each image.

Unfortunately, only 4 diseases labels are included in the Fitzpatrick17k dataset,

despite the fact that it also has a high number of photos What makes the woundpictures unique is that they are scattered throughout several regions on each image

rather than being distinctly centered or grouped Because of this, it is challenging to

precisely categorize and locate the injuries In particular, correctly recognizing andcategorizing lesions is one of the many obstacles that arise when processing pictures

containing these heterogeneously spread lesions for research purposes, additionally, the image was taken of the whole body and not on a surface of the skin as expected.

For this reason, after careful consideration and feasibility assessment, we

decided to conduct the main study on the dataset from ISIC2018 This data set not

only provides a full range of 7 disease labels, but also provides more uniformity and focus on the location of wounds on each image, making classification and

positioning more natural and effective more exactly

- 7 labels disease of skin cancer include:

+ Melanoma (mel): A melanoma is a tumor produced by the malignant

transformation of melanocytes

+ Melanocytic nevus (nv): is indicate for a mole Nevi can appear

anywhere on the body They are benign (non-cancerous) and typically

do not require treatment [19].

+ Basal cell carcinoma (bcc): is most often a benign form of skin cancer

caused by exposure to ultraviolet (UV) light However, it’s the most

frequently occurring form of all skin cancers, with more than 3 million people developing BCC in the U.S every year [20]

+ Actinic keratosis (akiec): which is sometimes called solar keratosis, is

usually benign [21]

+ Benign keratosis (bk1): is a common noncancerous (benign) skin

growth [22].

+ Dermatofibroma(df): are referred to as benign fibrous histiocytomas

of the skin, superficial/cutaneous benign fibrous histiocytomas, or

common fibrous histiocytoma [23]

+ Vascular lesion (vasc): Vascular lession are subdivided into benign,

locally aggressive or borderline and malignant tumors [24]

Figure 3 1.1 Data Overview

Our training data consists of 10015 images of ISIC 2018, stored as JPG files torepresent the images Additionally, we also use a CSV file to add information aboutthe images during training This CSV file can contain information such as image

labels, file name about the image Besides, this dataset has problem is not balance,

so the method to solve it in this project is use weighted sampler.

Test data includes 1512 images, which are also stored as JPG files for ease ofreading and image processing Similar to the training data, we also have an

accompanying CSV file, which provides additional information about the images

during testing This information can help us compare prediction results after the

model has been trained with test data.

3.2 Data Inpaint

Before feeding the data into the model for training, we perform a series of datatransformations, called data inpaint The purpose of this part is to remove hairs,feathers, and image noise components that lead to reduced model performance

Convert Image to Gray Scale

Morphological filtering

Find the hair countours

intensify the hair countours

Inpaint the original image

Figure 3 2.1 Data InaintProcess Illustration

+ Convert Image to Gray Scale: First, the skin image will be converted from the

RGB color space to a grayscale image by using cv2.cvtColor( src,

cv2.COLOR_RGB2GRAY )

+ Morphological filtering [27]: a kernel created for use in morphological

filtering cv2.getStructuringElement (1, (17,17))

+ Find the hair countours: Use Black Hat [26] filtering (cv2.MORPH_BLACKHAT)

on the gray image to find the hairline Black Hat operation helps highlight brightdetails on a dark background

+ Intensify the hair countours: A certain threshold (threshold) is applied to the

resulting image from the previous step to create a mask, where pixels with values

higher than the threshold will be considered hairlines,

cv2.threshold(blackhat,10,255,cv2.THRESH BINARY)

+ Inpaint the original image: This function cv2.INPAINT_TELEA will use the

information available in the image to try to "predict" the values of the missing area (defined in the mask) based on what is available around that area.

During the process of preparing data for the model, the CustomDataset class will

be responsible for performing subsequent image processing after the augmentation

process Specifically, at this stage, the image will undergo a resize step to fit the desired input size and then be converted into tensor form through the process of

using ToTensorV2()

e Resize(): After applying transformations (augmentation), some deep learning

libraries need the input image size to be fixed Therefore, before injecting images

into the model, a resize step can be performed to ensure that all images are the same size This resizing use a fixed size to normalize the size of all images in the dataset

e ToTensorV2(): The final step in data preparation is to convert the image from a

regular image format to tensor format using ToTensorV2() Tensor is the dataformat that machine learning models often use for processing and training, so this

conversion is an important step to be able to feed the data into the model.

As aresult, after these steps, each image in the dataset will be ready to be included

in the machine learning model through the dataset class

3.4 MFSNet Implementation

Figure 3.4.1 MF SNet Implementation lllustration

Data after inpaitned carefully before applying the main modules The data isincluded in the MFSNET model, passing a series of short-circuit neural network

modules Each of these modules consists of many short -layer layers with the

function of analyzing and extracting hidden characteristics from input data, is used

self.rfb4 1 = RFB_modified(2048, channel), self.ra4 conv5 =

BasicConv2d (256, n_class, kernel _size=1) and self.aggl =

aggregation(channel,n class) present for RA module and PPD Module These

short -sized layers operate in parallel to search for different patterns, edges and localcharacteristics in the image, thereby creating a typical representation of the image

After the data has passed through all the modules, the MFSNET model creates a detailed segment map This map marks location, boundaries, and identifies the

objects in the image, providing accurate information about the location and type ofobject

MFSNET Evaluation:

With MFSNet, this project we use Sensitivity(Sen) to evaluate performance.

It determines the percentage of pixels that belong to a correctly classified object,

identified as the desired object class This is an important measurement to evaluate

the ability of the object recognition model, indicating the level of accuracy in

detecting and locating objects in the image.

Sensitivit True Positive

ensitivity = ————————

y True Positive + False Negative

+ True Positive: present for object which is predicted correct.

+ False Negative: present for pixel belong object which is predicted

incorrect

So, the result after we use MFSNet on dataset ISIC2018 which is used in

this project reach about 0.952 compare with original model by authors [7] is 0.954 3.5 EfficientNet Implementation

Data processing begins with the augmentation phase This is the process bywhich we expand and transform the original data to create a more diverse data set,

helping the deep learning model understand variations and increase generalization

ability The data after the augmentation stage was put into a pre-processing step to

ensure size, then passed through the MFSNet model, an important step to extract

important information of wounds in skin diseases from image data , this information

becomes an important basis for building the next model We used the MFSNet model to preprocess the data before feeding it into the main model EfficientNet-B2

[25] The MFSNet model is applied to extract high-quality features from input data,creating a basic feature representation that the main model can use for learning andprediction adjust them to create accurate predictions for the desired classes

After obtaining feature representations from MFSNet, we fed them into theEfficientNet-B2 model, a version of the EfficientNet family of models, pre-trained

on a large amount of image data from different sources which is with 1408 features.

This helps make optimal use of features extracted from data, facilitating modellearning and prediction with high performance To optimize learning from these

features, we added three linear layers nn.Linear(model fc.in features, 512)

,nn.Linear(512, 256), nn.Linear(256, 7) These layers have the function of

representing relationships complexity between features, creating a multidimensional space to classify classes of output data Using these linear layers helps the model

learn important information from the extracted features and adjust them to create

accurate predictions for the desired classes.

The final result returned from the model is a tensor of size [32, 7], where 32 is

the number of samples and 7 is the number of output classes, corresponding to the

classes or labels we want to predict

Figure 3 5.1 Training Processing Illustration

The training process uses a previously modified efficientNet model, containing

many layers and weights that are adjusted during the training process To update the model's weights, we use the Stochastic Gradient Descent (SGD) optimization

algorithm, configured with a learning rate of 0.001 and momentum of 0.9,optimizer = optim.SGD (filter (lambda Pp: b.requires_ grad,

model.parameters()), lIr=0.001, momentum=0.9) The loss function is an

indispensable component in the training process In this case, the Cross EntropyLoss function is used, which is suitable for the classification problem This

function helps measure the distance between the model's prediction and the actual label, thereby optimizing the model to minimize this difference, criterion =

nn.CrossEntropyLoss() To adjust the optimizer's learning rate over time,training process use the Exponential Learning Rate Scheduler exp _1r_scheduler

process, the Batch Normalization classes in the model are frozen by setting the

requires_grad of their parameters to False This helps maintain model stability and

accuracy during training.

The results of the training process include prediction and loss training Train loss, calculated from the Cross Entropy loss function between the prediction and

the actual label on the training data These two metrics help evaluate and measure

the model's performance during training The output goes through softmax, softmax outputs = F.softmax(outputs, dim=1) detach (), the softmax function

is applied on outputs with a dimension length of 1 (dim=1), which means it will

calculate the probability of each class in the outputs When this probability is calculated, we obtain a new tensor with the same size as the previous outputs but the value is converted to the corresponding positive probability.

! + | Dataloader

test_acc fest_loss

Figure 3 6.1 Test Processing Illustration

During this testing process, the model that has been trained on the training datawill be used and during the testing process, the model will be applied to predict on

the test data 1512 images In this process, the loss function is also used similarly to

the training process, the image reading process in both training and testing stages isthe same to ensure correct label prediction The results of the testing process includeprediction and loss test - that is, the loss value calculated from the Cross Entropy

loss function between the prediction and the actual label on the test data.

3.6 Evaluation And Conclusion

3.6.1 Accuracy

The frequency with which a machine learning model accurately predicts theresult is measured by its accuracy By dividing the total number of guesses by the

number of right forecasts skin disease , accuracy may be computed [13].

Cons: Since accuracy gives the model's ability to forecast all categories equal

weight, it is less beneficial if the classes in the data are imbalanced

True Positive + True Negative

A =

mm All Predictions

3.6.2 Precision Precision is a metric that measures how often a machine learning model correctly predicts the positive class Calculate precision by dividing the number of

correct positive predictions (true positives) by the total number of instances themodel predicted as positive (both true and false positives) [14]

Precisi True Positiue

recision = ———————

True Positive + False Positive

3.6.3 Recall

Recalling that some measurement of machine model frequency measurementsaccurately determine the active (real positive) cases from all actual positive models

in the data set.

Recall True Positives

ecall = sa

True Positives + False Negatives

3.6.4 F1-Score

The F-score is a way of combining the precision and recall of the model, and it

is defined as the harmonic mean of the model’s precision and recall.

FL =2 Precision * Recall

score = (Precision + Recall)

This is result after process model :

Accuracy Precision Recall F1-Score 0.82 0.711 0.77 0.71

- Accuracy: This is the percentage that separates the total number of data points

from the number of accurate forecasts The accuracy of the model, or theproportion of accurate forecasts to total predictions, was around 82%

- Precision: Ratio between the number oftrue positive predictions and the total

number of positive predictions In this case, about 71.1% of the positive

predictions were identified correctly

- Recall: The ratio between the number of true positive predictions and the

actual total number of positive cases For the dataset, about 77% of the true positives were correctly identified by the model.

- The FI-Score is approximately 71%, providing a composite figure for the

model's overall performance in balancing Precision and Recall

Chapter 4 Analysis and Design The Application

Both patient and

doctor must to log in Book an before use function appointment

status

Figure 4.2.1.1 Usecase Diagram for Patient

«extend»

-Delete result

ooo view Profile

Doctor

wextend» _ schedule

«eAtends eee View list V2 View History

1 Patient People who use all fuction for role patient

2 Doctor People who use all function for role doctor

Table 4.2.2.1 List Actor4.2.3 List of Use-case

Patient’s Role Book an appointment Make appointment for patient

View Medicine Show medicine suggest by

doctorProfile View and set up profile for

patient

Scan Skm Scan Skin or take photo to

predict diseases

Scan QR Code to | Check QR Code of appointment

check appointment |to set status done for

status appointment

Doctor’s Role | Create Schedule Create medical examination

schedule

View Profile View profile doctor

Prescribe medication | Create prescriptions for patients View list schedule View list schedule after create

Table 4.2.3.1 List of UseCase4.2.4 Usecase specification and activity diagram

4.2.4.1 Book an appointment

Usecase name |_ Sax Book an appointment

Description Make an appointment for patient

Trigger Patient click to button has icon shedule

Pre-condition 1 Patient’s devices must connect to Internet

2 Not included two patient book appointment at

the same time.

Post-condition Patient confirm pop up complete make appointment

Basic flow 0 Patient click on button has icon schedule

1 System display screen book appointment

function

2 Patient choose date

3 Patient choose start time

4 System search schedule base date and start

time which available from doctor.

5 Display doctor meet schedule available

6 Patient choose doctor

7 Confirm create appointment

8 System store data to database appointment

9 Display pop up successful Alternation flow

Search schedule available

Get doctor to dislay sereen doctor

Choose Doctor

Display pop up successful

Descrtiption View and set up profile patient

: : : lộ

| Trigger Patient must click on button Q |

| Pre-condition Patient’s device must connect to Internet |

Patient’s account existed

Post-condition Patient gets screen profile with information profile

database PATIENT

System display information profile

Patient select finish process or updateprofile

Patient enter new information profile.

System validate format.

System store data.

Alternative flow 4a Patient choose update profile Use case

continue step 5

4b Patient choose cancel Finish process

6a System validate format succuess, use casecontinue to next step 7

6b System validate format fail Usecasecontinue step 5

Table 4.2.4.2 View And Update Profile