Extensive Endpoint Service Availability Testing (EESAT)
2002
1 Background
Cellular network quality in Malaysia is measured with EESAT with the purpose of benchmarking the Quality of Service (QoS) for cellular networks. The test was conducted in the Klang Valley from 30 August 2002 through 1 October 2002, involving five cellular networks:
a. Celcom (019) b. DiGi (016) c. Maxis (012) d. TimeCel (017) e. TMTouch (013)
EESAT could provide consumers with a common platform to gauge cellular networks performance. This is part of the Malaysian Communications and Multimedia Commission’s ongoing effort to keep consumers better informed.
Furthermore, this would encourage cellular network operators to improve their networks which could lead to a better level of service quality.
2 Methodology
Measurements were conducted with benchmarking tools that could generate calls automatically and uniformly for every cellular network. EESAT was conducted during weekdays between 8:00am and 7:00pm. In principle, EESAT involved sequence of test calls, defined in two methods of testing:
a. Drive test b. Static test
The drive test involves sequences of test calls made in a vehicle along selected routes which covers main roads, public access and hot spot areas. There are nine clusters identified for the drive test in the Klang Valley:
Cluster Name Area
Metro KL Kuala Lumpur City Centre including Bangsar, Damansara Heights and Pudu
North KL Gombak, Selayang, Sentul, Segambut, Kepong, Batu Caves, Sg. Buloh and Rawang
East KL Keramat, Setapak, Ampang, Hulu Kelang and
Pandan
South KL Cheras, Sungai Besi, Kajang, Bandar Baru Bangi, Serdang and Seri Kembangan North PJ Petaling Jaya (North of Federal Highway),
Subang and Damansara
South PJ Petaling Jaya (South of Federal Highway), Subang Jaya, USJ, Puchong and Kinrara Shah Alam Shah Alam area from Padang Jawa to Shah
Alam Stadium and NKVE to KESAS Highway Klang Klang Town, Port Klang, Westport, Meru,
Pandamaran and Kapar
MSC Putrajaya, Cyberjaya, KLIA and Nilai Town
Drive test is performed to assess network performance from a moving subscribers’ point of view. On average, 200 test calls were made for each cellular network for every cluster. All test calls were intra-network calls to its own test number.
Static test involves sequences of test calls made in a stationary position at selected locations which comprising public access, business and commercial areas. Ten locations were identified for static test in the Klang Valley:
Locations Central Market
Kuala Lumpur City Center (KLCC) The Store, Klang Town
MidValley Megamall
Kuala Lumpur International Airport (KLIA) USJ Taipan, Subang Jaya
Bangunan MPPJ, Petaling Jaya Dataran Putra, Putrajaya
The Street Mall, Cyberjaya Kompleks PKNS, Shah Alam
Static measurement test is intended to investigate and observe the network reliability and availability in terms of a subscriber’s perception at the selected public access area. On average, 200 calls were made for each cellular network for each location. The calls were segregated to 40 test calls to own test number
and 40 test calls to each of the other cellular network test numbers. Thus, the static test would also involve inter-network call handling.
3 EESAT Results
It is important to note that results of both the drive and static tests delineate cellular networks performance based on the selected routes and locations at the specific period of time only.
3.1 Service Coverage Performance
Service coverage performance is defined as signal strength received by subscribers’ mobile phones while they are having conversation or are in a dedicated mode. Technically, coverage performance is measured with the parameter Received Signal Level (RxLev) with units in dBm. The availability of service coverage for drive test is defined that a network must achieve a minimum of -98dBm of signal strength, and -82dBm of signal strength for static test.
Performance is based on the percentage of calls that have the minimum signal strength required. The performances of all networks for drive test are as shown below:
RxLev > -98dBm - Klang Valley 99.95
99.70
99.77
99.32
99.82
99.00 99.10 99.20 99.30 99.40 99.50 99.60 99.70 99.80 99.90 100.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
Percentage (%)
Drive Test
Celcom Digi Maxis TimeCel TMTouch Signal strength minimum
of -98dBm
99.82% 99.77% 99.95% 99.32% 99.70%
The performances of all networks for static test are as shown below:
RxLev > -82dBm - Klang Valley All
99.99 100.00 100.00 99.99 100.00
0.00 20.00 40.00 60.00 80.00 100.00 120.00
M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019
Percentage (%)
Static Test
Celcom Digi Maxis TimeCel TMTouch Signal strength minimum
of -82dBm
100% 100% 99.99% 99.99% 100%
3.2 Signal Quality Performance
Signal quality performance is defined as quality of voice transmission while subscribers are using their mobile phones. Excellent signal quality performance would guarantee no missing words or “ping pong” sound. Technically, this parameter is a measurement of Bit Error Rate (BER) of the signal received by subscribers. Bit Error Rate is then converted to a value ranging from 0 to 7. The parameter used is Received Signal Quality (RxQual). The result would give information of the radio quality performance as below:
RxQual Radio Quality Categorization
0-3 Excellent 4-5 Good 6-7 Poor
Performance is based on the percentage of calls that have the minimum signal quality of 5. Table below shows the performances of all networks for drive test:
RxQual <=5 - Klang Valley
97.74
99.04
98.22
98.90
98.31
97.00 97.50 98.00 98.50 99.00 99.50
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
Percentage (%)
Drive Test
Celcom Digi Maxis TimeCel TMTouch Signal quality (RxQual) =
5 and above
98.31% 98.22% 97.74% 98.90% 99.04%
The table below shows the performances of all networks for static test:
RxQual <=5 - Klang Valley All
99.62 99.63
99.50
99.58
99.54
99.40 99.45 99.50 99.55 99.60 99.65
M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019
Percentage (%)
Static Test
Celcom Digi Maxis TimeCel TMTouch Signal quality (RxQual) =
5 and above
99.54% 99.50% 99.62% 99.58% 99.63%
3.3 Speech Quality
Speech quality measurement in EESAT is conducted to emulate speech quality as perceived by the human ear. The parameter used in this test is the Mean Opinion Score (MOS). MOS is a subjective speech quality measurement, which usually involves convening listening panel to assess speech quality, as recommended in ITU-T Recommendation P.800. However, in this EESAT exercise, MOS is obtained through the objective speech measurement method called Perceptual Speech Quality Measure (PSQM), based on ITU-T Recommendation P.861. The scale is defined from 1 to 5. Definition of MOS scale would be as below:
MOS scale Speech quality
1 Bad 2 Poor 3 Fair 4 Good 5 Excellent
In this EESAT, MOS is measured in the drive test only. Performance is defined on the percentage of calls with the MOS value of 3 and above. Below are results of speech quality performance for all cellular networks:
MOS >= 3 - Klang Valley
98.45 98.56
96.73 96.91 96.86
95.50 96.00 96.50 97.00 97.50 98.00 98.50 99.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
Percentage (%)
Drive Test
Celcom Digi Maxis TimeCel TMTouch MOS = 3 and above 96.86% 96.73% 98.45% 96.91% 98.56%
3.4 Dropped Call Rate
Call drop is defined as a call that is disconnected prematurely during the 60- second holding period. A call drop would possibly be caused by weak signal strength, interference, congestion and other possible reasons. Table below
indicates percentage of calls that was disconnected prematurely during the drive test for each cellular network:
Dropped Call rate - Klang Valley
1.60
1.00
0.48 0.52
0.21
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 1.80
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
Percentage (%)
Drive Test
Celcom Digi Maxis TimeCel TMTouch Dropped Call Rate 0.21% 0.48% 1.60% 0.52% 1.00%
The table below indicates percentage of calls that was disconnected prematurely during the static test for each cellular network:
Dropped Call Rate - Klang Valley All
0.15
0.05 0.05 0.05
0.00 0.00
0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18
M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019
Percentage (%)
Static Test
Celcom Digi Maxis TimeCel TMTouch Dropped Call Rate 0.00% 0.05% 0.15% 0.05% 0.05%
3.5 Congestion Rate
Congestion is defined as failure to establish connection between caller and receiver after a call attempt has been made even when service coverage is good.
This is due to the fact that resources in the network are insufficient to provide services for all subscribers in the area: Table below shows the congestion rate of all cellular networks for drive test:
Drive Test
Celcom Digi Maxis TimeCel TMTouch Congestion Rate 0.27% *NA 0.38% 0.40% 0.35%
* DiG’'s result did not indicate its actual performance due to test number problem during the drive test
Blocked Call Rate - Klang Valley
0.38 0.35 NA
0.40 0.27
0.00 5.00 10.00 15.00 20.00 25.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
Percentage (%)
Table below shows the congestion rate of all cellular networks for static test
:
Blocked Call Rate - Klang Valley All
0.00
0.80
0.00 0.00
0.15
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90
M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019
Percentage (%)
Static Test
Celcom Digi Maxis TimeCel TMTouch Congestion Rate 0.15% 0.00% 0.00% 0.00% 0.80%
3.3 Call Setup Time
Call Setup Time is the total time measured from sending in the called party number to the time the other party is ringing. Performance is based on percentage of calls that have Call Setup Time less than 15 seconds. Table below indicates the performances of all cellular networks for drive test:
Call Setup Tim e < 15 Sec - Klang Valley
100.00 100.00 100.00 100.00 100.00
0.00 20.00 40.00 60.00 80.00 100.00 120.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
Percentage (%)
Drive Test
Celcom Digi Maxis TimeCel TMTouch Call Setup Time less
than 15 seconds
100% 100% 100% 100% 100%
The table below indicates the performances of all cellular networks for static test:
Call Setup Time < 15 Sec - Klang Valley All 100.00
99.52
100.00 100.00 100.00
99.20 99.30 99.40 99.50 99.60 99.70 99.80 99.90 100.00 100.10
M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019
Percentage (%)
Static Test
Celcom Digi Maxis TimeCel TMTouch Call Setup Time less
than 15 seconds
100% 100% 100% 100% 99.52%
4. Historical Comparison
The first EESAT was conducted in March and April 2001 covering six regions, namely Central, Northern, Southern, Eastern, Sabah and Sarawak. The test was conducted manually with the presence of representatives from network operators.
Only static test was conducted and six parameters were identified to be measured during the test, as follows:
a. Connection delay b. Successful Call Attempt c. Correct destination
d. Ability to hold the line connection (60 seconds) e. Line Clarity; and
f.
Service availabilityThe next EESAT was conducted from 19 November 2001 to 13 December 2001.
However, this EESAT was conducted only in the Klang Valley. This was the first automated EESAT where two methods of testing were conducted: static test and drive test. External consultant was appointed to conduct the test. Seven main parameters were measured in the test, as follows:
a. Downlink received signal level b. Downlink received signal quality c. Dropped Call Rate
d. Blocked Call rate e. Speech Quality (SQI) f. Handover Success Rate
g.
Call Setup TimeThe most recent EESAT was conducted from 30 August 2002 through 1 October 2002. It was also conducted in Klang Valley and involved static test and drive test.
The same parameters were measured with two changes made:
a. Previous EESAT required calls terminated to another mobile phone number, but recent EESAT required calls terminated to the exclusive test number
b.
For Speech Quality measurement, scale used was Mean Opinion Score (MOS), which consists of uplink and downlink measurement, as compared to previous EESAT which used Speech Quality Index (SQI)5. Results Comparison
Comparison on results could be made between EESAT done in Quarter 3 2002 and EESAT done in Quarter 4 2001. All graphs below show comparisons of results between the two tests:
Drive Test_Rx Lev - 2001/2002
99.17
99.49
99.23
98.72
99.94 99.95
99.70 99.77
99.32
99.82
98.00 98.20 98.40 98.60 98.80 99.00 99.20 99.40 99.60 99.80 100.00 100.20
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
RX Lev_2001 RX Lev_2002
Static Test_RX Lev sub_2001/2002
99.65
93.93
99.91
94.05
99.96
99.99 100.00 100.00 99.99 100.00
90.00 91.00 92.00 93.00 94.00 95.00 96.00 97.00 98.00 99.00 100.00 101.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 RX Lev_2001 RX Lev_2002
Drive Test RX Qual_sub - 2001/2002
95.36
98.81
95.47
97.20
97.69 97.74
99.04
98.22
98.90
98.31
93.00 94.00 95.00 96.00 97.00 98.00 99.00 100.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 RX Qual_2001 RX Qual_2002
Static Test_RX Qual sub_2001/2002
99.02
99.26
98.98
98.94
99.29
99.62 99.63
99.50
99.58
99.54
98.40 98.60 98.80 99.00 99.20 99.40 99.60 99.80
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 RX Qual_2001 RX Qual_2002
Drive Test_Handover Success Rate - 2001/2002
95.90
94.10
98.49
95.79
98.46
96.59 96.51
98.90
95.01
98.29
91.00 92.00 93.00 94.00 95.00 96.00 97.00 98.00 99.00 100.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
HOSR_2001 HOSR_2002
Static Test_Handover Success Rate_2001/2002
99.70
91.34
99.54
98.96
99.93
97.56 97.88
100.00
90.17
100.00
84.00 86.00 88.00 90.00 92.00 94.00 96.00 98.00 100.00 102.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 HOSR_2001 HOSR_2002
Drive Test_Dropped Call Rate_2001/2002
7.50
0.67 0.98
1.45 1.60 1.83
1.00
0.48 0.52
0.21 0.00
1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019
DCR_2001 DCR_2002
Static Test_Dropped Call Rate_2001/2002
4.30
0.04
3.02
3.41
2.23
0.15 0.05 0.05 0.05 0.00
0.00 0.50 1.00 1.50 2.00
2.50 3.00 3.50 4.00 4.50 5.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 DCR_2001 DCR_2002
Static Test_Blocked Call Rate_2001/2002
4.21
3.61
1.53
4.27
1.32
0.00
0.80
0.00 0.00 0.15
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 BCR_2001 BCR_2002
Drive Test_Blocked Call Rate_2001/2002
4.21 3.61
1.53
4.27
1.32
0.38 0.35
NA 0.40 0.27
0.00 5.00 10.00 15.00 20.00 25.00
MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 BCR_2001 BCR_2002
Drive Test_Call Setup Time <15 sec - 2001/2002
99.26
99.95 99.94
98.60
100.00
100.00 100.00 100.00 100.00 100.00
97.50 98.00 98.50 99.00 99.50 100.00 100.50
MAXIS_012 DIGI_016 CELCOM_019
CST<15sec_2001 CST<15sec_2002
Static Test_Call Setup Time <15 sec -2001/2002
98.45
99.42
99.33
98.80
99.11 100.00
99.52
100.00 100.00 100.00
97.50 98.00 98.50 99.00 99.50 100.00 100.50
MAXIS_012 TCSB_013 DIGI_016 TIMECEL_017 CELCOM_019 CST<15sec_2001 CST<15sec_2002
Most networks recorded improvements in Received Signal Level and Received Signal Quality in Quarter 3 2002, which gave indication of the improvement in coverage areas of cellular network. Dropped Call Rate also improved with the exception for TMTouch, which recorded higher Dropped Call Rate compared to previous EESAT. Improvement could also be seen for Blocked Call Rate and Call Setup Time. However, this could not be interpreted as actual improvement as the previous EESAT exercise had calls terminated to mobile phone numbers and recent EESAT had calls terminated to the exclusive test numbers.
6. Future Plan
Benchmarking would give an overview of the network performance of cellular networks in Malaysia. The exercise will be extended to other places in Malaysia in the near future. EESAT would be an ongoing process due to the fact that all operators optimise their networks at different period of time to cope with capacity demands and extensive network growth. Each cellular network operator is obliged to ensure their quality of service is at the best. Essentially, this would also benefit Malaysia in terms of the development of communications and multimedia industry and ensuring the level of competitiveness in the region.
