The rapid paradigm changes during the COVID-19 pandemic with regards to telecommunication services consumption and the instrumental role of connectivity in our daily lives have demonstrated the importance of Quality of Experience (QoE) as a competitive advantage or service differentiator. This article deals with how to carry out Service Traffic Measurement in a real environment to grasp the true customer experience and enhance the quality of service provided.
This article was originally published by MedUX.
QoE has been simplified and commonly related to network performance in the past. However, end users’ perception, satisfaction and QoE include the complete end-to-end system effects (client, terminal, network, services infrastructure, servers etc.).
Ensuring and providing the best-in-class QoE, both for fixed and mobile broadband services, is a critical, indispensable part of the recent story. Improving QoE will be a major challenge considering that some factors are beyond the control of Telecommunication Operators or need collaboration with third party actors from a multi-provider internet environment. It is an end-to-end process assessing the full-service path between source and destination of the communication.
This is one of the main reasons why at MedUX we take measuring both network and end-to-end services performance from the end-user perspective seriously, based on both objective and subjective indicators that contextualize and characterize the telecommunication services.
A similar term to QoE is QoS (Quality of Service). Among service providers, network operators and equipment manufacturers the term QoS has been in use for several decades[1]. QoS entails performance metrics, which are independent from the user perception, such as throughput, delay, jitter, and packet loss, among other indicators. QoS work is based on technical performance (i.e., it is mainly technology-centered) whereas QoE is based on end-user behaviour (it is user-centered).
MedUX unique measurement and analytics ecosystem proposes a model where technical measurements are transformed into perceptive indicators considering the human response to broadband service events, improvements and degradations. Our objective is to assess the QoS perceived by the user, estimate QoE from values obtained by making end-to-end real services measurements from the end-user perspective in a real environment.
Figure 1 MedUX unique customer-centric approach
In this regard, MedUX is also working on a methodology to develop a global QoS management model for next generation networks, taking advantage of big data and machine learning (ML). Our approach is based on the use of supervised and unsupervised ML techniques to identify network and service performance anomalies, as well as which KPIs are most relevant for the users. Our methodology links performance indicators with perceived QoE and provides information about the areas where corrective actions are required.
Our purpose is to obtain a real and independent view of broadband network performance from the end-user perspective in real-time, to provide telecom operators and other industry players a source of truth about how their Internet services are really performing, as well as benchmarking their positioning with their competition.
This article belongs to a monographic series about QoE and our approach to help telecom operators, regulators and other interested parties understand, measure and improve it:
- Introduction to MedUX measurement and testing methodology (this article)
- Mobile broadband benchmarking methodology
- Fixed broadband benchmarking methodology
- Launch of European measurement initiatives in new countries (RO, HU, CZ)
- MedUX QoE Scorecard methodology
- MedUX QoE seals, certificates and awards (NEW)
- Subjective customer feedback collection and NPS (NEW)
- AI/ML powered proactive QoE assurance (NEW)
Hereby, we introduce MedUX unique broadband measurement and testing methodology principles, that ensure that the data we collect is a reliable true representation of actual fixed and mobile network performance and experience. For more detailed insights don’t hesitate to contact us.
Introduction to MedUX Measurement and Testing Methodology
At MedUX measuring network and services performance means to collect and provide precise, trustworthy, and fair information and insights on the customer-perceived network quality (QoE), while covering a wide range of fixed and mobile services and allowing deep technical analysis based on the smallest details.
MedUX accurately measures both fixed and mobile broadband networks by collecting a vast amount of Key Performance Indicators (KPIs) to create large comparable datasets and know how networks are really performing from the end-user perspective without the need for integration with Telecom Operators. This provides a fair, transparent and independent evaluation of all networks.
For this purpose, we have developed a proprietary technology, our own measurement devices -so-called MedUX HOME, for fixed networks testing; and MedUX MOBILE, for mobile networks testing- that collect data independently from the access technology (e.g., xDSL/FTTx/HFC, Wi-Fi 5 and 6 and 2G/3G/4G/5G). MedUX plug&play testing units are vendor-agnostic and all-router compatible, based on a non-intrusive and reliable approach.
MedUX testing devices gather real-time network performance QoE metrics, through active and independent measurements of both technical parameters and real services, such as web browsing, streaming, cloud gaming or cloud storage, to understand the user’s real experience.
So, What are MedUX Main Benefits and Use Cases?
- Network and Services Performance Monitoring: continuously monitor networks and services to collect relevant performance and experience information and identify network problems and issues affecting end-users. We can help with network optimization, troubleshooting and technology developments.
- Benchmarking and Competitive Intelligence: elaborate comparative studies on the performance and experience of telecommunication networks and services, including scoring and ranking of access technologies, commercial products or international peers. We can help with promotional campaigns, independent quality certifications, network improvements and new technology deployments.
- Pre-launch testing and assurance: execute controlled and automated testing to fully test network upgrades and releases across core or customer premise equipment, in the lab, in the production network before massive testing, and in the network (live). We can help with new technology testing and deployments.
- Assessment and Independent Validation: transform your network change management and assurance capabilities with vendor-agnostic performance and customer experience visibility. It can support new technology deployments and vendors selection/negotiation as networks evolve.
- Regulatory compliance: obtain an independent measurement for regulatory purposes with basic QoS metrics (e.g. contracted speed, availability, reliability), and beyond with QoE, to manage regulatory risks, obligations enforcement, customer complaints and law of evidence. Our information is key to provide visibility into the QoS/QoE and network performance delivered by all network operators, and to shape future policies.
All our quality monitoring, benchmarking, testing and compliance management solutions are available for Regulators, Governments and Telecom Operators alike, but customized according to their needs and particular use case.
How do We Measure
MedUX big data analytics and insights provide competitive intelligence, service management, optimization and monitoring capabilities, helping Telcos, Regulators and Digital businesses with their quality of experience challenges.
- Benchmarking services against competitors without integration
MedUX offers managed services without integrating robots or any hardware or software into the operators’ networks. This way MedUX guarantees independence from industry players, enables benchmarking capabilities and shortens time-to-insight and results.
- Testing telecom networks 24/7
MedUX runs 24/7 pre-installed software on our testing probes that measure dozens of broadband performance and experience metrics of fixed and mobile broadband services to provide both historical and real-time network-wide visibility.
We help our clients gain visibility into network and service health indicators from the end-customer perspective to quickly localize service degradation and assess the impact of incidents. With MedUX analytics and insights, organizations can transform their service monitoring capabilities to redefine a customer-centric strategy as networks evolve.
- Focusing on the most used digital services
Increased latency or packet loss and lower throughputs surely represent bad QoE. However, the true customer experience depends on the service that the customer is using.
MedUX monitors the quality of the most used services, for example OTT (Over The Top) applications, such as social networks, messaging platforms, web browsing, video streaming, or gaming, from the end-user perspective. MedUX tests focus on the application layer to help understand the deterioration of service KPIs that impact end-user perception.
- Getting the true network-related Customer Experience
MedUX provides an accurate view of the QoS and its impact on the Customer Experience, focusing on the combination of at least the following basic network and service performance testing indicators to identify the QoE from the end-user perspective:
-Reliability (testing success rate): how consistent a network performs over time, by validating if the test has been successfully completed in a reasonable time.
-Service/Network Availability: how available the network and services are (network and service uptime), indicating the availability in terms of connectivity and access to the network from the end-user’s perspective.
-Matching the promise: indicates (in percentage value) to what extent the Quality of Service (throughput measure) received by the end user matches with that offered by the Telecom Operator in its commercial offer.
-Value for speed: how much data can be transferred (network capacity), emulating typical customer experience download/ upload based on a single- and/or multi-thread download/upload test.
-Accessibility (network responsiveness): how long it takes to access the network, while observing the network performance in terms of the transmission parameters and response time (packet delays, delay variation and packet loss).
-Services and OTT experience:
Web Browsing Experience (including web loading time, bits and number of resources)
Video Streaming (including average speed, buffering, streaming interruptions and video resolution, among others)
Cloud Gaming (including performance indicators for the most important gaming servers worldwide)
Cloud Storage performance (including service availability, transmission speed and success rate), and OTTs performance (Facebook, Twitter, Whatsapp).
- Gathering user feedback and perception
It is very complex to infer customer experience solely based on network performance parameters (e.g. latency, throughput, packet loss, etc.). In addition to having a wide and service-centric testing scope, we also ask our panellists (fixed broadband benchmarking collaborators) to participate in surveys and rate their providers.
With this new data source, we can gauge customer satisfaction and specific user perception feedback and compare that information with performance data to get a full picture of customer experience.
- Transforming QoS into a synthetic QoE score
Our MedUX QoE Score is an indicator aimed at easing operational and management performance reviews:
– Aggregating the complete set of network and service performance indicators (KPIs) into a single QoE score.
-Giving visibility on all statistics, parameters and weighting criteria to ensure a complete understanding of outcomes and potential improvement areas.
-Offering comparability across all relevant dimensions such as operator, access technology, geographical area or commercial offers.
Our scorecard uses fully customizable mathematical models to emulate end-user subjective evaluation of services. We basically convert our performance and service indicators into several MedUX Opinion Score (MOS) that are then aggregated into a single QoE score. In the following, an academic example of subjective testing for the assessment of Gaming QoE[2] for a set of well-known games based on MOS trajectories when considering three resolutions (A blue 1080p, B red 720p and C grey 480p) and varying bitrates for different games.
Our Statistical Approach
We follow one of the most intensive and accurate ways of measuring and understanding network performance, which delivers unrivalled detail and accuracy in a cost-efficient manner. MedUX collects an extensive dataset around internet performance across many telecommunication operators, geographies, access technologies, commercial offers, and speed tiers.
Our statistical model is focused on providing statistically representative results at local, regional or national level, or any other relevant dimension, depending on client needs and granularity requirements. It is a question of statistical analysis as to the number of data points needed to establish a reliable statistical sample for each analysis dimension or segment of the data. Selecting a statistically significant sample can ensure that the resulting QoS/QoE measurements reflect the service provider’s true performance with a certain degree of confidence that will allow for such measurement to be admissible under the law of evidence.
Our methodology is based on international best practices for sample plan design. The practical concern is understanding the minimum sample size and number of measuring devices to align with confidence interval and error margin targets.
In fixed broadband benchmarking and monitoring, based on our MedUX HOME solution, our approach is similar to the TV audience measurement (e.g. Nielsen), a model inspired by traditional statistical sampling and international standards defined by the International Telecommunications Union (ITU) and European Telecommunications Standards Institute (ETSI).
We aim at having transparent information on fixed networks and services performance by collecting in-field measurements across a representative sample of a given country where appropriate. We use the terms “cluster” or “universe” to refer to the sub-segments of the sample, which share specific characteristics or attributes such as service provider, technology, or geographical area, among others.
The number of tests per relevant geographical area and period of time, but most importantly, the number of observation points per cluster are the defining parameters of the model.
Our measurement model is based on traditional statistical concepts around the Law of Large Numbers (LoLN) and Central Limit Theorem (CLT). The LoLN states that the average of a random sample of a large population will tend to the average of the entire population. The central limit theorem says that the sum or average of a sufficiently large size of a random variable is approximately normally distributed regardless of the distribution of the population.
These statistical models are used to define the size of the sample, meaning the number of observation points and number of devices to be deployed per cluster or universe. It enables that the measuring outcome will be within the X% points (margin of error) of the average measurement of the real population value during Y% of the time according to the confidence level.
For example, for a typical deployment with 82% confidence level and a 10% margin of error requiring 45 observation points and devices collecting QoS/QoE KPIs every hour means that our statistics will be within 10 percentage points of the real population value 90% of the time.
In mobile broadband benchmarking and monitoring, based on our MedUX MOBILE solution, our approach is inspired by standards defined by the ITU and ETSI.
We aim at having transparent information on mobile networks and services performance by collecting in-field measurements across a representative sample of a given country.
Our mobile measurement methodologies available to reflect the true mobile customer experience:
- Indoor and outdoor testing
- Walk and drive tests
- Unattended probes
- Crowdsourced data collection
- Stationary and mobile scenarios
Covering a wider area, using more measuring devices or measuring over a longer period could provide more statistically representative results of the targeted area and region’s population.
As per best practices, we usually follow a two-step methodology that can be used to monitor the QoS/QoE at a national level:
- Stratification: can be used to calculate the number of geographical areas (e.g., cities, municipalities, towns, districts or roads) to be covered during a measurement campaign to get results that represent the network QoS at a national level.
- Simple random sampling: can then be used to calculate the number of measurements to perform at each of the geographical areas that were selected through sampling methods.
In addition to this, we may also use hypothesis testing based on the measurement results for all strata to ensure that the results obtained from the selected sample are meaningful from a statistical perspective. For regulatory purposes, it can be used to determine whether the service provider complies with the threshold established by the regulator, so no penalty would be imposed.
Below some of the international standards inspiring MedUX methodology:
- Recommendation ITU-T E.802 Amd.2 (06/2018), Framework and methodologies for the determination and application of QoS parameters
- Recommendation ITU-T E.806 (06/2019), Framework and methodologies for the determination and application of QoS parameters
- Recommendation ITU-T E.840 (06/2018), Statistical framework for end-to-end network performance benchmark scoring and ranking
- ETSI TR 103 559 V1.1.1 (2019-08), Speech and multimedia Transmission Quality (STQ); Best practices for robust network QoS benchmark testing and scoring
- ETSI TS 102 250-6 V1.3.1 (2019-11), Speech and Multimedia Transmission Quality (STQ); QoS Aspects for Popular Services in Mobile Networks; Part 6: Post Processing and Statistical Methods.
- ETSI EG 202 057-4 V1.2.1 (2008-07), Speech Processing, Transmission and Quality Aspects (STQ); User related QoS parameter definitions and measurements; Part 4: Internet access
About MedUX
MedUX is a Spanish multinational company that specializes in the measurement, monitoring and improvement of the user experience and quality of telecommunications, fixed, mobile and television networks.
Founded in 2014, the company has a presence in more than 20 countries, mainly in Europe and Latin America but also in Africa and the Middle East. In addition, it has a team of approximately 50 people at its headquarters in Madrid (Spain) and offices in Miami (USA), Bogota (CO), Mexico City (MEX), San Jos (CR) and Zaragoza (Spain).
Today, the company monitors and compares the services and user experience of more than 60 operators around the world, and its clients include large national and international telecommunications groups, such as Vodafone, Orange, Telefónica, Millicom, AméricaM vil/Claro and AT&T, and government entities, including Mineco (Spain), SUTEL (Costa Rica), TRA (UAE), SIGET (El Salvador) and CRC (Colombia).
MedUX is always working to improve the user’s quality of service and quality of experience, especially at a time when telecommunications and network performance are more important than ever. Get in touch with us at [email protected] if you need further information or request our demo to find out how you can gain end-to-end network visibility and real-time insights and performance statistics, to anticipate and address network issues.
