DxOMark Mobile test protocol and scores

The DxOMark Mobile test protocol takes a different approach to our sensor and lens test protocols. Rather than looking at a discrete component (sensor, lens) the Mobile protocol assesses the performance of the imaging pipeline in its entirety, including lens, sensor, camera control, and image processing. This means that still image quality evaluation in the Mobile protocol is based on the camera’s JPG output rather than on RAW image data as for sensors and lenses. The protocol also covers the test device’s performance in video mode, and scoring is calculated from both objective measurements and from perceptual evaluation of more than 50 challenging and realistic indoor and outdoor scenes by a panel of imaging experts.

Testing for most image quality criteria is undertaken with the smartphone camera in its default (auto) mode. We intervene manually only to activate the bokeh (portrait) mode, the flash, and the zoom feature.

The new protocol includes the evaluation of camera performance in a wide range of real-life situations.

We designed a dedicated studio setup for the new bokeh test.

DxOMark Mobile: What’s new for 2017?

With the mobile sector being a major driver of innovation in imaging, and smartphone camera technology rapidly evolving since the original DxOMark Mobile test protocol launched in 2012, the time has come for an update. The new version of DxOMark mobile takes into account the latest developments in mobile imaging technology and adds the following components:

  • The new zoom test analyzes image output at three subject distances and in several test situations (outdoors, indoors, and in the lab). Key criteria for judging the results are resolution and image detail, noise, color reproduction, and image artifacts.
  • A new bokeh test analyzes the camera’s ability to create a pleasing background blur while keeping the image’s main subject in focus. The test evaluates the strength, smoothness, and transition of the blur effect. It also looks at distortion on portrait subjects, artifacts, and the image processing’s depth estimation capabilities.
  • When testing the autofocus and sharpness, the updated test protocol now takes into account motion in the scene and the speed of image capture. Sharpness on moving subjects with a moving photographer is measured in a simulation setup in the lab. A new autofocus test evaluates not only accuracy but also speed and texture, and we now undertake noise tests shooting handheld as well as shooting with the device mounted on a tripod.
  • For the updated low-light analysis, we take still image samples at very low light levels between 1 and 20 Lux. For our video quality analysis, we record all low-light footage in between 5 and 20 Lux. Overall light levels for most photo tests now range from 1 lux to 1000 lux, and color temperature from 2300 to 6504 Kelvin.
  • We have expanded our DxOMark Mobile video analysis to include objective measurements in addition to perceptual evaluation for most test criteria. For many tests — for example, white balance, exposure, and noise, we now also measure how quickly and smoothly the camera adapts to changing light conditions during recording.

Perceptual analysis is undertaken by a panel of experts using image quality rulers.

A new studio test measures the camera’s speed of capture — that is, the time between pressing the shutter and image capture.

DxOMark Mobile Scores

The DxOMark Mobile Photo and Video sub-scores and the overall score are generated from a combination of objective lab measurements and perceptual analyses. The following test criteria feed into the Photo sub-score:

The Video sub-score is calculated from the following criteria sub-scores:

  • Exposure
  • Color
  • Texture and noise
  • Autofocus
  • Artifacts
  • Stabilization

Scores are presented in the format as shown below.

DxOMark Mobile
DxOMark Mobile

For a better idea of how the updated protocol impacts device scoring in practice, please have a look at our 2012 vs. 2017 score comparisons for a range of devices (including the Google Pixel, Nokia 808 and iPhone 7 Plus) that we tested using both test protocols.

  • Viacheslav Dynin


    What about the total score? Can it be more than 100?

    How overall score of HTC U11 can be 90, if max score HTC achieved on autofocus is ONLY 90, and other one less..

    If mobile phone achieved from different criteria 90 (Color, texture, noise and etc), overall will be 90 too, or 150?!

    If maximum score is 100, it means that, two cameras can be better than average, for example HTC U11 (perfect) and another one, mobile X from future (better than HTC U11, for example in two times, score in differrent criteria will be 95, for exampe).. And what will be it total score?! HTC 90 and Mobile X 92, or 95?! One mobile better than another in two times, but different in score only 5 points?!

    Thanks a lot

    • Lars Rehm

      Hi there, yes, the total score can be higher than 100, we just need to wait a little longer for the cameras to get even better πŸ™‚

  • Correct! Bokeh and zoom are the future of mobile photography! Very sad for the new scores of my HTC U11 πŸ™

  • Nomaned

    This smells really bad, you’re going to favor the iphone

  • Nomaned

    Zoom y bokeh ? In serius, this is to favor iphones

    • MrWalker1000

      convenient timing for the new iphones too.

  • Trugvang Ericksson

    How much did Apple shell out this time πŸ™

    Bokeh ..eeh and not resolution and low iso performance… Seems DXO-mark is going down the bias hill?

    • Jsmith8899

      Bokeh has existed on android forever though…right?

  • Kambis

    Optical stabilization should also enter into the Photo score. How many people are going to shoot with tripod like in the test scenario?

  • Marynarz

    Why i dont see Microsoft Lumia 950XL??? Because is te best mobile camera ine the world ? πŸ˜€ πŸ˜€ πŸ˜€

    • Dharmik Pipaliya

      Agree with yuhh brother. Totally nailed it..πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘ŒπŸ‘Œ 🀘🀘

      • babu pink

        Lumia 950 and XL are NOT taken for review by most of the people. I bet these will score at least 90 in their test.

  • ConCor

    Zoom score? A zoom lens is optional on a phone. I suggest you deduct points of the iphone for a lack of a wide angle lens then as well. This rating system is designed for an iphone.

  • Pfelelep_écrit_en_UTF-8

    We would learn a lot if we could compare perfs between mobiles and a cheap compact camera.

  • MasterElwood

    How do you do the video testing? what settings? 1080p? UHD? Just OIS or with EIS on? HDR on/off/auto?

    Thanx in A.

  • I don’t understand why you’re basing everything on JPG output. For all you know, the camera could take 30 photos of your stills scenes and remove all noise and extract extra detail for your tests to be fooled. I found that some of the finest cameras I’ve seen JPGs from have very noisy sensors when you actually look at the RAW file. Why test the results of computational photography and denoising algorithms on a phone when you are so meticulous about RAWs on everything else?

    • Klunst

      Because they are testing phones, not sensors ?

      • They should include raw sensor quality for those of us who shoot raw on the phone and edit ourselves. A JPG is pointless and mushy in a lot of cases.

  • Sywak

    Can you explain how you reach 90 with these results? (90 80 98 70 65 81 50 21 30)

    • Hello,
      The global DxOMark Mobile score for a smartphone is not based on just a simple average of all the subcategories’ scores. Rather, to compute this score, we first apply a specific weight to all Photo and Video subset scores to make sure that the final score correlates to the user’s experience.
      To define the weights, we rank subcategories from highest to lowest importance as perceived by the human visual system.
      Once we have this weighted-average intermediate score, we apply a linear and continuous mapping function to ensure that the best devices tested under the previous protocol achieve roughly the same Photo and Video scores with the new protocol. Use of this mapping function (as opposed to simple averaging) means that the global scores end up being higher than the subcategory scores.
      Please read this article to know more about it: https://www.dxomark.com/dxo

      • Sywak

        Ok, I see now, thanks for the reply !

We are hiring!