Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/development' into development
Browse files Browse the repository at this point in the history
  • Loading branch information
@vnemes
vnemes committed May 27, 2017
2 parents bf35499 + 50bcaa7 commit 74c81a1
Showing 1 changed file with 61 additions and 19 deletions.
80 changes: 61 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,59 @@
# AndroidBenchmark
## Introduction

## Activities
The *AndroidBenchmark* is an Android Application useful for assessing the performance of any device which runs on the Android Platform. It consists of a comprehensive suite of tests which evaluate the most important components of a device, such as the CPU, Internal Storage and the Wireless Adapter.

We have decided to start this project firstly because [Smartphones have become the most used mean to access the internet](https://www.theguardian.com/technology/2015/aug/06/smartphones-most-popular-way-to-browse-internet-ofcom), so there must exist a standardized way to evaluate and classify their performance and secondly, since [Android now the world's most popular operating system](http://www.networkworld.com/article/3187011/mobile-wireless/android-is-now-the-worlds-most-popular-operating-system.html) this platform seemed like the perfect fit to collect the biggest data set.

## Design and Implementation
### Benchmarks

The application consists of 7 benchmarks, each and every one of them testing a different functionality of the device under test.
The benchmarks implemented are as follows:
+ CPU Benchmark
This benchmark evaluates the performance of the CPU. It consists of 3 smaller benchmarks:
<br>*Integer arithmetic*: Stresses the ALU by doing lots of integer computations.
<br>*Floating point*: Stresses the FPU by doing lots of floating point computations.
<br>*Digits of PI*: Puts caching and multithreading to the test by computing digits of PI, using the
[Bailey–Borwein–Plouffe formula](https://en.wikipedia.org/wiki/Bailey%E2%80%93Borwein%E2%80%93Plouffe_formula) in a parallel manner.
+ Hashing Benchmark
<br>This benchmark computes hashes using the BCrypt Algorithm. [The implementation](http://www.mindrot.org/projects/jBCrypt/) of the BCrypt class was provided by Damien Miller.
+ Network Benchmark
<br>Measures download speed by downloading part of a large file from <http://www.engineerhammad.com>.
+ Files Benchmark
<br>Evaluates the read/write performance of the device using a fixed 4kb buffer, writing 16 files of 64mb each.

### Firebase Authentification and Database

When first logging into the application, the device will be assigned an unique id used for anonymously auhenticating it in the Firebase System. By using that id the Application can query the *Firebase Database* by taking SnapShots at certain locations where dataChangeListeners are triggered.
Using these SnapShots, the application retrieves the scores of an user for a particular benchmark, all its scores, or all the scores of all the users on a particular benchmark.

Data is posted in two locations in the Firebase Database, one is the path which is only accessible by the user, and the other one where all users post their scores for a particular benchmark.

## State of the Art

Since the Android Operating System is so widely used, obviously there exist other Benchmark Applicatins which assess the performance of the device. Two of the most popular ones are:

+ [*AnTuTu Benchmark*](https://play.google.com/store/apps/details?id=com.antutu.ABenchMark&hl=en) - A full suite of benchmarks which evaluate the CPU, RAM, GPU and the IO of a device. <br>
The most interesting of its features is the *UX* metric, which assesses the *User experience* through multithreaded tasks and runtime application evaluation (on Dalvik or ART)

+ [*3DMARK*](https://play.google.com/store/apps/details?id=com.futuremark.dmandroid.application&hl=en) - One of the most popular Video benchmarks on PC is also availible on Android. This benchmark evaluates the performance of device in respect to 3D graphic rendering and CPU workload processing capabilities.
<br>At the moment of this writing, 3DMark consists of 4 tests on Android, 2 for Flagship devices and 2 for older devices comparison. Each tests assesses the performance of the device by rendering a multiple scenes with shaders, particle effects, post processing and other GPU & CPU intensive effects added.

Both of these benchmarks have been tested on over 3000 devices and have over 100000 values in their datasets.

### Comparison with AnTuTu

| Features | AndroidBenchmark | AnTuTu |
| :-------------: |:-----------------:| :-----:|
| UX | No | Yes |
| CPU | Yes | Yes |
| RAM | No | Yes |
| Storage IO | Yes | Yes |
| GPU | No | Yes |
| Network | Yes | No |

## Usage: Activities

+ *BaseActivity* is the superclass that all other activities extend.
The class handles basic UI tasks such as inflating the drawer and header Views and setting the respectve onItemSelected and *onClickListeners* listeners.
Expand All @@ -16,24 +69,13 @@ General information about the benchmark are presented in the field above the *Ru

+ *ScoreActivity* is the Activity that displays the user the score achieved after running the benchmark, metrics about the benchmark and a leaderboard of devices and their respective scores from the Database.

## Firebase Authentification and Database
## Results
| - | - |
|---|---|
![scores image](https://s7.postimg.org/5chb0ichn/capture.png "Preliminary benchmark results") | After running the benchmark on several devices, the performance assessment is very similar to the results found on more popular benchmarks (i.e. AnTuTu Benchmark) |

When first logging into the application, the device will be assigned an unique id used for anonymously auhenticating it in the Firebase System. By using that id the Application can query the *Firebase Database* by taking SnapShots at certain locations where dataChangeListeners are triggered.
Using these SnapShots, the application retrieves the scores of an user for a particular benchmark, all its scores, or all the scores of all the users on a particular benchmark.
One interesting fact we have found our during our beta deployment is that the Google Pixel Smartphone has a hashing performance (evaluated using BCrypt) several orders of magnitude higher than any other device we have tested so far. We are still analysing the cause of this discrepancy (perhaps hardware accelerated BCrypt hash computation?)

Data is posted in two locations in the Firebase Database, one is the path which is only accessible by the user, and the other one where all users post their scores for a particular benchmark.
## Conclusions

## About the benchmarks

+ CPU Benchmark
This benchmark evaluates the performance of the CPU. It consists of 3 smaller benchmarks:
<br>*Integer arithmetic*: Stresses the ALU by doing lots of integer computations.
<br>*Floating point*: Stresses the FPU by doing lots of floating point computations.
<br>*Digits of PI*: Puts caching and multithreading to the test by computing digits of PI, using the
[Bailey–Borwein–Plouffe formula](https://en.wikipedia.org/wiki/Bailey%E2%80%93Borwein%E2%80%93Plouffe_formula) in a parallel manner.
+ Hashing Benchmark
<br>This benchmark computes hashes using the BCrypt Algorithm. [The implementation](http://www.mindrot.org/projects/jBCrypt/) of the BCrypt class was provided by Damien Miller.
+ Network Benchmark
<br>Measures download speed by downloading part of a large file from <http://www.engineerhammad.com>.
+ Files Benchmark
<br>Evaluates the read/write performance of the device using a fixed 4kb buffer, writing 16 files of 64mb each.
We feel that while developing this project we have learned a lot about the particularities of the Android Application Development Process, Android & Firebase specific *Best Pactices*, and last but not least a better understanding of the inner workings on how the complex system we carry in our pocket works and interacts with the environment, be it through sensors or the Internet.

0 comments on commit 74c81a1

Please sign in to comment.