DESIGNDOC.md

Sift Android SDK

Mobile Android SDK Software Design Documentation

Table of Contents

1 Overview

2 High Level Class Diagram

3 Data Models

• 3.1 AndroidDeviceLocation
• 3.2 AndroidAppState
• 3.3 AndroidDeviceProperties
• 3.4 MobileEvent

4 Modules

• 4.1 SIFT
• 4.2 SIFT IMPL
• 4.3 QUEUE

5 Flow Chart

1 Overview

The sift-android is an Android SDK developed only for mobiles. Java will be used as the programming language and Android Studio will be used as the IDE. SDK will be supporting a minimum of Android 4.1 (Jelly Bean).

The Android-specific features used are: SharedPreferences, Executors, PackageManager, BatteryManager, Location, NetworkInterface and TelephonyManager. It uses Android's SharedPreferences to manage configurations of the Sift object andpersisting collected events and config data to disk. TheSDK uses PackageManager, BatteryManager, Location and NetworkInterface for collecting AppState details. The Device properties details are collected with the help of TelephonyManager and PackageManager along with Build details. In particular, event collecting, appending and uploading are handled on a separate thread with the help of Executors.

A high-level block diagram is shown

1. Android app loads the SDK with the Sift configurations.
2. The sift SDK will collect and send events to the Sift server when there are events to upload.

This document describes the data models,classes for handling events, and specific flows, respectively. For more information on Android technologies, the Android documentation contains a wealth of information on all Android programming topics and technologies.

2 High Level Class Diagram

classDiagram
class AndroidDeviceLocation {
  +Double latitude
  +Double longitude
 }

 class AndroidAppState {
   +String activityClassName
   +AndroidDeviceLocation location
   +String sdkVersion
   +Double batteryLevel
   +Long batteryState
   +Long batteryHealth
   +Long plugState
   +List~String~ networkAddresses
  }

  class AndroidDeviceProperties {
   +String appName
   +String appVersion
   +String sdkVersion
   +String mobileCarrierName
   +String mobileIsoCountryCode
   +String deviceManufacturer
   +String deviceModel
   +String deviceSystemVersion
   +String androidId
   +String buildTags
   +List~String~ evidenceFilesPresent
   +List~String~ evidencePackagesPresent
   +List~String~ evidenceProperties
   +List~String~ evidenceDirectoriesWritable
  }

  class MobileEvent {
   +Long time
   +String userId
   +String installationId
   +AndroidDeviceProperties androidDeviceProperties
   +AndroidAppState androidAppState
  }

  AndroidDeviceLocation ..> AndroidAppState
  AndroidAppState ..> MobileEvent
  AndroidDeviceProperties  ..> MobileEvent

Loading

3 Data Models

The data models used in this SDK are auto-generated from 'yaml' files. The yaml to POJO conversion is handled by the jsonSchema2Pojo gradle plugin. Which uses 'yamlschema' as input and generates POJO for collecting and uploading AppState and DeviceProperties events.

3.1 AndroidDeviceLocation

The AndroidDeviceLocation consist of the following information:

• latitude : {type: number, required: false}
  • Which indicates the latitude of the collected location.
• longitude : {type: number, required: false}
  • Which indicates the longitude of the collected location.

Class diagram of AndroidDeviceLocation

  classDiagram
  class AndroidDeviceLocation{
    +Double latitude
    +Double longitude

    +withLatitude(latitude)
    +withLongitude(longitude)
    +toString() string
    +hashCode() int
    +equals() boolean
  }

Loading

3.2 AndroidAppState

The AndroidAppState collects the following informations:

• activity_class_name : {type: string, required: false}
  • The activity class name indicates the current activity/fragment class name from where the data are collected.
• location : {type: AndroidDeviceLocation, required: false}
  • The location consists of collective information of latitude, longitude, accuracy and the time at which data was collected as shown in the section 3.1. (Have data only if the sift configuration and permissions are enabled)
• sdk_version : {type: string, required: false}
  • The sdk version indicates the current Sift SDK version which is used.
• battery_level : {type: number, required: false}
  • The current battery level, from 0 to 1 (84% indicates 0.84)
• battery_state : {type: integer, required: false}
  • The current status constant of the battery, from 1 to 5
    • Constant Value: 1 -> BATTERY_STATUS_UNKNOWN
    • Constant Value: 2 -> BATTERY_STATUS_CHARGING
    • Constant Value: 3 -> BATTERY_STATUS_DISCHARGING
    • Constant Value: 4 -> BATTERY_STATUS_NOT_CHARGING
    • Constant Value: 5 -> BATTERY_STATUS_FULL
• battery_health : {type: integer, required: false}
  • The battery health indicates the current health constant, from 1 to 7
    • Constant Value: 1 -> BATTERY_HEALTH_UNKNOWN
    • Constant Value: 2 -> BATTERY_HEALTH_GOOD
    • Constant Value: 3 -> BATTERY_HEALTH_OVERHEAT
    • Constant Value: 4 -> BATTERY_HEALTH_DEAD
    • Constant Value: 5 -> BATTERY_HEALTH_OVER_VOLTAGE
    • Constant Value: 6 -> BATTERY_HEALTH_UNSPECIFIED_FAILURE
    • Constant Value: 7 -> BATTERY_HEALTH_COLD
• plug_state : {type: integer, required: false}
  • The plug state indicates whether the device is plugged in to a power source; 0 means it is on battery, other constants are different types of power sources.
    • Constant Value: 1 -> BATTERY_PLUGGED_AC
    • Constant Value: 2 -> BATTERY_PLUGGED_USB
    • Constant Value: 4 -> BATTERY_PLUGGED_WIRELESS
• network_addresses : {type: array, items: {type: string}, required: false}
  • The network addresses indicate the list of IP addresses of the current device in which the SDK is running.

Class diagram of AndroidAppState

3.3 AndroidDeviceProperties

The AndroidDeviceProperties collects the following information:

• app_name : {type: string, required: false}

  • The app name indicates the name of the application in which the sift SDK is used.

• app_version : {type: string, required: false}

  • The app version indicates the current version name of the application in which the sift SDK is used.

• sdk_version : {type: string, required: false}

  • The sdk version indicates the current version of the sift SDK that has been used in the application.

• mobile_carrier_name : {type: string, required: false}

  • The mobile carrier name indicates the alphabetic name of the current registered network operator.

• mobile_iso_country_code : {type: string, required: false}

  • It indicates the ISO-3166-1 alpha-2 country code equivalent for the SIM provider's country code.

• device_manufacturer : {type: string, required: false}

  • The device manufacturer indicates the manufacturer of the product/hardware.

• device_model : {type: string, required: false}

  • The device model indicates the end-user-visible name for the end product.

• device_system_version : {type: string, required: false}

  • It indicates the user-visible operating system version string. E.g., "1.0" or "3.2.6".

• android_id : {type: string, required: false}

  • The Android id indicates the 64-bit number (expressed as a hexadecimal string) unique to each device.

• build_tags : {type: string, required: false}

  • The build tags indicate the comma-separated tags describing the build, like "unsigned,debug".
  • If Build.TAGS contains "test-keys", then it is a rooted device.

• evidence_files_present : {type: array, items: {type: string}, required: false}

  • The evidence file present may contain a list of files that are known to indicate rooted devices.
  • If it is an empty list then the device is not a rooted device.

• evidence_packages_present : {type: array, items: {type: string}, required: false}

  • The evidence package present may contain a list of packages that are known to indicate rooted devices.
  • If it is an empty list then the device is not a rooted device.

• evidence_properties : {type: array, items: {type: string}, required: false}

  • The evidence property may contain a list of dangerous properties that indicate rooted devices.
  • If it is an empty list then the device is not a rooted device.

• evidence_directories_writable : {type: array, items: {type: string}, required: false}

  • It may contain a list of path to common system directories which have write permissions that indicate rooted devices.
  • If it is an empty list then the device is not a rooted device.

Class diagram of AndroidDeviceProperties

classDiagram
class AndroidDeviceProperties {
  +String appName
  +String appVersion
  +String sdkVersion
  +String mobileCarrierName
  +String mobileIsoCountryCode
  +String deviceManufacturer
  +String deviceModel
  +String deviceSystemVersion
  +String androidId
  +String buildTags
  +List~String~ evidenceFilesPresent
  +List~String~ evidencePackagesPresent
  +List~String~ evidenceProperties
  +List~String~ evidenceDirectoriesWritable

  +withAppName(appName)
  +withAppVersion(appVersion)
  +withSdkVersion(sdkVersion)
  +withMobileCarrierName(mobileCarrierName)
  +withMobileIsoCountryCode(mobileIsoCountryCode)
  +withDeviceManufacturer(deviceManufacturer)
  +withDeviceModel(deviceModel)
  +withDeviceSystemVersion(deviceSystemVersion)
  +withAndroidId(androidId)
  +withBuildTags(buildTags)
  +withEvidenceFilesPresent(evidenceFilesPresent)
  +withEvidencePackagesPresent(evidencePackagesPresent) 
  +withEvidenceProperties(evidenceProperties)
  +withEvidenceDirectoriesWritable(evidenceDirectoriesWritable)
  +toString() string
  +hashCode() int
  +equals() boolean
}

Loading

3.4 MobileEvent

The MobileEvent mainly collects the following information:

• time : {type: integer, required: true}
  • It indicates the time (in ms since the unix epoch) that this event occurred.
• user_id : {type: string, required: false}
  • It indicates the time (in ms since the unix epoch) that this event occurred.
• installation_id : {type: string, required: false}
  • The installation id indicates the 64-bit number (expressed as a hexadecimal string) unique to each device.
• android_device_properties : {type: AndroidDeviceProperties, required: false}
  • The android device property indicates the device related properties as mentioned in section 3.3
• android_app_state : {type: AndroidAppState, required: false}
  • The android app state indicates the application related datas as mentioned in section 3.2.

Class diagram of MobileEvent

4 Modules

The SDK also has a number of classes that deal with event collecting, saving and uploading to Sift server.

4.1 SIFT

This is a utility class of the sift client library which handles the application-level code for interacting with the framework for collecting, saving and uploading events. This class sets up and holds references to the Gson, SiftImpl and event collectors, including AppStateCollector and DevicePropertiesCollector.

It has a Config class to setup the account details with constructor to initialise the values:

• Config(accountId, beaconKey, serverUrlFormat, disallowLocationCollection)
  • accountId : {type: string}
    • Your account ID; defaults to null.
  • beaconKey : {type: string}
    • Your beacon key; defaults to null.
  • serverUrlFormat : {type: string}
    • The location of the API endpoint; defaults to https://api3.siftscience.com/v3/accounts/%s/mobile\_events
  • disallowLocationCollection : {type: boolean}
    • Whether to allow location collection; defaults to false.

Also the Config class provide a builder class to initialize the configuration data:

• withAccountId(accountId)
• withBeaconKey(beaconKey)
• withServerUrlFormat(serverUrlFormat)
• withDisallowLocationCollection(disallowLocationCollection)

Sift.Config.Builder()
	.withAccountId("YOUR_ACCOUNT_ID")
	.withBeaconKey("YOUR_BEACON_KEY")
	.build()

The builder build() method will execute the Config() constructor with provided values.

Following are the static API to interact with SDK:

• open(context, config, activityName)
  • @param context the Activity context
  • @param config the Sift.Config object
  • @param activityName the Activity
  • Should call in the onCreate() callback of each Activity.
  • It creates the Sift singleton instance and collectors if they do not exist, and passes along the current Activity context.
  • For your application's main Activity, make sure to provide a Sift.Config object as the second parameter.
  • If you are integrating per-Activity rather than at the Application level, you can specify the name that will be associated with each Activity event (defaults to the class name of the embedding Activity).
  • There are overloaded methods below for your convenience.
    • open(context, activityName)
    • open(context, config)
    • open(context)
• collect()
  • Should call Sift.collect() after the Sift.open() call in each Activity.
  • Which executes a runnable task to collect SDK events for Device Properties and Application State.
• pause()
  • Should call Sift.pause() in the onPause() callback of each Activity.
  • Which persists the instance state to disk and disconnects location services.
• resume(context, activityName)
  • Should call Sift.resume() in the onResume() callback of each Activity.
  • It will try to reconnect the location services if configuration and permissions are enabled.
  • If you provide a non-null activity name as a parameter then it will set the current activity class name as the name provided, otherwise it will set the simple name of the underlying class.
  • There is an overloaded method for your convenience.
    • resume(context)
• close()
  • Call Sift.close() in the onDestroy() callback of each Activity.
  • It persists the instance state to disk and disconnects location services.
• setUserId(userId)
  • It will set the provided id as the userId inside the sift instance.
• unsetUserId()
  • Which removes any presetted useId to null

4.2 SIFT IMPL

This class is the implementation of Sift instance which sets up and holds the references to the Sift.Config, queues, task manager and uploader. Where Sift.Config consist of account related data, queues consist of events related to App State and Device Properties, task manager handles the execution of runnable tasks, and uloader handles the uploading events to the sift server. It generates the queue configuration to decide the event acceptance and uploading criteria. Like accepting the same event after 1 hour, upload events when more than 8 and so on.

It provides the implementation of interfaces like UserIdProvider and UploadRequester in Queue and ConfigProvider in Uploader.

This class mainly handles the following task:

• Archive/Save all of the sift instance states to the disk using shared preference, which includes Sift.Config, user Id, app state queue and device properties queue.
• Unarchive/Restore all the sift instance states(Sift.Confi, user Id, and queues) from disk.
• Appends the collected event to the App State queue and Device Properties queue.
• Setting Sift.Config.
• Setting user Id.

These tasks runon a separate executor, so that if any largeamounts of data does not affect the main thread.

To execute those task it provide the following instance API:

• save()

  • Which invokes the task manager to execute the archiving task.

• stop()

  • Which invokes the task manager to terminate the executor.

• appendAppStateEvent(event)

  • Which invokes the task manager to execute the append task with App State queue identifier and provided event.

• appendDevicePropertiesEvent(event)

  • Which invokes the task manager to execute the append task with Device Properties queue identifier and provided event.

• upload(events)

  • Which invokes the uploader to execute the upload task with a list of collected events.

• setConfig(config)

  • Sets the configuration for the Sift instance
  • Which invokes the task manager to execute the set config task with provided configuration.

• getConfig()

  • Return the configuration for the Sift instance
  • It will provide the non-nul Sift.Config if available, otherwise unarchive the configuration from shared preference.

• setUserId(userId)

  • Sets the user ID for the Sift instance.
  • Which invokes the task manager to execute the set user Id task with specified userId.

• getUserId()

  • Return the user ID for the Sift instance.

• unsetUserId()

  • Unsets the user ID for the Sift instance.
  • Which invokes the task manager to execute the set user Id task with null value.

• createQueue(identifier, config)

  • It will check whether the queue already exists with the identifier, if so it will throw the exception. Otherwise it will create a new queue with provided config, and object of userIdProvider and uploadProvider.

• getQueue(identifier)

  • It will return the specified queue from the available queues map.

Whenever a new object is created the constructor will invoke the task manager to execute the unarchive task to restore the Sift.Config, User Id, and Queues from disk if any.

The UploadRequester interface overridden method requestUpload(events) will invoke the SiftImpl.upload(events). Similarly ConfigProvider interface overridden method getConfig() will invoke SiftImpl.getConfig()

It holds an enum class ArchinveKey which serves as a key provider for the data persistence on the disk. There are three enums for 3 types of key, one for Sift.Config, one for userId, and one for queue. Also provide two static methods:

getKeyForQueueIdentifier(identifier): {type: string}

• Which returns a string with combination of queue key and identifier.

getQueueIdentifier(key): {type: string}

• Which returns null if the provided combination is not starting with a queue key, otherwise it will return the identifier string from the combination.

4.3 QUEUE

This class is for holding events until they are ready for upload to the sift server. Basically there are two separate queues, one for AppState events and other for DeviceProperties events in order to collect and upload the events independently.

Whenever an event is collected and tries to add either in the App State or Device Properties queue, it will append and upload depending on the queue's batching policy. The queue's batching policy is controlled by the queue configuration and state of the queue.

The queue configuration depends on the following factors:

• acceptSameEventAfter : {type: long}
  • Time after which an event that is basically the same as the most recently appended event can be appended again.
• uploadWhenMoreThan : {type: int}
  • Max queue depth before flush and upload request.
• uploadWhenOlderThan : {type: long}
  • Max queue age before flush and upload request.

Which can be initialized through the queue config builder class, which provide the following methods:

• withAcceptSameEventAfter(acceptSameEventAfter): {type: long}
• withUploadWhenMoreThan(uploadWhenMoreThan): {type: int}
• withUploadWhenOlderThan(uploadWhenOlderThan): {type: long}

 Queue.Config.Builder()
	.withUploadWhenMoreThan(8)
	.withUploadWhenOlderThan(TimeUnit.MINUTES.toMillis(1))
	.build();

The DeviceProperties queue is configured as:

• acceptSameEventAfter: 1 hour
• uploadWhenMoreThan: 0
• uploadWhenOlderThan: 1 minute

The AppState queue is configured as:

• uploadWhenMoreThan: 8
• uploadWhenOlderThan: 1 minute

This class holds the state of the queue with the following attributes:

• config : {type: Queue.Config}
  • The configuration of the queue which decides the batching policy.
• queue : {type: List< MobileEvent>}
  • The list of collected events as of now to be uploaded depending on the policy.
• lastEvent : {type: MobileEvent}
  • The recent event added to the queue.
• lastUploadTimestamp: {type: long}
  • The time at which recent upload was carried on.

This class have the following methods:

• archive()

  • Which return json string of the current queue state with the help of Gson.

• unarchive(archive)

  • Where archive is the json string of queue state.
  • Which return queue state from provided json string.
  • It will return a new object of queue state, If the provided json string is null or any exception in json syntax.

• getConfig()

  • Which returns the reference value of current Queue.Config.

• flush()

  • Which shifts the queue into a temporary queue and clears the current queue.
  • Finally returns the temporary queue.

• isReadyForUpload(now)

  • Where now is the current timestamp.
  • The queue uploading policy is based on:

• state.queue.size() > config.uploadWhenMoreThan

• now > state.lastUploadTimestamp + config.uploadWhenOlderThan

If any one of the uploading policies were satisfied, then the queue will be flushed and uploaded.

• append(event)

  • Where event is the collected event to be appended and uploaded depending on the queue batching policies.
  • The queue appending policy is based on the following criteria:

• config.acceptSameEventAfter > 0

• state.lastEvent != null

• Current time < state.lastEvent.time + config.acceptSameEventAfter

• state.lastEvent and current event are basically the same

If all of the above conditions are met, then the queue will drop the collected event as a duplicate event. Otherwise the event will be appended to the queue and the state.lastEvent updated with the new event.

• After appending it will check whether the event isReadyForUpload(now), if ready then request the uploader to upload the events.

The queue constructor expect the following parameters:

• archive : {type: string}
  • The archived state of the queue in json, if it is a non-null value then the state of the newly created queue will be initialized with the saved one. Otherwise the queue will create a new state.
• userIdProvider : {type: UserIdProvider}
  • The reference variable of UserIdProvider interface which holds overridden getUserId() method.
  • Which is used at the time of appending events, whenever there is a lack of userId in the collected event then it will assign the value from userIdProvider.getUserId()
• uploadRequester : {type: UploadRequester}
  • The reference variable of UploadRequester interface which holds overridden requestUpload(events) method.
  • Which is used at the time of uploading events, whenever the collected events are ready for upload then it will request uploader with list of events as uploadRequester.requestUpload(flush())
• config : {type: Queue.Config}
  • The queue configuration which decides the batching policy.

4.4 APP STATE COLLECTOR

This module collects the App State events which consist of battery information, location information and network address. The battery information includes current battery level, battery status, battery health and battery plug state. Similarly the location information includes latitude, longitude, accuracy and time at which the location collected. Also the network address holds the ip addresses of the device.

This class expects the Sift instance and context, depending on the sift instance configuration data it will initialise location provider and location request if disallowLocationCollection is false (default value).

It provides the following instance methods:

• setActivityName(activityName)
  • Which will assign the current activity class name as the provided name.
• collect()
  • This will request location updates if Sift.Config disallowLocationCollection is false and location permission is enabled by the user which was requested by your application (NB: Sift will not request permissions that are not granted by the user from your application). Otherwise it will collect the rest of app state events without location information.
  • The location collection is managed by FusedLocationProviderClient. The location request is configured as PRIORITY_HIGH_ACCURACY with one minute interval and 10 seconds fastest interval.
  • If location permission is granted by the user then first of all it will fetch the last known location from FusedLocationClient and which is saved in lastLocation reference.
  • Once the onLocationResult callback is triggered, it will update the current location, remove location updates and collect the remaining app state events.
  • If any of the lastLocation or location have non-null value then at the time of app state event collection the AndroidDeviceLocation model will be updated with either acquired new location or last known location depending on the availability and priority.
  • The battery information is collected from ACTION_BATTERY_CHANGED intent and with the help of BatteryManager. The battery level is calculated on the value from BatteryManager.EXTRA_LEVEL over BatteryManager.EXTRA_SCALE. The status value is identified based on the integer value from BatteryManager.EXTRA_STATUS, the plug state integer value from BatteryManager.EXTRA_PLUGGED and health integer value from BatteryManager.EXTRA_HEALTH
  • The Ip addresses are irritated from the host address provided by the InetAddress under NetworkInterface.
  • After collecting these app state event information it will invoke the swift instance method appendAppStateEvent(event) with MobileEvent which is occupied with AndroidAppState data model.
• disconnectLocationServices()
  • This will try to remove ongoing location updates if any.
• reconnectLocationServices()
  • It will request location updates if Sift.Config with disallowLocationCollection is false, location permissions are enabled, fused location clients have non-null reference, and there are no ongoing location updates.

4.5 DEVICE PROPERTIES COLLECTOR

This module collects device properties events which includes app name, app version, sdk version, android id, device manufacturer, device model, mobile carrier name, mobile iso country name, device system version, build brand, build device, build fingerprint, build hardware, build product, build tag, evidence file present, evidence package present, evidence properties and evidence directories writable.

It provides only one instance method:

• collect()
  • It will collect the app name and the app version from PackageManager application info and package info respectively.
  • Using TelephonyManager mobile carrier name is assigned by the network operator name and mobile carrier iso country code is collected from sim country iso.
  • Other build and device informations are collected from android.os.Build provider:
    • The device manufacturer detail is collected from Build.MANUFACTURER which provides the manufacturer of the product/hardware.
    • The device model from Build.MODEL which provides the end-user-visible name for the end product.
    • The build tag from Build.TAGS which provides comma-separated tags describing the build, like "unsigned,debug".
  • It will check and collect evidence from the device which indicates whether the device is rooted with the following different approaches:
    • existingRootFiles()
      • Checks for files that are known to indicate root.
      • Will return a list of super user paths, if any.
    • existingRootPackages()
      • Checks for packages that are known to indicate root.
      • Will return a list of package names which are known root apps, dangerous apps and root cloaking, if any.
    • existingDangerousProperties()
      • Checks system properties for any dangerous properties that indicate root.
      • Will return a list of dangerous properties that indicate root, If any.
    • existingRWPaths()
      • This method checks if any of the system paths that should not be writable are writable (in rooted devices you can change the write permissions on common system directories).
      • Will return all restricted system paths that are writable, if any.
  • The following are different methods to detect whether the device is rooted.
    • If Build.TAGS contains "test-keys", then it's rooted.
    • If any of the other evidence* methods return a non-empty list, then it's rooted.

4.6 TASK MANAGER

This module is a wrapper around executor that abstracts exception handling and shutdown logic. It will create a single-threaded executor that can schedule commands to run after a given delay, or requests for immediate execution with zero delay, or to execute periodically.

This class provide three instance methods as follow:

• submit(task)
  • Where the task is a Runnable task that is to be executed.
  • It will invoke the executor's submit() method with the provided runnable task for immediate execution.
  • If the task cannot be submitted for execution then it will throw a RejectedExecutionException.
  • The runnable tasks which are requested for immediate execution are: archiving, unarchiving, appending, setting user Id and setting sift config
• schedule(task, delay, unit)
  • Where:
    • task is a Runnable task that is to be executed.
    • delay is the time from now to delay execution.
    • unit is the time unit of the delay parameter.
  • It will create and execute a one-shot action that becomes enabled after the given delay by invoking executor's schedule() method.
  • If the runnable task cannot be scheduled for execution then it will throw a RejectedExecutionException.
  • All uploading tasks are scheduled execution with an incrementing delay upto there retries.
• shutdown()
  • Will invoke executor's shutdown method to initiate an orderly shutdown in which previously submitted tasks are executed, but no new tasks will be accepted. Invocation has no additional effect if already shut down.
  • This method does not wait for previously submitted tasks to complete execution.

5 Flow Chart