General Data Index Query System

1. Performance Metrics-General Query

For performance data indicators, it is often necessary to perform multi-dimension and multi-indicator cross query, so an indicator query method is provided, which is described in JSON language as follows.

Examples

An example of an indicator trend chart:

{
  "timePeriod": 180,
  "endTime":"2020-02-19 23:10"
  "datasource": "APP_BASE_DATA",
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent",
    "throughput"
  ],
  "dimensions": [
    "timeStr"
  ],
  "orderByExprs": "slowLoadPercent desc",
  "havingExprs": "viewDuration>10",
  "render":"csv",
  "top": {
    "metric": "loadDuration",
    "direction": "desc",
    "dimensions": [
      "mobileAppId"
    ],
    "limit": 5
  }
}

Property description

property	description	Mandatory
timePeriod	Start Time-End Interval Minutes	yes
endTime	End time, format YYYY-MM-DD HH: mm, not specified as current time	no
datasource	Metric data source	yes
metrics	Indicator item	yes
filters	Filter condition, only filter1 and filter2 are supported	yes
dimensions	Grouping dimensions	no
orderByExprs	Sort expression, must be an item in metric	no
havingExprs	Grouping filter expression, meaning refers to SQL having, the field must be metric	no
render	Renderer, support list, CSV, chart	no
top	Secondary query means to query based on top first, and then return the value in. It can be understood as an in subquery, which is commonly used in the topN trend chart of the report	no

DSL->SQL

Equivalent to the example converted to SQL, non-standard SQL mode:

SELECT 
  [dimensions],
  [metric]
FROM
  [datasource]
WHERE
  [time]<=[endtime] AND [time]>[endTime-timePeriod] AND [filters] AND [top.dimensions] IN (topQuery)
GROUP BY [dimensions]
HAVING [havingExprs]
ORDER BY [orderByExprs]

Renderer

Render models, currently supports the following:

• list
• csv
• chart
• object

list

Simplified JSON format:

[
    {
        "requestCount": "879",
        "uriId": "1",
        "requestType": "2",
        "trafficConsumption": "3",
        "responseTime": "3172",
        "networkErrorCount": "83",
        "requestTypeName": "CDN-Self-service",
        "slowCount": "170",
        "httpErrorRate": "8.3",
        "networkErrorRate": "9.44",
        "networkSpeed": "1",
        "uriName": "/user",
        "throughput": "14.65",
        "httpErrorCount": "73"
    }
]

csv

CSV table format:

timeStr,View load milliseconds,Slow load percentage%,Slow rendering percentage%,View rendering milliseconds
2019-4-23 8:00:00,1525,0,0,2430
2019-4-26 8:00:00,987,0,0,2460
2019-4-28 8:00:00,978,0,0,2521
2019-4-30 8:00:00,286,0,0,286
2019-5-5 8:00:00,835,0,0,892
2019-5-6 8:00:00,517,0,0,685
2019-5-10 8:00:00,111,0,0,292
2019-5-13 8:00:00,80,0,0,216
2019-5-14 8:00:00,589,0,0,652

object

Different from the list mode, the unit and format string of the indicator will be output:

[
    {
        "loadDuration": {
            "format": "987",
            "label": "View loading",
            "unit": "milliseconds",
            "value": 987
        },
        "slowLoadPercent": {
            "format": "0",
            "label": "Slow load percentage",
            "unit": "%",
            "value": 0
        },
        "timeStr": 1556236800000,
        "throughput": {
            "format": "0",
            "label": "Slow rendering percentage",
            "unit": "%",
            "value": 0
        },
        "viewDuration": {
            "format": "2460",
            "label": "View rendering",
            "unit": "milliseconds",
            "value": 2460
        }
    }
]

chart

Data style in hightchart format:

{
    "series": [
        {
            "data": [
                {
                    "x": 1556236800000,
                    "y": 987,
                    "tooltip": {
                        "data": [
                            {
                                "unit": "milliseconds",
                                "title": "View loading",
                                "value": 987
                            },
                            {
                                "unit": "%",
                                "title": "Slow load percentage",
                                "value": 0
                            },
                            {
                                "unit": "%",
                                "title": "Slow rendering percentage",
                                "value": 0
                            },
                            {
                                "unit": "milliseconds",
                                "title": "View rendering",
                                "value": 2460
                            }
                        ],
                        "title": "04-26 08:00--04-27 08:00"
                    }
                }
            ],
            "name": "View loading",
            "type": "line",
            "yAxis": 0,
            "zIndex": 1
        }
    ],
    "yAxis": [
        {
            "i18n": "app.unit.ms",
            "opposite": false,
            "tickUnit": "milliseconds"
        },
        {
            "i18n": "app.unit.percent",
            "opposite": true,
            "tickUnit": "%"
        }
    ]
}

Query example

Time trend chart

{
  "timePeriod": 43200,
  "datasource": "APP_BASE_DATA",
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent",
    "throughput"
  ],
  "dimensions": [
    "timeStr"
  ],
  "havingExprs": "viewDuration>10"
}

TOP5 time trend chart

{
  "timePeriod": 43200,
  "datasource": "APP_BASE_DATA",
  "filters": [
    {
      "name": "regionId",
      "value": [4811,4812,4813,4814,4815],
      "operator": "IN"
    }
  ],
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent"
  ],
  "dimensions": ["timeStr"]
}

Average time consumption of TOP10 in descending order of slow loading rate > 10

{
  "timePeriod": 43200,
  "datasource": "APP_BASE_DATA",
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent",
    "throughput"
  ],
  "dimensions": [
    "mobileAppId"
  ],
  "orderByExprs": "slowLoadPercent desc",
  "havingExprs": "viewDuration>10",
  "limit":10
}

Endpoint

Method: POST
Type: RAW
URL: {{host}}/adhoc/query

Headers

Key	Value
Accept	application/json, text/plain, /
Content-Type	application/json

Body

{
    "timePeriod": {{timePeriod}},
    "endTime": {{endTime}},
    "datasource": "APP_NETWORK_DATA",
    "metrics": [
        "responseTime",
        "successCount"
    ],
    "dimensions": [
        "timeStr",
        "hostId"
    ],
    "filters": [
        {
            "name": "mobileAppId",
            "value": [
                {{mobileAppId}}
            ],
            "operator": "IN"
        },
        {
            "name": "hostId",
            "value": [
                0
            ],
            "operator": "!="
        }
    ],
    "top": {
        "metric": "throughput",
        "direction": "desc",
        "dimensions": [
            "hostId"
        ],
        "limit": 5
    },
    "limit": -1,
    "render": "object"
}

2. User experience (launch, page, action)

Data source name

APP_UX_DATA

Dimension

Indicator item	Expand the dimensions	Meaning
timeStr
moduleType		Data type
mobileBusinessId	mobileBusinessName	Business line
mobileAppId	mobileAppName	Application
mobileAppVersionId	mobileAppVersionName	Application version
manufacturerId	manufacturerName	Device
manufacturerModelId	manufacturerModelName	Device model
osId	osName	Operating system
osVersionId	osVersionName	Operating system version
channelId	channelName	Channel
countryId	countryName	Country
regionId	regionName	Province
cityId	cityName	City
carrierId	carrierName	Carrier
connectTypeId	connectTypeName	Connect type
launchType		Launch type
actionType		Operation type
actionId	actionName	Operation name
actionIdGetView	actionViewName	Page name
actionViewId		Page Name
viewType		Page type
viewId	viewName	View Name
viewCompositeId	viewCompositeName	Page name
countyId	countyName	District

Indicators

Indicator item	Meaning
launchCount	launchCount
launchInitCount	Startup initialization count
launchCreateCount	Startup create count
launchLoadViewCount	Startup page load count
launchSlowCount	Slow startup count
abnormalLaunchCount	Abnormal startup
launchCrashRate	Startup crash rate
launchNetErrorRate	Startup network error rate
launchHttpErrorRate	Startup HTTP error rate
normalLaunchCount	Normal startup count
launchCrashCount	Startup crash count
launchRequestCount	Request count
launchRequestErrorCount	Startup request error count
launchRequestHttpErrorCount	Startup HTTP error count
launchRequestNetErrorCount	Startup network error count
launchSlowPercent	Slow initial startup percentage
launchStartupTime	Average startup time
launchInitTime	Average startup initialization time
launchCreateTime	Average startup create time
launchLoadViewTime	Average startup page load time
launchStartupTime50	Startup time 50th percentile
launchStartupTime75	Startup time 75th percentile
launchStartupTime95	Startup time 95th percentile
launchStartupTime99	Startup time 99th percentile
viewCount	Page count
normalViewCount	Normal page count
viewSlowInteractiveCount	Slow interactive page count
viewSlowAppearCount	Slow first screen page count
viewInteractiveTime	Average interactive time
viewAppearTime	Average first paint time
viewInteractiveTimeSum	Interactive time
viewAppearTimeSum	First screen time
slowInteractiveRate	Slow interactive time percentage
slowAppearRate	Slow first screen time percentage
viewInteractiveTime50	50th Percentile Time to Interactive
viewInteractiveTime75	75th Percentile Time to Interactive
viewInteractiveTime95	95th Percentile Time to Interactive
viewInteractiveTime99	99th Percentile Time to Interactive
viewAppearTime50	50th Percentile First Screen Time
viewAppearTime75	75th Percentile First Screen Time
viewAppearTime95	95th Percentile First Screen Time
viewAppearTime99	99th Percentile First Screen Time
viewInteractiveTimeHistogram	Distribution of Page Interactive Times
viewAppearTimeHistogram	Distribution of First Screen Times
actionCount	Operation count
actionFluencyCount	Smooth Operations Count
actionCrashCount	Crash Operations Count
actionFailureCount	Failure Operations Count
actionAnrCount	ANR Operations Count
actionSlowCount	Slow Operations Count
actionFluencyErrorCount	Smooth Operations Error Count
actionAnrErrorCount	ANR Operations Error Count
actionSlowErrorCount	Slow Operations Error Count
actionAvailableCount	Usable Operations Count
actionDurationTime	Average Operation Time
actionBlockTime	Average Operation Blocking Time
actionFluencyRate	Smooth Operations Percentage
actionSlowRate	Slow Operations Percentage
actionAnrRate	ANR Operations Percentage
actionCrashRate	Crash Operations Percentage
actionFailureRate	Failure Operations Percentage
actionaAvailability	Operation Usability
actionDurationTimeHistogram	Distribution of Operation Duration
actionBlockTimeHistogram	Distribution of Blocking Duration
actionDurationTime50	50th Percentile Operation Time
actionDurationTime75	75th Percentile Operation Time
actionDurationTime95	95th Percentile Operation Time
actionDurationTime99	99th Percentile Operation Time

Endpoint

Method: POST
Type: RAW
URL: {{host}}/adhoc/query

Headers

Key	Value
Accept	application/json, text/plain, /
Content-Type	application/json

Body

{
    "timePeriod": {{timePeriod}},
    "endTime": {{endTime}},
    "datasource": "APP_UX_DATA",
    "metrics": [
        "launchStartupTime"
    ],
    "dimensions": [
       
    ],
    "filters": [
        {
            "name": "countryId",
            "value": [
                48
            ],
            "groupType": "region"
        },
        {
            "name": "mobileAppId",
            "value": [
                {{mobileAppId}}
            ],
            "operator": "IN"
        },
        {
            "name": "launchType",
            "value": [
                1
            ],
            "operator": "IN"
        },
        {
            "name": "moduleType",
            "value": [
                1
            ],
            "operator": "="
        }
    ],
    "render": "list",
    "orderByExprs": "launchStartupTime desc"
}

3. Network Request-Performance Metric

Data source name

APP_NETWORK_DATA

Explain: The extended dimension cannot be filtered or queried.

Dimension

Indicator item	Expand the dimensions	Meaning
timeStr
mobileAppId	mobileAppName	Application
mobileAppVersionId	mobileAppVersionName	Application version
mobileBusinessId	mobileBusinessName	Business line
manufacturerId	manufacturerName	Device
manufacturerModelId	manufacturerModelName	Device model
osId	osName	Operation system
osVersionId	osVersionName	Operation system version
countryId	countryName	Country
regionId	regionName	Province
cityId	cityName	City
carrierId	carrierName	Carrier
connectTypeId	connectTypeName	Connect type
hostId	hostName	Domain name
uriId	uriName	Request URL
errorType	errorTypeName	Error type
errorCode	errorCodeName	Error code
cdnId	cdnName	CDN vendor
hostIp		Server IP
hostCountryId	hostCountryName	Server country
hostRegionId	hostRegionName	Server province
hostCityId	hostCityName	Server city
hostCarrierId	hostCarrierName	Server carrier
bytesType	bytesTypeLabel	Package size
protocolType		Protocol type
channelId	channelName	Channel
hostIpType	hostIpTypeName	Server IP type
requestType	requestTypeName	Service type
launchType		Launch type
countyId	countyName	District
hostCountyId	hostCountyName	District

Indicators

Indicator item	Meaning
requestCount	Request count
successCount	Success request count
passCount	Normal request count
slowCount	Slow request count
slowRequestRate	Slow request percentage
errorCount	Error count
httpErrorCount	HTTP error count
networkErrorCount	Network error count
dnsCount	DNS count
tcpCount	TCP count
sslCount	SSL count
responseTimeTotal	Total response time
networkTimeTotal	Total network time
responseTime	Average response time
dnsTime	Average DNS time
connectTime	Average TCP time
sslTime	Average SSL time
firstPacketTime	Average firstPacket time
remainPacketTime	Average remainPacket time
localQueueTime	Average localQueue time
networkTime	Average network time
throughput	Throughput Rate
networkSpeed	Transmission Rate
networkTrafficSpeed	Network Transmission Rate
totalTrafficConsumption	Total Data Consumption
trafficConsumption	Average Data Transfer
bytesSend	bytesSend
bytesReceived	bytesReceived
httpErrorRate	HTTP error rate
networkErrorRate	Network error rate
requestErrorRate	Request error rate
pingSuccessCount	ping success count
pingTime	Network latency
packetLossRate	Packet loss rate
reuseRate	Multiplexing rate
availability	Availability

Endpoint

Method: POST
Type: RAW
URL: {{host}}/adhoc/query

Headers

Key	Value	Description
Accept	application/json, text/plain, /
Content-Type	application/json

Body

{
    "timePeriod": {{timePeriod}},
    "endTime": {{endTime}},
    "datasource": "APP_NETWORK_DATA",
    "metrics": [
        "responseTime",
        "successCount"
    ],
    "dimensions": [
        "timeStr",
        "hostId"
    ],
    "filters": [
        {
            "name": "mobileAppId",
            "value": [
                {{mobileAppId}}
            ],
            "operator": "IN"
        },
        {
            "name": "hostId",
            "value": [
                0
            ],
            "operator": "!="
        }
    ],
    "top": {
        "metric": "throughput",
        "direction": "desc",
        "dimensions": [
            "hostId"
        ],
        "limit": 5
    },
    "limit": -1,
    "render": "list"
}