通用数据指标查询系统

1. 性能指标-通用查询

由于性能数据指标，经常要做多维度多指标交叉查询，所以提供一个指标查询方式，该方式采用JSON语言方式进行描述如下方式所示。

示例

一个指标趋势图的示例：

{
  "timePeriod": 180,
  "endTime":"2020-02-19 23:10"
  "datasource": "APP_BASE_DATA",
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent",
    "throughput"
  ],
  "dimensions": [
    "timeStr"
  ],
  "orderByExprs": "slowLoadPercent desc",
  "havingExprs": "viewDuration>10",
  "render":"csv",
  "top": {
    "metric": "loadDuration",
    "direction": "desc",
    "dimensions": [
      "mobileAppId"
    ],
    "limit": 5
  }
}

属性说明

property	description	是否必选
timePeriod	开始时间-结束时间 间隔分钟数	yes
endTime	截止时间，格式YYYY-MM-DD hh:mm,不指定为当前时间	no
datasource	指标数据源	yes
metrics	指标项	yes
filters	过滤条件，只支持filter1 and filter2	yes
dimensions	分组维度	no
orderByExprs	排序表达式，必须为metric中的项	no
havingExprs	分组过滤表达式，意义参考sql having，字段必须为metric	no
render	渲染器，支持list，csv，chart	no
top	二次查询，意为，先基于top进行查询，然后将返回值in，可以理解为in 子查询,常用在报表的topN趋势图中	no

DSL->SQL

相当于转换为SQL的示例，非标准SQL模式：

SELECT 
  [dimensions],
  [metric]
FROM
  [datasource]
WHERE
  [time]<=[endtime] AND [time]>[endTime-timePeriod] AND [filters] AND [top.dimensions] IN (topQuery)
GROUP BY [dimensions]
HAVING [havingExprs]
ORDER BY [orderByExprs]

渲染器

渲染模型，目前支持以下：

• list
• csv
• chart
• object

list

简化的json格式：

[
    {
        "requestCount": "879",
        "uriId": "1",
        "requestType": "2",
        "trafficConsumption": "3",
        "responseTime": "3172",
        "networkErrorCount": "83",
        "requestTypeName": "CDN-自身服务",
        "slowCount": "170",
        "httpErrorRate": "8.3",
        "networkErrorRate": "9.44",
        "networkSpeed": "1",
        "uriName": "/user",
        "throughput": "14.65",
        "httpErrorCount": "73"
    }
]

csv

csv 表格格式：

timeStr,视图加载毫秒,慢加载占比%,慢展现占比%,视图展现毫秒
2019-4-23 8:00:00,1525,0,0,2430
2019-4-26 8:00:00,987,0,0,2460
2019-4-28 8:00:00,978,0,0,2521
2019-4-30 8:00:00,286,0,0,286
2019-5-5 8:00:00,835,0,0,892
2019-5-6 8:00:00,517,0,0,685
2019-5-10 8:00:00,111,0,0,292
2019-5-13 8:00:00,80,0,0,216
2019-5-14 8:00:00,589,0,0,652

object

和list不同的模式为，会将指标的单位和格式化字符串输出：

[
    {
        "loadDuration": {
            "format": "987",
            "label": "视图加载",
            "unit": "毫秒",
            "value": 987
        },
        "slowLoadPercent": {
            "format": "0",
            "label": "慢加载占比",
            "unit": "%",
            "value": 0
        },
        "timeStr": 1556236800000,
        "throughput": {
            "format": "0",
            "label": "慢展现占比",
            "unit": "%",
            "value": 0
        },
        "viewDuration": {
            "format": "2460",
            "label": "视图展现",
            "unit": "毫秒",
            "value": 2460
        }
    }
]

chart

hightchart 格式的数据样式：

{
    "series": [
        {
            "data": [
                {
                    "x": 1556236800000,
                    "y": 987,
                    "tooltip": {
                        "data": [
                            {
                                "unit": "毫秒",
                                "title": "视图加载",
                                "value": 987
                            },
                            {
                                "unit": "%",
                                "title": "慢加载占比",
                                "value": 0
                            },
                            {
                                "unit": "%",
                                "title": "慢展现占比",
                                "value": 0
                            },
                            {
                                "unit": "毫秒",
                                "title": "视图展现",
                                "value": 2460
                            }
                        ],
                        "title": "04-26 08:00--04-27 08:00"
                    }
                }
            ],
            "name": "视图加载",
            "type": "line",
            "yAxis": 0,
            "zIndex": 1
        }
    ],
    "yAxis": [
        {
            "i18n": "app.unit.ms",
            "opposite": false,
            "tickUnit": "毫秒"
        },
        {
            "i18n": "app.unit.percent",
            "opposite": true,
            "tickUnit": "%"
        }
    ]
}

查询示例

时间趋势图

{
  "timePeriod": 43200,
  "datasource": "APP_BASE_DATA",
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent",
    "throughput"
  ],
  "dimensions": [
    "timeStr"
  ],
  "havingExprs": "viewDuration>10"
}

TOP5 时间趋势图

{
  "timePeriod": 43200,
  "datasource": "APP_BASE_DATA",
  "filters": [
    {
      "name": "regionId",
      "value": [4811,4812,4813,4814,4815],
      "operator": "IN"
    }
  ],
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent"
  ],
  "dimensions": ["timeStr"]
}

TOP10 按慢加载率降序平均耗时>10

{
  "timePeriod": 43200,
  "datasource": "APP_BASE_DATA",
  "metrics": [
    "loadDuration",
    "viewDuration",
    "slowLoadPercent",
    "throughput"
  ],
  "dimensions": [
    "mobileAppId"
  ],
  "orderByExprs": "slowLoadPercent desc",
  "havingExprs": "viewDuration>10",
  "limit":10
}

Endpoint

Method: POST
Type: RAW
URL: {{host}}/adhoc/query

Headers

Key	Value
Accept	application/json, text/plain, /
Content-Type	application/json

Body

{
    "timePeriod": {{timePeriod}},
    "endTime": {{endTime}},
    "datasource": "APP_NETWORK_DATA",
    "metrics": [
        "responseTime",
        "successCount"
    ],
    "dimensions": [
        "timeStr",
        "hostId"
    ],
    "filters": [
        {
            "name": "mobileAppId",
            "value": [
                {{mobileAppId}}
            ],
            "operator": "IN"
        },
        {
            "name": "hostId",
            "value": [
                0
            ],
            "operator": "!="
        }
    ],
    "top": {
        "metric": "throughput",
        "direction": "desc",
        "dimensions": [
            "hostId"
        ],
        "limit": 5
    },
    "limit": -1,
    "render": "object"
}

2. 用户体验（启动、页面、操作）

数据源名称

APP_UX_DATA

维度

列名	扩展维度	含义
timeStr
moduleType		数据类型
mobileBusinessId	mobileBusinessName	业务线
mobileAppId	mobileAppName	应用
mobileAppVersionId	mobileAppVersionName	应用版本
manufacturerId	manufacturerName	设备
manufacturerModelId	manufacturerModelName	设备型号
osId	osName	系统
osVersionId	osVersionName	系统版本
channelId	channelName	渠道
countryId	countryName	国家
regionId	regionName	省份
cityId	cityName	城市
carrierId	carrierName	运营商
connectTypeId	connectTypeName	接入方式
launchType		启动类型
actionType		操作类型
actionId	actionName	操作名称
actionIdGetView	actionViewName	所属页面
actionViewId		所属页面
viewType		页面类型
viewId	viewName	视图名称
viewCompositeId	viewCompositeName	页面名称
countyId	countyName	区县

指标

指标项	含义
launchCount	启动次数
launchInitCount	启动初始化次数
launchCreateCount	启动构建次数
launchLoadViewCount	启动页面加载次数
launchSlowCount	启动慢次数
abnormalLaunchCount	异常启动
launchCrashRate	启动崩溃率
launchNetErrorRate	启动网络错误率
launchHttpErrorRate	启动HTTP错误率
normalLaunchCount	正常启动次数
launchCrashCount	启动崩溃次数
launchRequestCount	请求次数
launchRequestErrorCount	启动错误次数
launchRequestHttpErrorCount	启动http错误次数
launchRequestNetErrorCount	启动网络错误次数
launchSlowPercent	慢首次启动占比
launchStartupTime	平均启动时间
launchInitTime	平均启动初始化时间
launchCreateTime	平均启动构建时间
launchLoadViewTime	平均启动页面加载时间
launchStartupTime50	启动时间50分位值
launchStartupTime75	启动时间75分位值
launchStartupTime95	启动时间95分位值
launchStartupTime99	启动时间99分位值
viewCount	页面次数
normalViewCount	正常页面次数
viewSlowInteractiveCount	页面慢可交互次数
viewSlowAppearCount	页面慢首屏次数
viewInteractiveTime	平均可交互时间
viewAppearTime	平均首屏时间
viewInteractiveTimeSum	可交互时间
viewAppearTimeSum	首屏时间
slowInteractiveRate	慢可交互占比
slowAppearRate	慢首屏占比
viewInteractiveTime50	可交互时间50分位值
viewInteractiveTime75	可交互时间75分位值
viewInteractiveTime95	可交互时间95分位值
viewInteractiveTime99	可交互时间99分位值
viewAppearTime50	首屏时间50分位值
viewAppearTime75	首屏时间75分位值
viewAppearTime95	首屏时间95分位值
viewAppearTime99	首屏时间99分位值
viewInteractiveTimeHistogram	页面可交互时间分布
viewAppearTimeHistogram	首屏时间分布
actionCount	操作次数
actionFluencyCount	操作流畅次数
actionCrashCount	操作崩溃次数
actionFailureCount	操作失败次数
actionAnrCount	操作卡顿次数
actionSlowCount	操作慢次数
actionFluencyErrorCount	操作流畅错误次数
actionAnrErrorCount	操作卡顿错误次数
actionSlowErrorCount	操作慢错误次数
actionAvailableCount	操作可用次数
actionDurationTime	平均操作时间
actionBlockTime	平均操作阻塞时间
actionFluencyRate	流畅操作占比
actionSlowRate	慢操作占比
actionAnrRate	卡顿占比
actionCrashRate	崩溃操作占比
actionFailureRate	失败操作占比
actionaAvailability	操作可用性
actionDurationTimeHistogram	操作耗时分布
actionBlockTimeHistogram	阻塞耗时分布
actionDurationTime50	操作时间50分位值
actionDurationTime75	操作时间75分位值
actionDurationTime95	操作时间95分位值
actionDurationTime99	操作时间99分位值

Endpoint

Method: POST
Type: RAW
URL: {{host}}/adhoc/query

Headers

Key	Value
Accept	application/json, text/plain, /
Content-Type	application/json

Body

{
    "timePeriod": {{timePeriod}},
    "endTime": {{endTime}},
    "datasource": "APP_UX_DATA",
    "metrics": [
        "launchStartupTime"
    ],
    "dimensions": [
       
    ],
    "filters": [
        {
            "name": "countryId",
            "value": [
                48
            ],
            "groupType": "region"
        },
        {
            "name": "mobileAppId",
            "value": [
                {{mobileAppId}}
            ],
            "operator": "IN"
        },
        {
            "name": "launchType",
            "value": [
                1
            ],
            "operator": "IN"
        },
        {
            "name": "moduleType",
            "value": [
                1
            ],
            "operator": "="
        }
    ],
    "render": "list",
    "orderByExprs": "launchStartupTime desc"
}

3. 网络请求-性能指标

数据源名称

APP_NETWORK_DATA

说明：扩展维度不可进行过滤和查询。

维度

列名	扩展维度	含义
timeStr
mobileAppId	mobileAppName	应用
mobileAppVersionId	mobileAppVersionName	应用版本
mobileBusinessId	mobileBusinessName	业务线
manufacturerId	manufacturerName	设备
manufacturerModelId	manufacturerModelName	设备型号
osId	osName	系统
osVersionId	osVersionName	系统版本
countryId	countryName	国家
regionId	regionName	省份
cityId	cityName	城市
carrierId	carrierName	运营商
connectTypeId	connectTypeName	接入方式
hostId	hostName	请求域名
uriId	uriName	请求地址
errorType	errorTypeName	错误类型
errorCode	errorCodeName	错误码
cdnId	cdnName	CDN厂商
hostIp		服务端IP
hostCountryId	hostCountryName	服务端国家
hostRegionId	hostRegionName	服务端省份
hostCityId	hostCityName	服务端城市
hostCarrierId	hostCarrierName	服务端运营商
bytesType	bytesTypeLabel	包大小
protocolType		协议类型
channelId	channelName	渠道
hostIpType	hostIpTypeName	服务端IP类型
requestType	requestTypeName	服务类型
launchType		启动类型
countyId	countyName	区县
hostCountyId	hostCountyName	区县

指标

指标项	含义
requestCount	请求次数
successCount	成功请求次数
passCount	正常请求次数
slowCount	慢请求次数
slowRequestRate	慢请求占比
errorCount	错误次数
httpErrorCount	HTTP错误次数
networkErrorCount	网络错误次数
dnsCount	DNS次数
tcpCount	TCP次数
sslCount	SSL次数
responseTimeTotal	总响应时间
networkTimeTotal	总网络时间
responseTime	平均响应时间
dnsTime	平均DNS时间
connectTime	平均TCP时间
sslTime	平均SSL时间
firstPacketTime	平均首包时间
remainPacketTime	平均剩余包时间
localQueueTime	平均客户端时间
networkTime	平均网络时间
throughput	吞吐率
networkSpeed	传输速率
networkTrafficSpeed	网络传输速率
totalTrafficConsumption	总流量消耗
trafficConsumption	平均传输数据量
bytesSend	上行流量消耗
bytesReceived	下行流量消耗
httpErrorRate	HTTP错误率
networkErrorRate	网络错误率
requestErrorRate	请求错误率
pingSuccessCount	ping成功次数
pingTime	网络延时
packetLossRate	丢包率
reuseRate	复用率
availability	可用性

Endpoint

Method: POST
Type: RAW
URL: {{host}}/adhoc/query

Headers

Key	Value	Description
Accept	application/json, text/plain, /
Content-Type	application/json

Body

{
    "timePeriod": {{timePeriod}},
    "endTime": {{endTime}},
    "datasource": "APP_NETWORK_DATA",
    "metrics": [
        "responseTime",
        "successCount"
    ],
    "dimensions": [
        "timeStr",
        "hostId"
    ],
    "filters": [
        {
            "name": "mobileAppId",
            "value": [
                {{mobileAppId}}
            ],
            "operator": "IN"
        },
        {
            "name": "hostId",
            "value": [
                0
            ],
            "operator": "!="
        }
    ],
    "top": {
        "metric": "throughput",
        "direction": "desc",
        "dimensions": [
            "hostId"
        ],
        "limit": 5
    },
    "limit": -1,
    "render": "list"
}