This commit is contained in:
Isuru Samarathunga
2025-11-01 00:11:13 +05:30
commit 9d85e3d822
5723 changed files with 1758962 additions and 0 deletions

View File

@ -0,0 +1,69 @@
# Purgatorio
[link on wikipedia](https://en.wikipedia.org/wiki/Purgatorio)
![user interface](images/user_interface.png)
Purgatorio is a visualization tool for simpleperf traces. It's based on [libsimpleperf](https://source.corp.google.com/android/system/extras/simpleperf/;l=1?q=simpleperf&sq=package:%5Eandroid$),
[Bokeh](https://bokeh.org/) and [D3 flamegraphs](https://github.com/spiermar/d3-flame-graph).
The main difference from [Inferno](https://source.corp.google.com/android/system/extras/simpleperf/scripts/inferno/;l=1) is that Purgatorio focuses on visualizing system-wide traces (the ones with the `-a` argument) on a time-organized sequence, and allow the user to interact with the graph by zooming, hovering on samples and visualize a flame graph for specific samples (be it restricted by time interval, set of threads or whatever subset).
## Obtaining the sources
git clone sso://user/balejs/purgatorio
## Getting ready
**NOTE**: In theory it should work on most OSes, but Purgatorio has been tested on gLinux only. Any feedback, recommendations and patches to get it work elsewhere will be welcome (balejs@).
Purgatorio tends to be self-contained, but Bokeh and some of its dependencies aren't shipped with the default python libraries, hence they require to be installed with pip3. Assuming they already have python3 installed, Purgatorio hopefuls should follow these steps:
$ sudo apt-get install python3-pip
$ pip3 install jinja2 bokeh pandas
Run `python3 purgatorio.py -h` for a list of command-line arguments.
## Example
One can trace a Camera warm launch with:
$ adb shell simpleperf record --trace-offcpu --call-graph fp -o /data/local/camera_warm_launch.data -a
[launch camera here, then press ctrl + c]
$ adb pull /data/local/camera_warm_launch.data
And then run:
python3 purgatorio.py camera_warm_launch.data
If you get lots of "Failed to read symbols" messages, and backtraces in the diagram don't show the symbols you're interested into, you might want to try [building a symbols cache](https://chromium.googlesource.com/android_ndk/+/refs/heads/master/simpleperf/doc/README.md#how-to-solve-missing-symbols-in-report) for the trace, then run purgatorio again with:
python3 purgatorio.py camera_warm_launch.data -u [symbols cache]
# Purgatorio interface
The Purgatorio User Interface is divided in three areas:
## Main Graph
It's the area to the left, including process names and color-coded dots grouped by process. It's used to navigate throungh the trace and identify samples of ineterest. By hovering on a sample (or set of samples) their callstacks will be visualized over the graph. When selecting a et of samples, their aggregated data will be visualized in the other sections of the UI. Multiple sections of the graph can be aggregated by holding down the [ctrl] button during selection.
The toolbox to the right can be used to configure interactions with the graph:
![toolbox description](images/toolbox.png)
## Flame graph
The flame graph is located in the upper right portion. Once samples are selected in the main graph, the flame graph will show an interactive visualization for their aggregated callstacks. In this case the selection included mostly samples for com.google.android.GoogleCamera
![flame graph](images/flame_graph.png)
It's possible to select a given stack entry to zoom on it and look at entry deeper in the callstack
![flame graph](images/flame_graph_zoomed.png)
When studiyng system issues it's often useful to visualize an inverted callstack. This can be done by clicking the related check box. The graph below is the same as in the first flame graph above, but with call stack inverted. In this case, inverted visualization directly points at [possible issues with io](http://b/158783580#comment12)
![inverted flame graph](images/inverted_flame_graph.png)
## Sample table
It's located in the lower right and counts samples by thread (for direct flame graphs) or symbol (for inverted flame graphs). Table columns can be sorted by clicking on their respective layers, and selecting specific lines filters the contents of the flame graph to the selected threads or symbols. Multiple lines can be selected at the same time.
![table](images/table.png)

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 186 KiB

View File

@ -0,0 +1,306 @@
#!/usr/bin/env python3
#
# Copyright (C) 2021 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import argparse
import bisect
import jinja2
import io
import math
import os
import pandas as pd
from pathlib import Path
import re
import sys
from bokeh.embed import components
from bokeh.io import output_file, show
from bokeh.layouts import layout, Spacer
from bokeh.models import ColumnDataSource, CustomJS, WheelZoomTool, HoverTool, FuncTickFormatter
from bokeh.models.widgets import DataTable, DateFormatter, TableColumn
from bokeh.models.ranges import FactorRange
from bokeh.palettes import Category20b
from bokeh.plotting import figure
from bokeh.resources import INLINE
from bokeh.transform import jitter
from bokeh.util.browser import view
from functools import cmp_to_key
# fmt: off
simpleperf_path = Path(__file__).absolute().parents[1]
sys.path.insert(0, str(simpleperf_path))
import simpleperf_report_lib as sp
from simpleperf_utils import BaseArgumentParser
# fmt: on
def create_graph(args, source, data_range):
graph = figure(
sizing_mode='stretch_both', x_range=data_range,
tools=['pan', 'wheel_zoom', 'ywheel_zoom', 'xwheel_zoom', 'reset', 'tap', 'box_select'],
active_drag='box_select', active_scroll='wheel_zoom',
tooltips=[('thread', '@thread'),
('callchain', '@callchain{safe}')],
title=args.title, name='graph')
# a crude way to avoid process name cluttering at some zoom levels.
# TODO: remove processes from the ticker base on the number of samples currently visualized.
# The process with most samples visualized should always be visible on the ticker
graph.xaxis.formatter = FuncTickFormatter(args={'range': data_range, 'graph': graph}, code="""
var pixels_per_entry = graph.inner_height / (range.end - range.start) //Do not rond end and start here
var entries_to_skip = Math.ceil(12 / pixels_per_entry) // kind of 12 px per entry
var desc = tick.split(/:| /)
// desc[0] == desc[1] for main threads
var keep = (desc[0] == desc[1]) &&
!(desc[2].includes('unknown') ||
desc[2].includes('Binder') ||
desc[2].includes('kworker'))
if (pixels_per_entry < 8 && !keep) {
//if (index + Math.round(range.start)) % entries_to_skip != 0) {
return ""
}
return tick """)
graph.xaxis.major_label_orientation = math.pi/6
graph.circle(y='time',
x='thread',
source=source,
color='color',
alpha=0.3,
selection_fill_color='White',
selection_line_color='Black',
selection_line_width=0.5,
selection_alpha=1.0)
graph.y_range.range_padding = 0
graph.xgrid.grid_line_color = None
return graph
def create_table(graph):
# Empty dataframe, will be filled up in js land
empty_data = {'thread': [], 'count': []}
table_source = ColumnDataSource(pd.DataFrame(
empty_data, columns=['thread', 'count'], index=None))
graph_source = graph.renderers[0].data_source
columns = [
TableColumn(field='thread', title='Thread'),
TableColumn(field='count', title='Count')
]
# start with a small table size (stretch doesn't reduce from the preferred size)
table = DataTable(
width=100,
height=100,
sizing_mode='stretch_both',
source=table_source,
columns=columns,
index_position=None,
name='table')
graph_selection_cb = CustomJS(code='update_selections()')
graph_source.selected.js_on_change('indices', graph_selection_cb)
table_source.selected.js_on_change('indices', CustomJS(args={}, code='update_flamegraph()'))
return table
def generate_template(template_file='index.html.jinja2'):
loader = jinja2.FileSystemLoader(
searchpath=os.path.dirname(os.path.realpath(__file__)) + '/templates/')
env = jinja2.Environment(loader=loader)
return env.get_template(template_file)
def generate_html(args, components_dict, title):
resources = INLINE.render()
script, div = components(components_dict)
return generate_template().render(
resources=resources, plot_script=script, plot_div=div, title=title)
class ThreadDescriptor:
def __init__(self, pid, tid, name):
self.name = name
self.tid = tid
self.pid = pid
def __lt__(self, other):
return self.pid < other.pid or (self.pid == other.pid and self.tid < other.tid)
def __gt__(self, other):
return self.pid > other.pid or (self.pid == other.pid and self.tid > other.tid)
def __eq__(self, other):
return self.pid == other.pid and self.tid == other.tid and self.name == other.name
def __str__(self):
return str(self.pid) + ':' + str(self.tid) + ' ' + self.name
def generate_datasource(args):
lib = sp.ReportLib()
lib.ShowIpForUnknownSymbol()
if args.usyms:
lib.SetSymfs(args.usyms)
if args.input_file:
lib.SetRecordFile(args.input_file)
if args.ksyms:
lib.SetKallsymsFile(args.ksyms)
lib.SetReportOptions(args.report_lib_options)
product = lib.MetaInfo().get('product_props')
if product:
manufacturer, model, name = product.split(':')
start_time = -1
end_time = -1
times = []
threads = []
thread_descs = []
callchains = []
while True:
sample = lib.GetNextSample()
if sample is None:
lib.Close()
break
symbol = lib.GetSymbolOfCurrentSample()
callchain = lib.GetCallChainOfCurrentSample()
if start_time == -1:
start_time = sample.time
sample_time = (sample.time - start_time) / 1e6 # convert to ms
times.append(sample_time)
if sample_time > end_time:
end_time = sample_time
thread_desc = ThreadDescriptor(sample.pid, sample.tid, sample.thread_comm)
threads.append(str(thread_desc))
if thread_desc not in thread_descs:
bisect.insort(thread_descs, thread_desc)
callchain_str = ''
for i in range(callchain.nr):
symbol = callchain.entries[i].symbol # SymbolStruct
entry_line = ''
if args.include_dso_names:
entry_line += symbol._dso_name.decode('utf-8') + ':'
entry_line += symbol._symbol_name.decode('utf-8')
if args.include_symbols_addr:
entry_line += ':' + hex(symbol.symbol_addr)
if i < callchain.nr - 1:
callchain_str += entry_line + '<br>'
callchains.append(callchain_str)
# define colors per-process
palette = Category20b[20]
color_map = {}
last_pid = -1
palette_index = 0
for thread_desc in thread_descs:
if thread_desc.pid != last_pid:
last_pid = thread_desc.pid
palette_index += 1
palette_index %= len(palette)
color_map[str(thread_desc.pid)] = palette[palette_index]
colors = []
for sample_thread in threads:
pid = str(sample_thread.split(':')[0])
colors.append(color_map[pid])
threads_range = [str(thread_desc) for thread_desc in thread_descs]
data_range = FactorRange(factors=threads_range, bounds='auto')
data = {'time': times,
'thread': threads,
'callchain': callchains,
'color': colors}
source = ColumnDataSource(data)
return source, data_range
def main():
parser = BaseArgumentParser()
parser.add_argument('-i', '--input_file', type=str, required=True, help='input file')
parser.add_argument('--title', '-t', type=str, help='document title')
parser.add_argument('--ksyms', '-k', type=str, help='path to kernel symbols (kallsyms)')
parser.add_argument('--usyms', '-u', type=str, help='path to tree with user space symbols')
parser.add_argument('--output', '-o', type=str, help='output file')
parser.add_argument('--dont_open', '-d', action='store_true', help='Don\'t open output file')
parser.add_argument('--include_dso_names', '-n', action='store_true',
help='Include dso names in backtraces')
parser.add_argument('--include_symbols_addr', '-s', action='store_true',
help='Include addresses of symbols in backtraces')
parser.add_report_lib_options(default_show_art_frames=True)
args = parser.parse_args()
# TODO test hierarchical ranges too
source, data_range = generate_datasource(args)
graph = create_graph(args, source, data_range)
table = create_table(graph)
output_filename = args.output
if not output_filename:
output_filename = os.path.splitext(os.path.basename(args.input_file))[0] + '.html'
title = os.path.splitext(os.path.basename(output_filename))[0]
html = generate_html(args, {'graph': graph, 'table': table}, title)
with io.open(output_filename, mode='w', encoding='utf-8') as fout:
fout.write(html)
if not args.dont_open:
view(output_filename)
if __name__ == "__main__":
main()

View File

@ -0,0 +1,66 @@
<!DOCTYPE html>
<html lang="en">
<head>
<title>{{ title }}</title>
<script
src="https://code.jquery.com/jquery-3.5.1.min.js"
integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0="
crossorigin="anonymous"></script>
<meta charset="utf-8">
{{ resources }}
<style>
{% include 'styles.css' %}
</style>
<link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/npm/d3-flame-graph@4.0.6/dist/d3-flamegraph.css">
<script
src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"
integrity="sha256-VazP97ZCwtekAsvgPBSUwPFKdrwD3unUfSGVYrahUqU="
crossorigin="anonymous"></script>
<script type="text/javascript" src="https://d3js.org/d3.v4.min.js"></script>
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/d3-flame-graph@4.0.6/dist/d3-flamegraph.min.js"></script>
</head>
<body>
<div id="help_dialog" class="dialog">
<div class="dialog_area">
<span class="dialog_close">&times;</span>
<p> <b>Main plot (upper left):</b> pan with click+mouse movement, zoom in/out with the mouse
wheel, hover on sample clusters to see backtraces. Select samples with the rectangular
selection tool or by clicking on them. Select holding shift to add or ctrl+shift to
remove samples to or from the selection. Different tools can be enabled/disabled from
the toolbox.</p>
<p><b>Flame graph (upper right):</b> click on specific items to zoom in.</p>
<p><b>Sample table (lower right):</b> select processes to filter in the Flame graph.</p>
</div>
</div>
<div class="top_right">
<button id="help_button" class="help" text-align="right">HELP</button>
</div>
<div class="left"> {{ plot_div.graph }} </div>
<div class="middle_right">
<div id="flame"/>
</div>
<div class="bottom_right">
<div style="display: flex; justify-content: space-around">
<div>
<label for="regex">Filter by regex:</label>
<input type="text" id="regex" oninput="update_selections()"/>
</div>
<div>
Invert callstack <input type="checkbox" id="inverted_checkbox" onclick="update_selections()">
</div>
</div>
{{ plot_div.table }}
</div>
<script>{% include 'main.js' %}</script>
{{ plot_script }}
</body>
</html>

View File

@ -0,0 +1,245 @@
/*
* Copyright (C) 2021 The Android Open Source Project
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
function generateHash (name) {
// Return a vector (0.0->1.0) that is a hash of the input string.
// The hash is computed to favor early characters over later ones, so
// that strings with similar starts have similar vectors. Only the first
// 6 characters are considered.
const MAX_CHAR = 6
var hash = 0
var maxHash = 0
var weight = 1
var mod = 10
if (name) {
for (var i = 0; i < name.length; i++) {
if (i > MAX_CHAR) { break }
hash += weight * (name.charCodeAt(i) % mod)
maxHash += weight * (mod - 1)
weight *= 0.70
}
if (maxHash > 0) { hash = hash / maxHash }
}
return hash
}
function offCpuColorMapper (d) {
if (d.highlight) return '#E600E6'
let name = d.data.n || d.data.name
let vector = 0
const nameArr = name.split('`')
if (nameArr.length > 1) {
name = nameArr[nameArr.length - 1] // drop module name if present
}
name = name.split('(')[0] // drop extra info
vector = generateHash(name)
const r = 0 + Math.round(55 * (1 - vector))
const g = 0 + Math.round(230 * (1 - vector))
const b = 200 + Math.round(55 * vector)
return 'rgb(' + r + ',' + g + ',' + b + ')'
}
var flame = flamegraph()
.cellHeight(18)
  .width(window.innerWidth * 3 / 10 - 20) // 30% width
.transitionDuration(750)
.minFrameSize(5)
.transitionEase(d3.easeCubic)
.inverted(false)
.sort(true)
.title("")
//.differential(false)
//.elided(false)
.selfValue(false)
.setColorMapper(offCpuColorMapper);
function update_table() {
let inverted = document.getElementById("inverted_checkbox").checked
let regex
let graph_source = Bokeh.documents[0].get_model_by_name('graph').renderers[0].data_source
let table_source = Bokeh.documents[0].get_model_by_name('table').source
let graph_selection = graph_source.selected.indices
let threads = graph_source.data.thread
let callchains = graph_source.data.callchain
let selection_len = graph_selection.length;
if (document.getElementById("regex").value) {
regex = new RegExp(document.getElementById("regex").value)
}
table_source.data.thread = []
table_source.data.count = []
table_source.data.index = []
for (let i = 0; i < selection_len; i ++) {
let entry = "<no callchain>"
if (regex !== undefined && !regex.test(callchains[graph_selection[i]])) {
continue;
}
if (inverted) {
let callchain = callchains[graph_selection[i]].split("<br>")
for (let e = 0; e < callchain.length; e ++) {
if (callchain[e] != "") { // last entry is apparently always an empty string
entry = callchain[e]
break
}
}
} else {
entry = threads[graph_selection[i]]
}
let pos = table_source.data.thread.indexOf(entry)
if(pos == -1) {
table_source.data.thread.push(entry)
table_source.data.count.push(1)
table_source.data.index.push(table_source.data.thread.length)
} else {
table_source.data.count[pos] ++
}
}
table_source.selected.indices = []
table_source.change.emit()
}
function should_insert_callchain(callchain, items, filter_index, inverted) {
for (t = 0; t < filter_index.length; t ++) {
if (callchain[0] === items[filter_index[t]]) {
return true
}
}
if (filter_index.length > 0) {
return false
}
return true
}
function insert_callchain(root, callchain, inverted) {
let root_pos = -1
let node = root
node.value ++
for (let e = 0; e < callchain.length; e ++) {
let entry = callchain[e].replace(/^\s+|\s+$/g, '')
let entry_pos = -1
for (let j = 0; j < node.children.length; j ++) {
if (node.children[j].name == entry) {
entry_pos = j
break
}
}
if (entry_pos == -1) {
node.children.push({name: entry, value:0, children:[]})
entry_pos = node.children.length - 1
}
node = node.children[entry_pos]
node.value ++
}
}
function update_flamegraph() {
let inverted = document.getElementById("inverted_checkbox").checked
let root = {name: inverted ? "samples" : "processes", value: 0, children: []}
let graph_source = Bokeh.documents[0].get_model_by_name('graph').renderers[0].data_source
let graph_selection = graph_source.selected.indices
let callchains = graph_source.data.callchain
let graph_threads = graph_source.data.thread
let table_source = Bokeh.documents[0].get_model_by_name('table').source
let table_selection = table_source.selected.indices
let table_threads = table_source.data.thread
let regex
if (document.getElementById("regex").value) {
regex = new RegExp(document.getElementById("regex").value)
}
for (let i = 0; i < graph_selection.length; i ++) {
let thread = graph_threads[graph_selection[i]]
let callchain = callchains[graph_selection[i]].split("<br>")
callchain = callchain.filter(function(e){return e != ""})
if (regex !== undefined && !regex.test(callchains[graph_selection[i]])) {
continue;
}
if (callchain.length == 0) {
callchain.push("<no callchain>")
}
callchain.push(thread)
if (!inverted){
callchain = callchain.reverse()
}
if (should_insert_callchain(callchain, table_threads, table_selection)) {
insert_callchain(root, callchain)
}
}
if (root.children.length == 1) {
root = root.children[0]
}
d3.select("#flame")
.datum(root)
.call(flame)
}
var help_dialog = document.getElementById("help_dialog");
document.getElementById("help_button").onclick = function() {
help_dialog.style.display = "block";
}
window.onclick = function(event) {
if (event.target == help_dialog) {
help_dialog.style.display = "none";
}
}
document.getElementsByClassName("dialog_close")[0].onclick = function() {
help_dialog.style.display = "none";
}
function update_selections() {
update_flamegraph()
update_table()
}

View File

@ -0,0 +1,133 @@
/*
* Copyright (C) 2021 The Android Open Source Project
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
body {
font-family: sans-serif;
}
::-webkit-scrollbar {
width: 1em;
}
::-webkit-scrollbar-track {
box-shadow: inset 0 0 0.1em white;
border-radius: 0.5em;
}
::-webkit-scrollbar-thumb {
background: lightgrey;
border-radius: 0.5em;
}
div.left {
position:fixed;
top: 0;
left: 0;
width: 70%;
height: 100%;
}
div.top_right {
position: fixed;
top: 0;
right: 0;
text-align: right;
width: 30%;
height: 1em;
}
div.middle_right {
position:fixed;
top: 2%;
right: 0;
width: 30%;
height: 78%;
overflow-y: scroll;
}
div.bottom_right {
position: fixed;
width: 30%;
height: 20%;
bottom: 0;
right: 0;
}
button {
border: none;
outline: none;
padding: 0;
background: white;
font-size: 0.5em;
}
button.help:before
{
content: '?';
display: inline-block;
font-weight: bold;
text-align: center;
font-size: 1.4em;
width: 1.5em;
height: 1.5em;
line-height: 1.6em;
border-radius: 1.2em;
margin-right: 0.3em;
color: GoldenRod;
background: white;
border: 0.1em solid GoldenRod;
}
button.help:hover:before
{
color: white;
background: GoldenRod;
}
.dialog {
display: none;
position: fixed;
z-index: 1;
left: 0;
top: 0;
width: 100%;
height: 100%;
overflow: auto;
background-color: rgba(0,0,0,0.4);
}
.dialog_area {
background-color: white;
margin: 20% auto;
border: 0.05em solid gray;
border-radius: 0.5em;
padding-left: 0.5em;
padding-right: 0.5em;
width: 50%;
}
.dialog_close {
color: darkgray;
float: right;
font-size: 2em;
font-weight: bold;
}
.dialog_close:focus,
.dialog_close:hover {
cursor: pointer;
color: black;
}