Skip to main content
XenonStack

TestOps

XenonStack Insights Performance Profiling: Tools, Types and Go pprof Guide

and | Published | Updated

Performance profiling guide: monitoring names the symptom, a benchmark repeats it, pprof finds the hotspot, a profile diff proves the fix.

Key takeaway

  • Performance profiling finds the code that costs CPU, memory and wait time. Learn profile types, Go pprof commands, tools and safe production use.

Performance profiling measures where a running program spends its CPU time, memory and waiting time, so that you fix the code that actually causes a slowdown instead of guessing. A profile is a set of sampled call stacks. It tells you which functions are hot, which allocate most, and which block on locks. It does not tell you whether the system meets a target, which is the job of performance testing.

This guide uses Go's pprof tooling for the worked commands and compares it with other profilers. It is written for engineering leads and architects who need a repeatable way to find bottlenecks, decide how much profiling is safe in production, and prove that a fix worked. For the neighbouring practices, see performance monitoring, benchmarking and performance tuning.

What is the difference between profiling, monitoring, benchmarking and tuning?

They answer different questions and work best in sequence. Monitoring tells you that something is slow, a benchmark gives you a repeatable measurement, a profile shows why, and tuning is the change you make afterwards.

PracticeQuestion it answersTypical outputWhere to read more
MonitoringIs the system slow right now?Metrics and alerts over timePerformance monitoring
BenchmarkingHow fast is this code under a fixed test?Repeatable timings and allocationsBenchmarking
ProfilingWhich code causes the time or memory use?Sampled call stacks, flat and cumulative costThis post
TuningWhat change removes the cost?A code or configuration change, then a re-measurePerformance tuning
Performance testingDoes the system meet its target under load?Pass or fail against a service levelPerformance testing

What kinds of profile can you collect?

According to the Go diagnostics documentation (go.dev/doc/diagnostics), the runtime provides several predefined profiles. Other languages and tools have equivalents under different names.

ProfileWhat it showsUse it when
CPUWhere the program spends time actively using the CPU, not sleeping or waiting for I/OCPU use is high or a function is slow
HeapMemory allocation samples, for current and historical memory useMemory grows or you suspect a leak
GoroutineStack traces of all current goroutinesGoroutine counts keep rising
BlockWhere goroutines block on synchronisation primitives (enable with runtime.SetBlockProfileRate)Work is waiting but the CPU is idle
MutexLock contention (enable with runtime.SetMutexProfileFraction)CPU is underused because of lock contention
ThreadcreateParts of the program that create new OS threadsThread counts are unexpectedly high

The same page adds that the execution tracer, viewed with go tool trace, is better for latency and parallelism problems, while profiling tools are better for hot spots.

How does performance profiling work in Go?

Go ships the profiler with the toolchain. You collect a profile file, then read it with go tool pprof. The Go blog post on profiling (go.dev/blog/pprof) describes three ways to switch it on: flags in a standalone program, the standard flags of go test, and the net/http/pprof package for servers.

Two-column diagram mapping symptoms (high CPU, growing memory, waiting with idle CPU, latency spikes) to CPU profile, heap profile, block and mutex profiles, and execution trace.

Profile a benchmark with go test

Write a benchmark function in a file ending in _test.go, then run it with the profile flags. The -run=^$ pattern skips ordinary tests so that only the benchmark is measured.

go test -run=^$ -bench=. -benchmem -cpuprofile=cpu.out -memprofile=mem.out

This writes the CPU profile to cpu.out and the memory profile to mem.out. Open either with go tool pprof:

go tool pprof cpu.out
go tool pprof mem.out

Profile a running service

According to the net/http/pprof documentation (pkg.go.dev/net/http/pprof), importing the package for its side effect registers handlers under /debug/pprof/, and the seconds parameter sets how long a CPU profile runs. The example below collects a 30-second CPU profile from a service listening on localhost:6060.

import _ "net/http/pprof"

go tool pprof "http://localhost:6060/debug/pprof/profile?seconds=30"

Explore the result

Inside the pprof prompt, the commands described in the Go blog post are the ones you will use most:

  • top N lists the N most expensive functions, and top N -cum sorts them by cumulative cost.
  • list FUNC shows the source lines of a function annotated with their cost.
  • web writes an SVG call graph and opens it.
  • --inuse_objects on a memory profile reports object counts instead of sizes.

The pprof project documentation (github.com/google/pprof) adds a browser interface. Running pprof with the -http flag starts a local web server with graph, flame graph, top, source and peek views.

How do you read flat and cumulative columns?

Per the Go blog post, the first two columns of top output (flat and flat%) show the number of samples in which the function itself was running, as a count and a percentage. The third column (sum%) is the running total of the listing. The last two columns (cum and cum%) count samples in which the function appeared on the stack, either running or waiting for a function it called to return.

ColumnMeaningWhat a large value tells you
flat, flat%Samples where this function was the one runningThe function's own code is the cost
sum%Running total down the listingHow few functions account for most of the samples
cum, cum%Samples where the function was on the stack at allThe function, or something it calls, is the cost

A function with high cum but low flat is usually a caller. Follow it down to the callee with high flat. A function with high flat is where to read the source first.

How do you prove a fix worked?

Profile, change one thing, then profile again under the same conditions. The Go blog post follows this loop through a worked example. The pprof documentation describes the -diff_base option for comparing two profiles, where percentages are relative to the total of the base profile and some entries may be negative. Collect both profiles with the same duration and workload, otherwise the comparison misleads.

The Go diagnostics page also warns that some diagnostic tools interfere with each other. Its example is that precise memory profiling skews CPU profiles. Collect one profile type at a time.

Which performance profiling tools should you consider?

The right tool depends on the language and on whether you need CPU, memory or lock data. The descriptions below come from each project's own pages.

ToolWhat it does (per its own page)FitsStatus or caution
Go pprofReads CPU, heap, block, mutex and other profiles from Go programs (Go diagnostics)Go services and benchmarksPart of the Go toolchain
Valgrind (Callgrind, Cachegrind, Massif, DHAT)Callgrind adds call-graph information to cache simulation, Massif samples heap use over time, DHAT analyses heap allocations (valgrind.org)C and C++ programsIts page gives roughly 20 to 100 times slowdown for Cachegrind and about 20 times for Massif, so use it in test, not production
OProfileA statistical profiler for Linux that profiles running code at low overhead (oprofile.sourceforge.io)System-wide Linux profilingIts page says the legacy profiler was removed in version 1.0.0 in favour of the Linux Kernel Performance Events Subsystem
gprofThe GNU profiler, which reports a flat profile and a call graph (GNU binutils manual)C and C++ programs built with GNU toolsConfirm usage steps in the manual for your binutils version
Linux perfThe Go diagnostics page notes that perf can profile Go programs, including cgo and kernel codeMixed native and Go workloadsNeeds Linux kernel support

Is it safe to profile in production?

Mostly yes, with limits. The Go diagnostics page states that it is safe to profile programs in production, but that enabling some profiles, such as the CPU profile, adds cost, so you should expect a performance downgrade and measure the profiler's overhead before enabling it. It suggests profiling a randomly selected replica periodically, and collecting one profile at a time.

There is also a governance point. The pprof endpoints expose internals of the running program. The net/http/pprof page does not discuss access control, so we recommend binding them to localhost or an internal network, restricting who can call them, and reviewing profile files before sharing them, since stack traces can include function and package names. Treat that as a house rule to confirm with your security team, not as a vendor claim.

Can profiles also improve the build itself?

Yes. According to the Go profile-guided optimisation guide (go.dev/doc/pgo), the compiler can use a CPU pprof profile from representative runs to make better optimisation decisions, such as inlining frequently called functions. PGO is supported starting in Go 1.20, a profile named default.pgo in the main package directory is picked up automatically by go build from Go 1.21 (the default is -pgo=auto), and the guide reports around 2 to 14 per cent improvement on a representative set of programs as of Go 1.22. It stresses that the profile must be representative of production, and that microbenchmarks are usually poor candidates. Confirm the current figure and behaviour on the guide before you rely on it.

How should an enterprise adopt performance profiling?

  1. Start from a symptom. Use monitoring and load tests to name the slow endpoint or job before profiling.
  2. Make a benchmark. A repeatable benchmark gives you a before and after.
  3. Profile in test first. Collect CPU and heap profiles on a representative build.
  4. Add production profiling carefully. Sample one replica, one profile type at a time, with internal-only access.
  5. Fix one hotspot, then re-profile. Record the diff and keep the profile files with the change.
  6. Keep the evidence. Store profiles and benchmark results next to the pull request so reviewers can see the effect.

Teams that run AI agents on top of these services should profile the services the agents call, because slow tools and retrieval steps show up as slow agent runs. ElixirClaw is an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control, and AgentQA provides quality assurance for agent workflows and regressions, according to its own page.

When is profiling not the right tool?

SituationWhy profiling is not enoughBetter first step
You do not yet know which request is slowA profile of the whole process is noisyMonitoring and tracing
The system must meet a target under loadA profile shows cost, not whether the target is metPerformance testing
Code is fast but the wait is on a database or network callCPU profiles hide time spent waitingBlock profiles and the execution tracer, or tracing
Benchmarks run a tiny part of the applicationResults may not match production behaviourProfile a representative workload

To map one performance decision your teams and agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

FAQ

What is performance profiling?

Performance profiling measures where a running program spends CPU time, memory and waiting time, using sampled call stacks, so you can find the functions that cause a slowdown.

What do flat and cum mean in pprof?

Per the Go blog, flat counts samples in which the function itself was running. Cum counts samples in which the function was on the stack, either running or waiting for a called function to return.

Is it safe to profile in production?

According to the Go diagnostics page, it is safe to profile programs in production, but some profiles, such as the CPU profile, add cost. Measure the overhead first and profile one replica at a time.

How is profiling different from benchmarking?

A benchmark gives a repeatable measurement of how fast code runs. A profile shows where inside that code the time or memory goes. Use a benchmark to prove a fix and a profile to find it.

Which profiling tool should I use?

Use pprof for Go, Valgrind tools such as Callgrind or Massif for C and C++ in test, and system-wide tools such as perf on Linux. Check each tool's own documentation for current status.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue