Fundamentals
Latency and queue theory
https://thenewstack.io/an-introduction-to-queue-theory-why-disaster-happens-at-the-edges/
- focus on distribution and outliers, not average
Queueing theory, latency, throughput
http://blog.tacertain.com/latency-throughput-and-escalators/
- Little’s Law and escalators
Performance at capacity
https://blog.nelhage.com/post/systems-at-capacity/
- congestion collapse
- queues don’t help with peak throughput
- backpressure / flow control
- rate limiting
- load shedding
Load testing / load shedding at AWS
https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/
Load testing
https://www.marcobehler.com/guides/load-testing#_how_to_load_test
- great introduction
- server + multiple loaders + probe
-
- load testing vs limit testing
- check for errors, check for expected rates
API load testing tools
https://www.npmjs.com/package/autocannon
https://github.com/giltene/wrk2
https://k6.io/blog/load-testing-using-github-actions/
Lock convoys
https://davekilian.com/lock-convoys.html
- Excessive contention on locks
- Forming queues
- High latency, low throughput
Perf engineering advice
https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/
- compression
- I/O
- fsync
I/O and CPU Linux benchmarks
http://www.kroah.com/log/blog/2020/09/18/fast-kernel-builds/
Linux performance
https://www.brendangregg.com/linuxperf.html
Techniques
https://tech.popdata.org/Flame-Graphs-Performance-Tuning-Made-Easy/
- Flame graphs
Tools
https://thume.ca/2023/12/02/tracing-methods/
- eBPF tracing
- Perfetto visualization
Micro benchmarking tools
http://ykarroum.com/2022/05/30/true-cost-list/
Profilers
https://github.com/mstange/samply
https://hackingcpp.com/cpp/tools/profilers.html
https://github.com/janestreet/magic-trace
https://github.com/brendangregg/perf-tools
Tracy profiler
https://github.com/wolfpld/tracy
memory allocation traces
https://github.com/matt-kimball/allocscope
AMD-specific profiler
https://www.amd.com/en/developer/uprof.html
- frame and sampling profiler
- CPU and GPU