### Install PCP on Debian/Ubuntu

Source: https://pcp.io/download.html

Installs the Performance Co-Pilot package using apt-get and starts the necessary services.

```bash
apt-get install pcp
invoke-rc.d pmcd start
invoke-rc.d pmie start
invoke-rc.d pmlogger start

```

--------------------------------

### Install PCP REST API and Valkey Service

Source: https://pcp.io/docs/guide.html

Enables and starts both the pmproxy and valkey services. This is typically done before configuring a web interface like Grafana.

```bash
# systemctl enable --now pmproxy valkey

```

--------------------------------

### Install PCP on Fedora/RHEL

Source: https://pcp.io/download.html

Installs PCP using yum and starts services. Includes commands for RHEL 6 and RHEL 7/Fedora.

```bash
yum install pcp
# RHEL 6
service pmcd start
service pmlogger start
service pmie start
# RHEL 7 and later / Fedora
systemctl start pmcd
systemctl start pmie
systemctl start pmlogger

```

--------------------------------

### Start PCP Services and Rebuild PMNS

Source: https://pcp.io/faq.html

Use this command sequence to start PCP services and rebuild the Performance Metrics Name Space (PMNS) if it appears empty after a source build and 'make install'.

```bash
# source /etc/pcp.conf
# touch $PCP_VAR_DIR/pmns/.NeedRebuild
# $PCP_RC_DIR/pcp start

```

--------------------------------

### Install PCP Glider on Windows

Source: https://pcp.io/glider.html

Install PCP Glider using the MSI installer and run the post-installation script.

```bash
C:> msiexec /i pcp-glider-*.msi
C:> cd \Glider\scripts
C:> .\postinst.bat
```

--------------------------------

### Starting the Performance Metrics Inference Engine (PMIE)

Source: https://pcp.io/papers/pcp-presentation/index.html

Start the PMIE service manually.

```bash
# systemctl start pmie
```

--------------------------------

### Enable and start PCP related services

Source: https://pcp.io/papers/pcp-presentation/index.html

Enables and starts the pmproxy, grafana-server, and redis services.

```bash
# systemctl enable pmproxy grafana-server redis
```

```bash
# systemctl start pmproxy grafana-server redis
```

--------------------------------

### Enabling the Performance Metrics Inference Engine (PMIE)

Source: https://pcp.io/papers/pcp-presentation/index.html

Enable the PMIE service to start automatically on system boot.

```bash
# systemctl enable pmie
```

--------------------------------

### Install PCP REST API Service

Source: https://pcp.io/docs/guide.html

Enables and starts the Performance Metrics Proxy Daemon (pmproxy) service. This provides a REST API for accessing performance data.

```bash
# systemctl enable --now pmproxy

```

--------------------------------

### Install Basic PCP Tools and Services

Source: https://pcp.io/docs/guide.html

Installs essential PCP tools and enables the Performance Metrics Collector Daemon (pmcd) and logger (pmlogger) on systemd-based distributions.

```bash
# yum install pcp  # or apt-get or dnf or zypper
# systemctl enable --now pmcd pmlogger

```

--------------------------------

### Install PCP on Fedora/RHEL

Source: https://pcp.io/papers/pcp-presentation/index.html

Installs the pcp-zeroconf package on Fedora or RHEL-based systems.

```bash
# dnf install pcp-zeroconf
```

--------------------------------

### Enable and Start PMIE Service

Source: https://pcp.io/docs/guide.html

Enable the Performance Metrics Inference Engine (PMIE) service to start automatically and run it immediately.

```bash
# systemctl enable --now pmie
```

--------------------------------

### Install PCP on Debian/Ubuntu

Source: https://pcp.io/papers/pcp-presentation/index.html

Installs the pcp-zeroconf package on Debian or Ubuntu-based systems.

```bash
# apt-get install pcp-zeroconf
```

--------------------------------

### Enable and Start PCP Time Series Services

Source: https://pcp.io/docs/guide.html

Enable and start the necessary PCP services for fast, scalable time-series querying: `pmlogger`, `pmproxy`, and `valkey`.

```bash
# systemctl enable --now pmlogger pmproxy valkey
```

--------------------------------

### Install PostgreSQL PMDA

Source: https://pcp.io/docs/guide.html

Installs the PostgreSQL Performance Metrics Domain Agent (PMDA) by navigating to its directory and running the Install script.

```bash
# cd /var/lib/pcp/pmdas/postgresql
# ./Install

```

--------------------------------

### Replay System Metrics in Top-like Window

Source: https://pcp.io/docs/guide.html

Use 'pcp atop' to replay system metrics in a top(1)-like window. '-b' specifies the start time, and '-r' enables replay mode.

```bash
$ pcp atop -b 09:00 -r acme.com/20140902
```

```bash
$ pcp -S @09:00 -a acme.com/20140902 atop
```

--------------------------------

### Install Additional PCP Monitoring Tools

Source: https://pcp.io/docs/guide.html

Installs optional packages on the monitoring host to extend the set of available monitoring tools, including documentation and graphical interfaces.

```bash
# yum install pcp-doc pcp-gui pcp-system-tools  # or apt-get or dnf or zypper

```

--------------------------------

### Start pmlogger service

Source: https://pcp.io/papers/pcp-presentation/index.html

Starts the pmlogger service for collecting performance data.

```bash
# systemctl start pmlogger
```

--------------------------------

### Install Grafana and Redis for PCP

Source: https://pcp.io/papers/pcp-presentation/index.html

Installs Grafana and Redis, which can be used with PCP for web interface and data storage.

```bash
# dnf install grafana-pcp redis
```

--------------------------------

### Fetching Container Metrics with pminfo

Source: https://pcp.io/papers/pcp-presentation/index.html

Use pminfo with the --fetch option to retrieve metrics from within containers. No PCP installation is needed inside the container.

```bash
# pminfo --fetch containers.name containers.state.running
```

--------------------------------

### PMIE Script Example

Source: https://pcp.io/docs/guide.html

Define a PMIE rule to check memory usage, test its syntax, run it against an archive, and print a message if a condition is met.

```bash
$ cat pmie.ex
bloated = (mem.util.used > 5 Gbyte) -> print "%v memory used on %h!";
$ pmie -C pmie.ex
$ pmie -t 1min -c pmie.ex -S @09:00 -T @10:00 -a acme.com/20140820
```

--------------------------------

### Get detailed information about free memory metric

Source: https://pcp.io/papers/pcp-presentation/index.html

Uses pminfo with -dtf flags to display detailed information about the 'mem.freemem' metric.

```bash
# pminfo -dtf mem.freemem
```

--------------------------------

### Get detailed information about network interface bytes metric

Source: https://pcp.io/papers/pcp-presentation/index.html

Uses pminfo with -dtTf flags to display detailed information about the 'network.interface.in.bytes' metric, including instance data.

```bash
# pminfo -dtTf network.interface.in.bytes
```

--------------------------------

### Get Metric Descriptor with pminfo

Source: https://pcp.io/faq.html

Retrieve detailed information about a specific metric, such as 'filesys.free', using the 'pminfo -md' command. This includes its PMID, data type, instance domain, semantics, and units.

```bash
$ pminfo -md filesys.free
filesys.free PMID: 60.5.3
  Data Type: 64-bit unsigned int  InDom: 60.5 0xf000005
  Semantics: instant  Units: Kbyte
```

--------------------------------

### Explore Filesystem Metrics with pminfo

Source: https://pcp.io/faq.html

Use the 'pminfo' command to list available filesystem metrics. This command helps in understanding the metrics exposed by the PCP system for filesystem-related performance data.

```bash
$ pminfo filesys
filesys.capacity
filesys.used
filesys.free
filesys.maxfiles
filesys.usedfiles
filesys.freefiles
filesys.mountdir
filesys.full
```

--------------------------------

### Display All Available Process Metrics

Source: https://pcp.io/docs/guide.html

Use `pminfo` to list all available performance metrics related to processes.

```bash
$ pminfo proc
```

--------------------------------

### Dump System Metrics in Free-like Fashion

Source: https://pcp.io/docs/guide.html

Use 'pcp ... free' to dump system metrics in a free(1)-like fashion at a specific historical time offset. '-O' specifies the offset.

```bash
$ pcp -a acme.com/20140902 -O @10:02 free
```

--------------------------------

### Get current values of free memory metric

Source: https://pcp.io/papers/pcp-presentation/index.html

Uses pmval to retrieve and display multiple samples of the 'mem.freemem' metric.

```bash
# pmval mem.freemem
```

--------------------------------

### Display first 3 performance metrics

Source: https://pcp.io/papers/pcp-presentation/index.html

Uses pminfo to display the first three performance metrics available on the system.

```bash
# pminfo | head -n3
```

--------------------------------

### Replay Performance Metrics with GUI

Source: https://pcp.io/docs/guide.html

Use 'pmchart' to replay performance metrics with a GUI application. '-t' sets the default interval, and '-S' and '-T' define the time range. Use File->New Chart to select metrics.

```bash
$ pmchart -t 2sec -S @09:00 -T @10:00 -a acme.com/20140902
```

--------------------------------

### Dump System Metrics with Fixed Columns

Source: https://pcp.io/docs/guide.html

Use 'pmdumptext -Xlimu' to dump system metrics averaged over an interval with fixed columns. The -X option enables fixed columns.

```bash
$ pmdumptext -Xlimu -t 10m -S @09:00 -T @10:00 -a acme.com/20140902 mem.util.used
```

--------------------------------

### Monitor Process Creation and Memory Usage with Timestamps

Source: https://pcp.io/docs/guide.html

Track live process creation rates and free/used memory with timestamps and output in GBs. Use -p for process metrics, -b for byte units, -t for interval, and -o for output format.

```bash
$ pmrep -p -b GB -t 2sec -o csv kernel.all.sysfork mem.util.free mem.util.used
```

--------------------------------

### Display Enabled Performance Metrics

Source: https://pcp.io/docs/guide.html

List all enabled performance metrics on a host along with a brief description. This is useful for discovering available metrics.

```bash
$ pminfo -t
```

--------------------------------

### Check PCP Configuration

Source: https://pcp.io/docs/guide.html

Use 'pcp -a' to check the PCP configuration at the time an archive was created.

```bash
$ pcp -a acme.com/20140902
```

--------------------------------

### Enable and Restart pmfind Service

Source: https://pcp.io/docs/guide.html

Enables the dynamic host discovery service (pmfind) and restarts it to begin monitoring discovered metric sources.

```bash
# systemctl enable pmfind
# systemctl restart pmfind

```

--------------------------------

### Display Enabled Performance Metrics

Source: https://pcp.io/docs/guide.html

Use 'pminfo -a' to display all enabled performance metrics at the time an archive was created.

```bash
$ pminfo -a acme.com/20140902
```

--------------------------------

### Import and Visualize iostat Data

Source: https://pcp.io/docs/guide.html

Import iostat data to a PCP archive using 'iostat2pcp' and visualize it with 'pmchart'. Ensure iostat data is first captured to a file.

```bash
$ iostat -t -x 2 > iostat.out
```

```bash
$ iostat2pcp iostat.out iostat.pcp
```

```bash
$ pmchart -t 2sec -a iostat.pcp
```

--------------------------------

### Monitor Multiple Live Metrics on a Remote Host

Source: https://pcp.io/docs/guide.html

Monitor live CPU load, memory usage, and disk write operations on a remote host at a set interval. Use -X for text output, -l for limits, -i for instance filtering, -m for metadata, and -u for unique instances.

```bash
$ pmdumptext -Xlimu -t 2sec 'kernel.all.load[1]' mem.util.used disk.partitions.write -h acme.com
```

--------------------------------

### Display Process Metrics from Archive

Source: https://pcp.io/docs/guide.html

Use `pminfo` with the `-a` flag to display process-related metrics from a specified archive.

```bash
$ pminfo proc -a acme.com/20140902
```

--------------------------------

### Monitor System Metrics in a Top-like Window

Source: https://pcp.io/docs/guide.html

View system metrics in an interactive, top-like interface. This provides a real-time overview of system activity.

```bash
$ pcp atop
$ pcp htop
```

--------------------------------

### Configure and Monitor Hot Processes

Source: https://pcp.io/docs/guide.html

Configure the `hotproc` namespace to monitor processes by name on the fly and display collected metrics.

```bash
# pmstore hotproc.control.config 'fname == "java"'
# pminfo -f hotproc
```

--------------------------------

### Fetch Metric Values with pminfo

Source: https://pcp.io/faq.html

Display the values for a specific metric, like 'filesys.free', along with their associated instances using the 'pminfo -f' command. This shows the current state of the metric across different instances.

```bash
$ pminfo -f filesys.free
filesys.free
  inst [0 or "/dev/root"] value 3498272
  inst [1 or "/dev/hda3"] value 20106
  inst [2 or "/dev/hda5"] value 7747420
  inst [3 or "/dev/hda2"] value 368432
```

--------------------------------

### Manually Rebuild PMNS

Source: https://pcp.io/faq.html

This is a brute-force method to rebuild the Performance Metrics Name Space (PMNS) by navigating to the PMNS directory and executing the Rebuild script. Use this if the standard startup method does not resolve an empty PMNS.

```bash
# cd $PCP_VAR_DIR/pmns
# ./Rebuild -du

```

--------------------------------

### Define Derived Metrics Configuration

Source: https://pcp.io/docs/guide.html

Creates a configuration file for defining custom derived performance metrics using arithmetic expressions. These metrics can then be used with PCP utilities.

```bash
$ cat ./pcp-derive-metrics.conf

disk.dev.avqsz = disk.dev.read_rawactive + disk.dev.write_rawactive disk.dev.avrqsz = 2 * rate(disk.dev.total_bytes) / rate(disk.dev.total) disk.dev.await = 1000 * (rate(disk.dev.read_rawactive) + rate(disk.dev.write_rawactive)) / rate(disk.dev.total)

```

--------------------------------

### Monitor Performance Metrics with a GUI

Source: https://pcp.io/docs/guide.html

Visualize performance metrics using a graphical user interface. This tool allows for selecting and viewing metrics in charts. Use -t for the default interval and -h for hosts.

```bash
$ pmchart -t 2sec -h acme1.com -h acme2.com
```

--------------------------------

### Monitor System Metrics in an iostat-like Fashion

Source: https://pcp.io/docs/guide.html

Observe system metrics in an interface similar to iostat, focusing on I/O statistics. Use -t for the interval.

```bash
$ pmiostat -t 2sec
```

--------------------------------

### Dump System Metrics in Iostat-like Fashion

Source: https://pcp.io/docs/guide.html

Use 'pmiostat' to dump system metrics in an iostat(1)-like fashion, averaged over a specified interval.

```bash
$ pmiostat -t 1h -a acme.com/20140902
```

--------------------------------

### Count total number of performance metrics

Source: https://pcp.io/papers/pcp-presentation/index.html

Uses pminfo and wc to count the total number of performance metrics available.

```bash
# pminfo | wc -l
```

--------------------------------

### Discover PCP pmcd Services

Source: https://pcp.io/docs/guide.html

Discovers PCP pmcd services on the local network using the pmfind command.

```bash
$ pmfind -s pmcd

```

--------------------------------

### Probing Network Interface Status Globally with pmprobe

Source: https://pcp.io/papers/pcp-presentation/index.html

Use pmprobe to list available network interfaces and their status.

```bash
# pmprobe -I network.interface.up
```

--------------------------------

### Enable pmlogger service

Source: https://pcp.io/papers/pcp-presentation/index.html

Enables the pmlogger service to collect performance data over time.

```bash
# systemctl enable pmlogger
```

--------------------------------

### Dump System Metrics in Sar-like Fashion

Source: https://pcp.io/docs/guide.html

Use 'pmstat' to dump system metrics in a sar(1)-like fashion, averaged over a specified interval and time range.

```bash
$ pmstat -t 10m -S @09:00 -T @10:00 -a acme.com/20140902
```

--------------------------------

### Fetch PDU Tracing Information

Source: https://pcp.io/faq.html

Use pminfo with the -D PDU flag to enable PDU tracing for kernel metrics. This helps in understanding the low-level communication details.

```bash
$ pminfo -D PDU kernel.all.cpu
```

```bash
$ pminfo -D PDU -fdT kernel.all.load
```

--------------------------------

### Display Detailed Metric Information and Values

Source: https://pcp.io/docs/guide.html

Show detailed information about a specific performance metric, including its current values. Use -f for format, -d for details, -m for metadata, and -T for values.

```bash
$ pminfo -dfmtT disk.partitions.read
```

--------------------------------

### Display Detailed Metric Information

Source: https://pcp.io/docs/guide.html

Use 'pminfo -df' to display detailed information about a specific performance metric within an archive.

```bash
$ pminfo -df mem.freemem -a acme.com/20140902
```

--------------------------------

### Calculate Average Performance Metrics

Source: https://pcp.io/docs/guide.html

Use 'pmlogsummary' to calculate average values of performance metrics within a specified time range, including min/max values and times. Table-like formatting is enabled with -HlfiImM.

```bash
$ pmlogsummary -HlfiImM -S @09:00 -T @10:00 acme.com/20140902 disk.partitions.write mem.freemem
```

--------------------------------

### Monitor Derived Metric with Hostname

Source: https://pcp.io/docs/guide.html

Monitors the 'disk.dev.avqsz' derived metric using pmval, specifying a target host 'acme.com'.

```bash
$ pmval -t 2sec -f 3 disk.dev.avqsz -h acme.com

```

--------------------------------

### Set a dynamic configuration value

Source: https://pcp.io/papers/pcp-presentation/index.html

Uses pmstore to dynamically set a configuration value for the 'hotproc' PMDA, specifying a Java executable name.

```bash
# pmstore hotproc.control.config 'fname = "java"'
```

--------------------------------

### Monitor System Metrics from Multiple Hosts

Source: https://pcp.io/docs/guide.html

Collect and display system metrics from multiple hosts simultaneously in a sar-like fashion. Use -t for interval and -h for host specification.

```bash
$ pmstat -t 2sec -h acme1.com -h acme2.com
```

--------------------------------

### Check Archive Details

Source: https://pcp.io/docs/guide.html

Use 'pmdumplog -L' to check the host, timezone, and time period covered by a PCP archive.

```bash
$ pmdumplog -L acme.com/20140902
```

--------------------------------

### Monitor Live Disk Write Operations

Source: https://pcp.io/docs/guide.html

Track live disk write operations per partition at a specified interval. Use -f for fixed-point notation and -t for the interval.

```bash
$ pmval -t 2sec -f 3 disk.partitions.write
```

--------------------------------

### Probing Network Interface Status in Containers with pmprobe

Source: https://pcp.io/papers/pcp-presentation/index.html

Use pmprobe with the --container option to check the status of network interfaces within a specific container.

```bash
# pmprobe -I --container sharp_feynman network.interface.up
```

--------------------------------

### Monitor Process Resource Usage

Source: https://pcp.io/docs/guide.html

Use `pmdumptext` to monitor CPU time, memory usage (RSS), and thread count for a specific process.

```bash
$ pmdumptext -Xlimu -t 2sec 'proc.psinfo.utime[1234]' 'proc.memory.rss[1234]' 'proc.psinfo.threads[1234]'
```

--------------------------------

### Set PCP Derived Config and Monitor Metric

Source: https://pcp.io/docs/guide.html

Sets the PCP_DERIVED_CONFIG environment variable to use the custom derived metrics file and then monitors the 'disk.dev.avqsz' metric using pmval.

```bash
$ PCP_DERIVED_CONFIG=./pcp-derive-metrics.conf; export PCP_DERIVED_CONFIG
$ pmval -t 2sec -f 3 disk.dev.avqsz

```

--------------------------------

### Define and Monitor Combined Memory Metric

Source: https://pcp.io/docs/guide.html

Defines a new derived metric 'mem.util.allcache' on the command line by combining existing memory metrics and then monitors it along with other memory metrics using pmrep.

```bash
$ pmrep -t 2sec -p -b MB -e "mem.util.allcache = mem.util.bufmem + mem.util.cached + mem.util.slab" mem.util.free mem.util.allcache mem.util.used

```

--------------------------------

### Replay vmstat-like Metrics

Source: https://pcp.io/docs/guide.html

Use 'pmrep' to replay vmstat-like metrics from an archive. Metrics can be defined in pmrep.conf. -A specifies the archive interval, -t the reporting interval, and -Z the timezone.

```bash
$ pmrep -a acme.com/20140902 -A 5min -t 5min -Z UTC :vmstat
```

--------------------------------

### Summarize Performance Metric Differences

Source: https://pcp.io/docs/guide.html

Use 'pmdiff' to summarize differences between performance metrics in two archives over specified time periods. Grep for '+' to find values that were zero in the first period.

```bash
$ pmdiff -S @02:00 -T @03:00 -B @09:00 -E @10:00 acme.com/20140902 acme.com/20140901
```

--------------------------------

### Configure PMIE for Remote Host Monitoring

Source: https://pcp.io/docs/guide.html

Add a remote host to the PMIE control configuration to enable monitoring of metrics from that collector host.

```bash
# echo acme.com n PCP_LOG_DIR/pmie/acme.com -c config.acme.com >> /etc/pcp/pmie/control
# systemctl restart pmie
```

--------------------------------

### Import and Visualize sar Data

Source: https://pcp.io/docs/guide.html

Import sar data to a PCP archive using 'sar2pcp' and visualize it with 'pmchart'. Sar logs are typically found under /var/log/sysstat on Debian/Ubuntu.

```bash
$ sar2pcp /var/log/sa/sa15 sar.pcp
```

```bash
$ pmchart -t 2sec -a sar.pcp
```

--------------------------------

### Monitor Network Interface Metrics

Source: https://pcp.io/docs/guide.html

Use `pmrep` to monitor all outgoing network metrics for a specified interface.

```bash
$ pmrep -i wlan0 -v network.interface.out
```

--------------------------------

### Monitor System Metrics in a Sar-like Manner

Source: https://pcp.io/docs/guide.html

Analyze system metrics using a System Activity Report (SAR) style interface. This is useful for historical performance analysis.

```bash
$ pcp atopsar
```

--------------------------------

### Monitor Derived Metric from Archive

Source: https://pcp.io/docs/guide.html

Monitors the 'disk.dev.await' derived metric using pmval, reading data from a specific archive file 'acme.com/20140902'.

```bash
$ pmval -t 2sec -f 3 disk.dev.await -a acme.com/20140902

```

--------------------------------

### Check PCP Services and Archives

Source: https://pcp.io/docs/guide.html

Perform a basic health check for PCP services on a remote host or analyze historical data from a local archive.

```bash
$ pcp -h munch
$ pcp -a /var/log/pcp/pmlogger/smash/20190909
```

--------------------------------

### Configure Remote Archive Collection

Source: https://pcp.io/docs/guide.html

Enables centralized archive collection on the monitoring host by adding a collector host to the pmlogger control file and restarting the service.

```bash
# echo acme.com n n PCP_LOG_DIR/pmlogger/acme.com -r -T24h10m -c config.acme.com >> /etc/pcp/pmlogger/control
# systemctl restart pmlogger

```