Matt's *Time Log*
Collecting file-integrity-operator memory profile data
The OpenShift file-integrity-operator can be made to collect Golang pprof data from the daemonSet pods that it spawns. So this hack is not for a production system, but useful for debugging. After checking out the source from the file-integrity-operator repo, run:
$ sed -i 's/\/\/\"--pprof=true\"\,/\"--pprof=true\"\,/' pkg/controller/fileintegrity/fileintegrity_controller.go
Make sure you are using Go 1.15, have podman, and are logged in to the cluster as admin/kube-admin. I use gvm to manage Go versions:
$ gvm use go1.15.4
Now using version go1.15.4
$ oc whoami
kube:admin
Build the containers locally and push them to your cluster, using the deploy-local make target:
$ make deploy-local
Creating 'openshift-file-integrity' namespace/project
namespace/openshift-file-integrity created
/home/user/.gvm/pkgsets/go1.15.4/global/bin/operator-sdk build quay.io/file-integrity-operator/file-integrity-operator:latest --image-builder podman
INFO[0002] Building OCI image quay.io/file-integrity-operator/file-integrity-operator:latest
STEP 1: FROM registry.access.redhat.com/ubi8/go-toolset AS builder
STEP 2: USER root
--> Using cache 0e28419c126b2a6d798d17542464c0cde179a2681c6025c9c133f4c6bc0060f9
--> 0e28419c126
...
Switch to the openshift-file-integrity namespace and deploy your FileIntegrity CR. For this example, I’ll use the default CR that is provided by the operator repo.
$ oc project openshift-file-integrity
Now using project "openshift-file-integrity" on server "...".
$ oc create -f deploy/crds/fileintegrity.openshift.io_v1alpha1_fileintegrity_cr.yaml
fileintegrity.fileintegrity.openshift.io/example-fileintegrity created
Confirm that the operator’s resources are available:
$ oc get all
NAME READY STATUS RESTARTS AGE
pod/aide-ds-example-fileintegrity-86l9r 1/1 Running 0 66s
pod/aide-ds-example-fileintegrity-k9rgk 1/1 Running 0 66s
pod/aide-ds-example-fileintegrity-n2qgf 1/1 Running 0 66s
pod/aide-ds-example-fileintegrity-pcwn6 1/1 Running 0 66s
pod/aide-ds-example-fileintegrity-sfg7w 1/1 Running 0 66s
pod/aide-ds-example-fileintegrity-vtk5v 1/1 Running 0 66s
pod/file-integrity-operator-7f7576f89b-mvf6v 1/1 Running 0 4m27s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/file-integrity-operator-metrics ClusterIP 172.30.119.252 <none> 8383/TCP,8686/TCP 3m47s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/aide-ds-example-fileintegrity 6 6 6 6 6 <none> 66s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/file-integrity-operator 1/1 1 1 11m
NAME DESIRED CURRENT READY AGE
replicaset.apps/file-integrity-operator-7f7576f89b 1 1 1 11m
NAME IMAGE REPOSITORY TAGS UPDATED
imagestream.image.openshift.io/file-integrity-operator image-registry.openshift-image-registry.svc:5000/openshift-file-integrity/file-integrity-operator latest 11 minutes ago
$ oc get fileintegrities -o json | jq '.items[].status'
{
"phase": "Active"
}
Now our scans have been running for some time, and we want to capture a snapshot of the memory usage from one of the daemonSet pods at a point in time. To do this, use the following script:
#!/bin/bash
set -xe
oc exec -it $1 -- go tool pprof \-top \-cum http://localhost:6060/debug/pprof/heap
oc cp openshift-file-integrity/$1:/root/pprof/pprof.file-integrity-operator.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz ./pprof-$(date +%s)-$1.pb.gz
oc exec -it $1 -- rm -f /root/pprof/pprof.file-integrity-operator.alloc_objects.alloc_space.inuse_objects.inuse_space*
The script takes a single argument, the pod name. Let’s pick one out to capture:
$ oc adm top pod
NAME CPU(cores) MEMORY(bytes)
aide-ds-example-fileintegrity-86l9r 0m 130Mi
aide-ds-example-fileintegrity-k9rgk 0m 155Mi
aide-ds-example-fileintegrity-n2qgf 0m 145Mi
aide-ds-example-fileintegrity-pcwn6 0m 130Mi
aide-ds-example-fileintegrity-sfg7w 0m 122Mi
aide-ds-example-fileintegrity-vtk5v 0m 129Mi
file-integrity-operator-7f7576f89b-mvf6v 0m 17Mi
$ getpprof.sh aide-ds-example-fileintegrity-86l9r
+ oc exec -it aide-ds-example-fileintegrity-86l9r -- go tool pprof -top -cum http://localhost:6060/debug/pprof/heap
Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap
Saved profile in /root/pprof/pprof.file-integrity-operator.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
File: file-integrity-operator
Type: inuse_space
Time: Apr 12, 2021 at 6:01pm (UTC)
Showing nodes accounting for 512.03kB, 100% of 512.03kB total
flat flat% sum% cum cum%
512.03kB 100% 100% 512.03kB 100% bytes.(*Buffer).String (inline)
0 0% 100% 512.03kB 100% encoding/json.(*decodeState).object
0 0% 100% 512.03kB 100% encoding/json.(*decodeState).unmarshal
0 0% 100% 512.03kB 100% encoding/json.(*decodeState).value
0 0% 100% 512.03kB 100% encoding/json.Unmarshal
0 0% 100% 512.03kB 100% github.com/PuerkitoBio/purell.NormalizeURL
0 0% 100% 512.03kB 100% github.com/PuerkitoBio/urlesc.Escape
0 0% 100% 512.03kB 100% github.com/go-openapi/jsonreference.(*Ref).parse
0 0% 100% 512.03kB 100% github.com/go-openapi/jsonreference.New (inline)
0 0% 100% 512.03kB 100% github.com/go-openapi/spec.(*Ref).fromMap
0 0% 100% 512.03kB 100% github.com/go-openapi/spec.(*Schema).UnmarshalJSON
0 0% 100% 512.03kB 100% github.com/go-openapi/spec.MustLoadSwagger20Schema (inline)
0 0% 100% 512.03kB 100% github.com/go-openapi/spec.Swagger20Schema
0 0% 100% 512.03kB 100% github.com/go-openapi/spec.init.4
++ date +%s
+ oc cp openshift-file-integrity/aide-ds-example-fileintegrity-86l9r:/root/pprof/pprof.file-integrity-operator.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz ./pprof-1618250487-aide-ds-example-fileintegrity-86l9r.pb.gz
tar: Removing leading `/' from member names
+ oc exec -it aide-ds-example-fileintegrity-86l9r -- rm -f '/root/pprof/pprof.file-integrity-operator.alloc_objects.alloc_space.inuse_objects.inuse_space*'
The resulting archive on your local system can then be analyzed using go tool pprof:
$ go tool pprof -http=:6061 ./pprof-1618250487-aide-ds-example-fileintegrity-86l9r.pb.gz
Serving web UI on http://localhost:6061
© 2021 Matt's *Time Log* ― Github