Running Postgres in OpenShift with persistence volume is failing with user permissions issues

[Wisw]

Container platform

OCP 4

Version

I have tried to deploy the Postgres with 15 and its failing to be able to run with volume mounts in OCP 4.7 (latest).

OS version of the container image

RHEL 8

Bugzilla, Jira

Description

I have tried the below quote blocks along with multiple other settings with various securityContext params for container & pods specs. None of them are being acceptable as they are being blocked by SecurityContextConstraints I presume default to OCP.

Also I have seen that the volumes in OCP is yet to fixed as per line 72 of Dockerfile in below link.

https://catalog.redhat.com/software/containers/rhel8/postgresql-15/63d29a05fd1c4f5552a305b3?image=678fe0ad976ab6d74f5e4074&architecture=arm64&container-tabs=dockerfile

The git issue seem not relevant as the issue mentioned in Dockerfile is already closed, so I'm lost. Also tried to use the other $HOME directory for the volume by building and then felt it to be pointless as fix-permissions script is resolving to postgres user which is running with root privilege.

Need assistance with this.

apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-pv
labels:
app: postgres
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: localblock-sc
volumeMode: Filesystem
hostPath:
path: /var/lib/pgsql/data
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-node-01

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-pvc
namespace: supervity
labels:
app: postgres
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
volumeName: postgres-pv
storageClassName: localblock-sc
volumeMode: Filesystem

apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: test
labels:
app: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
creationTimestamp: null
labels:
app: postgres
spec:
volumes:
- name: postgres-pvc
persistentVolumeClaim:
claimName: postgres-pvc
securityContext:
fsGroup: 26
containers:
- resources: {}
terminationMessagePath: /dev/termination-log
name: container
env:
- name: POSTGRESQL_USER
value: test
- name: POSTGRESQL_PASSWORD
value: verySecureDBtestpswd
- name: POSTGRESQL_DATABASE
value: supervity
ports:
- containerPort: 5432
protocol: TCP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: postgres-pvc
mountPath: /var/lib/pgsql/data
readOnly: false
terminationMessagePolicy: File
image: 'registry.redhat.io/rhel8/postgresql-15:1-90'
securityContext: {}
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node-01
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600

Reproducer

No response

[phracek]

Hi @Wisw, Thanks for reporting this issue. Can you please provide reproducing steps? What kind of cluster do you use?
Can you please use also some newer clusters like 4.14+?

Do you use some shared clusters?

What is the output in OpenShift POD?

[Wisw]

Nope its not a shared cluster, currently we are using latest built OCP 4.17.x variant trial cluster on premises.

The pod is crashing with error at initialization of DB stating that fix-permissions is failing with operation not permitted.

[phracek]

I will try to focus on it during the next three weeks. It blocks us as well. It looks like initContainer is not present in template files. Please be patient.

[phracek]

@Wisw May I have a last question? How do you deploy postgresql-container on cluster? What commands do you use? Do you use any templates / imagestreams / helm charts? Thanks

[Wisw]

@phracek For a testing scenario I tried to deploy it standalone mode via deployment, svc, pvc & pv (using a localblock/hostPath for the StorageClass). Already shared the template above with all of those.

I was also thinking of a problem that its being blocked by an SCC in OCP. Created a custom SCC and tried but it failed as the initialization script consumed by postgres user tries to own the storage folder which we define dynamically via supported env.

[phracek]

@phracek For a testing scenario I tried to deploy it standalone mode via deployment, svc, pvc & pv (using a localblock/hostPath for the StorageClass). Already shared the template above with all of those.

I was also thinking of a problem that its being blocked by an SCC in OCP. Created a custom SCC and tried but it failed as the initialization script consumed by postgres user tries to own the storage folder which we define dynamically via supported env.

@Wisw Sorry for bothering you again with the question, which deployment? Link please?

We currently use for testing:

• Deployment Ephemeral https://github.com/sclorg/postgresql-container/blob/master/examples/postgresql-ephemeral-template.json
• Deployment Persistent https://github.com/sclorg/postgresql-container/blob/master/examples/postgresql-persistent-template.json
• Or Helm Charts https://github.com/sclorg/helm-charts/tree/main/charts/redhat/redhat-postgresql-persistent/src/templates

And commands that you use for deploying. I need a reproducer.

[Wisw]

@phracek having lost all hope, we went ahead with trying to deploy postgres:15 via NFS backed PV as mentioned in https://people.redhat.com/aweiteka/docs/preview/20170510/install_config/persistent_storage/persistent_storage_nfs.html#install-config-persistent-storage-persistent-storage-nfs . That worked!

Earlier we are trying to deploy in a standlone manner using the image backed by Dockerfile : https://catalog.redhat.com/software/containers/rhel9/postgresql-15/63f763f779eb1214c4d6fcf6?gs=&q=postgre&container-tabs=dockerfile
Especially what worried in our case is the commented part between lines: 76 & 86.

Had it been a regular docker or docker-compose it would have not been an issue at all maybe. Ours was in OCP 4.7.x and SecurityContextConstraints although customized to get data persisted backed by PV using hostPath/localPath modes of StorageClasses. It didn't fit even with SCC hackery.

Had to procure NFS based/backed PV. That worked.

I wish there were an option to choose a specific userID & groupID for the Postgres in OCP scenario at least and then let that postgres user use them to gain ownership of storage data folders like it did as part of initialization script.

Perhaps we tried deploying this variant: Deployment Persistent https://github.com/sclorg/postgresql-container/blob/master/examples/postgresql-persistent-template.json

In our case we just tried with RedHat image and not use any example template maybe.

[phracek]

@Wisw Ok, thanks for response. But I guess, this issue is not related to Postgresql-container alone but using by your written Deployment. In case you will find any issue with https://github.com/sclorg/postgresql-container/blob/master/examples/postgresql-persistent-template.json feel free to file a PR that is going to fix it, otherwise we can close this issue.

Sorry we do not provide all combinations that can be used in OpenSHift 4.X. What do you thing?

[phracek]

Perhaps we tried deploying this variant: Deployment Persistent https://github.com/sclorg/postgresql-container/blob/master/examples/postgresql-persistent-template.json

In our case we just tried with RedHat image and not use any example template maybe.

@Wisw Please let me know if this works for you.

[phracek]

@Wisw One more thing that was raised in my mind. In our postgresql-container test suite we tested the templates as in our Internal OpenShift 4 cluster as in IT shared cluster.
In case you have a template that is feasible to test during our tests, feel free to send a PR with the template here: https://github.com/sclorg/postgresql-container/tree/master/examples and enhancethe test suite here https://github.com/sclorg/postgresql-container/tree/master/examples and the alone test here: https://github.com/sclorg/postgresql-container/blob/master/test/test-lib-postgresql.sh#L15