Issue related to the External-IP and loadBalancer in a manged Kubernetes deployment

[drvgautam]

Dear Alexandre,

I was attempting to deploy the ABCDesktop (kubernetes-version-3.0) for testing my setup with a few of my own desktop applications. Everything worked fine when tested locally in my Ubuntu (22.04) machine. I then thought to deploy the setup in a DigitalOcean Kubernetes (DOK) cluster (I currently have a 3 nodes cluster). Because the DOK is a managed Kubernetes, so it was straight forward to use the context and deploy using the install.sh script (https://raw.githubusercontent.com/abcdesktopio/conf/main/kubernetes/install.sh), just for the testing purpose. To access the deployed ABCDesktop from an external IP, I changed the NGINX service type from nodePort to loadBalancer in the abcdesktop.yaml, as below. Saved and applied the chnages using 'kubectl apply -f abcdesktop.yaml'. I can now access the abcdesktop from 'http://<EXTERNAL-IP>', but I am unable to sign-in using login usernames and passwords from the OLDAP test samples. 'Sign-in Anonymously also fails. I get the error message:'undefined'. Any suggestions as to what I may be doing wrong here would be very much appreciated. Following are some of the details.

changes in the nginx service block within the abcdesktop.yaml

kind: Service
apiVersion: v1
metadata:
  name: nginx
  namespace: abcdesktop
spec:
  type: LoadBalancer
  selector:
    run: nginx-od
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30443
    name: http

some additional information about the status of services and nodes in my deployment.

$kubectl get service -w

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xx.yyy.w.z   <none>        443/TCP   50d

$kubectl get nodes -o=wide

 NAME                             STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP       OS-IMAGE                         KERNEL-VERSION           CONTAINER-RUNTIME
        mkp-kubernetes-node-pool-q2r66   Ready    <none>   50d   v1.25.4   xx.yyy.www.zzz   lll.mmm.nnn.ppp   Debian GNU/Linux 11 (bullseye)   5.18.0-0.deb11.4-amd64   containerd://1.4.13
        mkp-kubernetes-node-pool-q2r6l   Ready    <none>   50d   v1.25.4   sss.ppp.uuu.vvv   iii.jjj.kkk.lll    Debian GNU/Linux 11 (bullseye)   5.18.0-0.deb11.4-amd64   containerd://1.4.13
        mkp-kubernetes-node-pool-q2r6t   Ready    <none>   50d   v1.25.4   aaa.bbb.ccc.ddd   uuu.vvv.www.kkk    Debian GNU/Linux 11 (bullseye)   5.18.0-0.deb11.4-amd64   containerd://1.4.13

$kubectl get services -n abcdesktop

NAME        TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)           AGE
desktop     ClusterIP      None             <none>            <none>            9h
memcached   ClusterIP      xx.yyy.www.zz    <none>            11211/TCP         9h
mongodb     ClusterIP      aa.bbb.cc.dd     <none>            27017/TCP         9h
nginx       LoadBalancer   ll.jjj.kkk.mmm   <EXTERAL-IP>      80:30443/TCP      9h
openldap    ClusterIP      uu.vvv.www.zzz   <none>            389/TCP,636/TCP   9h
pyos        ClusterIP      ss.ooo.pp.kk    <none>             8000/TCP          9h
speedtest   ClusterIP      ii.jjj.kkk.ll    <none>            80/TCP            9h

[drvgautam]

Just to add a bit more information, I tried to look into the log of my openldap pod in the real time. Following is the log, which seems okay and the user 'Phillip J. Fry' succeeded in signing in successfully. However, in reality, I ended up getting the same error 'undefined' in my abcdesktop browser window.

$kubectl logs -n abcdesktop openldap-od-747846c8dd-4rl72 -c openldap -f

6454d2b3 conn=1005 fd=18 ACCEPT from IP=10.244.0.10:42217 (IP=0.0.0.0:10389)
6454d2b3 conn=1005 op=0 BIND dn="cn=Philip J. Fry,ou=people,dc=planetexpress,dc=com" method=128
6454d2b3 conn=1005 op=0 BIND dn="cn=Philip J. Fry,ou=people,dc=planetexpress,dc=com" mech=SIMPLE ssf=0
6454d2b3 conn=1005 op=0 RESULT tag=97 err=0 text=
6454d2b3 conn=1005 op=1 SRCH base="" scope=0 deref=3 filter="(objectClass=*)"
6454d2b3 conn=1005 op=1 SRCH attr=subschemaSubentry +
6454d2b3 conn=1005 op=1 SEARCH RESULT tag=101 err=0 nentries=1 text=
6454d2b3 conn=1005 op=2 SRCH base="cn=Subschema" scope=0 deref=3 filter="(objectClass=subschema)"
6454d2b3 conn=1005 op=2 SRCH attr=objectClasses attributeTypes ldapSyntaxes matchingRules matchingRuleUse dITContentRules dITStructureRules nameForms createTimestamp modifyTimestamp * +
6454d2b3 conn=1005 op=2 SEARCH RESULT tag=101 err=0 nentries=1 text=
6454d2b3 conn=1005 op=3 SRCH base="ou=people,dc=planetexpress,dc=com" scope=2 deref=3 filter="(&(objectClass=inetOrgPerson)(cn=philip j. fry))"
6454d2b3 conn=1005 op=3 SRCH attr=cn distinguishedName
6454d2b3 conn=1005 op=3 SEARCH RESULT tag=101 err=0 nentries=1 text=
6454d2b3 conn=1005 op=4 SRCH base="ou=people,dc=planetexpress,dc=com" scope=2 deref=3 filter="(&(objectClass=inetOrgPerson)(cn=philip j. fry))"
6454d2b3 conn=1005 op=4 SRCH attr=cn sn description employeeType givenName jpegPhoto mail ou title uid distinguishedName displayName name
6454d2b3 conn=1005 op=4 SEARCH RESULT tag=101 err=0 nentries=1 text=
6454d2b3 conn=1005 op=5 SRCH base="ou=people,dc=planetexpress,dc=com" scope=2 deref=3 filter="(&(objectClass=inetOrgPerson)(cn=philip j. fry))"
6454d2b3 conn=1005 op=5 SRCH attr=memberOf
6454d2b3 conn=1005 op=5 SEARCH RESULT tag=101 err=0 nentries=1 text=
6454d2b3 conn=1005 op=6 UNBIND
6454d2b3 conn=1005 fd=18 closed

[alexandredevely]

Hello,

Thank you for using abcdesktop.io, i don't use the DigitalOcean Kubernetes cluster. It's the first time.
Your NGINX service change from nodePort to loadBalancer, it is correct and it should works.

Did you update the od.config file option ?

# set the external URL service if you use a reverse proxy
# default_host_url : 'https://external.domain.org'
default_host_url : 'http://localhost'

You may have top replace http://localhost this your loadBalancer ip address or your FQDN ( if you set one) .

Your ldap log is clear, thank you, you have to go in deep : can you read the pyos log file ?

Get the pod name of pyos

kubectl get pods -l name=daemonset-pyospods -n abcdesktop

You should read

NAME                   READY   STATUS    RESTARTS   AGE
daemonset-pyos-64x6n   1/1     Running   0          63m
daemonset-pyos-llz8r   1/1     Running   0          63m

Then read the log of a (or all) pyos pods

kubectl logs daemonset-pyos-64x6n -n abcdesktop

The pod create process should hang with an error.

Can you also check if your user pod is created ?

kubectl get pods -n abcdesktop

I use the Philip J. Fry login

NAME                            READY   STATUS    RESTARTS       AGE
daemonset-pyos-64x6n            1/1     Running   0              72m
daemonset-pyos-llz8r            1/1     Running   0              72m
fry-262f2                       4/4     Running   0              15s

kubectl describe pods -l access_userid=fry -n abcdesktop

Thank you for your feedback
Alexandre

[alexandredevely]

Hello Vinay,

I have installed a kubernetes cluster on digitalocean.com too.

Before using a type: LoadBalancer, I would like to confirme with you that a simple port-forward works.

% kubectl get pods -n abcdesktop
NAME                            READY   STATUS    RESTARTS   AGE
daemonset-nginx-4bvhv           1/1     Running   0          26m
daemonset-nginx-mz2pf           1/1     Running   0          26m
daemonset-nginx-v9762           1/1     Running   0          26m
daemonset-pyos-9ttbb            1/1     Running   0          26m
daemonset-pyos-lcmgt            1/1     Running   0          26m
daemonset-pyos-mph5p            1/1     Running   0          26m
memcached-od-5b58d595fb-dk6d7   1/1     Running   0          26m
mongodb-od-6496b46dbd-lrx9j     1/1     Running   0          26m
openldap-od-67db7ddd8d-j2h64    1/1     Running   0          26m
speedtest-od-6d9c7675f-xfwgg    1/1     Running   0          26m

Choose one of nginx front, and run the port forward command line

For example

% kubectl port-forward daemonset-nginx-v9762 --address 0.0.0.0  80:80 -n abcdesktop
Forwarding from 0.0.0.0:80 -> 80

Then open your web browser, you should get the home page, login using LDAP auth or Anonymous auth should works.

Then login, and you get a pod user.

For the first time, you may get a time out error, if all container image can not be downloaded in less than 180 seconds on the worker node.

% kubectl get pods -n abcdesktop                                                   
NAME                            READY   STATUS    RESTARTS   AGE
daemonset-nginx-4bvhv           1/1     Running   0          40m
daemonset-nginx-mz2pf           1/1     Running   0          40m
daemonset-nginx-v9762           1/1     Running   0          40m
daemonset-pyos-9ttbb            1/1     Running   0          40m
daemonset-pyos-lcmgt            1/1     Running   0          40m
daemonset-pyos-mph5p            1/1     Running   0          40m
fry-2cf3a                       4/4     Running   0          4m15s
memcached-od-5b58d595fb-dk6d7   1/1     Running   0          40m
mongodb-od-6496b46dbd-lrx9j     1/1     Running   0          40m
openldap-od-67db7ddd8d-j2h64    1/1     Running   0          40m
speedtest-od-6d9c7675f-xfwgg    1/1     Running   0          40m

The user fry gets the abcdesktop pod fry-2cf3a

See you,

Alexandre

[alexandredevely]

Hello Vinay,

To use a LoadBalancer in digitalocean.

Could you please update the nginx service section in abcdesktop.yaml file

---
kind: Service
apiVersion: v1
metadata:
  name: nginx
  namespace: abcdesktop
spec:
  type: LoadBalancer
  selector:
    run: nginx-od 
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    name: http
---

After the a kubectl apply -f abcdesktop.yaml command , wait for the pending state

% kubectl get services -n abcdesktop
NAME        TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
desktop     ClusterIP      None             <none>        <none>            57m
memcached   ClusterIP      10.245.247.50    <none>        11211/TCP         57m
mongodb     ClusterIP      10.245.198.243   <none>        27017/TCP         57m
nginx       LoadBalancer   10.245.172.53    <pending>     80:30443/TCP      57m
openldap    ClusterIP      10.245.109.131   <none>        389/TCP,636/TCP   57m
pyos        ClusterIP      10.245.94.15     <none>        8000/TCP          57m
speedtest   ClusterIP      10.245.67.168    <none>        80/TCP            57m

You get the ip address for your LoadBalancer

% kubectl get services -n abcdesktop
NAME        TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)           AGE
desktop     ClusterIP      None             <none>         <none>            61m
memcached   ClusterIP      10.245.247.50    <none>         11211/TCP         61m
mongodb     ClusterIP      10.245.198.243   <none>         27017/TCP         61m
nginx       LoadBalancer   10.245.172.53    161.35.246.4   80:30443/TCP      61m
openldap    ClusterIP      10.245.109.131   <none>         389/TCP,636/TCP   61m
pyos        ClusterIP      10.245.94.15     <none>         8000/TCP          61m
speedtest   ClusterIP      10.245.67.168    <none>         80/TCP            61m

Then open your web browser to reach the abcdesktop service.
In case, the loadbalancing service give me the ip address 161.35.246.4

Login using Philip J. Fry

And you should get the fry desktop

See you,

Alexandre

[drvgautam]

Dear Alexandre,

Thank you very much for the help in figuring out the issue. I confirm that the DigitalOcean deployment of the abcdesktop works now perfectly fine as you illustrated in the above when I used the version-3.0(https://github.com/abcdesktopio/conf/blob/main/kubernetes/abcdesktop-3.0.yaml). I was somehow using an older version from (https://raw.githubusercontent.com/abcdesktopio/conf/main/kubernetes/abcdesktop.yaml). From the older version, I had two of the pods (as shown below in the Status column) failing to start. I have no moved on to the version-3, so I do not have those old logs from the pyos and mongodb pods to share here, but I can surely go back and try to reproduce those if you need me to report for debugging purpose. Otherwise, I am happy with the successful deployment using the version 3.0! To import applications in the loadBalancer mode, I use the loadBalancer without the nodePort (curl -X PUT -H 'Content-Type: text/javascript' http://<EXTERNAL-IP>/API/manager/image -d @abcsample.json ) and then everything works fine.

kubectl get pods -n abcdesktop

NAME                            READY   STATUS             RESTARTS          AGE
daemonset-nginx-l6gxq           1/1     Running            0                 13h
daemonset-nginx-sqgrs           1/1     Running            0                 13h
daemonset-nginx-xmn86           1/1     Running            0                 13h
daemonset-pyos-mvmxj            0/1     CrashLoopBackOff   161 (44s ago)     13h
daemonset-pyos-vbhzz            0/1     CrashLoopBackOff   0                 22h
daemonset-pyos-vv7qq            0/1     CrashLoopBackOff   0                 22h
memcached-od-bdcbbcb74-7rkl4    1/1     Running            0                 22h
mongodb-od-5f57999d58-544z7     0/1     CrashLoopBackOff   157 (4m26s ago)   13h
openldap-od-747846c8dd-4rl72    1/1     Running            0                 13h
speedtest-od-6b5f8584f5-qvzfm   1/1     Running            0                 13h

The next I am setting up a MinIO storage facility and aiming to map the ABCDesktop user's home directory to enable a dedicated (persistent?) data storage to make things meaningful with respect to my application that needs a data persistence and exchange. While looking into the abcdesktop installation information, I found some information regarding the persistentVolume . Does the 'pyos' version 3 support such thing? Your insightful suggestion on this side would be very much appreciated.

thanks a lot.
regards,
Vinay

[alexandredevely]

Hello Vinay,

Thank you for your message, and for this feedback.
You have updated the application pull url path, to use the new loadbalancer address, this is great thank you.

'pyos' version 3 only supports HostPath ( for a kubernetes limitation, subpath is not supported in ephemeral container, and ephemeral container need to know the user home directory path )
Note, you have data persistence with host path, the default design is based on NFS mount point.

But i like your design with Minio to store user volume.
So i will try to add a persistentVolumeClaim for each user, and them bypass the subpath usage for ephemeral container.
If it works and does not take so log time to create a persistentVolumeClaim mapped to a persistentVolume after the login process, we should support Minio and all S3, NFS storage backend ...

Thank you again for your comments and for this new design request with S3/Minio

Alexandre

[drvgautam]

Dear Alexandre,

Thanks again for considering to look into the minIO integration to enable persistentVolumeClaim. If it works, it will be very helpful feature. I will wait for this feature.
I am facing one more issue related to enabling external OAuth provider in my ABCDesktop kubernetes deployment. Based on the documentation, I added my external authentication providers n the authmanager dictionary within the od.config. I delested, created and restarted the configmap, but I did not get the sign-in buttons of my added authenticators activated. I was hoping to get the buttons, similar to for example what demo.abcdesktop.io shows. Am I missing something here to get the external auth providers activated ?

After I make the changes in the od.conf, I delete and restart the configmap using the following three commands,
$kubectl delete configmap abcdesktop-config -n abcdesktop
$kubectl create configmap abcdesktop-config --from-file=od.config -n abcdesktop
$kubectl rollout restart daemonset daemonset-pyos -n abcdesktop

my authmanager dictionary in the od.config looks like as below

#I have set the ldpconf-enabled = False and the external providers enabled
authmanagers: {
  'external': {
    'providers': {
       'materials-marketplace': { 
        'displayname': 'abcdesktop-materials-marketplace', 
        'enabled': True,
        'client_id':     '<MYCLIENT_ID>', 
        'client_secret': '<MYCLIENT_SECRET>', 
        'dialog_url': 'https://github.com/login/oauth/authorize?client_id={client_id}&redirect_uri={callback_url}&scope={scope}&state={state}',
        'auth_url': 'https://github.com/login/oauth/access_token?client_id={client_id}&client_secret={client_secret}&code={code}&redirect_uri={callback_url}&state={state}',
        'userinfo_url': 'https://api.github.com/user/emails',
        'callback_url': 'https://abcdesktop.materials-marketplace.eu/callback',
       },
       'github': {
    	'displayname': 'Github',
    	'enabled': True,
    	'client_id': 'CLIENT_ID_FROM_GITHUB_APP_REG',
    	'client_secret': 'CLIENT_SECRET_FROM_GITHUB_APP_REG',
    	'redirect_uri_prefix': 'http://<EXTERNAL-IP>/hub/oauth_callback',
    	'redirect_uri_querystring': 'http://<EXTERNAL-IP>/hub/oauth_callback?provider=github',
    	'authorization_base_url': 'https://github.com/login/oauth/authorize',
    	'token_url': 'https://github.com/login/oauth/access_token',
    	'userinfo_url': 'https://api.github.com/user',
    	'userinfo_method': 'GET',
    	'userinfomap': {
        	'username': 'login',
        	'name': 'name',
        	'email': 'email'
    	},
    	'scope': 'read:user'
    }
      }
     },
  'metaexplicit': {}, 
  'explicit': {
    'show_domains': True,
    'providers'   : { 
	'planet': { 
		'config_ref': 'ldapconfig',  
		'enabled': False } 
	}
  } 
}

# Note serviceaccount is optional
ldapconfig : {
        'planet': {
                'default'       : True,
                'ldap_timeout'  : 15,
                'ldap_protocol' : 'ldap',
                'ldap_basedn'   : 'ou=people,dc=planetexpress,dc=com',
                'servers'       : [ 'openldap.abcdesktop.svc.cluster.local' ],
                'secure'        : False,
                'serviceaccount': { 'login': 'cn=admin,dc=planetexpress,dc=com', 'password': 'GoodNewsEveryone' },
                'policies': {
                        'acls': None,
                        'rules' : {
				'rule-dummy' : {
				 	'conditions' : [ {'boolean':True, 'expected':True } ],
                                        'expected' : True,
                                        'label':'labeltrue'
				},
                                'rule-ship': {
                                        'conditions' : [ { 'memberOf': 'cn=ship_crew,ou=people,dc=planetexpress,dc=com',   'expected' : True  } ],
                                        'expected' : True,
                                        'label':'shipcrew'
                                },
                                'rule-test': {
                                        'conditions' : [ { 'memberOf': 'cn=admin_staff,ou=people,dc=planetexpress,dc=com', 'expected' : True  } ],
                                        'expected' : True,
                                        'label': 'adminstaff'
                                }
                        }
                } } }

[alexandredevely]

Hello,

Some entries are missing in documentation ( I have to fix it, thank you )

Here is a provider dict with the materials-marketplace connect button.

abcweb.domain.com is an abcdesktop web site
the 'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth'

'providers': {
      'materials-marketplace': { 
        'displayname': 'abcdesktop-materials-marketplace', 
        'enabled': True,
        'client_id': 'xxx', 
        'client_secret': 'xxx',
        'userinfo_auth': True,
        'scope': [ 'https://www.googleapis.com/auth/userinfo.email',  'openid' ],
        'userinfo_url': 'https://www.googleapis.com/oauth2/v1/userinfo',
        'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth',
        'redirect_uri_querystring': 'manager=external&provider=google',
        'authorization_base_url': 'https://accounts.google.com/o/oauth2/v2/auth',
        'token_url': 'https://oauth2.googleapis.com/token',
        'policies': { 'acl'  : { 'permit': [ 'all' ] } }
      },
      'google': { 
        'displayname': 'Google', 
        'enabled': True,
        'client_id': 'xxx', 
        'client_secret': 'xxx',
        'userinfo_auth': True,
        'scope': [ 'https://www.googleapis.com/auth/userinfo.email', 'openid' ],
        'userinfo_url': 'https://www.googleapis.com/oauth2/v1/userinfo',
        'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth',
        'redirect_uri_querystring': 'manager=external&provider=google',
        'authorization_base_url': 'https://accounts.google.com/o/oauth2/v2/auth',
        'token_url': 'https://oauth2.googleapis.com/token',
        'policies': { 'acl'  : { 'permit': [ 'all' ] } }
      },
      'orangeconnect': {
        'displayname': 'DIOD Orange Connect',
        'enabled': True,
        'basic_auth': True,
        'scope': [ 'openid', 'profile', 'email' ],
        'userinfo_auth': True,
        'client_id': 'xxx',
        'client_secret': 'xxx',
        'userinfo_url': 'https://auth.tech.orange/oauth2/userinfo',
        'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth',
        'redirect_uri_querystring': 'manager=external&provider=orangeconnect',
        'authorization_base_url': 'https://auth.tech.orange/oauth2/authorize',
        'token_url': 'https://auth.tech.orange/oauth2/token',
	      'explicitproviderapproval': 'AD',
        'policies': { 'acl'  : { 'permit': [ 'all' ] } }
      },
      'orange': {       
        'displayname': 'Orange', 
        'enabled': True,
        'basic_auth': True,
        'userinfo_auth': True,
        'scope' : [ 'openid', 'form_filling' ],
        'client_id': 'xxx',
        'client_secret': 'xxx',
        'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth',
        'redirect_uri_querystring': 'manager=external&provider=orange',
        'authorization_base_url': 'https://api.orange.com/openidconnect/fr/v1/authorize',
        'token_url': 'https://api.orange.com/openidconnect/fr/v1/token', 
        'userinfo_url': 'https://api.orange.com/formfilling/fr/v1/userinfo',
        'policies': { 'acl'  : { 'permit': [ 'all' ] } }
      },
      'facebook': { 
        'displayname': 'Facebook', 
        'enabled': True,
        'userinfo_auth': True,
        'client_id': 'xxx', 
        'client_secret': 'xxx', 
        'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth',
        'redirect_uri_querystring': 'manager=external&provider=facebook',
        'authorization_base_url': 'https://www.facebook.com/dialog/oauth',
        'userinfo_url': 'https://graph.facebook.com/v2.6/me?fields=picture.width(400),name',
        'token_url': 'https://graph.facebook.com/v2.3/oauth/access_token',
        'userinfomap': {
            '*': '*',
            'picture': 'picture.data.url'
        },
        'policies': { 'acl'  : { 'permit': [ 'all' ] } }
      },
      'github': {
        'displayname': 'Github',
        'enabled': True,
        'basic_auth': True,
        'userinfo_auth': True,
        'scope' : [ 'read:user' ], 
        'client_id': 'xxx',
        'client_secret': 'xxx',
        'redirect_uri_prefix' : 'https://abcweb.domain.com/API/auth/oauth',
        'redirect_uri_querystring': 'manager=external&provider=github',
        'authorization_base_url': 'https://github.com/login/oauth/authorize',
        'token_url': 'https://github.com/login/oauth/access_token',
        'userinfo_url': 'https://api.github.com/user',
        'policies': { 'acl'  : { 'permit': [ 'all' ] } }
      } 
    }

Thank you

Regards,
Alexandre

[drvgautam]

Great, thank you very much.

regards,
Vinay

[alexandredevely]

Hello Vinay,

Thank you for your message. The last abcdesktopio/oc.pyos:3.0 image has now a persistentVolumeClaim support for pod and for ephemeralcontainer without subPath option.
You can define persistentVolumeClaim using hostPath or using csidriver via storageClass parameter.

Using hostPath

In your od.config file, define the new entries desktop.homedirectorytype desktop.persistentvolumespec desktop.persistentvolumeclaimspec

• desktop.homedirectorytype: 'persistentVolumeClaim'
• desktop.persistentvolumespec: create a new volume for the user's homeDir, for persistentVolume hostPath.
• desktop.persistentvolumeclaimspec: create a new volume claim for the user's homeDir

# set to persistentVolumeClaim
desktop.homedirectorytype: 'persistentVolumeClaim'

desktop.persistentvolumespec: {
            'storageClassName': '',
            'capacity': { 'storage': '1Gi' },
            'accessModes': [ 'ReadWriteMany' ], 
            'hostPath': { 'path': '/mnt/abcdesktop_volumes/{{ provider }}/{{ userid }}' } }

desktop.persistentvolumeclaimspec: {
            'storageClassName': '',
            'resources': { 
              'requests': { 
                'storage': '1Gi'
              } 
            },
            'accessModes': [ 'ReadWriteMany' ] }

desktop.persistentvolumespec support template values. For example '/mnt/abcdesktop_volumes/{{ provider }}/{{ userid }}'.

• {{ provider }} is the provider's name templated value.
• {{ userid }} is the user's id templated value.

The user's home directory inside the pod is located on host to /mnt/abcdesktop_volumes/{{ provider }}/{{ userid }}. The directory is created automatically by kubernetes.

The /mnt/abcdesktop_volumes/ content lists the provider name.

$ ls -la /mnt/abcdesktop_volumes/
total 20
drwxr-xr-x   5 root root 4096 mai   12 12:40 .
drwxr-xr-x 106 root root 4096 mai   11 11:34 ..
drwxr-xr-x   3 root root 4096 mai   12 12:40 anonymous
drwxr-xr-x   3 root root 4096 mai   12 12:39 github
drwxr-xr-x   5 root root 4096 mai   12 12:40 google

For provider google, all users are listed.

$ ls -la /mnt/abcdesktop_volumes/google/
total 20
drwxr-xr-x  5 root root 4096 mai   12 12:40 .
drwxr-xr-x  5 root root 4096 mai   12 12:40 ..
drwxr-x--- 16 2048 2048 4096 mai   12 12:39 103464335761332102620
drwxr-x--- 16 2048 2048 4096 mai   12 12:40 112026272437223559761
drwxr-x--- 16 2048 2048 4096 mai   12 12:39 114102844260599245242

For provider google, the user home directory for 103464335761332102620

$ ls -la /mnt/abcdesktop_volumes/google/103464335761332102620/
total 76
drwxr-x--- 16 2048 2048 4096 mai   12 12:39 .
drwxr-xr-x  5 root root 4096 mai   12 12:40 ..
-rw-------  1 2048 2048   71 mai   12 12:39 .Xauthority
-rw-rw-r--  1 2048 2048   12 janv. 27 18:36 .Xresources
drwxr-x---  3 2048 2048 4096 mai   12 12:39 .cache
drwxr-x---  6 2048 2048 4096 mai   12 12:39 .config
drwxrwxr-x  3 2048 2048 4096 janv. 27 18:36 .gconf
-rw-r-----  1 2048 2048    0 mai   12 12:39 .gtk-bookmarks
-rw-rw-r--  1 2048 2048  564 janv. 27 18:36 .gtkrc-2.0
drwxr-x---  3 2048 2048 4096 mai   12 12:39 .local
drwxr-x---  2 2048 2048 4096 mai   12 12:39 .store
drwxr-x---  2 2048 2048 4096 mai   12 12:39 .wallpapers
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Desktop
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Documents
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Downloads
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Music
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Pictures
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Public
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Templates
drwxr-x---  2 2048 2048 4096 mai   12 12:39 Videos

The template values can be one of them :

var	description
cn	Common Name
uid	user id
gid	group id
uidNumber	user id number
gidNumber	group id number
homeDirectory	homeDirectory
loginShell	loginShell
description	description
groups	groups
gecos	gecos
provider	provider
protocol	protocol
providertype	providertype
name	user name
userid	user id
locale	user's locale
template tag value	tag value set by auth rules

Note: hostPath supports file permissions and the pod's init commands chown or chmod can be used.

Using CSI for S3 k8s-csi-s3

I was looking for a csi S3 driver for kubernetes https://kubernetes-csi.github.io/docs/drivers.html but
I can't find an official AWS support for S3 as a CSI.

I've only found two fuse drivers :

• from ctrox here https://github.com/ctrox/csi-s3.git [ I can't make it work ]
• from Yandex-cloud here https://github.com/yandex-cloud/k8s-csi-s3

Please, let me know if you can find an official driver for S3 as a CSI.

Install a CSI for S3

I use the https://github.com/yandex-cloud/k8s-csi-s3 as a CSI for S3 with minio backend.

Follow https://github.com/yandex-cloud/k8s-csi-s3 setup guide and test with the sample pod to make sure that fuse mount the S3 file system.

Update storageclass.yaml

Add bucket: abcdesktop to use an existing bucket

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-s3
provisioner: ru.yandex.s3.csi
parameters:
  mounter: geesefs 
  # you can set mount options here, for example limit memory cache size (recommended)
  options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666"
  # to use an existing bucket, specify it here:
  bucket: abcdesktop
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system

Update od.config

In your od.config file, define the entry desktop.persistentvolumeclaimspec

• desktop.homedirectorytype: 'persistentVolumeClaim' to use the persistentVolumeClaim features.
• desktop.persistentvolumespec: None to skip the persistent volume provisioning.
• desktop.persistentvolumeclaimspec create a new volume claim for the user's homeDir, the storageClassName csi-s3

# set to persistentVolumeClaim
desktop.homedirectorytype: 'persistentVolumeClaim'
desktop.persistentvolumespec: None
desktop.persistentvolumeclaimspec: {
            'storageClassName': 'csi-s3',
            'resources': { 
              'requests': { 
                'storage': '1Gi'
              } 
            },
            'accessModes': [ 'ReadWriteOnce' ] }

The mounter options

By default the storageclass use mounter: geesefs. geesefs does not store file permissions and the init commands chown or chmod exit with no zero value, then the pod does not start.
All files belongs to root, but with correct permissions options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666".

Update the 'init' in desktop.pod dict

'init': { 
    'image': 'busybox',
    'enable': True,
    'pullpolicy':  'IfNotPresent',
    'securityContext': {
        'runAsUser':   0,
        'runAsGroup':  0 
    },
    'acl':  { 'permit': [ 'all' ] },
    'command':  [ 'sh', '-c',  'chown {{ uidNumber }}:{{ gidNumber }} ~ || true && chmod 750 ~ || true' ] 
  },

I can't make it works with another mounter like mounter: s3fs, but it should work too and support the file permissions.

kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                             STORAGECLASS               REASON   AGE
pvc-81a65ed9-b98e-462c-86c6-36c89c3d4f1b   1Gi        RWO            Delete           Bound    abcdesktop/github-12896316-96cb5                  csi-s3                              2m46s

kubectl get pvc -n abcdesktop
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
github-12896316-96cb5                  Bound    pvc-81a65ed9-b98e-462c-86c6-36c89c3d4f1b   1Gi        RWO            csi-s3                     2m21s

On the minio console you get the objects

Feel free to give me your feedback. I hope that k8s-csi-s3 is stable enough.

Alexandre

[alexandredevely]

Hi Vinay,

The is an issue on k8s-csi-s3 yandex-cloud/k8s-csi-s3#45

We need to add --setuid 0 to the options, to fix the umount command.
This StorageClass yaml file works fine on digitalocean cloud service.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-s3
provisioner: ru.yandex.s3.csi
parameters:
  mounter: geesefs
  # you can set mount options here, for example limit memory cache size (recommended)
  options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666 --setuid 0"
  # to use an existing bucket, specify it here:
  bucket: abcdesktop
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system

[alexandredevely]

Hi Vinay,

Digitalocean provides a storageClassName: do-block-storage for the kubernetes cluster.
It works fine, with good performance, better than hosting a S3 gateway.

I share the od.config file, the

# set to persistentVolumeClaim
desktop.homedirectorytype: 'persistentVolumeClaim'
desktop.persistentvolumespec: None
desktop.persistentvolumeclaimspec: {
            'storageClassName': 'do-block-storage',
            'resources': {
              'requests': {
                'storage': '1Gi'
              } 
            },
            'accessModes': [ 'ReadWriteOnce' ] }

See you,

Alexandre

[drvgautam]

Dear Alexandre,

Thank you for adding persistentVolumeClaim feature in the abcdesktopio/oc.pyos:3. This is excellent and works flawlessly. I struggled with hosting a S3 gateway, but the do-block-storage works very nicely. Thank you very much for the help in setting up the abcdesktop on cloud cluster.

In the meantime, I also tried to enable ssl certificates using a rather cumbersome approach, which still has some issues, but I am debugging it. I may seek your advice if I do not succeed. I am using ingress-controller, cert-manager, nginx-ingress resource and in the end I use a networkpolicy to enable communication between the the two loadBalancers (ingress-controller and abcdesktop-nginx) pods to communicate to allow passing the security http01 challenge. This tool chain looks a bit complex and I am not sure if this is the right way to do this.

thanks again.
regards,
Vinay

[alexandredevely]

Dear Vinay,

I've just run a test with an external domain, get a SSL certificat with Letsencrypt. Digital ocean does it for you. The SSL is established to the digital ocean (HTTPS) LoadBalancer, then incoming request goes into nginx HTTP.
You need to add annotations to the nginx services like :

---
kind: Service
apiVersion: v1
metadata:
  name: nginx
  namespace: abcdesktop
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
    service.beta.kubernetes.io/do-loadbalancer-protocol: "https"
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: "xxxxxx-yyyy-zzzzz-aaaaa-bbbbbbbbbbb"
    service.beta.kubernetes.io/do-loadbalancer-disable-lets-encrypt-dns-records: "true"
spec:   
  type: LoadBalancer
  selector:
    run: nginx-od
  ports:  
  - protocol: TCP
    port: 80 
    targetPort: 80
    name: http 
  - protocol: TCP
    port: 443  
    targetPort: 80
    name: https
---

See you
Alexandre

[drvgautam]

Dear Alexandre,
Thanks for this suggestion and sorry for a delayed reply. It worked fine with the SSL certificate setup using this approach. My abcdesktop deployment is working fine now ( webpage: https://promo-abcloudtop.io). However, I have just a few of the applications running that I previously added (before adding SSL certificate and forwarding) using
curl -X PUT -H 'Content-Type: text/javascript' http://LOADBALANCER_IP/API/manager/image -d @APPLICATION_NAME.d.3.0.json.
Now when I try to add new applications using using the domain name in place of the IP
(curl -X PUT -H 'Content-Type: text/javascript' https://promo-abcloudtop.io/API/manager/image -d @APPLICATION_NAME.d.3.0.json),
I get the following error.

{"status": 403, "message": "ip address is denied"}.
It seems I can no more access the 'manager' API end point. I checked my firewalls in the Digital Ocean and it seems okay; my IP address should be allowed to access the endpoint. I was reading this documentation related to the controllers https://www.abcdesktop.io/3.0/config/controllers/, but could not figure out what's wrong.

regards,
Vinay