Running kubernetes distributed example not working

General Discussions
#1

Hello, I followed the kubernestes distributed example to run agola in minukube, all the pods are up and running:

NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
default       agola-configstore-6b695fd9ff-qptsn          1/1     Running   5          14m
default       agola-configstore-6b695fd9ff-rpl7h          1/1     Running   5          14m
default       agola-executor-576778ff84-qknbh             1/1     Running   0          14m
default       agola-executor-576778ff84-vptfx             1/1     Running   0          14m
default       agola-gateway-others-79cbd6cc5f-p86wq       1/1     Running   0          14m
default       agola-gateway-others-79cbd6cc5f-wjpb4       1/1     Running   0          14m
default       agola-gitserver-869f8bbfb8-cn2c9            1/1     Running   0          14m
default       agola-runservice-8f44c4d47-vp89f            1/1     Running   5          14m
default       agola-runservice-8f44c4d47-wlqs2            1/1     Running   5          14m
default       minio-6f44ccc59-qf45k                       1/1     Running   0          14m

However, when I access agola web expose URL to register, I got a blank page. Looking at the chrome development tools, I can see:

curl 'http://192.168.64.20:30002/api/v1alpha/remotesources' -H 'Connection: keep-alive' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36' -H 'Accept: */*' -H 'Referer: http://192.168.64.20:30002/register' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.9,vi;q=0.8' -H 'Cookie: lang=en-US; i_like_gitea=82076592b485c935; _csrf=6mbbW4svRvaXmHhSLY8ta_FXNPU6MTU4MTUwMjgyNDIyMjcwODA3Mg' --compressed --insecure

Looking at the gateway pods logs:

2020-02-12T11:01:15.209Z	ERROR	scheduler/scheduler.go:116	err: failed to get running runs:
    agola.io/agola/internal/services/scheduler.(*Scheduler).approve
        /agola/internal/services/scheduler/scheduler.go:133
  - Get http://agola-runservice:4000/api/v1alpha/runs?asc=&phase=running: dial tcp 10.97.93.3:4000: connect: connection refused

So I attach to the runservice pod, and check if port 4000 is opened, but it does not.

How can I fix it?

#2

@cuonglm Please provide your configuration and how you are accessing the web interface.

What is a blank page? Do you see the agola top bar? If so you should first create a remote source and then click on register to register an user.

Your browser is calling the agola gateway on http://192.168.64.20:30002. Is it right? Are you sure your browser can access that url?

This error at startup is normal since the runservice component could not yet be started when the scheduler component tries to access it so it’s not related.

1 Like
#3

@sgotti

What is a blank page? Do you see the agola top bar? If so you should first create a remote source and then click on register to register an user.

Thanks, it’s exactly what I see. I haven’t created remoteresource yet.

#4

Yeah, you should first create a remote source. You can take a look at https://agola.io/tryit .

#5

Hi @sgotti, I use this command to create remotesource:

./bin/agola --token "admintoken" --gateway-url http://192.168.64.20:30002 remotesource create \
--name gitea \
--type gitea \
--api-url 192.168.64.20 \
--auth-type oauth2 \
--clientid a5043718-c471-4399-a3b7-4fc93839e051 \
--secret vgsgyuJmQZoxIEO9EH2CJBsJX5qWXMCs2ujgzaPQg90= \
--skip-ssh-host-key-check

It returns an error:

2020-02-12T23:01:48.460+0700	INFO	cmd/remotesourcecreate.go:116	creating remotesource
2020-02-12T23:01:48.471+0700	FATAL	cmd/remotesourcecreate.go:34	err: failed to create remotesource: internal server error

The gateway log:

2020-02-12T16:01:48.527Z	INFO	action/remotesource.go:112	creating remotesource
2020-02-12T16:01:48.529Z	ERROR	api/remotesource.go:66	err: failed to create remotesource:
    agola.io/agola/internal/services/gateway/action.(*ActionHandler).CreateRemoteSource
        /agola/internal/services/gateway/action/remotesource.go:115
#6

@cuonglm You should first use the basic example provided https://agola.io/tryit since I’m quite sure you have something misconfigured in your k8s distributed config.

So you should at first make it work with the try it demo and then, when it works, please provide the whole config and component logs.

#7

The example provided https://agola.io/tryit can not work in Mac, because it assumes the docker bridge gateway is 172.17.0.1, in my case, it’s 192.168.64.1

I actually did follow the example, created a remoteresource successfully, but then I still stuck because going to 127.0.0.1:8000 just failed, because the page call to 172.17.0.1:8000 for api.

you have something misconfigured in your k8s distributed config.

Can you point me out where? Since when all the pods are up and running, I assume it should work.

#8

The problem is the agola-configstore is not listening on port 4002:

root@agola-gateway-others-7866556766-2bhdz:/# telnet agola-configstore 4002
Trying 10.98.22.112...
telnet: Unable to connect to remote host: Connection refused

kubectl exec -it agola-configstore-659887b86d-sgtfn bash                                                                                                                                                                     ──(Wed,Feb12)─┘
root@agola-configstore-659887b86d-sgtfn:/# ss -tnlp
Netid                           State                           Recv-Q                           Send-Q                                                     Local Address:Port                                                       Peer Address:Port
root@agola-configstore-659887b86d-sgtfn:/#
#9

@sgotti I can run the example successfully now. I copy config.yml in the container and edit it with correct information.

Thanks for your supports.

PS: it’s great if we can make the examples with k8s work, too.

#10

@cuonglm I’m not sure if this has been solved or not and what was missing. We are using a distributed environment so it definitely works. If it doesn’t work please provide the whole config you used or it’s quite impossible to help you.

Probably you misconfigured something, please provide the k8s configuration you used.

#11

Hi @sgotti, here’s my agola.yml:

apiVersion: v1
kind: Service
metadata:
  name: agola-gateway
spec:
  ports:
    - port: 8000
      nodePort: 30002
  selector:
    app: agola
    component: gateway-others
  type: LoadBalancer

---

# The service for internal components communication with the runservice.
apiVersion: v1
kind: Service
metadata:
  name: agola-runservice
spec:
  ports:
    - port: 4000
  selector:
    app: agola
    component: runservice

---

# The service for internal components communication with the configstore.
apiVersion: v1
kind: Service
metadata:
  name: agola-configstore
spec:
  ports:
    - port: 4002
  selector:
    app: agola
    component: configstore

---

# The service for internal components communication with the gitserver.
apiVersion: v1
kind: Service
metadata:
  name: agola-gitserver
spec:
  ports:
    - port: 4003
  selector:
    app: agola
    component: gitserver

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: agola
data:
  config.yml: |
    gateway:
      # The api url that clients will call
      # Change this to the exposed "agola" service IP or dns name
      apiExposedURL: "http://192.168.64.20:30002"
      # The web interface url that clients will use
      # Change this to the exposed "agola" service IP or dns name
      webExposedURL: "http://192.168.64.20:30002"
      runserviceURL: "http://agola-runservice:4000"
      configstoreURL: "http://agola-configstore:4002"
      gitserverURL: "http://agola-gitserver:4003"

      web:
        listenAddress: ":8000"
      tokenSigning:
        # hmac or rsa (it possible use rsa)
        method: hmac
        # key to use when signing with hmac
        key: supersecretsigningkey
        # paths to the private and public keys in pem encoding when using rsa signing
        #privateKeyPath: /path/to/privatekey.pem
        #publicKeyPath: /path/to/public.pem
      adminToken: "admintoken"

    scheduler:
      runserviceURL: "http://agola-runservice:4000"

    notification:
      webExposedURL: "http://192.168.64.20:30002"
      runserviceURL: "http://agola-runservice:4000"
      configstoreURL: "http://agola-configstore:4002"
      etcd:
        endpoints: "http://etcd:2379"

    configstore:
      dataDir: /mnt/agola/local/configstore
      etcd:
        endpoints: "http://etcd:2379"
      objectStorage:
        type: s3
        # example with minio
        endpoint: "http://minio-service:9000"
        bucket: configstore
        accessKey: minio
        secretAccessKey: minio123
      web:
        listenAddress: ":4002"

    runservice:
      #debug: true
      dataDir: /mnt/agola/local/runservice
      etcd:
        endpoints: "http://etcd:2379"
      objectStorage:
        type: s3
        # example with minio
        endpoint: "http://minio-service:9000"
        bucket: runservice
        accessKey: minio
        secretAccessKey: minio123
      web:
        listenAddress: ":4000"

    executor:
      dataDir: /mnt/agola/local/executor
      # The directory containing the toolbox compiled for the various supported architectures
      toolboxPath: ./bin
      runserviceURL: "http://agola-runservice:4000"
      web:
        listenAddress: ":4001"
      activeTasksLimit: 2
      driver:
        type: kubernetes

    gitserver:
      dataDir: /mnt/agola/local/gitserver
      gatewayURL: "http://agola-gateway:8000"
      web:
        listenAddress: ":4003"

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agola-gateway-others
spec:
  replicas: 2
  selector:
    matchLabels:
      app: agola
      component: gateway-others
  template:
    metadata:
      labels:
        app: agola
        component: gateway-others
    spec:
      containers:
      - name: agola
        image: sorintlab/agola
        command:
          - /bin/agola
          - serve
          - "--config"
          - /mnt/agola/config/config.yml
          - "--components"
          - gateway,scheduler,notification
        env:
        ports:
          - containerPort: 8000
        volumeMounts:
          - name: config-volume
            mountPath: /mnt/agola/config
          - name: agola-localdata
            mountPath: /mnt/agola/local
      volumes:
          - name: config-volume
            configMap:
              name: agola
          - name: agola-localdata
            emptyDir: {}
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agola-runservice
spec:
  replicas: 2
  selector:
    matchLabels:
      app: agola
      component: runservice
  template:
    metadata:
      labels:
        app: agola
        component: runservice
    spec:
      containers:
      - name: agola
        image: sorintlab/agola
        command:
          - /bin/agola
          - serve
          - "--config"
          - /mnt/agola/config/config.yml
          - "--components"
          - runservice
        env:
        ports:
          - containerPort: 4000
        volumeMounts:
          - name: config-volume
            mountPath: /mnt/agola/config
          - name: agola-localdata
            mountPath: /mnt/agola/local
      volumes:
          - name: config-volume
            configMap:
              name: agola
          - name: agola-localdata
            emptyDir: {}

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agola-executor
spec:
  replicas: 2
  selector:
    matchLabels:
      app: agola
      component: executor
  template:
    metadata:
      labels:
        app: agola
        component: executor
    spec:
      containers:
      - name: agola
        image: sorintlab/agola
        command:
          - /bin/agola
          - serve
          - "--config"
          - /mnt/agola/config/config.yml
          - "--components"
          - executor
        env:
        ports:
          - containerPort: 4001
        volumeMounts:
          - name: config-volume
            mountPath: /mnt/agola/config
          - name: agola-localdata
            mountPath: /mnt/agola/local
      volumes:
          - name: config-volume
            configMap:
              name: agola
          - name: agola-localdata
            emptyDir: {}

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agola-configstore
spec:
  replicas: 2
  selector:
    matchLabels:
      app: agola
      component: configstore
  template:
    metadata:
      labels:
        app: agola
        component: configstore
    spec:
      containers:
      - name: agola
        image: sorintlab/agola
        command:
          - /bin/agola
          - serve
          - "--config"
          - /mnt/agola/config/config.yml
          - "--components"
          - configstore
        env:
        ports:
          - containerPort: 4002
        volumeMounts:
          - name: config-volume
            mountPath: /mnt/agola/config
          - name: agola-localdata
            mountPath: /mnt/agola/local
      volumes:
          - name: config-volume
            configMap:
              name: agola
          - name: agola-localdata
            emptyDir: {}

---

# The gitserver. Since it'll primarily store temporary git build data the
# simple way to deploy it is to use a deployment with 1 replica and an emptyDir
# volume. A statefulset with 1 replica and a persistent volume will be a better
# alternative.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agola-gitserver
spec:
  # Don't increase the replicas
  replicas: 1
  selector:
    matchLabels:
      app: agola
      component: gitserver
  template:
    metadata:
      labels:
        app: agola
        component: gitserver
    spec:
      containers:
      - name: agola
        image: sorintlab/agola
        command:
          - /bin/agola
          - serve
          - "--config"
          - /mnt/agola/config/config.yml
          - "--components"
          - gitserver
        env:
        ports:
          - containerPort: 4003
        volumeMounts:
          - name: config-volume
            mountPath: /mnt/agola/config
          - name: agola-localdata
            mountPath: /mnt/agola/local
      volumes:
          - name: config-volume
            configMap:
              name: agola
          - name: agola-localdata
            emptyDir: {}
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: agola
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - "*"

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: agola
  namespace: default
rules:
- apiGroups:
  - ""
  - "coordination.k8s.io"
  resources:
  - nodes
  - pods
  - pods/exec
  - configmaps
  - leases
  - secrets
  verbs:
  - "*"

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: agola
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: agola
subjects:
- kind: ServiceAccount
  name: default
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: agola
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: agola
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
#12

@cuonglm I’m not sure what you changed in the config but if the configstore isn’t starting listening you should check why. Usually the configstore won’t start listening if it cannot connect to etcd and won’t log anything so I’m quite sure you have some communication issues with etcd.

#13

Thanks, so etcd should be the problem, because there’s no etcd config in default agola.yml.

I think it will be better if configstore log something, or event should not start without etcd?

From user perspective, it will be great if everything necessary to run is included in agola.yml, so user can just run kubectl apply -f to have the whole stack up and running.

PS; In my example above, I have to run minio myself, it is not included in default agola.yml as well.

#14

Etcd and ObjectStorage are external services with many possible types and options so are up to the user. It doesn’t makes sense to define them in agola.yml since there’re many options. I see you deployed minio (also if not defined in agola.yml), so the same should be done for etcd.

The distributed example is complex and requires the user to understand how agola works. Providing how to deploy your etcd or the object storage doesn’t makes much sense since there’re many options. The simple example uses embedded etcd and object storage but the distributed one cannot do this. What can be done is improving the documentation to explain this. Feel free to open a PR.

The reasons for this are complex and tied to the etcd client, needs time to improve it. Feel free to open an issue/PR.