cuonglm
February 12, 2020, 11:06am
1
Hello, I followed the kubernestes distributed example to run agola in minukube, all the pods are up and running:
NAMESPACE NAME READY STATUS RESTARTS AGE
default agola-configstore-6b695fd9ff-qptsn 1/1 Running 5 14m
default agola-configstore-6b695fd9ff-rpl7h 1/1 Running 5 14m
default agola-executor-576778ff84-qknbh 1/1 Running 0 14m
default agola-executor-576778ff84-vptfx 1/1 Running 0 14m
default agola-gateway-others-79cbd6cc5f-p86wq 1/1 Running 0 14m
default agola-gateway-others-79cbd6cc5f-wjpb4 1/1 Running 0 14m
default agola-gitserver-869f8bbfb8-cn2c9 1/1 Running 0 14m
default agola-runservice-8f44c4d47-vp89f 1/1 Running 5 14m
default agola-runservice-8f44c4d47-wlqs2 1/1 Running 5 14m
default minio-6f44ccc59-qf45k 1/1 Running 0 14m
However, when I access agola web expose URL to register, I got a blank page. Looking at the chrome development tools, I can see:
curl 'http://192.168.64.20:30002/api/v1alpha/remotesources' -H 'Connection: keep-alive' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36' -H 'Accept: */*' -H 'Referer: http://192.168.64.20:30002/register' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.9,vi;q=0.8' -H 'Cookie: lang=en-US; i_like_gitea=82076592b485c935; _csrf=6mbbW4svRvaXmHhSLY8ta_FXNPU6MTU4MTUwMjgyNDIyMjcwODA3Mg' --compressed --insecure
Looking at the gateway pods logs:
2020-02-12T11:01:15.209Z ERROR scheduler/scheduler.go:116 err: failed to get running runs:
agola.io/agola/internal/services/scheduler.(*Scheduler).approve
/agola/internal/services/scheduler/scheduler.go:133
- Get http://agola-runservice:4000/api/v1alpha/runs?asc=&phase=running: dial tcp 10.97.93.3:4000: connect: connection refused
So I attach to the runservice pod, and check if port 4000 is opened, but it does not.
How can I fix it?
sgotti
February 12, 2020, 11:13am
2
@cuonglm Please provide your configuration and how you are accessing the web interface.
cuonglm:
I got a blank page.
What is a blank page? Do you see the agola top bar? If so you should first create a remote source and then click on register to register an user.
Your browser is calling the agola gateway on http://192.168.64.20:30002 . Is it right? Are you sure your browser can access that url?
This error at startup is normal since the runservice component could not yet be started when the scheduler component tries to access it so it’s not related.
1 Like
cuonglm
February 12, 2020, 11:31am
3
@sgotti
What is a blank page? Do you see the agola top bar? If so you should first create a remote source and then click on register to register an user.
Thanks, it’s exactly what I see. I haven’t created remoteresource yet.
sgotti
February 12, 2020, 12:14pm
4
Yeah, you should first create a remote source. You can take a look at https://agola.io/tryit .
cuonglm
February 12, 2020, 4:05pm
5
Hi @sgotti , I use this command to create remotesource:
./bin/agola --token "admintoken" --gateway-url http://192.168.64.20:30002 remotesource create \
--name gitea \
--type gitea \
--api-url 192.168.64.20 \
--auth-type oauth2 \
--clientid a5043718-c471-4399-a3b7-4fc93839e051 \
--secret vgsgyuJmQZoxIEO9EH2CJBsJX5qWXMCs2ujgzaPQg90= \
--skip-ssh-host-key-check
It returns an error:
2020-02-12T23:01:48.460+0700 INFO cmd/remotesourcecreate.go:116 creating remotesource
2020-02-12T23:01:48.471+0700 FATAL cmd/remotesourcecreate.go:34 err: failed to create remotesource: internal server error
The gateway log:
2020-02-12T16:01:48.527Z INFO action/remotesource.go:112 creating remotesource
2020-02-12T16:01:48.529Z ERROR api/remotesource.go:66 err: failed to create remotesource:
agola.io/agola/internal/services/gateway/action.(*ActionHandler).CreateRemoteSource
/agola/internal/services/gateway/action/remotesource.go:115
sgotti
February 12, 2020, 4:34pm
6
@cuonglm You should first use the basic example provided https://agola.io/tryit since I’m quite sure you have something misconfigured in your k8s distributed config.
So you should at first make it work with the try it demo and then, when it works, please provide the whole config and component logs.
cuonglm
February 12, 2020, 4:43pm
7
The example provided https://agola.io/tryit can not work in Mac, because it assumes the docker bridge gateway is 172.17.0.1, in my case, it’s 192.168.64.1
I actually did follow the example, created a remoteresource successfully, but then I still stuck because going to 127.0.0.1:8000 just failed, because the page call to 172.17.0.1:8000 for api.
you have something misconfigured in your k8s distributed config.
Can you point me out where? Since when all the pods are up and running, I assume it should work.
cuonglm
February 12, 2020, 4:50pm
8
The problem is the agola-configstore is not listening on port 4002:
root@agola-gateway-others-7866556766-2bhdz:/# telnet agola-configstore 4002
Trying 10.98.22.112...
telnet: Unable to connect to remote host: Connection refused
kubectl exec -it agola-configstore-659887b86d-sgtfn bash ──(Wed,Feb12)─┘
root@agola-configstore-659887b86d-sgtfn:/# ss -tnlp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
root@agola-configstore-659887b86d-sgtfn:/#
cuonglm
February 12, 2020, 5:26pm
9
cuonglm:
The example provided https://agola.io/tryit can not work in Mac, because it assumes the docker bridge gateway is 172.17.0.1 , in my case, it’s 192.168.64.1
@sgotti I can run the example successfully now. I copy config.yml in the container and edit it with correct information.
Thanks for your supports.
PS: it’s great if we can make the examples with k8s work, too.
sgotti
February 13, 2020, 12:15pm
10
@cuonglm I’m not sure if this has been solved or not and what was missing. We are using a distributed environment so it definitely works. If it doesn’t work please provide the whole config you used or it’s quite impossible to help you.
Probably you misconfigured something, please provide the k8s configuration you used.
cuonglm
February 13, 2020, 12:45pm
11
Hi @sgotti , here’s my agola.yml:
apiVersion: v1
kind: Service
metadata:
name: agola-gateway
spec:
ports:
- port: 8000
nodePort: 30002
selector:
app: agola
component: gateway-others
type: LoadBalancer
---
# The service for internal components communication with the runservice.
apiVersion: v1
kind: Service
metadata:
name: agola-runservice
spec:
ports:
- port: 4000
selector:
app: agola
component: runservice
---
# The service for internal components communication with the configstore.
apiVersion: v1
kind: Service
metadata:
name: agola-configstore
spec:
ports:
- port: 4002
selector:
app: agola
component: configstore
---
# The service for internal components communication with the gitserver.
apiVersion: v1
kind: Service
metadata:
name: agola-gitserver
spec:
ports:
- port: 4003
selector:
app: agola
component: gitserver
---
apiVersion: v1
kind: ConfigMap
metadata:
name: agola
data:
config.yml: |
gateway:
# The api url that clients will call
# Change this to the exposed "agola" service IP or dns name
apiExposedURL: "http://192.168.64.20:30002"
# The web interface url that clients will use
# Change this to the exposed "agola" service IP or dns name
webExposedURL: "http://192.168.64.20:30002"
runserviceURL: "http://agola-runservice:4000"
configstoreURL: "http://agola-configstore:4002"
gitserverURL: "http://agola-gitserver:4003"
web:
listenAddress: ":8000"
tokenSigning:
# hmac or rsa (it possible use rsa)
method: hmac
# key to use when signing with hmac
key: supersecretsigningkey
# paths to the private and public keys in pem encoding when using rsa signing
#privateKeyPath: /path/to/privatekey.pem
#publicKeyPath: /path/to/public.pem
adminToken: "admintoken"
scheduler:
runserviceURL: "http://agola-runservice:4000"
notification:
webExposedURL: "http://192.168.64.20:30002"
runserviceURL: "http://agola-runservice:4000"
configstoreURL: "http://agola-configstore:4002"
etcd:
endpoints: "http://etcd:2379"
configstore:
dataDir: /mnt/agola/local/configstore
etcd:
endpoints: "http://etcd:2379"
objectStorage:
type: s3
# example with minio
endpoint: "http://minio-service:9000"
bucket: configstore
accessKey: minio
secretAccessKey: minio123
web:
listenAddress: ":4002"
runservice:
#debug: true
dataDir: /mnt/agola/local/runservice
etcd:
endpoints: "http://etcd:2379"
objectStorage:
type: s3
# example with minio
endpoint: "http://minio-service:9000"
bucket: runservice
accessKey: minio
secretAccessKey: minio123
web:
listenAddress: ":4000"
executor:
dataDir: /mnt/agola/local/executor
# The directory containing the toolbox compiled for the various supported architectures
toolboxPath: ./bin
runserviceURL: "http://agola-runservice:4000"
web:
listenAddress: ":4001"
activeTasksLimit: 2
driver:
type: kubernetes
gitserver:
dataDir: /mnt/agola/local/gitserver
gatewayURL: "http://agola-gateway:8000"
web:
listenAddress: ":4003"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: agola-gateway-others
spec:
replicas: 2
selector:
matchLabels:
app: agola
component: gateway-others
template:
metadata:
labels:
app: agola
component: gateway-others
spec:
containers:
- name: agola
image: sorintlab/agola
command:
- /bin/agola
- serve
- "--config"
- /mnt/agola/config/config.yml
- "--components"
- gateway,scheduler,notification
env:
ports:
- containerPort: 8000
volumeMounts:
- name: config-volume
mountPath: /mnt/agola/config
- name: agola-localdata
mountPath: /mnt/agola/local
volumes:
- name: config-volume
configMap:
name: agola
- name: agola-localdata
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: agola-runservice
spec:
replicas: 2
selector:
matchLabels:
app: agola
component: runservice
template:
metadata:
labels:
app: agola
component: runservice
spec:
containers:
- name: agola
image: sorintlab/agola
command:
- /bin/agola
- serve
- "--config"
- /mnt/agola/config/config.yml
- "--components"
- runservice
env:
ports:
- containerPort: 4000
volumeMounts:
- name: config-volume
mountPath: /mnt/agola/config
- name: agola-localdata
mountPath: /mnt/agola/local
volumes:
- name: config-volume
configMap:
name: agola
- name: agola-localdata
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: agola-executor
spec:
replicas: 2
selector:
matchLabels:
app: agola
component: executor
template:
metadata:
labels:
app: agola
component: executor
spec:
containers:
- name: agola
image: sorintlab/agola
command:
- /bin/agola
- serve
- "--config"
- /mnt/agola/config/config.yml
- "--components"
- executor
env:
ports:
- containerPort: 4001
volumeMounts:
- name: config-volume
mountPath: /mnt/agola/config
- name: agola-localdata
mountPath: /mnt/agola/local
volumes:
- name: config-volume
configMap:
name: agola
- name: agola-localdata
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: agola-configstore
spec:
replicas: 2
selector:
matchLabels:
app: agola
component: configstore
template:
metadata:
labels:
app: agola
component: configstore
spec:
containers:
- name: agola
image: sorintlab/agola
command:
- /bin/agola
- serve
- "--config"
- /mnt/agola/config/config.yml
- "--components"
- configstore
env:
ports:
- containerPort: 4002
volumeMounts:
- name: config-volume
mountPath: /mnt/agola/config
- name: agola-localdata
mountPath: /mnt/agola/local
volumes:
- name: config-volume
configMap:
name: agola
- name: agola-localdata
emptyDir: {}
---
# The gitserver. Since it'll primarily store temporary git build data the
# simple way to deploy it is to use a deployment with 1 replica and an emptyDir
# volume. A statefulset with 1 replica and a persistent volume will be a better
# alternative.
apiVersion: apps/v1
kind: Deployment
metadata:
name: agola-gitserver
spec:
# Don't increase the replicas
replicas: 1
selector:
matchLabels:
app: agola
component: gitserver
template:
metadata:
labels:
app: agola
component: gitserver
spec:
containers:
- name: agola
image: sorintlab/agola
command:
- /bin/agola
- serve
- "--config"
- /mnt/agola/config/config.yml
- "--components"
- gitserver
env:
ports:
- containerPort: 4003
volumeMounts:
- name: config-volume
mountPath: /mnt/agola/config
- name: agola-localdata
mountPath: /mnt/agola/local
volumes:
- name: config-volume
configMap:
name: agola
- name: agola-localdata
emptyDir: {}
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: agola
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: agola
namespace: default
rules:
- apiGroups:
- ""
- "coordination.k8s.io"
resources:
- nodes
- pods
- pods/exec
- configmaps
- leases
- secrets
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: agola
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: agola
subjects:
- kind: ServiceAccount
name: default
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: agola
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: agola
subjects:
- kind: ServiceAccount
name: default
namespace: default
sgotti
February 14, 2020, 8:29am
12
@cuonglm I’m not sure what you changed in the config but if the configstore isn’t starting listening you should check why. Usually the configstore won’t start listening if it cannot connect to etcd and won’t log anything so I’m quite sure you have some communication issues with etcd.
cuonglm
February 14, 2020, 9:00am
13
Thanks, so etcd should be the problem, because there’s no etcd config in default agola.yml.
I think it will be better if configstore log something, or event should not start without etcd?
From user perspective, it will be great if everything necessary to run is included in agola.yml, so user can just run kubectl apply -f to have the whole stack up and running.
PS; In my example above, I have to run minio myself, it is not included in default agola.yml as well.
sgotti
February 14, 2020, 9:08am
14
Etcd and ObjectStorage are external services with many possible types and options so are up to the user. It doesn’t makes sense to define them in agola.yml since there’re many options. I see you deployed minio (also if not defined in agola.yml), so the same should be done for etcd.
The distributed example is complex and requires the user to understand how agola works. Providing how to deploy your etcd or the object storage doesn’t makes much sense since there’re many options. The simple example uses embedded etcd and object storage but the distributed one cannot do this. What can be done is improving the documentation to explain this. Feel free to open a PR.
The reasons for this are complex and tied to the etcd client, needs time to improve it. Feel free to open an issue/PR.