David M. Karr (83) [Avatar] Offline
#1
So I'm following the instructions in chapter 2 to get the toy kubia image running in a pod. I already had a Docker Hub login because of some earlier experiments with pure Docker images.

As described in chapter 2 of the book, I built the toy "kubia" image, and I was able to push it to Docker Hub. I verified this again by logging into Docker Hub and seeing the image.

I'm doing this on Centos7.

I was able to build the image and push it to docker hub without any difficulty.

I then run the following to create the replication controller and pod running my image:

kubectl run kubia --image=davidmichaelkarr/kubia --port=8080 --generator=run/v1


I waited a while for statuses to change, but it never finishes downloading the image, when I describe the pod, I see something like this:

  Normal   Scheduled              24m                 default-scheduler  Successfully assigned kubia-25th5 to minikube
  Normal   SuccessfulMountVolume  24m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-x5nl4"
  Normal   Pulling                22m (x4 over 24m)   kubelet, minikube  pulling image "davidmichaelkarr/kubia"
  Warning  Failed                 22m (x4 over 24m)   kubelet, minikube  Failed to pull image "davidmichaelkarr/kubia": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)


So I then constructed the following command:

curl -v -u 'davidmichaelkarr:**' 'https://registry-1.docker.io/v2/'

Which uses the same password I use for Docker Hub (they should be the same, right?).

This gives me the following:

* About to connect() to proxy *** port 8080 (#0)
*   Trying **.**.**.**...
* Connected to *** (**.**.**.**) port 8080 (#0)
* Establish HTTP proxy tunnel to registry-1.docker.io:443
* Server auth using Basic with user 'davidmichaelkarr'
> CONNECT registry-1.docker.io:443 HTTP/1.1
> Host: registry-1.docker.io:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.docker.io
*       start date: Aug 02 00:00:00 2017 GMT
*       expire date: Sep 02 12:00:00 2018 GMT
*       common name: *.docker.io
*       issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
* Server auth using Basic with user 'davidmichaelkarr'
> GET /v2/ HTTP/1.1
> Authorization: Basic ***
> User-Agent: curl/7.29.0
> Host: registry-1.docker.io
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
< Date: Wed, 24 Jan 2018 18:34:39 GMT
< Content-Length: 87
< Strict-Transport-Security: max-age=31536000
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host *** left intact


I didn't know what else to try, so I asked about this on StackOverflow: https://stackoverflow.com/questions/48429591/kubectl-cant-connect-to-docker-registry-to-download-image

The answer I got talks about the need to set up a secret that I register in kubectl. I haven't started this yet. Is this just something that was missed in the book, or is there a reason why this might not have been required?
Marko Lukša (65) [Avatar] Offline
#2
A few things about pulling images:
- you only need imagePullSecrets when pulling from a private repo (this is explained somewhere in the book - most likely in the chapter about secrets);
- log into hub.docker.com and make sure your repo is marked as public, so you don't need to deal with imagePullSecrets at this point.
- I'm not sure if Docker Hub uses basic http auth; I've tried the curl command and it doesn't not work for me.

But the problem in your case is that the Docker daemon running in the minikube VM isn't using your internal proxy, and thus can't reach the registry at all. It's not an authentication problem.

You can confirm this by running the curl command from inside the minikube VM (run
minikube ssh
to log into it).

To fix the problem, make sure you run minikube with the following options:
--docker-env http_proxy=http://yourproxy:port --docker-env https_proxy=http://yourproxy:port --docker-env no_proxy=192.168.99.0/24







David M. Karr (83) [Avatar] Offline
#3
Hmm, this seemed promising, but it didn't make any difference. If I ran that command, and then did "minikube ssh", what should I see in that environment that would tell me whether or not these variables were set properly or not? I wouldn't expect to see anything in the simple "env" output, because those are intended to be docker-specific.
Marko Lukša (65) [Avatar] Offline
#4
Not sure how you can see the environment configured for the Docker daemon, but here's what I tried:

I ran minikube like this (pointed it to 8.8.8.8 instead of an actual proxy):

$ minikube start --docker-env http_proxy=http://8.8.8.8:88 --docker-env https_proxy=http://8.8.8.8:88 --docker-env no_proxy=192.168.99.0/24


Then, if I examine the minikube logs, I see that it's trying to use the configured proxy:

$ minikube logs | grep 8.8.8.8
Jan 25 20:49:41 minikube localkube[3536]: E0125 20:49:41.129124    3536 remote_image.go:108] PullImage "gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v1/_ping: http: error connecting to proxy http://8.8.8.8:88: dial tcp 8.8.8.8:88: i/o timeout


So, I suggest you try minikube logs to see if there's any trace of whether the proxy is being used or not. In any case, you should see the error preventing the image from being pulled.

Oh, maybe you need to delete the VM first with minikube delete... but I doubt that...
David M. Karr (83) [Avatar] Offline
#5
LOL. Your last "doubtful" comment was the key. When you told me I needed the "--docker-env" settings, I thought that "minikube stop" would do everything that was required to start over. I just did "minikube delete" and started over, and now the image was successfully downloaded and the container started. However, I also looked at "minikube logs" and grepped for "proxy", and it never found anything that looked like a connection that was going through a proxy.
Marko Lukša (65) [Avatar] Offline
#6
Yeah, if you just run minikube start, it uses the existing VM, so any flags you specify may or may not be honored. I've learned that any time minikube doesn't behave like it should, running minikube delete might be the solution.

I probably should have mentioned that in the book (and what to do if you need to use a proxy). Noted for 2nd edition.

Thank you for reporting this.

David M. Karr (83) [Avatar] Offline
#7
Hmm, looks like there are more issues related to this. Following the steps in the book, I then created the kubia-http LoadBalancer. When I tried to do "minikube service kubia-http" to get the host:port, I got this:

Error opening service: Could not find finalized endpoint being pointed to by kubia-http: Error validating service: Error getting service kubia-http: Get https://192.168.99.100:8443/api/v1/namespaces/default/services/kubia-http: Proxy Authentication Required


So this looks like it's seeing my https_proxy setting, but not my "no_proxy" setting, or perhaps the format I used isn't working. I found that the documentation for specifying IP addresses in no_proxy is ambiguous, and especially address ranges. I thought it would support CIDR, so I set it to "192.168.0.0/16", but for this piece that didn't appear to work.
Marko Lukša (65) [Avatar] Offline
#8
You probably need to set no_proxy in your local OS (the one you're running the minikube service command in).
David M. Karr (83) [Avatar] Offline
#9
That was already set. Doesn't make any difference.

Note that I found the following related issue for minikube, which I've commented on: https://github.com/kubernetes/minikube/issues/2453 .
Marko Lukša (65) [Avatar] Offline
#10
Hmm, isn't the Proxy Authentication Required message coming from your proxy? That would imply that the request to 192.168.99.100 is going to the proxy instead of to the minikube VM directly.

Have you tried using the actual IP address instead of the CIDR?

I've just ran the following, and it still wants to go through the proxy:

http_proxy=http://8.8.8.8 no_proxy=192.168.0.0/16 minikube service kubia


But, using the IP works:

http_proxy=http://8.8.8.8 no_proxy=192.168.99.100 minikube service kubia
Opening kubernetes service default/kubia in default browser...

Marko Lukša (65) [Avatar] Offline
#11
Ah, yes, using CIDR in no_proxy isn't supported in Go's net/http library, which minikube uses. See https://github.com/golang/go/issues/16704