Comments (27)
GCP Logging have the same issue
from runsd.
Http vs https shouldn't be the issue as we don't hijack those hostnames.
Can you run "dig cloudtrace.googleapis.com" or see what IP it resolves to in the container on Cloud Run?
from runsd.
I used your example code and results below.
Region europe-west1
$ dig +search A cloudtrace.googleapis.com
; <<>> DiG 9.16.11 <<>> +search A cloudtrace.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 33282
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cloudtrace.googleapis.com.google.internal. IN A
From logs
I0205 19:22:15.638194 1 dns.go:42] [dns] > Q0: type=A name=cloudtrace.googleapis.com.europe-west1.run.internal.
I0205 19:22:15.638336 1 dns.go:66] [dns] < type=A name=cloudtrace.googleapis.com.europe-west1.run.internal. is too short or long (need ndots=4; got=6), nxdomain
I0205 19:22:15.638900 1 dns.go:42] [dns] > Q0: type=A name=cloudtrace.googleapis.com.run.internal.
I0205 19:22:15.638920 1 dns.go:66] [dns] < type=A name=cloudtrace.googleapis.com.run.internal. is too short or long (need ndots=4; got=5), nxdomain
I0205 19:22:15.639314 1 dns.go:42] [dns] > Q0: type=A name=cloudtrace.googleapis.com.google.internal.
I0205 19:22:15.639333 1 dns.go:120] [dns] >> recursing type=A name=cloudtrace.googleapis.com.google.internal.
I0205 19:22:15.642319 1 dns.go:127] [dns] << recursed type=A name=cloudtrace.googleapis.com.google.internal. rcode=SERVFAIL answers=0 rtt=2.865112ms
from runsd.
Looks like ndots
config is a bit weird.
- Are you on the latest version?
- Any chance you can print out /etc/resolv.conf at the runtime of the container?
from runsd.
I'm asking these because I have an example container that runs runsd
+ has a curl form like this:
and when I run it, curl works fine:
from runsd.
- Yes, I'm using link with latest
ADD https://github.com/ahmetb/runsd/releases/latest/download/runsd /runsd
RUN chmod +x /runsd
ENTRYPOINT ["/runsd", "-v=5", "--", "/app"]
- I have added print endpoint to example code from this repo
func print(w http.ResponseWriter, req *http.Request) {
file, err := os.Open("/etc/resolv.conf")
if err != nil {
fmt.Fprintf(w, "dig failed: %v\n", err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
b, err := ioutil.ReadAll(file)
fmt.Fprint(w, string(b))
}
Output of file is
nameserver 127.0.0.1
nameserver ::1
search europe-west1.run.internal. run.internal. google.internal.
options ndots:4
Dig from example from this code repo
$ dig +search A cloudtrace.googleapis.com
; <<>> DiG 9.16.11 <<>> +search A cloudtrace.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 15816
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cloudtrace.googleapis.com.google.internal. IN A
Curl output
$ curl -sSLNv --http2 http://cloudtrace.googleapis.com
* Could not resolve host: cloudtrace.googleapis.com
* Closing connection 0
curl: (6) Could not resolve host: cloudtrace.googleapis.com
curl failed: exit status 6
from runsd.
Weird enough I have the exact example/
application working on my side:
/dig?domain=cloudtrace.googleapis.com
shows:
$ dig +search A cloudtrace.googleapis.com
; <<>> DiG 9.16.11 <<>> +search A cloudtrace.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59541
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cloudtrace.googleapis.com. IN A
;; ANSWER SECTION:
cloudtrace.googleapis.com. 253 IN A 172.217.214.95
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Feb 05 21:32:02 UTC 2021
;; MSG SIZE rcvd: 84
and /resolvconf
shows:
nameserver 127.0.0.1
nameserver ::1
search us-central1.run.internal. run.internal. google.internal.
options ndots:4
Do you spot any differences?
from runsd.
The only one difference I see is region us-central1 vs europe-west1 and in your screenshot there is no --http2
param in curl.
from runsd.
I updated my image, works with --http2 as well:
from runsd.
Do you mind trying another region?
from runsd.
Sure, on us-central1 not working too, the same results, only /etc/resolv.conf is different (as expected).
nameserver 127.0.0.1
nameserver ::1
search us-central1.run.internal. run.internal. google.internal.
options ndots:4
I'll try to run this on my private account, not corp one which is in organization but at this moment I don't see reason of this issue
from runsd.
I deployed it on my private account on private project and the same result, not working.
I was thinking that maybe it's Docker image issue but now I'm not sure. Image with example I'm building locally with docker-compose on wsl2.
First issue where we got errors from trace+logs was built on gitlab runner on GKE and it was completely different code (nodejs, slim image)
from runsd.
When I used dig with @8.8.8.8 I got results some results
$ dig +search A [email protected]
; <<>> DiG 9.16.11 <<>> +search A [email protected]
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 49689
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;cloudtrace.googleapis.com\@8.8.8.8. IN A
;; AUTHORITY SECTION:
. 86399 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2021020502 1800 900 604800 86400
;; Query time: 15 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Feb 05 22:00:35 UTC 2021
;; MSG SIZE rcvd: 126
;; Query time: 15 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Feb 05 22:00:35 UTC 2021
;; MSG SIZE rcvd: 126
from runsd.
that's because it didn't parse it correctly. :) @8.8.8.8 is a separate argument. :)
from runsd.
Oh, I was hoping that we moved closer with an issue:D For now I have no idea what's going on.
from runsd.
Can you perhaps try deploying to us-central1
.
I can't imagine what would cause this.
from runsd.
I did previously and on my private account it was us-central1.
I'll try build via Cloud Build and see if there is any difference
from runsd.
Building on Cloud Build does not change anything.
Startup
from runsd.
I cloned repo into cloudshell, slightly modified regions file to let me run this, built docker and dig and curl from example was working. I triend it also with runtime runsc to try gvisor but it's also working. Pushed this example into container registry, created revision on cloud run and nope, not working.
from runsd.
Hi, do you have any update regarding dns fix on google side?
edit:
Looks like it's working now. I have added this to our services and requests are passed correctly <3
Can you confirm that this fix on google side is stable?
from runsd.
Sorry, I've been debating internally defending this is a bug. So far I have not convinced people to fix this and it required me to go low level to prove that this is how DNS works and our bug makes it incompatible.
My only other option is to never proxy the queries to "google.internal." Zone in runsd resolve metadata.google.internal lookup myself. I am not sure what other side effects would be and it wouldn't be a long-term solution.
Maybe we can try.
from runsd.
At this moment it works on my 2 projects. One with hello example and one on my dev environment so maybe something has changed. But until it's not confirmed internally I won't use it.
I was considering using serverless connector as you mentioned earlier but it is at least $16 per month.
from runsd.
The problem wasnt resolving internal domains? It was resolving external domains I think? Example.com shouldn't be working?
from runsd.
Hm, internally it was not working too but maybe it was other reason and I just don't remember.
But yes, query to external domain still not working :(
Do you think that https://dns.google might help? Query this and cache when getting SERVFAIL or pattern does not match run.app. Just loose considerations.
https://dns.google/resolve?name=cloudtrace.googleapis.com&type=A
from runsd.
We can't blindly replace the nameserver we recurse into. People who use VPC Connector actually use internal DNS names of VMs and it works from Cloud Run. Thats why we have to proxy to 169.254.169.254. However the "google.Internal." search domain is giving us problems iterating over candidates. That's the problem.
from runsd.
I've just pushed a tag v0.0.0-rc.10
that I think fixes the problem. Give it a try @Mistic92 and close if it works.
from runsd.
It seems like the fix solved it.
from runsd.
Related Issues (20)
- CVE-2020-14040 Package: golang.org/x/text HOT 2
- Feeback on rc13 run as user HOT 4
- Problem with Listening gRPC Requests Over HTTP on Cloud Run HOT 4
- too many "context canceled" errors when client disconnects
- Unable to call service when going through a VPC connector HOT 2
- How to dns resolve to cloudRun services across different GCP projects HOT 1
- Shared VPC HOT 8
- Incremental Rollout HOT 3
- Correct or support Triggers for target Cloud Run service? HOT 2
- Ability to run on Compute Engine HOT 1
- 20+ seconds to resolve/connect to MongoDB atlas cluster HOT 3
- Automatic authentication and how that works HOT 1
- Usage without entrypoint subprocess HOT 1
- Use as a library in Go project? HOT 1
- Usage without entrypoint subprocess #44 HOT 4
- External domain getaddrinfo ENOTFOUND HOT 4
- Can't get proxy to work with gRPC server HOT 3
- Project state HOT 2
- Permission denied HOT 4
- Failing to integrate runsd into a dockerized next.js app which is deployed on cloud run due to error finding executable "runsd" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from runsd.