TL;DR
Try the curl
command listed below in Step 7. If that doesn't work, you likely have another issue besides your curl
command (such as a firewall or proxy).
Copying Browser Requests with cURL
If you can connect with a browser, one way to eliminate potential issues with your curl
command is to "copy" the desired transaction from your browser. Chrome, Firefox, and Opera all have a Developer console which allows this.
Open an new blank tab.
In Chrome and Firefox, press F12 to open the Developer console. In Opera, press Ctrl + Shift + I.
Look for the Network
option and select it.
Navigate to your desired URL e.g. https://registry.terraform.io/.well-known/terraform.json.
You should end up looking at something similar to the following:
ex. Network Transactions
![Firefox Developer Console - Network Transactions - Screenshot](../../I/static/images/d616b7c2406ff179a9dfeb01ef98950f61bd5766167d1a1a8dddac3b5e585604.png)
Select the desired transaction, right-click and select the option to Copy as cURL
(for Chrome, use Copy as cURL (cmd)
).
ex. Copy as cURL
![Firefox Developer Console - Network Transactions - Copy as cURL - Screenshot](../../I/static/images/bb407833c37302a63891bd5c6e13e6f248e65f5706f9a2e9aa33534442a3470d.png)
If you paste the contents of the clipboard, you should have an exact copy of the request for curl
:
curl "https://registry.terraform.io/.well-known/terraform.json" -H "authority: registry.terraform.io" -H "upgrade-insecure-requests: 1" -H "user-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" -H "sec-fetch-mode: navigate" -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "sec-fetch-site: none" -H "accept-encoding: gzip, deflate, br" -H "accept-language: en-US,en;q=0.9" --compressed
This curl
command should replicate the request made by the browser exactly.
Other Issues
Assuming that this doesn't solve the problem, then you will have to look into other reasons why this request might be failing, such as firewall/proxy issues or certificate issues.
Use the -k
or --insecure
option to ignore self-signed or invalid certificates with curl.
Consider using the --verbose
flag, as well as dumping HTTP headers with -D -
, with curl
to aid troubleshooting.
Emulating A Specific Browser
Based on your additional comment that your (generic) cURL requests are still failing, it's likely you will need to emulate the specific browser which can connect successfully.
Browsers identify themselves with a User Agent string associated with a given version of that browser. If you want to emulate the browser that can connect to your URL, you'll need to either:
Use the steps I give under Copying Browser Requests with cURL in the browser that can connect successfully.
Try replacing the user-agent
value in Step 7 with the value returned from a site like What Is My Browser (visit this link in the browser which has no trouble accessing your link).
Example (Option 2)
"Generic" User Agent (above)
-H "user-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
Specific User Agent (e.g. from What Is My Browser)
-H "user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
It's important to note that the first method (copying the request) is probably the most reliable, as it may catch headers other than the user-agent
that are used to authorize a request.
What is your curl Command-Line? – None – 2019-09-21T10:21:08.303
Please update the title, as I'm assuming you meant to type "Having an issue with..."? – JW0914 – 2019-09-27T10:38:27.763