Grab only the HTTP headers, instead of the entire HTTP response and content, from a URL:
curl -I $URL
To ensure you see the entire request/response chain, add the -L command line switch to follow redirects:
curl -IL $URL
If you maintain a lot of domain redirects, it is important to make sure they continually work. The following curl command can be used to loop through a text file containing URLs to check. The output for each check will be the final HTTP status code and URL.
curl -sL -w "%{http_code} %{url_effective}\\n" "$URL" -o /dev/null
For example, to verify https://old-site.com redirects to https://new-site.com:
curl -sL -w "%{http_code} %{url_effective}\\n" "https://old-site.com" -o /dev/null
Will output:
200 https://new-site.com
If you do local web development, you probably have to frequently access your website or web application from localhost. In most situations this shouldn’t be a problem, but for those situations where it is (such as testing Apache VirtualHosts or nginx Server Blocks), you can easily change the Host header with curl to tell the website or web application where you want to go.
curl -i -H 'Host: example.com' $URL
Alternatively, temporarily add a record in your workstation’s /etc/hosts file pointing the domain name of the URL you are accessing to 127.0.0.1.
Instead of changing your /etc/hosts file, force a curl request to go through an IP address that differs from what the hostname resolves to in public DNS.
curl -IL $HOSTNAME --resolve $HOSTNAME:$PORT:$SPECIFIC-IP
Create file curlloop.sh with the following content:
#!/bin/bash
url_path="${1}"
request_count=1
while true
do
echo -n "${request_count}: "
curl -sL -w "%{http_code} %{time_total} %header{cf-cache-status} %{url_effective}\\n" "${url_path}" -o /dev/null
sleep .5
((request_count++))
done
Set the executable permission:
chmod +x curlloop.sh
Run the shell script with the following command:
./curlloop.sh refcli.com
Example output:
curlloop.sh refcli.com
1: 200 0.341181 DYNAMIC https://refcli.com/
2: 200 0.158135 DYNAMIC https://refcli.com/
3: 200 0.169257 DYNAMIC https://refcli.com/
4: 200 0.179654 DYNAMIC https://refcli.com/
5: 200 0.185863 DYNAMIC https://refcli.com/
POST plain text data to URL:
curl -X POST -d 'plain-text' -H "Content-Type:text/plain" $URL
POST JSON data to URL:
curl -X POST -d '{"key1":"value1","key2":"value2"}' -H "Content-Type:application/json" $URL
Create a file called payload, put valid JSON in it, and reference it with the curl command using the following command:
curl -d @./payload -H "X-Auth: $TOKEN" "https://api.example.com/api/query"
If you are using the curl command within a shell script and want to pass JSON to it from a Bash variable in the same shell script, use the following method:
PAYLOAD='
[{
"auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "USERNAME",
"domain": {
"id": "default"
},
"password": "PASSWORD"
}
}
},
"scope": {
"project": {
"name": "PROJECT",
"domain": {
"id": "default"
}
}
}
}
}]'
RESULT=$(curl -s -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<< "$PAYLOAD")
Another method you can use is a HERE document. This eliminates the need for the $PAYLOAD Bash variable.
RESULT=$(curl -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<PAYLOAD
[{
"auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "USERNAME",
"domain": {
"id": "default"
},
"password": "PASSWORD"
}
}
},
"scope": {
"project": {
"name": "PROJECT",
"domain": {
"id": "default"
}
}
}
}
}]
PAYLOAD)
Many of the tools used today are built on top of some sort of API. Those tools abstract all of the granular details and information that comes through an API request and response. Sometimes it is useful to see what goes on behind the scenes, especially when it comes time to troubleshoot.
The following example uses the curl command to query the OpenStack API to show the details about a particular hypervisor managed by OpenStack Nova.
Before you can do anything with any API, you need a token. With OpenStack it’s no different. OpenStack Keystone is the identity and authentication service that is used to generate tokens for end users and services.
The Keystone v2 API is quickly being deprecated. I have only included it for posterity’s sake, so jump to the v3 API below to generate a token:
curl -s \
-X POST https://10.240.0.100:5000/v2.0/tokens \
-H "Content-Type: application/json" \
-d '{"auth": {"tenantName": "TENANT", "passwordCredentials":{"username": "USERNAME", "password": "PASSWORD"}}}'
What follows is the Keystone v3 API. You will need a valid username and password already stored in OpenStack Keystone to generate a token:
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "USERNAME",
"domain": { "id": "default" },
"password": "PASSWORD"
}
}
},
"scope": {
"project": {
"name": "PROJECT",
"domain": { "id": "default" }
}
}
}
}' \
https://10.240.0.100:5000/v3/auth/tokens
This will return a lot of JSON. Unfortunately, I did not make a copy of that JSON output when I originally created these notes. However, within that JSON will be a token that will be used in all of the subsequent commands. Let’s assume the token returned is 1234567890abcdefghijklmnopqrstuv.
Now that a token has been generated, I can begin querying the OpenStack Nova API for details on the particular hypervisor. For this example, I want details about compute01.local.lan, and I’m going to need its id:
curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/compute01.local.lan/search | python -m json.tool
Take note, the curl command is going to output JSON which can be difficult to read. I am piping the output to python -m json.tool to make it easier to read.
The above command returns the following JSON:
{
"hypervisors": [
{
"hypervisor_hostname": "compute01.local.lan",
"id": 1,
"state": "up",
"status": "enabled"
}
]
}
Now that I have the id, I can query the details about compute01.local.lan:
curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/1 | python -m json.tool
The above command returns the following JSON which provides all of the details for that particular hypervisor:
{
"hypervisor": {
"cpu_info": "{\"vendor\": \"Intel\", \"model\": \"SandyBridge\", \"arch\": \"x86_64\", \"features\": [\"ssse3\", \"pge\", \"avx\", \"clflush\", \"sep\", \"syscall\", \"vme\", \"dtes64\", \"tsc\", \"xsave\", \"vmx\", \"xtpr\", \"cmov\", \"pcid\", \"est\", \"pat\", \"monitor\", \"smx\", \"lm\", \"msr\", \"nx\", \"fxsr\", \"tm\", \"sse4.1\", \"pae\", \"sse4.2\", \"pclmuldq\", \"acpi\", \"tsc-deadline\", \"mmx\", \"osxsave\", \"cx8\", \"mce\", \"mtrr\", \"rdtscp\", \"ht\", \"dca\", \"lahf_lm\", \"pdcm\", \"mca\", \"pdpe1gb\", \"apic\", \"sse\", \"pse\", \"ds\", \"pni\", \"tm2\", \"aes\", \"sse2\", \"ss\", \"pbe\", \"de\", \"fpu\", \"cx16\", \"pse36\", \"ds_cpl\", \"popcnt\", \"x2apic\"], \"topology\": {\"cores\": 6, \"threads\": 2, \"sockets\": 2}}",
"current_workload": 0,
"disk_available_least": 752,
"free_disk_gb": 878,
"free_ram_mb": 117633,
"host_ip": "10.240.0.200",
"hypervisor_hostname": "compute01.local.lan",
"hypervisor_type": "QEMU",
"hypervisor_version": 2000000,
"id": 1,
"local_gb": 971,
"local_gb_used": 93,
"memory_mb": 128897,
"memory_mb_used": 11264,
"os-pci:pci_stats": [],
"running_vms": 5,
"service": {
"disabled_reason": null,
"host": "compute01",
"id": 25
},
"state": "up",
"status": "enabled",
"vcpus": 24,
"vcpus_used": 6
}
}
To interact with Google Cloud APIs using curl, you have to pass a token in the Authorization request header. This is tedious to do manually every time and the token will eventually expire, so you can create a command line alias to make this process easier. The generated token will map to the Google account you currently have authenticated with gcloud (verify which Google account is authenticated with the gcloud config list command).
The following command is copied from the Google Cloud Run Authenticating developers, services, and end users documentation:
alias gcurl='curl --header "Authorization: Bearer $(gcloud config config-helper --format=value\(credential.access_token\) --force-auth-refresh)"'