# curl

Get HTTP Headers Only

Grab only the HTTP headers, instead of the entire HTTP response and content, from a URL:

curl -I $URL

To ensure you see the entire request/response chain, add the -L command line switch to follow redirects:

curl -IL $URL

Check HTTP Redirects

If you maintain a lot of domain redirects, it is important to make sure they continually work. The following curl command can be used to loop through a text file containing URLs to check. The output for each check will be the final HTTP status code and URL.

curl -sL -w "%{http_code} %{url_effective}\\n" "$URL" -o /dev/null

For example, to verify https://old-site.com redirects to https://new-site.com:

curl -sL -w "%{http_code} %{url_effective}\\n" "https://old-site.com" -o /dev/null

Will output:

200 https://new-site.com

Pass Custom Host Header

If you do local web development, you probably have to frequently access your website or web application from localhost. In most situations this shouldn’t be a problem, but for those situations where it is (such as testing Apache VirtualHosts or nginx Server Blocks), you can easily change the Host header with curl to tell the website or web application where you want to go.

curl -i -H 'Host: example.com' $URL

Alternatively, temporarily add a record in your workstation’s /etc/hosts file pointing the domain name of the URL you are accessing to 127.0.0.1.

Force Request through a Specific IP Address

Instead of changing your /etc/hosts file, force a curl request to go through an IP address that differs from what the hostname resolves to in public DNS.

curl -IL $HOSTNAME --resolve $HOSTNAME:$PORT:$SPECIFIC-IP

Shell Script: curlloop.sh

Create file curlloop.sh with the following content:

#!/bin/bash

url_path="${1}"
request_count=1

while true
do
echo -n "${request_count}: "
curl -sL -w "%{http_code} %{time_total} %header{cf-cache-status} %{url_effective}\\n" "${url_path}" -o /dev/null
sleep .5
((request_count++))
done

Set the executable permission:

chmod +x curlloop.sh

Run the shell script with the following command:

./curlloop.sh refcli.com

Example output:

curlloop.sh refcli.com
1: 200 0.341181 DYNAMIC https://refcli.com/
2: 200 0.158135 DYNAMIC https://refcli.com/
3: 200 0.169257 DYNAMIC https://refcli.com/
4: 200 0.179654 DYNAMIC https://refcli.com/
5: 200 0.185863 DYNAMIC https://refcli.com/

POST Data and Content Type to URL

POST plain text data to URL:

curl -X POST -d 'plain-text' -H "Content-Type:text/plain" $URL

POST JSON data to URL:

curl -X POST -d '{"key1":"value1","key2":"value2"}' -H "Content-Type:application/json" $URL

Pass JSON Payload to curl Using a File, Bash Variable, or HERE Document

File

Create a file called payload, put valid JSON in it, and reference it with the curl command using the following command:

curl -d @./payload -H "X-Auth: $TOKEN" "https://api.example.com/api/query"

Bash Variable

If you are using the curl command within a shell script and want to pass JSON to it from a Bash variable in the same shell script, use the following method:

PAYLOAD='
[{
    "auth": {
        "identity": {
            "methods": ["password"],
            "password": {
                "user": {
                    "name": "USERNAME",
                    "domain": {
                        "id": "default"
                    },
                    "password": "PASSWORD"
                }
            }
        },
        "scope": {
            "project": {
                "name": "PROJECT",
                "domain": {
                    "id": "default"
                }
            }
        }
    }
}]'

RESULT=$(curl -s -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<< "$PAYLOAD")

HERE Document

Another method you can use is a HERE document. This eliminates the need for the $PAYLOAD Bash variable.

RESULT=$(curl -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<PAYLOAD
[{
    "auth": {
        "identity": {
            "methods": ["password"],
            "password": {
                "user": {
                    "name": "USERNAME",
                    "domain": {
                        "id": "default"
                    },
                    "password": "PASSWORD"
                }
            }
        },
        "scope": {
            "project": {
                "name": "PROJECT",
                "domain": {
                    "id": "default"
                }
            }
        }
    }
}]
PAYLOAD)

Using curl to Query an API

Many of the tools used today are built on top of some sort of API. Those tools abstract all of the granular details and information that comes through an API request and response. Sometimes it is useful to see what goes on behind the scenes, especially when it comes time to troubleshoot.

The following example uses the curl command to query the OpenStack API to show the details about a particular hypervisor managed by OpenStack Nova.

Get a token

Before you can do anything with any API, you need a token. With OpenStack it’s no different. OpenStack Keystone is the identity and authentication service that is used to generate tokens for end users and services.

The Keystone v2 API is quickly being deprecated. I have only included it for posterity’s sake, so jump to the v3 API below to generate a token:

curl -s \
     -X POST https://10.240.0.100:5000/v2.0/tokens \
     -H "Content-Type: application/json" \
     -d '{"auth": {"tenantName": "TENANT", "passwordCredentials":{"username": "USERNAME", "password": "PASSWORD"}}}'

What follows is the Keystone v3 API. You will need a valid username and password already stored in OpenStack Keystone to generate a token:

curl -i \
     -H "Content-Type: application/json" \
     -d '
        { "auth": {
            "identity": {
              "methods": ["password"],
              "password": {
                "user": {
                  "name": "USERNAME",
                  "domain": { "id": "default" },
                  "password": "PASSWORD"
                }
              }
            },
            "scope": {
              "project": {
                "name": "PROJECT",
                "domain": { "id": "default" }
              }
            }
          }
        }' \
     https://10.240.0.100:5000/v3/auth/tokens

This will return a lot of JSON. Unfortunately, I did not make a copy of that JSON output when I originally created these notes. However, within that JSON will be a token that will be used in all of the subsequent commands. Let’s assume the token returned is 1234567890abcdefghijklmnopqrstuv.

Now that a token has been generated, I can begin querying the OpenStack Nova API for details on the particular hypervisor. For this example, I want details about compute01.local.lan, and I’m going to need its id:

curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/compute01.local.lan/search | python -m json.tool

Take note, the curl command is going to output JSON which can be difficult to read. I am piping the output to python -m json.tool to make it easier to read.

The above command returns the following JSON:

{
    "hypervisors": [
        {
            "hypervisor_hostname": "compute01.local.lan",
            "id": 1,
            "state": "up",
            "status": "enabled"
        }
    ]
}

Now that I have the id, I can query the details about compute01.local.lan:

curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/1 | python -m json.tool

The above command returns the following JSON which provides all of the details for that particular hypervisor:

{
    "hypervisor": {
        "cpu_info": "{\"vendor\": \"Intel\", \"model\": \"SandyBridge\", \"arch\": \"x86_64\", \"features\": [\"ssse3\", \"pge\", \"avx\", \"clflush\", \"sep\", \"syscall\", \"vme\", \"dtes64\", \"tsc\", \"xsave\", \"vmx\", \"xtpr\", \"cmov\", \"pcid\", \"est\", \"pat\", \"monitor\", \"smx\", \"lm\", \"msr\", \"nx\", \"fxsr\", \"tm\", \"sse4.1\", \"pae\", \"sse4.2\", \"pclmuldq\", \"acpi\", \"tsc-deadline\", \"mmx\", \"osxsave\", \"cx8\", \"mce\", \"mtrr\", \"rdtscp\", \"ht\", \"dca\", \"lahf_lm\", \"pdcm\", \"mca\", \"pdpe1gb\", \"apic\", \"sse\", \"pse\", \"ds\", \"pni\", \"tm2\", \"aes\", \"sse2\", \"ss\", \"pbe\", \"de\", \"fpu\", \"cx16\", \"pse36\", \"ds_cpl\", \"popcnt\", \"x2apic\"], \"topology\": {\"cores\": 6, \"threads\": 2, \"sockets\": 2}}",
        "current_workload": 0,
        "disk_available_least": 752,
        "free_disk_gb": 878,
        "free_ram_mb": 117633,
        "host_ip": "10.240.0.200",
        "hypervisor_hostname": "compute01.local.lan",
        "hypervisor_type": "QEMU",
        "hypervisor_version": 2000000,
        "id": 1,
        "local_gb": 971,
        "local_gb_used": 93,
        "memory_mb": 128897,
        "memory_mb_used": 11264,
        "os-pci:pci_stats": [],
        "running_vms": 5,
        "service": {
            "disabled_reason": null,
            "host": "compute01",
            "id": 25
        },
        "state": "up",
        "status": "enabled",
        "vcpus": 24,
        "vcpus_used": 6
    }
}

Google Cloud curl, or gcurl

To interact with Google Cloud APIs using curl, you have to pass a token in the Authorization request header. This is tedious to do manually every time and the token will eventually expire, so you can create a command line alias to make this process easier. The generated token will map to the Google account you currently have authenticated with gcloud (verify which Google account is authenticated with the gcloud config list command).

The following command is copied from the Google Cloud Run Authenticating developers, services, and end users documentation:

alias gcurl='curl --header "Authorization: Bearer $(gcloud config config-helper --format=value\(credential.access_token\) --force-auth-refresh)"'

References