Large JSON Files are great for automated processing, but hard to read and understand. By default the AWS CLI presents its users with large to very large JSON output that is simply unusable for your everyday work. Thats why many people still rely on the AWS Console to get information about their deployed resources, when this is very unproductive. You have to leave your editor and terminal to go look up some information in the AWS Console that could be easily shown in your terminal as well.
The --query option allows you to use JMESPATH, a JSON query language, to limit and select the JSON attributes from an awscli command. This makes it easy to limit the JSON response to the data you’re actually interested in.
The following very simple example will only select the names of CloudFormation stacks from the larger aws cloudformation describe-stacks call.
# aws cloudformation describe-stacks --query "Stacks[].StackName"
[
"tslw-website",
"tslw-backend"
]
This together with the --output option and its text or table output give you a lot of control over how to present information from the AWS CLI. They can be easily chained together to build powerful tools, for example awsinfo is written completely in bash and uses the --query option extensively to give you quick access to the most important information on your AWS resources. You can also check out the awsinfo code to see many examples of JMESPath.
This makes it even clearer that the AWS CLI is great as a Shell SDK. You can easily write shell scripts or Makefiles that call the AWS CLI with the --query and --output option.
As the Query language behind the --query option JMESPath is a very powerful language. The following examples cover some of the more common use cases when used with the AWS CLI, but JMESPath can do way more than this. There is also a great tutorial and collection of examples on their page, though it can be a bit overwhelming at first.
The same as the example above where we just want to know the StackName for every Stack. [] (you can also use [*]) after Stacks will iterate over all stacks in the Stacks array and select the properties of each.
# aws cloudformation describe-stacks --query "Stacks[].StackName"
[
"tslw-website",
"tslw-backend"
]
This is important when you use --output text as it will put every StackName in its own line so they can be processed by other tools like xargs, sed, or awk.
# aws cloudformation describe-stacks --query "Stacks[].[StackName]"
[
[
"tslw-website"
],
[
"tslw-backend"
]
]
This is especially helpful when you use the --output table option as it allows you to name the columns of your table and thus decide on the order. We’re adding a number before every field as --output table sorts the columns by name. If you want to add numbers and a . separator you have to put the hash key in "". This example is taken directly from awsinfo where it is used for the awsinfo cfn command.
# aws cloudformation describe-stacks --query "Stacks[].{\"1.Name\":StackName,\"2.Status\":StackStatus,\"3.CreationTime\":CreationTime}"
[
{
"1.Name": "tslw-website",
"2.Status": "CREATE_COMPLETE",
"3.CreationTime": "2017-03-23T15:25:11.082Z"
},
{
"1.Name": "tslw-backend",
"2.Status": "UPDATE_COMPLETE",
"3.CreationTime": "2017-02-10T13:13:24.004Z"
}
]
JMESPath doesn’t have a shorthand syntax for selecting properties and reusing the property name as key (something like --query Stacks[].{StackName, StackStatus}), so you have to set the hash keys yourself, even if they should have the same name as the property already has.
This is especially helpful when you use the --output text option together with text processing tools like awk as it will print out each sublist on a tab separated line.
# aws cloudformation describe-stacks --query "Stacks[].[StackName, StackStatus, CreationTime]"
[
[
"tslw-website",
"CREATE_COMPLETE",
"2017-03-23T15:25:11.082Z"
],
[
"tslw-backend",
"UPDATE_COMPLETE",
"2017-02-10T13:13:24.004Z"
]
]
In this example we’re only interested in Stacks that match CREATE_COMPLETE, you can also use !=, <, > or many other comparison operators and functions
# aws cloudformation describe-stacks --query "Stacks[?StackStatus=='CREATE_COMPLETE'].StackName"
[
"tslw-website",
]
JMESPath supports many different functions that you can use in your query. They are especially helpful in Filters. In the following example we’re only interested in stacks where the StackName contains backend. Of course we could then combine this with a more complex query like above to select different properties and build a complex hash.
# aws cloudformation describe-stacks --query "Stacks[?contains(StackName,'backend')].StackName"
[
"tslw-backend",
]
This is used extensively in awsinfo to allow you to give only part of a resource name and match all resources that have the part in their name (or other properties depending on the resource).
In this example we’re sorting by CreationTime and you can see the results are sorted differently than in the example before.
# aws cloudformation describe-stacks --query "sort_by(Stacks,&CreationTime)[].{\"1.Name\":StackName,\"2.Status\":StackStatus,\"3.CreationTime\":CreationTime}"
[
{
"1.Name": "tslw-backend",
"2.Status": "UPDATE_COMPLETE",
"3.CreationTime": "2017-02-10T13:13:24.004Z"
},
{
"1.Name": "tslw-website",
"2.Status": "CREATE_COMPLETE",
"3.CreationTime": "2017-03-23T15:25:11.082Z"
}
]
Developing complex JMESPath queries can be quite slow when you have to call the AWS CLI constantly. To solve this there is the jp command line tool which is a Go implementation of JMESPath. Because you can run it on your local system it obviously speeds up development and debugging considerably.
In the following example we’re storing the JSON file from aws cloudformation describe-stacks in a stacks.json file first and then use jp to read the file and interpret the output.
# aws cloudformation describe-stacks > stacks.json
# jp -f stacks.json "Stacks[].StackName"
[
"tslw-website",
"tslw-backend",
]
As the JSON file is now read from the local filesystem this is of course very fast.
The official JMESPath page also has a browser JMESPath evaluation feature, but its a bit cumbersome to use in comparison to simply running in your local terminal.