Unlocking the Power of XML with xq and YAML with yq: A Beginner's Guide

XML (eXtensible Markup Language) has been a staple in data exchange formats for decades, particularly in enterprise systems, configuration files, and web services. However, navigating and processing XML data from the command line can be challenging without the right tools. Enter xq, a powerful command-line utility designed to make XML processing as simple and flexible as working with JSON. Similarly, yq is a robust command-line tool tailored for processing YAML (Yet Another Markup Language) files with the flexibility of jq.

If you're familiar with jq for JSON, xq and yq will feel like natural extensions for your XML and YAML processing needs.

What are xq and yq?

• xq: A command-line tool that extends the capabilities of jq to XML data. It allows you to convert XML to JSON, process it using the robust filtering and transformation capabilities of jq, and then convert it back to XML if needed.

• yq: A command-line YAML processor that leverages the power of jq for JSON, providing a similar set of tools for querying and manipulating YAML data.

Both tools provide a seamless way to handle XML and YAML data with the same ease and flexibility as JSON.

Why Use xq and yq?

Here are some compelling reasons to add xq and yq to your command-line toolkit:

• Simplified Data Processing: Both xq and yq leverage the power of jq to handle XML and YAML, allowing for concise and readable scripts.

• Flexible Data Transformation: Easily convert XML or YAML to JSON, manipulate it, and transform it back, giving you the best of both worlds.

• Powerful Filtering and Querying: Use jq's rich set of functions to filter, map, and transform XML and YAML data as if they were JSON.

• Automation Friendly: Ideal for use in shell scripts and automation tasks where XML or YAML data is prevalent.

Getting Started with xq and yq

Installation

Before you can start using xq or yq, you need to install yq, a lightweight and powerful command-line YAML/XML processor that comes bundled with xq support. Here’s how to install it on different operating systems:

• macOS: Use Homebrew to install yq with the following command:

brew install yq

• Linux: Download the binary from GitHub Releases and place it in your executable path:

wget https://github.com/mikefarah/yq/releases/download/v4.0.0/yq_linux_amd64 -O /usr/bin/yq
chmod +x /usr/bin/yq

• Windows: Download the executable from GitHub Releases and add it to your PATH.

Proof of Concept Examples for xq

Let's dive into some practical examples that demonstrate how to use xq for XML processing directly from the command line. These examples will help you understand how xq simplifies handling XML data, making it as straightforward as processing JSON with jq.

Example 1: Converting XML to JSON

Suppose you have an XML file named books.xml that looks like this:

Erik T. Ray
39.95



Elliotte Rusty Harold
29.95

To convert this XML to JSON using xq, run the following command:

xq . books.xml

This will output the XML content as JSON:

{
"library": {
"book": [
{
"title": "Learning XML",
"author": "Erik T. Ray",
"price": 39.95
},
{
"title": "XML in a Nutshell",
"author": "Elliotte Rusty Harold",
"price": 29.95
}
]
}
}

Example 2: Filtering XML Data

Let's say you want to extract only the titles of the books from the XML. You can use jq filtering capabilities directly on the converted JSON:

xq '.library.book[] | .title' books.xml

The output will be:

"Learning XML"
"XML in a Nutshell"

Example 3: Modifying XML Data

Suppose you want to update the price of "Learning XML" to 34.95. You can achieve this by converting the XML to JSON, modifying it, and converting it back to XML:

xq '(.. | .book? | select(.title == "Learning XML") | .price) |= 34.95' books.xml

This will output the modified XML:

Erik T. Ray
34.95



Elliotte Rusty Harold
29.95

Example 4: Converting Back to XML

After processing and modifying your XML data as JSON, you can convert it back to its XML form using xq. Suppose you want to output the modified JSON as XML:

xq --xml-output '(.. | .book? | select(.title == "Learning XML") | .price) |= 34.95' books.xml

Example 5: Combining XML Processing in a Shell Script

You can use xq in a shell script to automate XML processing tasks. Here is a simple script to find books cheaper than $35 and print their titles:

!/bin/bash

xq '.library.book[] | select(.price < 35) | .title' books.xml

Running this script will output:

"XML in a Nutshell"

Proof of Concept Examples for yq

Now, let’s explore some examples of using yq to handle YAML data. YAML is often used for configuration files and data serialization in applications. yq allows you to manipulate YAML files with ease.

Example 1: Reading YAML Data

Suppose you have a YAML file named config.yaml:

database:
host: localhost
port: 5432
username: user
password: pass

To read this YAML file and output its content, you can use:

yq eval . config.yaml

The output will be:

database:
host: localhost
port: 5432
username: user
password: pass

Example 2: Filtering YAML Data

If you want to retrieve only the host and port fields of the database configuration, use:

yq eval '.database | {host, port}' config.yaml

The output will be:

host: localhost
port: 5432

Example 3: Modifying YAML Data

To change the database username to admin, run:

yq eval '.database.username = "admin"' config.yaml

The output will reflect the updated username:

database:
host: localhost
port: 5432
username: admin
password: pass

Example 4: Adding Data to YAML

If you want to add a new field under database for timeout set to 30, you can do:

yq eval '.database.timeout = 30' config.yaml

The output will show the updated YAML:

database:
host: localhost
port: 5432
username: user
password: pass
timeout: 30

Example 5: Scripting with yq

Here’s a shell script to check if the timeout setting exists in the YAML and add it if it doesn’t:

!/bin/bash

if ! yq eval '.database.timeout' config.yaml > /dev/null; then
yq eval '.database.timeout = 30' config.yaml -i
fi

This script checks for the timeout setting and adds it if missing.

Conclusion

These examples demonstrate the versatility and power of xq and yq for handling XML and YAML data. With xq and yq, you can easily convert data to JSON, apply complex queries and transformations, and convert it back—all with simple command-line instructions. By integrating these tools into your toolkit, you gain the ability to manipulate XML and YAML data with the same ease and flexibility that jq provides for JSON, making them valuable tools for developers and data engineers alike.

Imported from rifaterdemsahin.com · 2024