Data Transformations in Azure with Liquid

As data is sent between two parties, or file contents need to be mapped from a source to target format, we can use a set of Azure services to process and map the data. A common way to achieve this is using the Liquid template language. This post is the first in a series about the Liquid language.

The Liquid Blog Series

Business Benefits and Use Cases

There are endless opportunities for transforming data from one format to another. ETL design tools such as Azure Data Factory, third-party tools such as Altova MapForce, or simply by writing XLST maps or running custom code. The tools require a runtime cost for the service and in most cases software licences or subscriptions. Custom code requires a container to execute the code in. Liquid allows us to use these transformations within a set of Azure tools without the need of extra services or licence costs.

I will outline a set of use cases where Liquid templates can help your business in this blog series.

THE LIQUID TEMPLATE LANGUAGE

The Liquid template language was created for the Shopify platform and contains a powerful set of features:

control flow
iteration
variables
templates
filters
data selectors

This allows us to, with relative ease, create advanced logic, formatting and data iterations. We can save this logic to a template and apply it within a logic flow. For example: processing a JSON body request within an API, transforming data within an XML file to an output JSON schema, or even formatting HTML or markup into flat file formats.

A common use case for Liquid is within Logic Apps. A workflow executes a mapping step from XML or JSON into XML, JSON or text. This is a low-code approach to data mapping, and templates can be reused between workflows for the same data mapping needs. Microsoft has an excellent documentation and tutorial on how to transform data with Logic Apps.

The downside is that you still need to write and understand the Liquid template language syntax. For developers, I highly recommend reading the syntax documentation, installing the Liquid extension for Visual Studio Code, which gives you syntax highlighting. I may also recommend using the LiquidJS Playground, which gives you some ability to test the template language.

				
					{% comment -%}
Short sample showing how iteration and data selectors with in Liquid
{% endcomment -%}
<ul id="products">
  {% assign products = data.payload.warehouse.products -%}
  {% for product in products -%}
    <li>
      <h2>{{product.name}}</h2>
      Only {{product.price | price }}

      {{product.description | prettyprint | paragraph }}
    </li>
  {% endfor -%}
</ul>

UNIT TESTS FOR LIQUID TEMPLATES

As we use developer tools and syntax to write Liquid templates, it would make sense to write unit test cases. We can ensure that the templates produce the correct result using a variety of data sources. I have not seen any useful implementations to achieve this level of quality assurance, so I have created an open source library that give Liquid unit test support.

More details can be found in the post article Unit Tests for Liquid Templates.

				
					// Arrange
var myObj = new MyObj( Title = "Title here");

// Act
var result = new LiquidParser()
    .SetContent(myObj, true) // 'true' => camelCase property names
    .Parse("{{ content.title }}")
    .Render();

// Assert
result.Should().NotBeEmpty("A result should have been returned");
result.Should().Be(myObj.Title, "The expected result should be returned");

LIVE PREVIEW SUPPORT

Using the LiquidJS Playground and the Liquid extension for Visual Studio Code still forces us to upload the template to a service, save changes, push example data and investigate the result. This is incredibly inconvenient while designing a large template with complex logic or deep data structures.Instead, I have created an open source project that runs a watcher agent that produces an output file while you compose the template. More details on this project, including a downloadable module for use with VS Code, or any other source code editor.

More details can be found in the post article Liquid Transformations with Live Preview.

Transforming API Requests

We can also use the Azure API Management service to change and transform data directly within the exposed endpoint. For some use cases, it is not needed to run any additional backend services or logic flows.

For a good intro to this approach, see Using Liquid Templates in API Management.