Multiple Partition Keys in Azure Cosmos DB (Part 3) – Azure Functions with Change Feed Trigger

Welcome to part 3 in this three-part series on using the Azure Cosmos DB change feed to implement a synchronized solution between multiple containers that store the same documents, each with a different partition key.

Start with part 1 for an overview of the problem we’re trying to solve. In a nutshell, we’re using the change feed to synchronize two containers, so that changes made to one container are replicated to the other. Thus the two containers have the same data in them, but each one is partitioned by a different partition key to best suit one of two common data access patterns. Part 1 presented a solution that queries the change feed directly. In part 2, I showed you how to use the Change Feed Processor (CFP) library instead, which provides numerous advantages over the low-level approach in part 1.

This post concludes the three-part series, and shows how to deploy the solution to execute in a serverless cloud environment using Azure Functions.

Building the Azure Functions Project

As you learned in part 2, the benefits of using the CFP library are clear. But we still need a way to deploy our host to run in Azure. We could certainly refactor the CfpLibraryHost console application as a web job. But the simplest way to achieve this is by using Azure Functions with a Cosmos DB trigger.

If you’re not already familiar with Azure Functions, they let you write individual methods (functions), which you deploy for execution in a serverless environment hosted on Azure. The term “serverless” here means without also having to create an Azure app service to deploy your host.

In the current ChangeFeedDemos solution, create a new Azure Functions project and name it CfpLibraryFunction. Visual Studio will offer up a selection of “trigger” templates to choose from, including the Cosmos DB Trigger that we’ll be using. However, we’ll just choose the Empty template and configure the project manually:

Next, add the Cosmos DB WebJob Extensions to the new project from the NuGet package Microsoft.Azure.WebJobs.Extensions.CosmosDB:

Now create a new class named MultiPkCollectionFunction in the CfpLibraryFunction project with the following code:

using Microsoft.Azure.Documents;
using Microsoft.Azure.Documents.Client;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;

namespace CosmosDbDemos.MultiPkCollection.ChangeFeedFunction
{
  public static class MultiPkFunction
  {
    private const string CosmosEndpoint = "https://<account-name>.documents.azure.com:443/";
    private const string CosmosMasterKey = "<account-key>";

    private static readonly DocumentClient _client;

    static MultiPkFunction()
    {
      _client = new DocumentClient(new Uri(CosmosEndpoint), CosmosMasterKey);
    }

    [FunctionName("MultiPkCollectionFunction")]
    public static void Run(
      [CosmosDBTrigger(
        databaseName: "multipk",
        collectionName: "byCity",
        ConnectionStringSetting = "CosmosDbConnectionString",
        LeaseCollectionName = "lease")]
      IReadOnlyList<Document> documents,
      ILogger log)
    {
      foreach (var document in documents)
      {
        var jDocument = JsonConvert.DeserializeObject<JObject>(document.ToString());

        if (jDocument["ttl"] == null)
        {
          var stateCollUri = UriFactory.CreateDocumentCollectionUri("multipk", "byState");
          _client.UpsertDocumentAsync(stateCollUri, document).Wait();
          log.LogInformation($"Upserted document id {document.Id} in byState collection");
        }
        else
        {
          var stateDocUri = UriFactory.CreateDocumentUri("multipk", "byState", document.Id);
          _client.DeleteDocumentAsync(stateDocUri, new RequestOptions
          {
            PartitionKey = new PartitionKey(jDocument["state"].Value<string>())
          }).Wait();
          log.LogInformation($"Deleted document id {document.Id} in byState collection");
        }
      }
    }
  }
}

Note that this is all the code we need. While the CFP library host project required less code than querying the change feed directly, it still required additional code to build and host the processor – plus you need a full Azure app service to deploy in the cloud.

Using an Azure Function, it’s super lean. We have the same minimal business logic as before, and still leverage all the CFP library goodness as before, but being “serverless,” we’re ready to deploy this as a standalone function in the cloud, without creating a full Azure app service. Plus, you can test the Azure Function locally using Visual Studio and the debugger, so that you can deploy to the cloud with confidence. Folks, it just doesn’t get much better than this!

The CosmosDBTrigger binding in the signature of the Run method causes this code to fire on every notification that get raised out of the CFP library that’s watching the byCity container for changes. The binding tells the trigger to monitor the byCity collection in the multipk database, and to persist leases to the container named lease. It also references a connection string setting named CosmosDbConnectionString to obtain the necessary endpoint and master key, which we’ll configure in a moment.

The actual implementation of the Run method is virtually identical to the ProcessChangesAsync method in our CFP library host project from part 2. The only difference is that we log output using the ILogger instance passed into our Azure Function, rather than using Console.WriteLine to log output in our CFP library host project.

Setting the connection string

Next we need to set the connection string for the trigger that we are referencing in the signature of the Run method. Azure Functions support configuration files similar to the way appsettings.json, app.config, or web.config is used for typical .NET applications. For Azure Functions, this file is named local.settings.json. Open this file and add the CosmosDbConnectionString setting that the trigger will use to monitor the byCity collection for changes:

{
  "IsEncrypted": false,
  "Values": {
    "CosmosDbConnectionString": "AccountEndpoint=https://<account-name>.documents.azure.com:443/;AccountKey=<account-key>;",
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet"
  }
} 

Testing locally

It’s very easy to test and debug our code locally, before deploying it to Azure.

Let’s see it work by running the two projects ChangeFeedDirect and CfpLibraryFunction at the same time. The ChangeFeedDirect console app provides an interactive menu for creating the database with the byCity, byState, and lease containers, and also for inserting, updating, and deleting documents in byCity container. The CfpLibraryFunction app, meanwhile, sits and waits to be notified of changes as they occur in the byCity container, so that they can be synchronized to the byState container.

Start the first console app (ChangeFeedDirect) and run the DB command to drop and recreate the database with empty byCity and byState containers. Then run CL to create the lease container.

Without closing the first console app, start the second one (CfpLibraryFunction). To do this, right-click the CfpLibraryFunction project in Solution Explorer and choose Debug, Start New Instance. You will see a console app open up which hosts the local webjob for our Azure function:

Wait a few moments until the local Azure Functions host indicates that the application has started:

Back in the first console app (ChangeFeedDirect), run the CD command to create three documents in the byCity container. Then run the UD command to update a document, and the DD command to delete another document (by setting its ttl property). Do not run SC to manually sync with the direct change feed queries from part 1. Instead, sit back and watch it happen automatically with the CFP library under control of the local Azure Functions webjob host:

Before we deploy to Azure, let re-initialize the database. Back in the first console app (ChangeFeedDirect), run the DB command to drop and recreate the database with empty byCity and byState containers. Then run CL to create the lease container.

Deploying to Azure

Satisfied that our code is working properly, we can now deploy the Azure function to run in the cloud.

Right-click the CfpLibraryFunction project and choose Publish.

Select Create New and then click Publish.

Assign a name for the deployed function, and select the desired subscription, resource group, hosting plan, and storage account (or just accept the defaults):

Now click Create and wait for the deployment to complete.

Over in the Azure portal, navigate to All Resources, filter on types for App Services, and locate your new Azure Functions service:

Click on the App Service, and then click Configuration:

Next, click New Application Setting:

This is the cloud equivalent of the local.settings.json file, where you need to plug in the same CosmosDbConnection string setting that you provided earlier when testing locally:

Now click Update, Save, and close the application settings blade.

You’re all done! To watch it work, click on MultiPkCollectionFunction, scroll down, and expand the Logs panel:

Back in the ChangeFeedDirect console app, run the CD command to create three documents in the byCity container. Then run the UD command to update a document, and the DD command to delete another document (by setting its ttl property). Do not run SC to manually sync with the direct change feed queries from part 1. Instead, sit back and watch it happen automatically with the CFP library running in our Azure function. Within a few moments, the Azure function trigger fires, and our code synchronizes them to the byState container:

What’s Next?

A few more pointers before concluding:

Conclusion

This concludes the three-post series on using the Cosmos DB change feed to synchronize two containers with different partition keys over the same data. Thanks for reading, and happy coding!

Advertisements

One Response to “Multiple Partition Keys in Azure Cosmos DB (Part 3) – Azure Functions with Change Feed Trigger”

  1. Multiple Partition Keys in Azure Cosmos DB (Part 2) – Using the Change Feed Processor Library | Lenni's Technology Blog Says:

    […] tune in to part 3, where I’ll conclude this three-part series by showing you how to deploy our solution to Azure […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: