Importing a random set of customers using Mockaroo and the import SDK with Deno

When evaluating a headless commerce system, one of the first things you should do is get your own data into it. Importing data can sometimes be a “chicken or the egg” situation, so you need to have some things configured, such as product attributes, for an import to really work well. In this article, I am going to show how you can use the commercetools import API from Deno to import a list of randomly generated customers from Mockaroo, how you can upload them in batches and how to monitor progress.

The Import API uploads data to commercetools Composable Commerce. Its asynchronous design is suitable to sync a large amount of resources from external systems into projects.

The Import API has the following advantages:

• Asynchronous
  You can send bulk data at once, and your systems can do something else while your data is imported asynchronously.

• Resource specific
  You can separately import products, product variants and prices.

• Automatic dependency handling
  The Import API automatically handles data dependencies such as the parental relations of categories.

• API-first
  Unlike other CLI tools and UI tools, it gives you a better integration opportunity with your modern application infrastructure.

When you want to import random customers into commercetools, you can follow the schema of CustomerImport.

Since this is a pretty simple schema, it almost asks to be generated by something in a pseudo random fashion. Here comes Mockaroo to the rescue.

Mockaroo is an excellent SaaS tool to generate demo data. I have been using it for a long time and it is one of those tools you would like to have in your tool belt. 

With Mockaroo, you can define a schema that follows the schema of ImportCustomer:

On top of that schema, you can create an API that takes some parameters, like the number of customers you would like to generate. And so a list of random customers is created:

This simple Mockaroo API script generates 30 customers by default, but when adding a count parameter, it will use the count, like in the example below:

https://my.api.mockaroo.com/customers.json?key=00aa64c0&count=5

Now that we have more demo data than we could ever consume, let's start loading those customers into commercetools. In the previous article, I wrote about the Deno SDK for commercetools. This SDK also exposes an Import SDK that can be used in the following way:

This small snippet uses the TypeScript Import SDK to list the available import containers:

You can create an Import container using the following snippet:

This will output the following result:

The Import container can handle imports of all sorts. You can upload a list of items to the container, where the container will provide the compute to do the actual import of the resources. A container can be queried for its status to check the progress of the import job.

First, lets kick-off an import job:

This small import script does the following:

1. Create an import container with key: “my-import-container”

2. Imports 5 random customers from Mockaroo

3. It sends the 5 customers to the import container

The response is:

This indicates that each record in the job is processing. If you check the container again a couple of seconds later with an operation ID, the following response would be shown:

This indicates that this single resource is now imported into commercetools. This also allows us to query it using the SDK:

This shows the generated customer imported in commercetools:

So far so good for the basic techniques. Now let's create a bulk importer that imports large lists of customers and then wait for the import to complete.

To help with importing a large dataset of any resource type, I created a small batch importer. The importer takes an array of objects to import, creates a container when needed and sends the data in chunks of 20 objects to the Import SDK. After that, it waits for the import SDK to complete by polling each operation. To make the batch importer resource type agnostic, you need to provide a callback that does the actual import of the requested resource.

The batch importer can be used in the following way:

This will import 22 random customers from Mockaroo into commercetools. It will split the import in a batch of 20 and a batch of 2. After the import is fired to commercetools, it will wait for the importer to complete all items:

The next article in this series will showcase how to prepare for the next import: Products. But before we can import products, we need to get the commercetools configuration setup correctly. 

Roll up your sleeves for some infrastructure as code!

To learn more about the creation of the Deno SDK, read the first article of the series: A Deno SDK for commercetools.