Creating File Generation Templates

DATPROF generator templates are Apache Velocity templates used to generate structured synthetic data files. A template can generate formats such as CSV, JSON, XML, SQL INSERT statements, or custom text-based files.

Templates combine static text with dynamic DATPROF generator expressions. These expressions produce values such as names, dates, IDs, email addresses, numbers, booleans, and values from predefined seed files.

A simple template can generate one record. A more advanced template can generate thousands of records, nested structures, optional fields, randomized lists, and sequential IDs.

In the following examples, I will use XML as output format, but that’s just for the example. At the end you will find an overview of all available generators with their syntax.

Simple example

Start by generating a first name:

$dpf.firstName

This outputs for example

Linna

Combine this with static text, for example:

<Person>
  <FirstName>$dpf.firstName</FirstName>
</Person>

This outputs

<Person>
  <FirstName>Linna</FirstName>
</Person>

or

<Person FirstName="$dpf.firstName"></Person

Outputs

<Person FirstName="Linna"></Person

<Person>
  <FirstName>$dpf.firstName</FirstName>
</Person>
<Person>
  <FirstName>$dpf.firstName</FirstName>
</Person>

This returns:

<Person>
  <FirstName>Linna</FirstName>
</Person>
<Person>
  <FirstName>Zejad</FirstName>
</Person>

Generator options

Some generators offer options, such as selecting a country or generating a random number between 18 and 50. You can use the UI option using Insert → Generator… to see the available options.

<Person>
  <FirstName>$dpf.firstName.languages("US","DE")/</FirstName>
  <Age>$dpf.randomInteger.min(18).max(50)</Age>
</Person>

Will return

<Person>
  <FirstName>Linna</FirstName>
  <Age>35</Age>
</Person>

Repeat

Generating 100 of these person objects by creating a template with a lot repeating lines is inefficient. We can use the repeat method instead.

Use the following to repeat specific text.

The following template

#@repeat (100)
  <Person>
    <FirstName>$dpf.firstName</FirstName>
  </Person>
#end

will return

<Person>
  <FirstName>Linna</FirstName>
</Person>
<Person>
  <FirstName>Zejad</FirstName>
</Person>
...

Nested repeats

Sometimes it’s necessary to create nested repeats. That can be easily done using the following template in which you use a #@repeat/#end inside another #@repeat/#end

#@repeat (2)
  <Person>
    <FirstName>$dpf.firstName</FirstName>
    <Childeren>
        #@repeat (2)
        <Child>$dpf.firstName</Child>
        #end
    </Childeren>
  </Person>
#end

Will return

<Person>
  <FirstName>Duddy</FirstName>
  <Childeren>
    <Child>Anamaria</Child>
    <Child>Krystle</Child>
  </Childeren>
</Person>
<Person>
  <FirstName>Tadeusz</FirstName>
  <Childeren>
    <Child>Sarmad</Child>
    <Child>Hanna-Louise</Child>
  </Childeren>
</Person>

Random repeat

In the above examples the repeat is always fixed. But maybe we want to randomly repeat between two numbers. We can combine the random number generator and the repeat method to randomly repeat text.

#@repeat (2)
  <Person>
    <FirstName>$dpf.firstName</FirstName>
    <Childeren>
        #@repeat ($dpf.randomInteger.min(1).max(5) )
        <Child>$dpf.firstName</Child>
        #end
    </Childeren>
  </Person>
#end

Output: see that one person has 3 children’s and the other one has two

<Person>
  <FirstName>Thanachai</FirstName>
  <Childeren>
    <Child>Claes-Christian</Child>
    <Child>Aayushma</Child>
    <Child>Toveline</Child>
  <Child>Farhiyo</Child>
  </Childeren>
</Person>
<Person>
  <FirstName>Mamadi</FirstName>
  <Childeren>
    <Child>Daud</Child>
    <Child>Bery</Child>
  </Childeren>
</Person>

Repeat with separator

Sometimes you want to repeat text separated by a character or other text, but not after the last item. For example, when building a list of JSON objects inside an array.

Use #separator(",") to set the separator. It separates each repeated item with the separator except the last.

For example:

#@repeat (3)
{ "firstname" : "$dpf.firstName"}#separator(",")
#end

This will output

{ "firstname" : "Mille"},
{ "firstname" : "Ana Felicia"},
{ "firstname" : "Aron"}

Controlling line-endings

If you want to prevent new lines, but keep a well formatted template, you can use the ## to prevent a new line.

Example without ##:

[#@repeat (3)$dpf.firstName#separator(",")#end]

Outputs

[ Carlos Mariano ,  Samreen ,  Khlosee  ]

Or more cleanly formated

[##
#@repeat (3) 
$dpf.firstName #separator(",")## 
#end
]

Also output

[Adília,Maria Maite,Henny-Ilona]

Sequences

If you need to create a sequence of numbers, we need to define the sequence outside the repeat, otherwise it will not generate sequential numbers, but constantly the same. In the example below a sequence $seq is defined outside the repeat, and used as just $seq inside the repeat.

#set ($seq = $dpf.sequence.start(10).step(2))
#@repeat (2)
  <Person id="$seq">
    <FirstName>$dpf.firstName.next</FirstName>
  </Person>
#end

Will produce

<Person id="12">
  <FirstName>Veenu</FirstName>
</Person>
<Person id="14">
  <FirstName>Jakkrit</FirstName>
</Person>

Unique number generators (SSN, BSN, IBAN, …)

Similar to sequences, when generating unique numbers inside a repeat, you will have to define them as a variable outside the repeat. If you are using the Insert + button to insert a generator snipper, this will be done automatically.

#set( $SSN_GENERATOR = $dpf.ssn.separator("-") )
#@repeat(300)
  $SSN_GENERATOR
#end

This returns:

001-01-0001
001-01-0002
001-01-0003
001-01-0004
001-01-0005
...

Regex-based values

Use $dpf.regex(...) when a value must match a specific pattern.

Examples:

$dpf.regex("[A-Z]{2}[0-9]{6}")
$dpf.regex("[2-9][0-9]{2}[2-9][0-9]{2}[0-9]{4}")
+1 $dpf.regex("[2-9][0-9]{2}[2-9][0-9]{2}[0-9]{4}")

Will outputs:

KY661284
4437147884
+1 2126242960

Regex generators are useful for custom codes, account numbers, phone numbers, license plates, and IDs with strict formats.

Transformations

You can also transform the output of the generator with a extra option to for example upper or lower case the value. For a full list of transformers use the autocompletion of generator.

<FirstName>$dpf.firstName.value.toUpperCase()</FirstName>
<FirstName>$dpf.firstName.value.toLowerCase()</FirstName>
<FirstName>$dpf.firstName.value.substring(0,1)</FirstName>

Will return

<FirstName>SALEBAN</FirstName>
<FirstName>maymun</FirstName>
<FirstName>G</FirstName>

Reusing generated values

There are two methods of using generated values twice.

1. Store the value of the generator inside a variable

2. Create a specific implementation of the generator

Store the value of the generator inside a variable

Using the #set, stores the generator's result in a variable for reuse. Remember to use .next to actually store the value of the generator. Otherwise you will only store the complete generator and calling the variable will generate new values every time you use the variable name.

#set ($firstName = $dpf.firstName.next)
<Person>
  <FirstName>$firstName</FirstName>
</Person>
<Person>
  <FirstName>$firstName</FirstName>
</Person>

This returns:

<Person>
  <FirstName>Linna</FirstName>
</Person>
<Person>
  <FirstName>Linna</FirstName>
</Person>

Create a specific implementation of the generator

You can create a specific instance of the generator. Use .value to retrieve the current value and .next to generate the next value.

#set ($firstName = $dpf.firstName)
<Person>
  <FirstName>$firstName</FirstName>
</Person>
<Person>
  <FirstName>$firstName.value</FirstName>
</Person>
<Person>
  <FirstName>$firstName.next</FirstName>
</Person>

This returns:

<Person>
  <FirstName>Linna</FirstName>
</Person>
<Person>
  <FirstName>Linna</FirstName>
</Person>
<Person>
  <FirstName>Veenu</FirstName>
</Person>

Generating dates, datetimes and times

Generating dates can be complex. Dates comes in many forms and formats.

To generate a date, you can use the following:

$dpf.date
$dpf.dateTime
$dpf.time

This will return a random day

1911-12-21
2011-07-29 22:12:08.000 Europe/Amsterdam
17:02:27

You can also generate a random date between two other dates / datetimes or times.

$dpf.date.min("1900-01-01").max("2000-12-31")
## or
$dpf.date.min(1900, 1, 1).max(2000,12,31)

$dpf.dateTime.min("1900-01-01T00:00:00Z").max("2099-12-31T23:59:59Z")
## or
$dpf.dateTime.min(1900, 1, 1, 0, 0, 0).max(2099, 12, 31, 23, 59, 59)

$dpf.time.min("00:00:00").max("23:59:59")

To get the current date, you can use the following

$dpf.date.now

To format a date, you can use the .format option.

$dpf.date.format("dd-MMMM-yyyy")

This will generate

28-July-1921

You can use the .format also in combination with other options, like:

$dpf.date.min("1900-01-01").max("2000-12-31").format("dd-MMMM-yyyy")

If you need to generate multiple dates inside a template you can also use the config option to set the default formatting that should be used:

$dpf.config.dateFormat("dd-MMM-yy")##
$dpf.date.min("1900-01-01").max("2000-12-31")
$dpf.date.min("1900-01-01").max("1920-12-31")
$dpf.date.min("1950-01-01").max("2010-12-31")

Calculating dates

Sometimes you need to create two dates offset from each other, such as a begin and end date.

The code below generates a variable called beginDate with a random date between January 1, 1940, and January 1, 2005.

To calculate an endDate, it generates a random date between beginDate plus 30 days and today minus 3 weeks.

#set($beginDate = $dpf.date.min("1940-01-01").max("2005-01-01").value)
"BeginDate": "$beginDate",
"EndDate": "$dpf.date.min($beginDate.plusDays(30)).max($dpf.date.now.minusWeeks(3))",

Conditional Logic

Use Velocity #if, #else, and #end to generate values conditionally. In the following example it will generate a random gender using the randomPick, based on the outcome it will call different generators.

Example:

#set($gender = $dpf.randomPick("Male", "Female"))
#if($gender == "Male")
  $dpf.firstNameMale.languages("US")
#else
  $dpf.firstNameFemale.languages("US")
#end
$gender

Using your own data

There could be cases in which you want to use your own seed data instead of being depended on the provided generators .

For small data sets, you can use the randomPick or weightedPick

$dpf.randomPick("Male", "Female")
$dpf.weightedPick.weights(6, 2, 2).options("Basic", "Standard", "Premium")

This will outputs

Female
Basic

You can also use your own data, for example a custom CSV file containing seed data.

In the following example values from three columns of a customers.csv will be read and used randomly to built a JSON object.

#set($csv = $dpf.csvFile("customers.csv"))
#@repeat(3)
{
  "firstname": "$csv.next.FIRST_NAME",
  "lastname": "$csv.value.LAST_NAME"
  "gender": "$csv.value.GENDER"
}#separator(",")
#end

Sequential mode

For some use cases you want to fully control in which order you want to use the data from the CSV file. With the .sequential mode, you can sequentially loop through the file. If all records are used, it will start from first record.

#set($csv = $dpf.csvFile("customers.csv").sequential)
#@repeat(3)
{
  "firstname": "$csv.next.FIRST_NAME",
  "lastname": "$csv.value.LAST_NAME"
  "gender": "$csv.value.GENDER"
}#separator(",")
#end

Useful utilities