--- version: "4.23" language: "en" --- # DATPROF Privacy Manual ![privacy_dark_full@8x.png](https://docs.datprof.com/__attachments/a_bd863a7c0bb3725eed5c3354d150fa71cbe6d11ef9b573cf941554fb5a0eaf86/privacy_dark_full@8x.png?cb=0b28090c42f8c874a0e71f7ca94de249) Privacy is a proprietary data-masking and generation tool used to help businesses maintain a high level of data-privacy compliance within their testing infrastructure. Privacy does this through an intuitive user interface that enables Privacy developers to link to an existing testing database, and create instructions to mask sensitive data, or to generate completely new sets of safe, synthetic data. This solution eliminates the need for manual database querying to update datasets, and gives testers clean, workable data that accurately portrays the production database without displaying sensitive data. For a more detailed look, please refer to [About DATPROF Privacy](https://docs.datprof.com/privacy/latest/about-datprof-privacy.md). ## In this manual In this manual you will find an exhaustive explanation of the features, functionality and options available to you as a Privacy developer. The manual aims to provide you with all the tools you need to use our software, and get the most out of it. It is, however, not a FAQ or knowledge base for known issues. For this, we have a separate knowledge base available on our documentation site. To navigate the manual, please use the left hand index to browse the articles, or use the search bar to search for a specific topic you'd like to know more about. If you feel any information is lacking we encourage you to contact us and describe what it is you're missing. The creation of high-quality material for our partners to use is important to DATPROF, and we're consistently working on expanding the amount of information we include with our software, and improving the readability of its content. Accompanying the manual, training material is available via the [https://datprof.atlassian.net/wiki/spaces/DPT](https://docs.datprof.com/privacy-training.md). The training material serves as a companion to this manual, and we suggest using both to gain a formative understanding of how to work with our software. This manual is continuously updated, and it is possible to select the version-appropriate material of the manual at the top-right of this page. --- version: "4.23" language: "en" --- # About DATPROF Privacy When testing with production data, security and privacy concerns must always be a top priority. Whether you're working with an internal testing team or outsourcing to an external party, it's crucial to avoid exposing sensitive data that could identify individuals or harm your business. Storing such data in easily accessible locations only increases the risk. By performing due diligence on data protection, you mitigate these risks, safeguard corporate information, and ensure compliance with evolving data privacy regulations. In short, data masking is not just important, it is becoming an essential requirement for businesses. Generally speaking, there are two ways to effectively comply with data privacy requirements. Data masking, where you take production data and change it so it can not be used to identify persons, or data generation, where you generate complete sets of data from scratch to emulate the production environment data. DATPROF Privacy can do both. ## The working principle of Privacy DATPROF Privacy does not process privacy-sensitive data directly. Instead, it works exclusively with database metadata. In simple terms, this means that DATPROF Privacy connects to the source database and uses information about its structure, such as table layouts and relationships, without accessing the actual sensitive data itself. This approach ensures privacy while still allowing for effective data masking and generation. Each deployment package created in DATPROF Privacy consists of a set of instructions that are executed directly on the database, ensuring that the data remains within the local environment. This approach guarantees that no sensitive data ever leaves the premises, maintaining privacy and security throughout the process. Imagine a source database with a table called "Customers" containing 15 columns. DATPROF Privacy can import the table and column names, such as "FIRST_NAME," and identify its datatype as VARCHAR, along with other relevant details. However, DATPROF Privacy does not access or know any of the actual values stored in the "FIRST_NAME" column. This is a crucial feature for maintaining data privacy, as the tool never handles or stores sensitive data. It only works with the structure and metadata of the database. DATPROF Privacy provides various methods for anonymizing, modifying, removing, or altering data. Its use is strictly intended for modifying test data, defined as data derived from production data or synthetic data sources, which can be replaced if lost or damaged. DATPROF Privacy is not recommended or supported for modifying production data, as this can lead to permanent data loss. ***Users should never alter data that cannot be easily restored and are always responsible for ensuring the correct use of DATPROF software.*** ## Using DATPROF Privacy Creating a masked database involves, in very general terms, performing the following steps: * Creating a new project. * Providing a connection to the target database. * Importing meta data from the target database. * Configuring masking functions, foreign keys and optional scripts. * Using DATPROF Runtime to generate a package. * Deploying the masking template. --- version: "4.23" language: "en" --- # AI Integration Disclaimer DATPROF Privacy offers an optional AI Assistant feature that integrates with the OpenAI API. This functionality is disabled by default and must be explicitly enabled by each individual user by providing their own OpenAI API key. No interaction with OpenAI is initiated unless a user has configured and actively uses this feature. When the AI Assistant is used, DATPROF Privacy sends the following information to the OpenAI API: * The user's prompt or question * Database type * Column name(s) * Data type(s) * Field length and precision No actual database records, field values, or personal data from the database are transmitted. Only technical metadata necessary to generate contextual assistance is shared. All communication with the OpenAI API is transmitted over encrypted HTTPS connections. OpenAI processes API data in accordance with its applicable API terms and privacy policies. As of the current OpenAI API policy, data submitted via the API is not used to train OpenAI's general models unless the customer has explicitly opted in. OpenAI may retain API data for a limited period for purposes such as abuse monitoring and security. Users are responsible for reviewing OpenAI's current terms of service and data usage policies to ensure compliance with their organizational and regulatory requirements. --- version: "4.23" language: "en" --- # Datatype and Function matrix ## Microsoft SQL Server |------------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | | BIGINT | X | X | X | | | X | X | | BINARY | | | X | | | X | X | | BIT | | X | X | | | X | X | | CHAR | X | X | X | | | X | X | | DATE | | X | X | X | X | X | X | | DATETIME | | X | X | X | X | X | X | | DATETIME2 | | X | X | X | X | X | X | | DECIMAL | X | X | X | | | X | X | | FLOAT | | X | X | | | X | X | | IMAGE | | | X | | | X | X | | INT | X | X | X | | | X | X | | INTEGER | X | X | X | | | X | X | | MONEY | | X | X | | | X | X | | NCHAR | X | X | X | | | X | X | | NTEXT | | X | X | | | X | X | | NUMERIC | X | X | X | | | X | X | | NVARCHAR | X | X | X | | | X | X | | REAL | | X | X | | | X | X | | SMALLDATETIME | | X | X | X | X | X | X | | SMALLINT | X | X | X | | | X | X | | SMALLMONEY | | X | X | | | X | X | | SQL_VARIANT | | | | | | | X | | TEXT | | | X | | | | X | | TIME | | X | X | | | X | X | | TIMESTAMP | | | | | | | X | | TINYINT | X | X | X | | | X | X | | UNIQUEIDENTIFIER | | | | | | | X | | VARBINARY | | | X | | | X | X | | VARCHAR | X | X | X | | | X | X | | XML | | | X | | | X | X | ## Oracle RDBMS |---------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | | BFILE | | | X | | | | X | | BINARY_DOUBLE | | | X | | | | X | | BINARY_FLOAT | | | X | | | | X | | BLOB | | | X | | | | X | | CHAR | X | X | X | | | X | X | | CLOB | | | X | | | | X | | DATE | | X | X | X | X | X | X | | FLOAT | | X | X | | | | X | | INTEGER | X | X | X | | | X | X | | LONG | | | | | | | X | | LONG RAW | | | | | | | X | | MLSLABEL | | | | | | | X | | NCHAR | X | X | X | | | X | X | | NCLOB | | | X | | | | X | | NUMBER | X | X | X | | | X | X | | NVARCHAR2 | X | X | X | | | X | X | | RAW | | | X | | | | X | | ROWID | | | | | | | X | | SDO_GEOMETRY | | | | | | | X | | TIMESTAMP | | X | X | X | X | X | X | | UROWID | | | | | | | X | | VARCHAR2 | X | X | X | | | X | X | | XMLTYPE | | | X | | | | X | ## DB2 LUW |------------------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | | BIGINT | X | X | X | | | X | X | | BINARY LARGE OBJECT | | | X | | | | X | | BLOB | | | X | | | | X | | CHARACTER | X | X | X | | | X | X | | CHARACTER LARGE OBJECT | | | X | | | | X | | CHARACTER VARYING | X | X | X | | | X | X | | CLOB | | | X | | | | X | | DATE | | X | X | X | X | X | X | | DBCLOB | | | X | | | | X | | DECFLOAT | | X | X | | | X | X | | DECIMAL | X | X | X | | | X | X | | DOUBLE | | X | X | | | X | X | | DOUBLE PRECISION | | | X | | | X | X | | GRAPHIC | | X | X | | | X | X | | INT | X | X | X | | | X | X | | INTEGER | X | | X | | | X | X | | LONG VARCHAR | X | X | X | | | | X | | LONG VARGRAPHIC | | X | X | | | X | X | | NUMERIC | X | X | X | | | X | X | | REAL | | X | X | | | X | X | | SMALLINT | X | X | X | | | X | X | | TIME | | X | X | | | X | X | | TIMESTAMP | | X | X | | | X | X | | VARCHAR | X | X | X | | | X | X | | VARGRAPHIC | | X | X | | | X | X | | XML | | | X | | | | X | ## DB2 iSeries |------------------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | | BIGINT | X | X | X | | | X | X | | BINARY LARGE OBJECT | | | X | | | | X | | BLOB | | | X | | | | X | | CHARACTER | X | X | X | | | X | X | | CHARACTER LARGE OBJECT | | | X | | | | X | | CHARACTER VARYING | X | X | X | | | X | X | | CLOB | | | X | | | | X | | DATE | | X | X | X | X | X | X | | DBCLOB | | | X | | | | X | | DECFLOAT | | X | X | | | X | X | | DECIMAL | X | X | X | | | X | X | | DOUBLE | | X | X | | | X | X | | DOUBLE PRECISION | | | X | | | X | X | | GRAPHIC | | X | X | | | X | X | | INT | X | X | X | | | X | X | | INTEGER | X | | X | | | X | X | | LONG VARCHAR | X | X | X | | | | X | | LONG VARGRAPHIC | | X | X | | | X | X | | NUMERIC | X | X | X | | | X | X | | REAL | | X | X | | | X | X | | SMALLINT | X | X | X | | | X | X | | TIME | | X | X | | | X | X | | TIMESTAMP | | X | X | | | X | X | | VARCHAR | X | X | X | | | X | X | | VARGRAPHIC | | X | X | | | X | X | | XML | | | X | | | | X | ## DB2 for z/OS |------------------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | | BIGINT | X | X | X | | | X | X | | BINARY LARGE OBJECT | | | X | | | | X | | BLOB | | | X | | | | X | | CHARACTER | X | X | X | | | X | X | | CHARACTER LARGE OBJECT | | | X | | | | X | | CHARACTER VARYING | X | X | X | | | X | X | | CLOB | | | X | | | | X | | DATE | | X | X | X | X | X | X | | DBCLOB | | | X | | | | X | | DECFLOAT | | X | X | | | X | X | | DECIMAL | X | X | X | | | X | X | | DOUBLE | | X | X | | | X | X | | DOUBLE PRECISION | | | X | | | X | X | | GRAPHIC | | X | X | | | X | X | | INT | X | X | X | | | X | X | | INTEGER | X | | X | | | X | X | | LONG VARCHAR | X | X | X | | | | X | | LONG VARGRAPHIC | | X | X | | | X | X | | NUMERIC | X | X | X | | | X | X | | REAL | | X | X | | | X | X | | SMALLINT | X | X | X | | | X | X | | TIME | | X | X | | | X | X | | TIMESTAMP | | X | X | | | X | X | | VARCHAR | X | X | X | | | X | X | | VARGRAPHIC | | X | X | | | X | X | | XML | | | X | | | | X | ## PostgreSQL |----------------------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | | BIGINT | X | X | X | | | X | X | | BIGSERIAL | | | X | | | | X | | BIT | | | X | | | | X | | BIT VARYING | | | X | | | | X | | BOOLEAN | | | X | | | | X | | BOX | | | X | | | | X | | BYTEA | | | X | | | | X | | CHAR | X | X | X | | | X | X | | CHARACTER | X | X | X | | | X | X | | CHARACTER VARYING | X | X | X | | | X | X | | CIDR | | | X | | | | X | | CIRCLE | | | X | | | | X | | DATE | | X | X | X | X | X | X | | DECIMAL | X | X | X | | | X | X | | DOUBLE_PRECISION | X | X | X | | | X | X | | FLOAT4 | X | X | X | | | X | X | | INET | | | X | | | | X | | INT | X | X | X | | | X | X | | INT2 | X | X | X | | | X | X | | INT4 | X | X | X | | | X | X | | INT8 | X | X | X | | | X | X | | INTEGER | X | X | X | | | X | X | | INTERVAL | | | X | | | | X | | JSON | | | X | | | | X | | LINE | | | X | | | | X | | LSEG | | | X | | | | X | | MACADDR | | | X | | | | X | | MONEY | | | X | | | | X | | NUMERIC | X | X | X | | | X | X | | PATH | | | X | | | | X | | POINT | | | X | | | | X | | POLYGON | | | X | | | | X | | REAL | X | X | X | | | X | X | | SERIAL | | | X | | | | X | | SMALLINT | X | X | X | | | X | X | | SMALLSERIAL | | | X | | | | X | | TEXT | X | X | X | | | X | X | | TIME | | | X | | | | X | | TIMESTAMP | | X | X | X | X | X | X | | TIMESTAMP WITHOUT TIMEZONE | | X | X | X | X | X | X | | TSQUERY | | | X | | | | X | | TSVECTOR | | | X | | | | X | | TXID_SNAPSHOT | | | X | | | | X | | UUID | | | X | | | | X | | VARCHAR | X | X | X | | | X | X | | XML | | | X | | | | X | ## MySQL | **Datatypes** | **Scramble** | **Shuffle** | **Blank** | **First day in month** | **First day in year** | **Lookup functions** | **Expression** | |---------------|--------------|-------------|-----------|------------------------|-----------------------|----------------------|----------------| | BIGINT | X | X | X | | | X | X | | BINARY | X | X | X | | | X | X | | BIT | | | X | | | | X | | BLOB | X | X | X | | | X | X | | CHAR | X | X | X | | | X | X | | DATE | | X | X | X | X | X | X | | DATETIME | | X | X | X | X | X | X | | DECIMAL | X | X | X | | | X | X | | DOUBLE | X | X | X | | | X | X | | FLOAT | X | X | X | | | X | X | | INT | X | X | X | | | X | X | | LINESTRING | | | X | | | | X | | MEDIUMINT | X | X | X | | | X | X | | POINT | | | X | | | | X | | POLYGON | | | X | | | | X | | SMALLINT | X | X | X | | | X | X | | TEXT | X | X | X | | | X | X | | TIME | | X | X | X | X | X | X | | TIMESTAMP | | X | X | X | X | X | X | | TINYINT | X | X | X | | | X | X | | VARBINARY | X | X | X | | | X | X | | VARCHAR | X | X | X | | | X | X | | YEAR | | X | X | X | X | X | X | ## Databricks |---------------|----------|---------|-------|--------------------|-------------------|------------------|------------| | Datatypes | Scramble | Shuffle | Blank | First day in month | First day in year | Lookup functions | Expression | | BIGINT | X | X | X | | | X | X | | BINARY | | | X | | | | X | | BOOLEAN | | | X | | | | X | | DATE | | X | X | X | X | X | X | | DECIMAL(p,s) | X | X | X | | | X | X | | DOUBLE | X | X | X | | | X | X | | FLOAT | X | X | X | | | X | X | | INT | X | X | X | | | X | X | | SMALLINT | X | X | X | | | X | X | | STRING | X | X | X | | | X | X | | TIMESTAMP | | X | X | X | X | X | X | | TIMESTAMP_NTZ | | X | X | X | X | X | X | | TINYINT | X | X | X | | | X | X | | STRUCT | | | X | | | | X | --- version: "4.23" language: "en" --- # Installation The installation is started by executing the installer "*PrivacySetup-\.exe"* The following screens will show up. ![DATPROF Privacy Setup 1.png](https://docs.datprof.com/__attachments/a_0a2d7891e43cf9cc8abfb92c9c7c5a34428097cf7a0fa53f22647c7dcc5cd5c2/DATPROF%20Privacy%20Setup%201.png?cb=b5664253503d856642402d3062fa40cd) Check **I accept the agreement** and press **Next**to continue. ![DATPROF Privacy Setup 2.png](https://docs.datprof.com/__attachments/a_923d47b3ca9f849b5b5ea14274d9f60229156b790bda9e88693d2b11f118aa06/DATPROF%20Privacy%20Setup%202.png?cb=0d824f303d32f6b415adf4f68be2bdf6) By default, DATPROF Privacy Setup installs to the "DATPROF" subfolder within "C:\\Program Files." However, you can choose a different installation folder if preferred. Press **Next**to continue. ![DATPROF Privacy Setup 3.png](https://docs.datprof.com/__attachments/a_e471cd10fa1056eb6ad5ca3bd03d5998fde68fd155963b2c71716d8dc3f4cc03/DATPROF%20Privacy%20Setup%203.png?cb=e09f48edc31d7d32a915cf4a08a94c60) By default, the setup will create a Start Menu entry. You can choose to edit the folder name or disable Start Menu folder creation altogether. Press Next to continue. ![DATPROF Privacy Setup 4.png](https://docs.datprof.com/__attachments/a_01143cee76aa09e9e83d495f71e4bc90e9d30fd5dada4b7fe6305f1b13044c22/DATPROF%20Privacy%20Setup%204.png?cb=e35ccef89fe208d00105c2ac3823ce8a) Check the box if you'd like Privacy Setup to create a desktop shortcut and click **Next**to continue. ![DATPROF Privacy Setup 5.png](https://docs.datprof.com/__attachments/a_2c7f603b8e45fbf1e90bca0e2dbafbca55fe7d5ad5a718ca01355a60a40286a1/DATPROF%20Privacy%20Setup%205.png?cb=e0e358a680c7cbb9ec1be1a11e5437a3) Review the previously specified settings, then click **Install** to begin installing DATPROF Privacy. ![DATPROF Privacy Setup 6.png](https://docs.datprof.com/__attachments/a_ed00c832a1ab652f3dfaadf909c27898fff0c7029eef9acb3a239138ff9d421e/DATPROF%20Privacy%20Setup%206.png?cb=cdc7a4123915667997f798080a8e8166) Setup is installing DATPROF Privacy. ![DATPROF Privacy Setup 7.png](https://docs.datprof.com/__attachments/a_b5f8699892bede0681115f4803583512e87aa829a81527a2ccc2caa5d8c2560d/DATPROF%20Privacy%20Setup%207.png?cb=93043783f44d12aafeb720db2cd1b92f) Setup has finished installing DATPROF Privacy. Press **Finish**to launch DATPROF Privacy. --- version: "4.23" language: "en" --- # Release information ## Privacy 4.23 | **Version** | 4.23.1 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** **Bugfixes:** * Fixed connection issue when opening project file from explorer * Fixed Windows Authentication issue for SQL Server *When generating Runtime applications Runtime 4.20 or higher is recommended* ***.*** | | **Date** | 2026-06-16 | | **Version** | 4.23.0 | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Cross-Database Deterministic Masking for Unique Generators * Masking support for JSON on Oracle, SQL Server, DB2 LUW * Additional JDBC Database Support (Custom JDBC Connection) **Bugfixes:** * Fixed issue on DB2 z/OS while inserting foreign keys in control table * Fixed Value Lookup issue on Postgres with null values * Fixed Value Lookup casting issue on SQL Server *When generating Runtime applications Runtime 4.20 or higher is recommended* ***.*** | | **Date** | 2026-06-08 | ## Privacy 4.22 | **Version** | 4.22.2 | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Fixed a crash saving a JSON Value Lookup without specifying lookup table When generating Runtime applications Runtime 4.19 or higher is recommended**.** | | **Date** | 2026-05-08 | | **Version** | 4.22.1 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Fixed generation issue when the same table is imported in two schema's When generating Runtime applications Runtime 4.19 or higher is recommended**.** | | **Date** | 2026-04-14 | | **Version** | 4.22.0 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Added UUID generator for Masking \& Generation **Bugfixes:** * Fixed issue with relative path to custom seed CSV files * Fixed password issue for Postgres when containing semicolons * Fixed pause \& resume issue When generating Runtime applications Runtime 4.19 or higher is recommended**.** | | **Date** | 2026-03-12 | ## Privacy 4.21 | **Version** | 4.21.0 | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Masking JSON inside Postgres** With this release, we introduce the ability to mask JSON elements stored in PostgreSQL databases using a wide range of generators. This includes support for deterministic masking and value lookup, ensuring that specific JSON attributes are masked consistently through translation or lookup tables. Using flexible JSON path expressions, you can precisely target individual JSON attributes for masking. Additionally, multiple conditional functions can be applied to mask different data elements within JSON objects based on defined rules. **AI Assistant** We are excited to introduce the first edition of our optional AI Assistant, now fully integrated into Privacy. The AI Assistant helps users generate seed values for the new **Value from List** generator and supports the creation of complex custom SQL expressions. [++Read more++](https://datprof99768.acemlnb.com/lt.php?x=4lZy~GDFJFOfD878_dy5VxFz1a-lvgHzvPozXXnDJXWg5536-ky7wudw1-3m-NFfx1GwZ5QWIned95.K0N5G) **Other improvement \& fixes** * Added *Value from list* generator that directly works with a list of items * Made the error screen resizable for improved usability * Fixed an issue where triggers were not correctly disabled (Oracle) * Fixed an issue with a missing database field in the connection editor (DB2 iSeries) * Fixed an issue involving composite foreign keys in PostgreSQL When generating Runtime applications Runtime 4.19 or higher is recommended**.** | | **Date** | 2026-02-24 | ## Privacy 4.20 | **Version** | 4.20.0 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Introducing Databricks support** You can now mask sensitive data and generate synthetic data directly inside Databricks, without exporting data or compromising security. This update enables in-place, fast, and secure processing with support for conditional masking using SQL filters and automatic Delta Table version handling to prevent access to historical unmasked data. Masking is deterministic and consistent across all sources, and metadata-driven templates keep rules synchronized. Plus, you can extend functionality with custom SQL, seed files, or functions to fit your unique data needs. **Other fixes** * Fixed issue with deployment logging contains data from multiple runs *When generating Runtime applications Runtime 4.17 or higher is recommended* ***.*** | | **Date** | 2025-10-20 | ## Privacy 4.19 | **Version** | 4.19.3 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** **Bugfixes:** * Fixed an issue preventing the application of functions on DB2 Boolean data types * Resolved auto-retry malfunction * Fixed bypass control not taking effect in Runtime * Addressed an issue when importing metadata from DB2 LUW * Improved handling of deadlocks on SQL Server ***When generating Runtime applications Runtime 4.15 or later is recommended.*** | | **Date** | 2025-09-12 | | **Version** | 4.19.2 | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** **Bugfixes:** * Upgraded Runtime agent with new Oracle JDBC drivers * Fixed missing logging lines for some temp tables * Fixed stuck Save DDL action on SQL Server due to blocked files. * Fixed issue with switching schema's in Masking tab ***When generating Runtime applications Runtime 4.15 or later is recommended.*** | | **Date** | 2025-06-13 | | **Version** | 4.19.1 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** **Bugfixes:** * Fixed an issue with the script manager always using velocity parsing ***When generating Runtime applications Runtime 4.13 or later is recommended.*** | | **Date** | 2025-05-21 | | **Version** | 4.19.0 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Table Labeling and Filtering * Column Profiling * Import Analyze Profiling Results * Bulk function wizard **Bugfixes:** * Resolved a rowcount issue for large tables (SQL Server) * Resolved an issue when retrying a single module * Resolved an issue with duplicate DPV_RID's with non-latin characters (SQL Server) * Improved performance when generating Runtime applications * Fixed connection issue with EntraID - Service Principal ***When generating Runtime applications Runtime 4.13 or later is recommended.*** | | **Date** | 2025-03-28 | ## Privacy 4.18 | **Version** | 4.18.1 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Fixed an issue for SQL Server with data containing special characters ***When generating Runtime applications Runtime 4.11 or later is recommended.*** | | **Date** | 2025-01-24 | | **Version** | 4.18.0 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Added [Application signing \& validation](https://docs.datprof.com/runtime/latest/applications) * Added an option to [determine a fixed port or range for the Runtime agent](https://docs.datprof.com/privacy/latest/direct-deployment#id-(4.18)DirectDeployment-RuntimeAgentPortRange) **Bugfixes:** * Fixed issue with seed files containing trailing white spaces * Fixed issue with running Privacy under a different user * Fixed connection issue with SQL Server (Missing jar file) * Fixed sync issue with DB2 LUW \& z/OS ***When generating Runtime applications Runtime 4.11 or later is recommended.*** | | **Date** | ​Jan 13, 2025 | ## Privacy 4.17 | **Version** | 4.17.5 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Resolved an issue where the regex generators were not functioning correctly. ***When generating Runtime applications Runtime 4.10 or later is recommended.*** | | **Date** | ​Nov 22, 2024 | | **Version** | 4.17.4 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Improved performance for seed list generators ***When generating Runtime applications Runtime 4.10 or later is recommended.*** | | **Date** | ​Nov 18, 2024 | | **Version** | 4.17.3 | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Added extra statistics step after non-conditional append (Oracle) ***When generating Runtime applications Runtime 4.10 or later is recommended.*** | | **Date** | ​Oct 14, 2024 | | **Version** | 4.17.2 | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Fixed issues with table partitioning for Postgres databases * Fixed issue with bypass on SQL Server due to missing certificate * Fixed issue with Tinytext data type on MySQL ***When generating Runtime applications Runtime 4.10 or later is recommended.*** | | **Date** | ​Sep 20, 2024 | | **Version** | 4.17.1 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Added support for Microsoft Entra ID authentication for SQL Server / Azure SQL * Changed from using ODBC to JDBC drivers for SQL Server connections * Added option to enter JDBC / [http://ADO.NET](http://ado.net/) properties for SQL Server connections * Updated the SaveDDL powershell script to support Entra ID authentication **Bugfixes:** * Fixed an issue that prevents opening a .dpv file directly from the explorer ***When generating Runtime applications Runtime 4.10 or later is recommended.*** | | **Date** | ​Jul 8, 2024 | ***Version 4.17 for DATPROF Privacy starts with bugfix version 1, as the initial build of this minor version was not published for customers due to small revisions needed.*** ## Privacy 4.16 | **Version** | 4.16.0 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Added option to make scripts executable per scenario (masking vs generation) * Added option to generate runtime package using command line * Updated SQL Server powershell module dependency (SQL Server) * Improve error handling in SaveDDL powershell script (SQL Server) * Using JDBC driver now for import and synchronization of meta data (DB2 LUW) **Bugfixes:** * Fixed an issue while using a combination of split limit \& custom dpv_rid (Oracle) * Fixed an issue with skipped scripts shown as success in the audit logging * Fixed an issue when adding a preconfigured generator * Fixed an issue with DPF_SCRIPTS not dropped after a run * Fixed an issue while filling the DPF_SCRIPTS table (SQL Server) * Fixed an issue with missing runtime-agent.log * Fixed an issue with not disabling auto-retry by leaving the field blank * Fixed an double schema import issue on DB2 z/OS * Fixed formatting issue for logged SQL statements ***When generating Runtime applications Runtime 4.9 or later is recommended.*** | | **Date** | ​Jul 8, 2024 | ## Privacy 4.15 | **Version** | 4.15.3 | |-----------------|--------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** Fixed an issue with duplicated rows due to collation edge-cases (SQL Server) | | **Date** | ​Jun 25, 2024 | | **Version** | 4.15.2 | |-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Fixed issue that Privacy is stuck in running state while agent is already done * Fixed connection issue with IBM DB2 z/OS when using direct connection * Fixed issue when using '{ }' inside a function condition | | **Date** | ​May 22, 2024 | | **Version** | 4.15.1 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Bugfixes:** * Preconfigured generators are now also enabled for Masking. * Fixed issue with deterministic flag on preconfigured generators not being saved correctly. | | **Date** | ​Apr 19, 2024 | | **Version** | 4.15.0 | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **New features:** * Added option to add preconfigured generators to centralize configuration * Added option to quickly enable and disable functions on table and schema level For more information on these features, please refer to the **General Functionality**chapter of this documentation. **Bugfixes and other changes:** * Removed XX and XXX from country seed file * Fixed a DB2-i issue with restoring foreign keys with combined columns * Fixed a DB2-i data generation issue with journaling * Fixed a copy and paste issue for Generation * Fixed an issue for Oracle on which insert fail on DPF_COLUMNS table * Fixed an issue with deterministic data masking on SQL Server on columns containing different casing. | | **Date** | ​Apr 5, 2024 | ## Privacy 4.14 | **Version** | 4.14.4 | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains a hotfix** * Fixed issue when dropping constraints and disabling triggers \[SQL Server\] * Fixed issue with creating index on LongText datatypes \[MySQL\] * Fixed issue with unsigned columns \[MySQL\] | | **Date** | ​Feb 27, 2024 | | **Version** | 4.14.3 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains bugfixes** * Fixed issue in conditional shuffle for Oracle introduced in 4.14.2 * Fixed "org.apache.velocity.runtime" error when 4.14.2 was installed over previous installation * Improved installer to prevent old files from previous installation interfering with current installation | | **Date** | ​Dec 8, 2023 | | **Version** | 4.14.2 | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains bugfixes** * Fixed issue while changing generator type overwriting custom function names * Fixed issue on DB2 LUW with restoring foreign keys with combined columns * Fixed issue for DB2 iSeries issue when saving a connection without a port defined * Fixed issue for generator functions where setting are not saved correctly * Fixed issue with generation for SQL Server with tables having identity columns * Fixed issue for DB2 iSeries sending out two start messages to the Runtime agent | | **Date** | ​Dec 8, 2023 | | **Version** | 4.14.1 | |-----------------|--------------------------------------------------------------------| | **Description** | **Privacy 4.14.1 is an internal release which was not published.** | | **Date** | Unreleased | | **Version** | 4.14 | |-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains new database support, features and bugfixes** **IBM DB2 z/OS (Mainframe) support** We are excited to bring full masking and generation support for IBM DB2 on z/OS (Mainframe). This new update gives large enterprises the option to modernize their test environments with representative and safe test data. * **License Note:**Using IBM DB2 z/OS requires an additional license. Please contact your sales representative to check if you have the right license to use this feature. **Other features and bugfixes** * We have upgraded our solution to .NET 6. All dependencies are now embedded and does not require extra installation of .NET dependencies * Added an option to enter and save a name/description for a connection. * Added a deprecation warning message when deploying in legacy mode * Fixed an issue in which the IBAN account generator was not deterministic * Fixed issues with the Random Date/Time generator when configured on a varchar field * Fixed an issue when upgrading an generated template in Runtime having duplicate schema mappings * Fixed a crash when opening the help documentation * Fixed an issue when using a disabled function and a translation table (DB2 LUW) * Fixed the 'open audit log' button **Important** : When generating Runtime applications with DATPROF Privacy 4.14, Runtime 4.7 or later is recommended**.** | | **Date** | ​Oct 9, 2023 | ## Privacy 4.13 | **Version** | 4.13.7 | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains bugfixes** * Resolved an issue for DB2 LUW involving gather statistics, a shuffle function and exclude nulls condition. * Resolved an issue for DB2 LUW involving gather statistics, a random lookup function. | | **Date** | ​Sep 29, 2023 | | **Version** | 4.13.6 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains bugfixes** * Resolved an issue for DB2 LUW in which gather statistics modules gives an error * Resolved an issue for MySQL where rows with null values were not restored correctly. | | **Date** | ​Sep 26, 2023 | | **Version** | 4.13.5 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains bugfixes** * Replaced seed list of Random word generator (US) * Resolved an execution issue when a function is disabled | | **Date** | ​May 10, 2023 | | **Version** | 4.13.4 | |-----------------|--------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains a fix for a single bug** * Resolved an duplicate key issue with multi column seed file generator | | **Date** | ​Apr 18, 2023 | | **Version** | 4.13.3 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains improvements and bugfixes** * Resolved an IComparable error during generation (introduced in 4.13.2) * Resolved an issue with resizing the treegrid in generation * Resolved an issue with the ValueLookup for DB2 on LUW | | **Date** | ​Mar 8, 2023 | | **Version** | 4.13.2 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains improvements and bugfixes** * Addressed an issue on Azure SQL where the database collation was set to Latin1_General_100_CI_AS_SC. * Resolved an issue where Post-SQL was not reset correctly when switching generator types. * Fixed an issue where XMLTYPE columns were truncated (metadata synchronization is required). * Improved performance by adding indexes to temporary tables in SQL Server. * Fixed a crash that occurred when synchronization caused issues in the Multi-Column Seed File Generator. * Resolved an issue where a project became corrupt after creating a Value from the Multi-Column Seed File Generator. * Fixed a crash that occurred when the project name contained trailing spaces. * Widened the default width of the Connection Editor to provide more space for connections. * Updated the Visualize Process Model to list function names instead of function types in the drop-down menu. | | **Date** | ​Mar 2, 2023 | | **Version** | 4.13.1 | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **This release contains improvements and bugfixes** * Added extra statistics on temporary tables when TempStatisticsPercentage is set (Oracle) * Fixed issue with meta data not being synced directly inside generation sets * Fixed issue when duplicate function names exist * Fixed issues with tables that are in other schema's than dbo (SQL Server) * Fixed issue with incorrect cast assignment (SQL Server) | | **Date** | ​Dec 16, 2022 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Version** | 4.13.0 | | **Description** | **This release contains bugfixes and improvements** * Added generation to process visualization * Added generation audit log * Added option to disable specific generation sets * Fixed issue in generation on columns that are NOT NULL * Fixed issue with bypassing spatial indexes on SQL Server * Fixed issue with restoring secondary XML indexes on SQL Server * Fixed UI issue that prevents selecting a table when a translation table is selected | | **Date** | ​Oct 6, 2022 | |-----------------|----------------------------------------------------------------------------------------------------------------------| | **Version** | 4.12.4 | | **Description** | **This release contains bugfixes** * Fixed issues on Oracle databases that use characters sets requiring orai18n.jar | | **Date** | ​Sep 2, 2022 | |-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Version** | 4.12.3 | | **Description** | **This release contains bugfixes and improvements** * Fixed characterset issues with custom seedfiles * Adding functions to not supported datatypes is disabled * Adding warning icons to columns with unsupported datatypes * Improved error message when custom seedfile could not be found * Fixed crash when first schema does not contain any table * Fixed padding option in Sequential Number generator * Fixed error when testing connections on DB2 iSeries when password is empty * Fixed generation issue on SQL Server when temp schema is not DBO * Prevent error when table inside generation set has no generators configured * Fixed error when generation set name contains special characters. * Fixed issue with foreign key generator when foreign key is removed | | **Date** | ​Jul 19, 2022 | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Version** | 4.12.2 | | **Description** | **This release contains only bugfixes** * Fixed error with Multi Column Seed file generator * Fixed SSL handshake failure on PostgreSQL / AWS Aurora databases | | **Date** | ​Jun 13, 2022 | |-----------------|--------------------------------------------------------------------------------------------------| | **Version** | 4.12.1 | | **Description** | **This release contains only bugfixes** * Fixed missing seedfile error with specific generators. | | **Date** | ​May 12, 2022 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Version** | 4.12 | | **Description** | **This release contains new improvements and bugfixes** * Database version is shown when testing a connection * Increased performance for MySQL 5 * Added ascending and descending options in table list * Added missing data types for Postgresql * Fixed sorting issue in table list * Fixed duplicating seedfiles in generated package * Fixed SQL Server issue with tables that are not stored in schema dbo * Fixed issue where Post SQL (Generation) was not saved * Fixed issue where the wrong schema was used for dropping the translation table **Important** : When generating Runtime applications with DATPROF Privacy 4.12, Runtime 4.2 or later is recommended**.** | | **Date** | ​May 11, 2022 | --- version: "4.23" language: "en" --- # Requirements ## Operating System **DATPROF Privacy requires one of the following:** * Microsoft Windows 7 or later * Microsoft Windows Server 2012 or later Additionally, the .NET Framework 4.8 must be installed, and the user installing the software must have the necessary permissions to install applications. ## Hardware * **CPU**: x86 1 GHz or more * **Memory**: At least 4 GB RAM * **Disk Space**: At least 1 GB of free disk space ### Supported Databases DATPROF follows the software lifecycle guidelines set by database vendors. This ensures compatibility and support with the latest versions of database systems while maintaining the stability and reliability of the tool. By aligning with the database vendors' lifecycle, DATPROF ensures that its solutions stay up-to-date with new features, security patches, and performance enhancements. | **Oracle** | Version 11.2 and later | | **Microsoft SQL Server** | Version 2008 (not for Runtime) Version 2012 Version 2014 Version 2016^\*^ Version 2017^\*^ Version 2019^\*^ Version 2022 and later^\*^ | | **DB2 LUW** | 10.5 and later | | **DB2 for i** | 7.2 7.3 7.4 | | **DB2 for z/OS** | 13.0 and later | | **PostgreSQL** | 9.5 and later | | **MySQL** | 8.0 | | **MariaDB** | 10.4 | | **Azure SQL** | Not applicable - SaaS solution. | | **Databricks** | Not applicable - SaaS solution. | |--------------------------|----------------------------------------------------------------------------------------------------------------------------------------| *\*Check the Powershell module remarks* ## Oracle RDBMS ### Permissions and Roles If the Privacy User is the TABLE_OWNER *Minimum System Privileges* * CREATE SESSION * CREATE TABLE *Minimum Object privileges* * EXECUTE permission on dbms_random package * EXECUTE permission on dbms_metadata package *Quota* * Unlimited tablespace If the Privacy User is not the same as the TABLE_OWNER these additional permissions are required: * SELECT on the tables to mask * REFERENCES on the tables to mask * ALTER on the tables to mask * CREATE ANY INDEX * DROP ANY INDEX * CREATE ANY TABLE * INSERT ANY TABLE * SELECT_CATALOG_ROLE When you create your own procedures, functions, use sequences, or reference views in your scripts, it is essential to ensure that the Privacy user is granted the necessary permissions to access and execute these elements. ## Microsoft SQL Server ### Permissions The user masking the database should have dbo permissions in the database targeted for data masking. ### Powershell module dependency To retrieve DDL information from Microsoft SQL Server, DATPROF Runtime uses a PowerShell module. Over time, Microsoft has replaced the original "SQLPS" module with the "SqlServer" module. As of version 4.0, the **SqlServer** module is included with DATPROF Privacy and no longer needs to be installed manually. Microsoft PowerShell version 5 or later is required to use the SqlServer module. 1. **Open a PowerShell window** on your system **as administrator**. 2. To check your current PowerShell version, run the following command: $psversiontable.Psversion 3. If the **major version** is lower than 5, you need to upgrade PowerShell. To do so, download and install **Windows Management Framework** version 5.x from the following link: [Windows Management Framework 5.1](https://www.microsoft.com/en-us/download/details.aspx?id=54616) **Linux systems** When using Runtime on a Linux system, PowerShell must be installed, and the **SQL Server PowerShell module** needs to be manually installed. To install PowerShell on a Linux system, refer to the Microsoft documentation: [Install PowerShell on Linux](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux). Once PowerShell is installed, follow these steps to install the SQL Server module: 1. Open a PowerShell window on your system (either as the user running Runtime or as root). 2. Execute the following command to install the required module: Install-Module -Name SqlServer This will install the necessary SQL Server module for PowerShell, enabling you to manage SQL Server from your Linux environment. ## DB2 LUW ### Requirements In prior versions of DATPROF Privacy, a specific IBM DB2 driver package was necessary as a manual installation, but as of version 4.16.0 of Privacy, this is no longer necessary as we now use a JDBC driver (db2jcc4.jar), which we bundle with our software. ### Permissions The user responsible for masking the database must have the following permissions in the targeted database: * **CONNECT**: Permission to establish a connection to the database. * **DATAACCESS**: Permission to access the data within the database. * **CREATETAB**: Permission to create tables, which is required during the data masking process. These permissions ensure that the user has the necessary access to perform data masking tasks effectively. ## DB2 iSeries ### Requirements To use DATPROF Privacy with DB2 iSeries, you should have one of the following installed: * **IBM i Access for Windows 7.1**, or * **IBM i Access Client Solutions** This software can be obtained from the [IBM website](https://www.ibm.com/support/pages/ibm-i-access-client-solutions). ### Permissions The user responsible for masking the database must have **DBADM** (Database Administrator) permissions in the database schema targeted for data masking. ## DB2 for z/OS (Mainframe) ### Requirements In order to use Privacy with DB2 for z/OS databases, the user must possess their own db2jcc_license_cisuz.jar file - This is the IBM Data Server for JDBC and SQLJ license file that allows access to the database through the JDBC protocol. This is a separate license that must be acquired through IBM. This file must then be placed in the **Drivers**sub-folder of the Privacy installation folder. If this is not done, the user is greeted by the following error upon trying to test a connection to the database: ![DATPROF Privacy Connection Failed db2jcc_license_cisuz.jar.png](https://docs.datprof.com/__attachments/a_dc978547dbccd8098db32dff5ee21a659656a62a8143619cc8f9d1cb8976d703/DATPROF%20Privacy%20Connection%20Failed%20db2jcc_license_cisuz.jar.png?cb=292c07f3510caf85e06d75a2d9cc7cd1) ### Permissions The user masking the database requires admin level permissions in the database. ## Azure SQL ## Permissions The user masking the database should have admin level permissions in the database targeted for data masking. ## PostgreSQL, MySQL and MariaDB ### Permissions You should at least have select permissions on the tables to import. To execute a scenario database owner permissions are required. --- version: "4.23" language: "en" --- # Deployment Options As of Privacy version 3.8.0 there is a distinction between ***Direct Deployment*** and ***Runtime Deployment.*** *Direct Deployment* means that you create and develop your data masking project in DATPROF Privacy and deploy it using DATPROF Privacy as well. *Runtime Deployment* means that you create and develop your data masking project in DATPROF Privacy and deploy it using DATPROF Runtime. As of Privacy version 4.0 *Direct Deployment* has been changed to use an embedded *Runtime Agent* by default. This means that the same package is generated as with *Runtime Deployment* but instead of uploading this package to DATPROF Runtime, a local agent is started that will execute the package directly. One major benefit of this new direct deployment scheme is that PostgreSQL is now supported with *Direct Deployment*. As of version 4.15.0 of Privacy, the ***Use legacy deployment*** option in the *Project* menu has been removed. | | **Direct Deployment** | **Direct Deployment (legacy)** | **Runtime Deployment** | |---------------------|-----------------------|--------------------------------|------------------------| | **Oracle** | x | x | x | | **MS SQL Server** | x | x | x | | **DB2 i** | x | x | x | | **DB2 Luw** | x | x | x | | **PostgreSQL** | x | | x | | **MySQL / MariaDB** | x | | x | | **Azure SQL** | x | | x | | **Databricks** | x | | x | --- version: "4.23" language: "en" --- # Direct Deployment As of Privacy version 4.0 the default *Direct Deployment* scheme will use an embedded Runtime Agent and will be called as such hereafter. The old scheme will be called *Legacy Deployment*. The following applies to both schemes unless stated otherwise. The deployment section allows the user to execute the configured masking functions on the target database (as specified in the *Connection Editor* ). Clicking **Start** will execute (after confirmation) the data masking functions. Data masking can be stopped or paused using the**Abort** and **Pause**buttons. ![DATPROF Privacy Run in Progress.png](https://docs.datprof.com/__attachments/a_ed6302080c381c0518eb24a514d24ea83409e4bf406f0938f2c2de97d4f9a0f2/DATPROF%20Privacy%20Run%20in%20Progress.png?cb=e885cc7be531ab0062b6041cf96e3d18) After the execution, the result of the masking is displayed. With the new scheme, not only the result of the masking functions are displayed but also the preparation and finalization. For each function, the number of masked records is shown. In the **Logging** tab the various executed steps and detailed results is shown. The log provides input for resolving any errors during the data masking process. Logging output is slightly different between the two schemes. The logging is also saved in a subdirectory of the Deployment folder of the project. The name of the subdirectory is the timestamp when the deployment was started. The contents is different between Direct Deployment and Direct Deployment (legacy). ## Settings Tab The **Settings Tab**shows Database and error handling options, depending on the deployment scheme different options are available. ### Direct Deployment ![DATPROF Privacy Settings Tab.png](https://docs.datprof.com/__attachments/a_a7ebefb9392dd4b342ed6707d0cedacfc4afbce6782bccb8d3cbd7770d9a790e/DATPROF%20Privacy%20Settings%20Tab.png?cb=5a5122579c35d8797a5b6dbac2f738af) **Runtime settings** * **Maximum parallel processes:** The maximum amount of parallel processes. The default is set to 8. * **Automatically retry errors:**You can specify the number of retries whenever errors occur. The default value is 10. * **Global salt:**This salt is used in addition to each generator specific salt when configured in deterministic mode. Changing this salt will affect the deterministic result the masking process. The global salt will also be available in Runtime as an environment parameter. * **Schema for temporary tables:** The name of the schema (User) to store the temporary tables. It's default in the current schema. ### Legacy Deployment As of version 4.15.0 of Privacy, this option has been removed. ![Legacy Deployment.png](https://docs.datprof.com/__attachments/a_09883361231b9812b84924691bcd0369260768d9ea2db5cf09d1b01dc0259e53/Legacy%20Deployment.png?cb=9413f4e9b263d146a1a4d1c3761934c1) **Database Settings.** For executing the data masking functions on the database, the user has a number of options which are all checked on by default: * **Bypass triggers:** With this option enabled, the application ensures that triggers are disabled during masking, so that adding, modifying and deleting records will cause no unwanted side effects. All these disabled triggers are enabled after completing the masking process. * **Bypass constraints:**With this option checked, the application ensures that constraints are disabled during masking, so the FK constraints do not cause errors during the masking process. All these disabled constraints are enabled after completing the masking process. This includes constraints on tables not in the template. * **Bypass indexes:**With this option checked, the application drops all indexes before masking to gain performance. Indexes are created after completing the data masking process. Indexes run sequentially per table. **Module Error handling.** * **Automatically retry errors:**You can specify the number of retries whenever errors occur. The default value is 10. * **Automatically skip errors:**Check this option to skip errors. This option is not checked by default. ## Advanced Tab Only available for the legacy deployment scheme. ![Deployment Options and Logging Advanced.png](https://docs.datprof.com/__attachments/a_2a31a7a9fb53251fec90ffd54f8295695a7e6386a7c938762ae085779fb35433/Deployment%20Options%20and%20Logging%20Advanced.png?cb=8b7d88619c8a994add077491cf19cfc2) **Automatic abort settings** * **Abort deployment after minutes:**You can specify the end time of the process. This might be helpful when you have limited time to run the masking process. * **Abort deployment after following time:**You can specify the exact moment to stop the process. This might be helpful when you are granted a limited time window to run the data masking process. **Runtime settings** * **Maximum parallel processes:** The maximum amount of parallel processes. The default is set to 8. * **Schema for temporary tables:** The name of the schema (User) to store the temporary tables. It's default in the current schema. ## Logging Tab The **Logging Tab** shows a scrollable area with the results of the last data masking run. ![DATPROF Privacy Logging Tab.png](https://docs.datprof.com/__attachments/a_5c552f1555dbe476f0af72fe0d0ae38bc19037ee0ec9197fd60a12f78635e660/DATPROF%20Privacy%20Logging%20Tab.png?cb=077cc10d45ce2a7bb2f4460e3cba3855) All logging is saved to a file, which you can open by clicking **Open log file...** An audit log is automatically generated after each run. You can open it by clicking **Open audit log...** ![DATPROF Privacy Audit Report.png](https://docs.datprof.com/__attachments/a_7080c58226e6acd0b0d7e9df8c2d0a0fac176b604f0ed1e3110482eb2532c753/DATPROF%20Privacy%20Audit%20Report.png?cb=07836358f6a9ce346ec8f1b4e9511a07) At the end of a run, an audit report is automatically generated and saved as **Auditlog.html** in a subfolder within the project's Deployment folder. This report includes details about the masked database, the user who performed the masking, and a record of the functions applied to each table and column. ## Comments Tab ![DATPROF Privacy Comments Tab.png](https://docs.datprof.com/__attachments/a_90c518f31e764929354f60df669ddde0f815c863817062dfde21cac1e55b4a6c/DATPROF%20Privacy%20Comments%20Tab.png?cb=f8b967b97e940502805cd735bd229849) The **Comments Tab** shows a editable text area where you can specify comments. These comments will also show up in **"Generate comments reports"** (Using Menu option **Project** *)* ## Options during the data masking process. When deploying the masking process the following options are available: * **Pause** : The process is temporarily stopped. The button is then temporarily replaced by a **Resume**button. Once this button is pressed, the run continues and the button is reset. * **Abort:** The process will be aborted. Aborting a run does **not**always stop a run gracefully. Because Privacy will perform a clean-up at the end of a run, aborting a run may leave artifacts from the run in the form of temporary tables. Usually, starting a new run cleans up any left behind artifacts from an aborted run. * **Error:** (Only available if an error occurs) By pressing this button a new screen opens with an overview of currently blocking errors. In this menu, an overview can be found of the offending modules' log text. ![Errors.png](https://docs.datprof.com/__attachments/a_8beaff87e68991b0d60c815201fc703688de48075d15417b313377cab086d7b5/Errors.png?cb=1925a9b16f995c3393539ef2fa820002) Most importantly, when an error is generated, the error(s) are shown in this dialog screen. The upper part of this dialog shows which modules have failed, with at the bottom a detailed description of what has gone wrong. If a run goes into error the user has time to change the underlying template, and either retry in the hope that this succeeds, or skip the module. This can be done by unfolding the **Action**drop-down box. * **None**: The default option, which aborts the process. * **Retry:**This option will retry the process. * **Skip**: Bypasses the error and continues with the remaining tasks. ![Error Actions.png](https://docs.datprof.com/__attachments/a_a95138857606d2b7ffdc7ee39468937592dc586f95c2834a5474920f7fd76389/Error%20Actions.png?cb=3496dc6594032e7092337a400e24165b) If the error is caused by an error during development it is required to adjust the code and then retry the process. DATPROF Privacy allows you to switch back to the development screen, modify the code, and then return to this screen to **Retry** the last process and to continue the run. To get there you should first press **Cancel**and continue from there. Every drop down action is confirmed by pressing **Apply**. ## Runtime Agent Port Range As of Privacy version **4.18**, you can manually specify a specific port or range of ports for use, providing greater flexibility and control over network configurations. This feature ensures better compatibility with various environments and minimizes the risk of port conflicts. If no port or range is manually specified, Privacy will default to using the first available port within the **5000-6000** range. To change the default port, you have two options: 1. Use the program argument `-port-ranges`. 2. Set the system environment variable `DPF_PORT_RANGES`. These options allow you to tailor Privacy's port usage to meet your specific requirements. You can specify multiple ports or ranges using a comma-separated format. This provides precise control over which ports are used. For example: 1024-2048,1234,1236,4000-4010 * `1024-2048` specifies a range of ports from 1024 to 2048. * `1234` and `1236` specify individual ports. * `4000-4010` specifies another range of ports from 4000 to 4010. ### Setting Ports with Program Arguments To manually configure a port using program arguments, follow these steps: 1. Open **Command Prompt** (CMD). 2. Navigate to the folder containing `Privacy.exe` using the `cd` command. 3. Start Privacy with a custom port by entering the following command: Privacy.exe -port-ranges 5002 ![DATPROF Privacy Port Range Program Arguments.png](https://docs.datprof.com/__attachments/a_2016b765fd7a848d9c8ff090f03a61f849217456f5b31b30befbbb0b752cb0e7/DATPROF%20Privacy%20Port%20Range%20Program%20Arguments.png?cb=21a9dad5f5d3ddeea41fe4b33c47c9d7) This command starts Privacy and sets the application to use port **5002**. Adjust the port number as needed to fit your network configuration. ### Setting Ports within System Environment Variables To configure ports using system environment variables, follow these steps: 1. **Open System Environment Variables** * On Windows, search for Environment Variables in the Start menu and select Edit the system environment variables. * Click Environment Variables in the System Properties window. 2. **Add a New System Variable** * Click New under the System variables section. * Set the Variable name to`DPF_PORT_RANGES`**.** * Set the Variable value to the desired port or range, e.g., `1024-2048,1234,1236,4000-4010`. 3. **Save and Apply** * Click OK to save the new variable. * Restart DATPROF Privacy to apply the changes. **Example Variable Value:** DPF_PORT_RANGES=1024-2048,1234,1236,4000-4010 ![DATPROF Privacy Port Range System Variable.png](https://docs.datprof.com/__attachments/a_1474711b34f5cea2733029b005fc75c5d519a5176fe4efe0dcbf762498595843/DATPROF%20Privacy%20Port%20Range%20System%20Variable.png?cb=1526194d38be0c1add84050abcd0ceb9) --- version: "4.23" language: "en" --- # Runtime Deployment ## Generating a package for Runtime To generate a package for Runtime to import and execute, follow these steps: * Open **Privacy**. * Navigate to the **top-left menu** of the screen. * Select **Project → Generate for Runtime...** ![DATPROF Privacy Generate for Runtime Project Menu.png](https://docs.datprof.com/__attachments/a_b151b30e85a93c689b032b520e5be7153dd51f6e794748b2d28b64570d1c8422/DATPROF%20Privacy%20Generate%20for%20Runtime%20Project%20Menu.png?cb=f9ea62a5c514eced3ba08293d5439d47) A dialogue screen will open, asking you to confirm. Press **"Generate"**. ![DATPROF Privacy Generate for Runtime.png](https://docs.datprof.com/__attachments/a_144184183d676d36f720c65f10e222c0c5452b03c23b7d7e6626c230d4f431b0/DATPROF%20Privacy%20Generate%20for%20Runtime.png?cb=89d229163050383e3173b0040dae4fb2) Privacy will now generate a package and open the Windows Explorer in the Privacy folder. Here, you can find the file as required by Runtime. ## Executing the package in Runtime For instructions on how to execute a package in Runtime, please refer to the Runtime Manual [++here++](https://docs.datprof.com/runtime/latest/applications). As the configuration of Runtime is separate to Privacy, this process requires an administrator to configure Runtime. --- version: "4.23" language: "en" --- # Tool Overview This chapter describes the global functionality which can be found in the title and menu bar of DATPROF Privacy. Following chapters give detailed descriptions of the core functionality of DATPROF Privacy. ## New project or existing project Once DATPROF Privacy is started a dialog is shown where the user can choose to create a new project or open an existing project. The user has three options: ![image-20240813-083559.png](https://docs.datprof.com/__attachments/a_8f01a27ac3e1d0d62275fac9a4a0b95a3bc21bfed24188b422e332d789f0c8c5/image-20240813-083559.png?cb=8455832235de00392c8c984c5c34f547) ### **New project** This option asks for a project name and location where the project should be saved. These fields are pre-filled with a default value. After confirmation, an empty project is created and opened in the *Project Settings* window #### **Recently opened projects** The right side of the dialog shows the *recently opened* projects. By clicking on one of these projects, the last saved version of the project is opened. #### **Open existing project** This option shows a dialog which can be used to open a saved project. ### Upgrade of projects When opening an existing project a check is done on the version number of the project file. Whenever this version is lower then the version DATPROF Privacy expects an upgrade is required. A dialog opens informing you an upgrade is required. By clicking the OK button a copy of the project and all the scripts is made to the sub-folder "backups" of the project. Then the project is upgraded and a message is shown the project has upgraded. You can find the backup in: **\\\backups\\BACKUP_\_\_\_\_\_\** When you click on Cancel the project remains unchanged. When a higher version number is detected the project cannot be opened. ## Menu Several features of the application can be started through the menu. The menu contains four options categories, each described in the following sub-chapters. ### File ![DATPROF Privacy File Menu.png](https://docs.datprof.com/__attachments/a_ae48755df63f9db2670e20633804342ea5b29f08e0c3d8d343cd490cb9ff74a1/DATPROF%20Privacy%20File%20Menu.png?cb=2056fa492919e59f7a3edaebf0da61fb) * **New Project:** Opens the dialog to create a new project. * **Open existing project:** Opens the dialog to open an existing project. * **Save**: Saves the changes of the current project. * **Save as**: Saves the changes of the current project as a new project. * **Close project**: Closes the current project, then opens the opening dialog. * **Exit**: Closes DATPROF Privacy. ### Project ![DATPROF Privacy Project Menu.png](https://docs.datprof.com/__attachments/a_beda698262a6477cde8abb09a118db55ac130136f2cae12cf46875563f67e8a7/DATPROF%20Privacy%20Project%20Menu.png?cb=b35f94c505faf002b9eea3484de124f3) #### Import and synchronize This option allows the user to import/synchronize database metadata through a wizard (which connects to the database over JDBC), or to import metadata through a DME-file. DME files are a generic DATPROF file format, which are generated by our software when exporting metadata. #### Export metadata to DME-file This option allows the user to export (parts of) a data model to a generic DATPROF file format. This file format can be used within the DATPROF product range to exchange the data model and metadata. #### Generate comments report This option parses all the information in the project, and distills it down into an HTML overview of the data used in the project. For instance, this will show all used functions, their names, comments, description, and other information relating to the project. #### Generate for Runtime This options generates a package with all template code, which can then be uploaded to DATPROF Runtime. When generation is complete, the Windows File Explorer is opened automatically in the directory in which the .zip package is generated. #### Runtime parameters The Runtime parameters menu enables the user to specify either string variables or boolean parameters. These variables are generally used by scripts defined in the **Script Manager** . This can help prevent having to hard-code scripts if a script is being used for for instance several different databases. Once a variable is defined, and saved by pressing the **Save**button, you can start calling it in your scripts. ![DATPROF Runtime Parameters.png](https://docs.datprof.com/__attachments/a_fa41a4b33ab38885b3854cdc9deb50727848d0a1a4853da5f1a66ed4fb0183e6/DATPROF%20Runtime%20Parameters.png?cb=b606c430fbe88b450ef2acd2ca4c2099) ![DATPROF Runtime Variables.png](https://docs.datprof.com/__attachments/a_41a95cf415418e3b7aeb0f9e1bbd075eac2d21e3110e65a8f12acdb85923030d/DATPROF%20Runtime%20Variables.png?cb=5bb4967b663602ce3c2b71bfa973b090) ##### Using Runtime Parameters in SQL scripts You can use parameters in your SQL script too by checking the **velocity script**option in the script manager. Then, to indicate your parameter you can use the following syntax: Syntax: ${dpf.parameters.} Example: ${dpf.parameters.MyFavouriteVariable} Output: MySuperSecurePassword You can also use a collection of standard parameters generated in each Privacy project. If you open Runtime, and look at the settings of a specific application, you can see what each parameter is called. ![DATPROF Runtime Application Parameters.png](https://docs.datprof.com/__attachments/a_0f2980e934f2dd63f1f2a12f4db55e74e64d2004b352e33aca211a862b3fdc50/DATPROF%20Runtime%20Application%20Parameters.png?cb=6bba6caceff11b98afaa2afb0d4e4363) Looking at the above example, we can expect something like the following: Syntax: ${dpf.parameters.} Example: ${dpf.parameters.DPF_TEMP_SCHEMA} Output: AL_T01 On top of this, you can also use the schema map which is configured in Runtime. This ensures that the user is able to dynamically mask, and that scripts also use dynamic features. An example below: Syntax: $dpf.db.format($dpf.parameters.DPF_SCHEMA_MAP.lookup("")) Example: $dpf.db.format($dpf.parameters.DPF_SCHEMA_MAP.lookup("GASWARE")) Output: AL_T01 For some databases, such as MS SQL Server, additional parameters may be required to execute this function. This example merely demonstrates that looking into the schema map is technically possible. ##### Using Runtime Parameters in OS scripts When calling a Runtime parameter from within an OS script, the normal method of calling a variable is sufficient, but annotation as explained in the SQL sub-chapter is also possible, if the **velocity script** option is toggled on. Syntax: %% When using Velocity: ${dpf.parameters.} Example: %MyFavouriteVariable% When using Velocity: ${dpf.parameters.MyFavouriteVariable} Output: MySuperSecurePassword #### Additional Scripts Opens the ***Script Manager*** where you can define scripts (on either a SQL-level or OS-level). For a complete overview, please refer to the **Script Manager**section of this page. #### Enable/disable functions This menu option allows the user to see an overview of all functions configured on the project, and arranges them in containers divided per **Schema / Table / Function**. To start, a user can unfold or fold a tier of functions by pressing the white triangle to the left of any row's checkbox. The user can then check or uncheck the checkbox to enable or disable functions per tier. For example, unchecking the **"Table:CUSTOMERS"**in the example below would immediately disable all functions on the table (in this case 4 different functions). The user can also individually select a function to enable/disable, or opt to disable the entire schema. On top of this, the user has a few options in the top menu: * **Enable all -**Enables all functions in the project. * **Disable all**- Disables all functions in the project. * **Expand all**- unfolds all levels of the function overview, displaying all functions. * **Collapse all**- folds all levels of the function overview, hiding all functions and tables. ![image-20240408-114739.png](https://docs.datprof.com/__attachments/a_cbad1b7d556b7f5fb8cf61b10c6e59d6ad8ff1c1d1cdeb97ca3000ff3dca4ae3/image-20240408-114739.png?cb=f7a108b33337964f82f12584ca23c915) #### Preconfigured Generators This option gives the user the ability to define generator expressions to later reuse multiple times in an existing project, therefore reducing the need for manual re-entry work by users. These are custom generators which are stored at the **project**level, thus allowing you to freely transfer the project between clients and retain the ability to view and modify generator specifications. Upon opening this menu, the user should see the following: ![image-20240408-072420.png](https://docs.datprof.com/__attachments/a_2fc5da331d92fa0ceb990d6845fa4d46206aa8b24d0a30e6d8d29ccb19d87f4d/image-20240408-072420.png?cb=b63c684642be2f4c1efde9cfe6867ad9) ##### Available Options The user is able to configure any generator available under the usual **Generate...** function tray. For a full overview of what each function type does, it's best to refer to the **Generation**chapter of this documentation. **Why not enable all functions?** The reason some functions from the **Masking** component of Privacy are not available is due to functions like the **Shuffle** and **Value Lookup** requiring the variable name of more than one column. ##### Adding a preconfigured generator To add a preconfigured generator, press **Add preconfigured generator**in the the top-left corner of the preconfigured generator screen. After this, an empty generator specification is created in the left-hand tray. Now, the user is able to create a generator in the same way as normally supported. ##### Removing a preconfigured generator To remove a preconfigured generator, simply select the generator from the left-hand tray and press **Delete preconfigured generator**in the top-left corner of the menu. One thing to note is that when removing a preconfigured generator that is already used in the project in one or multiple places, removing the specification will check the project for any remaining referencing functions, and warn the user if the project still has active functions. All functions which use this preconfigured generator must be deleted before deleting the overarching preconfigured generator. ![image-20240408-074929.png](https://docs.datprof.com/__attachments/a_f00358464156c22253e08c0e1ff9a530bc9c4c6903c975296eea1e70b09855fc/image-20240408-074929.png?cb=00675895a96b8da6d875c6e1e5baf5f4) ##### Modifying a preconfigured generator The user can freely modify a preconfigured generator by selecting it from the tray on the left-hand side and editing the function's **properties** . This will apply changes to all currently configured instances of this function in the project. If the user modifies the function type (say, from a **regular expression** to a **random** **date/time**function) then a blocking error will be thrown if any of the existing references to this generator would become invalid due to data type clashes. The user must then first remove invalid references before switching generator types. ![image-20240408-075506.png](https://docs.datprof.com/__attachments/a_023c70d939fbf049cd938c74a17db814de9332a9afc943ee2aba322e3b3f934f/image-20240408-075506.png?cb=9fec5bdef11b352f77bd063ab3ab78af) ##### Adding a configured generator to a column Once you've created a preconfigured generator, you can apply it to a column by selecting a column, adding a function, and selecting the **Generate...** option. After this, you'll see that at the very bottom of the available generator list, there's a header called **preconfigured** . Then, you can simply select your chosen generator, and save the function. All chosen generators are given the default name **GENERATOR_\**, which you can edit afterwards. At this point, this function works just like a regular function, and can be removed as such. Any modifications must be made within the preconfigured generators menu. #### Use legacy deployment As of version 4.15.0 of Privacy, this option has been removed. Enables the pre-4.0 direct deployment scheme. From version 4.0 onward, the default scheme for Direct Deployment will use the embedded Runtime Agent. This option should not be used unless instructed by DATPROF, as it exists to combat a few edge cases. Performance in this mode is worse than in the standard deployment, as well as less functionality being available. #### Bulk Function Wizard The Bulk Function Wizard allows you to quickly apply multiple functions across different columns and tables, significantly reducing manual effort and saving time. Instead of adding functions one by one, you can define a set of operations and apply them simultaneously to multiple fields within your dataset. This feature is particularly useful when working with large databases, where consistent masking need to be applied across multiple columns or tables. By streamlining repetitive tasks, the Bulk Function Wizard enhances efficiency and ensures consistency in data processing, making it an essential tool for large-scale data management. ![DATPROF Privacy Bulk Function Wizard.png](https://docs.datprof.com/__attachments/a_f034ca7fafab4d913b147cc6bdb71bf01dc3f2139cdcc0d366b010f703be89df/DATPROF%20Privacy%20Bulk%20Function%20Wizard.png?cb=7ca1ba954f5de714c3be250e2c792d56) #### Profile Overview The Profile Overview option provides a comprehensive summary of all the profiles configured within a project. It allows users to easily view and manage the different profiles that define specific settings, rules, or configurations for data handling, masking, or transformation tasks. With this feature, you can quickly identify and access each profile, ensuring that the correct settings are applied across various processes. The Profile Overview is an essential tool for efficiently managing profiles in complex projects, offering a clear, organized view that helps streamline workflow, maintain consistency, and enhance overall project management. ![DATPROF Privacy Profile Overview.png](https://docs.datprof.com/__attachments/a_0d3fae492b14beeddfc00c74018e7c62b0e0893db156ca437f014aec19dea5b2/DATPROF%20Privacy%20Profile%20Overview.png?cb=78a704cdbe98aa0859f3001d1b4a9df7) #### Import Profiles The Import Profiles option allows you to easily import profiles that were previously configured in DATPROF Analyze. By importing profiles, you can save time and effort by reusing pre-defined configurations, eliminating the need to recreate them from scratch. Whether you are working with Privacy or Subset, this option simplifies the process of applying standardized data management practices, ensuring uniformity in tasks such as data masking and subsetting across various environments. ![DATPROF Privacy Import Profiles.png](https://docs.datprof.com/__attachments/a_5c5e4b498069aa45574ba8d695ea62267fbd071d89f3f7e63aa997c327a5d573/DATPROF%20Privacy%20Import%20Profiles.png?cb=d8879871e42fe4061302442edd35eaa7) ### Visualize ![DATPROF Privacy Visualize Menu.png](https://docs.datprof.com/__attachments/a_093f101c81962f5e0600984376c3a19f308543df44c9a0ff0ef47fae33d03e60/DATPROF%20Privacy%20Visualize%20Menu.png?cb=135bae9bba6228450d19ad9c9c800a80) * **Visualize data model**: Shows a visual representation of the imported data model, including all (user defined) relations. * **Visualize process model:**Shows a visual representation of the order in which masking functions are going to be executed. ### Help ![DATPROF Privacy Help Menu.png](https://docs.datprof.com/__attachments/a_422c967fbad570003e224a2e55ccd0d463e9a01dbd7979ba6a0466ec4c6dc9ca/DATPROF%20Privacy%20Help%20Menu.png?cb=dd7c141d33c3b088a8bae474c50c8a2e) * **Help** *:*Opens the help-screen. * **Enter license key**: Opens a dialog which can be used to enter a purchased license key. * **Show security key** : Opens a window displaying a security key, [which can be imported into Runtime for application signing and verification](https://docs.datprof.com/runtime/latest/applications). * **About DATPROF Privacy** : Opens the *About Screen* of DATPROF Privacy. * **Debug**: Opens a Developer Logging with a list of threads. ## Generate for Runtime As of DATPROF Privacy version 3 you can separate development and deployment of a Privacy project. This is not available for all databases. Check the chapter Direct Deployment and Runtime Deployment to get the list of available deployment options per database. You can generate a package to be uploaded and installed in your DATPROF Runtime application. Under *Project -\> Generate for Runtime* you can click "Generate" and a zipfile named *\-\-priv.zip* is generated and saved in the subfolder *dpfgen* of your project. ![DATPROF Privacy Generate for Runtime.png](https://docs.datprof.com/__attachments/a_8d084e9902e55069472292017e0b8e83512f5a0ed4ccab0314383da4145ef0d6/DATPROF%20Privacy%20Generate%20for%20Runtime.png?cb=89d229163050383e3173b0040dae4fb2) This package contains the settings, process and all the code to execute the Privacy project. * DATPROF Runtime is a separate product not contained in the Privacy distribution. * Every "*Generate"* empties the *dpfgen* folder and removes existing packages. * Check the Datprof Runtime manual for the compatibility matrix of DATPROF Privacy and DATPROF Runtime ## Script Manager ### Overview In order to access the script manager, the user should press **Project → Additional scripts..** in the top-left corner of the Privacy interface. Here, the user is able to define custom scripts that can be used in combination with masking functions. Both SQL and OS-call scripts are supported. Using these scripts, the user can execute almost any database operation allowed by the database permissions. Care should be taken however that all SQL scripts are executed over a JDBC connection to the database, so results of queries may be slightly different from when testing these in a DBMS. For almost all use-cases this is not the case however, and can be resolved through some syntax changes. In addition to 'plain' SQL, a user can also use PL/SQL (*Oracle* ), Transact SQL (*SQL Server* ) , SQL PL (*DB2*) or PL/pgSQL (PostgreSQL) to define scripts. A script will only execute on the scenarios where it is enabled under the**Scenarios** option. By ticking either the Masking or Generation boxes the script can be (dis-)abled for either modes of deployment. If you want to completely disable a script, you can do this by unchecking both scenarios. The content of the script itself is stored in the **scripts**sub-folder found within the project folder. A SQL script is always executed as a single process and will never run parallel with other functions. This is done because it's difficult to predict which fields will be affected by these scripts, and as such running these parallel could enable the user to create scripts that cause deadlocks on the database. ![image-20240813-082442.png](https://docs.datprof.com/__attachments/a_cc3f93744242129e7149edcf00c385feeb040fc084ea8b58bd6997e0c923243a/image-20240813-082442.png?cb=cbabe3ff333315d2667f64131847e2ee) #### Batching of queries through the script manager Because the script manager functions differently from a native query editor like Microsoft's SQL Server Management Studio or DBeaver, it's not always possible to copy over a query one-to-one. One limitation is that batching queries in the Script Manager cannot be done with the standard separator symbol ( **;**). An example of an incorrect example can be found below. ![image-20231002-083155.png](https://docs.datprof.com/__attachments/a_374f6908ee4f500da8539902d4c4626874c022b75b6382b6524ee8d682dd723a/image-20231002-083155.png?cb=eb9e35d0f0fd2db53dcae6a5cca87e97) In order to execute multiple batches within one transaction (meaning, separate queries which will rollback if one of them fails), the user can specify a special separator command in the body of the query specified in the script manager. This special command is " **!DPF_QUERY_SEPARATOR! "** . If a complex script is being copied over from a database management tool, all query separator symbols can simply be replaced with the separator command, and should **not**remain in the query. This works for all supported database types. A correct example of the above query can be found in the image below. ![image-20231002-083346.png](https://docs.datprof.com/__attachments/a_3c99752d067f624e7fc5a8d521073b17928d47e0bd280c680122de24d029bae4/image-20231002-083346.png?cb=276f39e21cf1b296213f5cbed6d58554) ### Dependencies It's possible that the user requires a script to be executed during a specific moment during deployment. This can be configured by using **dependencies**. If a project is used for both **Generation** and **Masking**, configuring a dependency on a run process step will cause a script to be executed, regardless of which deployment mode you choose to use. This may lead to scripts being executed erroneously. To prevent this, it's prudent to split your generation and masking templates into different projects, or to disable any overlapping functions before starting a run. #### Masking The following script dependency options are available for the **Masking**chapter of Privacy: * Before or after any specific masking function * Before or after any other script * Before or after certain run process steps * **Before disable process:** This is the step that occurs after temporary tables for the run have been created, but before any database constraints are disabled. * **Before main process:**This step occurs after database constraints have been dropped or disabled, but before any masking functions have been executed. * **After main process:** This step occurs when the main masking portion of a run has been completed (all masking functions are done), but before constraints are re-enabled or recreated on the database. * **After enable process:** This step occurs after all constraints have been re-enabled or recreated on the database, but before the clean-up step where all utilized temporary tables are dropped. ##### The dependency editor In order to configure script dependencies, press **edit dependencies...** in the bottom-right corner of the script manager. This will take the user to the **Dependency editor** . An example is included below this paragraph. There are two possible dependency options, configuring another masking function or script as a **predecessor** causes the selected script to be executed before the script you're currently editing. Configuring another masking function or script as a **successor**causes the selected script to be executed after the script you're currently editing. A process visualization is displayed in the bottom half of the dependency editor. In order to assess the entire process, please refer to the **Visualize → Visualize Masking Process Model** or **Visualize → Visualize generation process model**options in the top-left corner of the Privacy application. When filtering through your masking functions, the user is able to search for functions by filtering on schema and table pairs. Any masking function displayed in this screen uses a set syntax. **Format:** TABLE.\ (\) **Example:** CUSTOMERS.Combine generated names (FULL_LAST_NAME) ![image-20230406-100025.png](https://docs.datprof.com/__attachments/a_00251b49f9be6edb00955e1f7936581ad44b8d49d3370d6c79d0238b00274dc6/image-20230406-100025.png?cb=17017e37528c1d7d58889b6f958fef80) #### Generation The following options are available for the **Generation**chapter of Privacy: * Before or after certain run process steps * **Before disable process:** This is the step that occurs after temporary tables for the run have been created, but before any database constraints are disabled. * **Before main process:**This step occurs after database constraints have been dropped or disabled, but before any masking functions have been executed. * **After main process:** This step occurs when the main masking portion of a run has been completed (all masking functions are done), but before constraints are re-enabled or recreated on the database. * **After enable process:** This step occurs after all constraints have been re-enabled or recreated on the database, but before the clean-up step where all utilized temporary tables are dropped. Due to a difference in how the main process is executed between **Generation** and **Masking**, dependency settings for scripts that would trigger before or after a specific generation set is executed are not available yet. ### Velocity For use in direct deployment and on Runtime you may choose to allow for **Velocity** syntax. By default this is disabled, To enable it select the checkbox "Velocity script". Velocity is not available in the legacy direct deployment scheme. Velocity is a Template engine internally used by DATPROF Runtime to parse code, both SQL scripts and OS call scripts. Velocity is an Apache project: . Using this template engine allows users to reference and/or specify Velocity properties in your own SQL code or OS Scripts. You can refer to properties by using the syntax *${identifier.property}* For DATPROF Runtime all DATPROF Runtime Velocity properties are available using the the **dpf.parameters** identifier. Example: *${dpf.parameters.DPF_RUN_DATE}* Scripts are saved in the 'Scripts' subfolder of the project. Bear in mind that Velocity parses # and ##. This might effect your code. To handle this situation you either adjust the code and avoid these characters, mark code as "Unparsed content" or disable Velocity for this script. For more information on the use of Velocity in Runtime check the "Velocity" chapter in the Runtime manual ( ) #### Example Script using Velocity including Comment line and Unparsed content. set LOG=${dpf.parameters.DPF_ARTIFACTS_DIR}\OutpUt-${dpf.parameters.DPF_RUN_ID}.log echo.>%LOG% echo Privacy Run Started at: ${dpf.parameters.DPF_RUN_DATE} >>%LOG% echo Project : ${dpf.parameters.DPF_PROJECT_NAME}>>%LOG% echo Environmnet: ${dpf.parameters.DPF_ENVIRONMENT_NAME}>>%LOG% echo Application: ${dpf.parameters.DPF_APPLICATION_NAME}>>%LOG% ## This line is ignored by Velocity #[[ echo ## End of script>> %LOG% ]]# ## Visualize data model Through the menu option **Visualize → Visualize data model**, a visual representation of the imported data model can be opened. The displayed data model contains all imported entities including all (custom added) relations. Tables which have one or more masking functions defined are highlighted in green. This window also offers functionality for saving or printing (parts of) the displayed diagram. ![DATPROF Privacy Visualize Data Model.png](https://docs.datprof.com/__attachments/a_d9aff8731211d7b300a9519f279e5305f38b47e7e7736d0ad4a673e70d7bf94d/DATPROF%20Privacy%20Visualize%20Data%20Model.png?cb=b71f80a77931e44bf922f608e478a3b6) ## Visualize process model Through the menu option **Visualize → Visualize process model**, the user can view the currently defined functions and scripts as well as the order they will be executed. Solid lines signify a dependency created by the user while a dashed line indicates a dependency created by DATPROF Privacy. DATPROF Privacy creates dependencies whenever two masking functions can target the same table at the same time. This can cause severe data corruption and deployment errors. A dependency created by DATPROF Privacy can be overruled by the user using the Dependency editor. A quick example: Imagine you've made two masking functions. One for FIRST_NAME and another for EMAIL_CONTACT. What you could do is anonymize the FIRST_NAME field (For example "Bob" gets shuffled to "Tim"), and then use this new name in a compound e-mail address (such as [Tim_TEST@gmail.com](mailto:Tim_TEST@gmail.com)). If, by some coincidence, the function that creates the e-mail address goes off before the function that changes FIRST_NAME you could get the following scenario: **FIRST_NAME: Tim** **EMAIL_CONTACT:** [**Bob_TEST@gmail.com**](mailto:Bob_TEST@gmail.com) in this case, you'd want to create a manual dependency that ensures FIRST_NAME always gets modified before EMAIL_CONTACT. ![DATPROF Privacy Visualitzation with Dependencies.png](https://docs.datprof.com/__attachments/a_d9b322bb8102f4b29c804f1207ef3252e58d60d0c27fa022bceadb0658969a9c/DATPROF%20Privacy%20Visualitzation%20with%20Dependencies.png?cb=991c12127df0acaf80f28e6d9fd31f73) This window also offers functionality for saving or printing (parts of) the displayed diagram. ## DME files The application allows the user to exchange data models using the *Data Model Export (DME)* functionality. This functionality saves the data model in a generic DATPROF file format. The resulting file can be used within the DATPROF product range. Through the menu option **Project → Export Metadata to DME file**, the project data model can be exported to a DME file. In the dialog the user can specify the tables (per schema) to be exported. The corresponding foreign keys can be exported too. Using DME Files has some limitations: * DME files do not support the use of multiple schemas * Foreign Keys with a condition are not exported ![DATPROF Privacy DME Exporter.png](https://docs.datprof.com/__attachments/a_9c79d16cd8ad86e472a7afdf0de2a9300cd4a4a75077ee560d10520df93b3fdd/DATPROF%20Privacy%20DME%20Exporter.png?cb=e93e94d1af66fa50cc4cf6a180d545aa) Using the menu option **Project → Import and Synchronize metadata from DME file** the user can import a data model from a DME file into their own project. --- version: "4.23" language: "en" --- # Using DATPROF Privacy DATPROF Privacy simplifies the process of managing, masking, and generating data within databases, enabling secure and efficient handling of sensitive information. This tool is designed to help organizations ensure compliance with data privacy regulations while maintaining data usability for testing and development purposes. Key topics covered include: * **Getting Started** Begin your journey with DATPROF Privacy by creating your first project. Learn how to establish a secure connection to your database and seamlessly import metadata to set the stage for effective data masking or generation. * **Masking a Database** Dive into the powerful masking features of DATPROF Privacy. Anonymize sensitive data while maintaining its usability, ensuring compliance with data privacy regulations and safeguarding your information. * **Generating Data** Unleash the potential of synthetic data generation. Discover how to create realistic, synthetic data to fill gaps or simulate real-world scenarios without using sensitive information. --- version: "4.23" language: "en" --- # Data Generation For certain use-cases it's not possible to mask existing data, and instead the user must generate new data. Generation makes this possible, giving the user the ability to generate any amount of rows for existing tables, while maintaining all of the existing constraints. It does this by allowing the user to create and configure different generations sets that generate new data. A **generation set**is a specific set of tables and their corresponding functions that can be configured to generate data that covers specific test data requirements. For example, it's possible to create a set that creates 10 rows of female customers between the age range 45 - 65, with specific contracts on specific products. This means that the user can generate as many sets of varying data for a table as is needed by creating multiple generation sets. ![image2020-11-10_12-12-38.png](https://docs.datprof.com/__attachments/a_01a8a8f1c9bdb985081ce556eb1009222f2f791ad4fb36d6e564a1e4b84dd1ca/image2020-11-10_12-12-38.png?cb=d2b7f42e338a2d402ae641b73e5b6497) ## Creating a new generation set To add a new generation set, click the **Add generation set**button in the toolbar, located in the top-left corner of the Privacy interface. This will create an empty generation set, which the user can then select and re-name to their chosen name. Additionally, the user can supply a description for the generation set. After a generation set has been added, selecting it will show a list of all imported columns within the table. In order to include columns in the generation set, check the tick-box next to the column name. Because rows are generated based off of this configuration, any columns not included in this selection will generate NULL values. ### Table settings ![image2020-11-10_14-11-32.png](https://docs.datprof.com/__attachments/a_0a76b1ad62735f48e0d98b810b0fc2267f424416dce1565134cb353ceb0df448/image2020-11-10_14-11-32.png?cb=18450d19335173c9b366639a811aaf0a) #### Row count It's possible to specify how many rows a specific generation set will generate by selecting the generation set and modifying the **Rows**value. By default, this is set to 100. #### Truncate Selecting this option will cause Privacy to execute a TRUNCATE TABLE statement on the table involved in the generation set. When using multiple generation sets on one table, this option should be left unchecked. ## Cloning generators sets and copying generator specifications Specifying different generators on a long list of columns and tables can be time consuming. Within generation it's possible to copy generation sets, which maintains the generation set specification. Afterwards, the user can modify the new generation set. To do this, select the generation set and click **Clone generation set**. Each table and their generator properties are copied to the cloned generator set. Care should be taken that certain (primary) keys or unique columns should be modified to prevent duplicate key issues. There is also an option to copy the generator specification of a table or column and paste it in a different generator set. Right click a table or column and select **Copy generator specification,** then right click a target table or column and click **Paste generator specification.**If you copy the complete generator specification of a table, the table on which you can paste it should contain the exact same columns. Otherwise the paste option is disabled, preventing the user from performing an illegitimate action. ## Foreign Key There are different methods to fill foreign key values! You can choose a constant if you want to refer to a specific value in the parent table. In that case you have to make sure that the referring key is present in that table. The **Foreign Key** generator however uses a lookup to actually get values from the parent table. There are two possible scenarios; * The parent table is part of the same generation set; in this case keys are generated that refer to the new parent records generated in the same generation set * The parent table is **not** part of the same generation set; in this case keys will be generated using current data content in the parent table ## Random versus Deterministic Masking All generators that use seed lists (for example; First name, Last name, Company, Job, User-Defined), can be randomly or deterministic replace the original values. When replacing original values in **Random mode** , each value will be replaced with a random value from the seed list. When the *Replace the same original values with the same generated value* is checked, it will make sure that multiple occurrences of the same original value, will be randomly replaced with the same seed value. In **Deterministic Mode**each occurrence of the original value will be replaced consistently with the same value. Each time that you execute the masking template the value will be masked consistently. When masking for example names in Oracle and SQL Server with different templates, you can achieve consistent results by using the same seed list(language), generator salt and global salt without the use of any translation tables. If you change the global salt in the Deployment - Settings this will affect all generators that are configured as deterministic. For each individual generator you can also specify a separate salt. **About Generators** The Generator options displayed in the drop-down list will reflect the datatype of the column in focus for masking. For instance, you cannot use the "First Name" generator on an integer column as this would violate the definition of the column. Likewise, it is not possible to generate a sequential number for a date-time column. Therefore, these functions will be filtered depending on which type of column you're working with. ## Standard Generator Functions The available generators as provided by Privacy differ slightly based on the data type of the underlying column. Because there are (database specific) variations on data types, we'll be grouping these together in a few groups. **Strings** (all data types focusing around text, from varchar fields to BLOBs), **numerical** and **date \& time** data types. | **Function name** | **Supported Data Types** | **Description** | |--------------------------------------------------------|-------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Constant Value | String, Numerical, Date \& Time | Creates a constant value in the generated dataset that is identical for every row. I.e. inputting **MyFavoriteValue**here will generate 'MyFavoriteValue' for every row in the resulting column's dataset. | | Random Date/Time | String, Date \& Time | Generates a random Date/Time value per row between a specified minimum and maximum datetime that corresponds with the underlying column's datatype. | | Sequential Date/Time | String, Date \& Time | Identical to the Random Date/Time generator except that this generator creates a sequential datetime for every row. Supplying a maximum datetime is optional. A number to increment the starting date is required, and can only accept whole numbers. Any unit of time to increment by can be chosen, from seconds to years. | | Random Decimal Number | String, Numerical (except for integer variants) | Generates a random decimal number between a supplied minimum and maximum value for every row. Using the **Scale**setting the decimal accuracy can be defined. For instance, a generator with a minimum value of 0 and a maximum value of 1000 using a scale of 4 might generate 132.4202. | | Random Whole Number | String, Numerical | Generates a random integer between a supplied minimum and maximum value for every row. | | Random String | String | Generates a random string of lower- and uppercase letters for every row. The minimum and maximum length for strings can be defined. | | Sequential Number | String, Numerical | Generates a sequential number for every row that starts at a specified value, and increments by the step value per row. Additionally, you can define a **Padding**for your generated integers. This is a set number that will be affixed to the generated integer. For example, using a padding of 3, and a start of 8 with a step of 2 will generate the following: **008 → 010 → 012** | | Names | String | The Names section contains multiple generators that generate data based on CSV files supplied by DATPROF. The available options are: * Brand * First Name * First Name (Male) * First Name (Female) * Last Name * Full Name * Company * Dinosaur * Random Word For some options (such as First Name) there are localization options which determine which CSV file is used. These can be toggled on/off with the **language(s)**drop-down menu. Multiple languages can be used simultaneously. | | 2 Letter Country Code / 3 Letter Country Code | String | Generates a random country code per row either in a 2 letter country code, or a 3 letter country code format. | | City | String | Generates a random city name per row. Here, you can specify for which countries you'd like to generate random names using the **Countries**drop-down menu. | | Country | String | Generates a random country name per row. The **Language(s)**option specifies in which language the country will be written. Multiple options can be enabled simultaneously. | | Street | String | Generates a random existing street name per row. The **Countries**option allows you to specify for which country street names will be generated. | | A-Number / GBA number (Dutch township security number) | String, Numerical | Generates a 10 digit GBA number per row. | | BSN | String, Numerical | Generates a valid Dutch social security number per row. | | Color | String | Generates a random color per row. (Ex. Baby Blue, Sky Blue, Soft White) | | Color Code | String | Generates a color hexcode (Both three-digit shorthand and six-digit full length hexcodes) per row. | | Credit Card Account Number | String, Numerical | Generates a random credit card account number. Using **Issuer(s)**, you must define one or multiple issuers to determine which syntax the generated account numbers adhere to. | | Currency Code | String | Generates a three letter currency code for every row. | | Currency Symbol | String | Generates a currency code for every row. | | Genre | String | Generates a media genre for every row. | | IBAN (International Bank Account Number) | String | Generates a valid IBAN number for every row. Using **Country Code(s)**you can specify which country codes you'd like your resulting IBAN codes to use. | | Job / Profession | String | Generates a profession for every row. Using the **Language(s)**you can specify which language(s) you want your resulting job names generated in. | | Military Rank | String | Generates a military rank name for every row. Using **Department(s)**you can specify which branches of the armed forces you'd like to include in your resulting dataset. | | SSN (US Social Security Number) | String | Generates a SSN for every row. You can specify how you want to separate your resulting numbers. You can choose one of the following formats: **None** (Example: 101010101, 003122142) **Dashed**(Example: 101-01-0101, 003-12-2142) **Spaced** (Example: 101 01 0101, 003 12 2142) | | User agent | String | Generates browser user agents like Mozilla/4.0 (compatible; MSIE 5.13; Mac_PowerPC), Opera/8.53 (Windows NT 5.2; U; en) per row. | ## Advanced Generator Functions ### SQL Expression The SQL expression generator is special in that it can use values from other columns. All other generators will generate it's values into a temporary table per generation set inside your database. The SQL expression on a column is used when the data in inserted back into the original table. That way a user can use the data that is generated in other columns. No dependency specification is required since SQL Expression rules will be the last to be applied to the table in focus. ### Value from seed file Sometimes you may wish to provide your own replacement data to the masking process. This could be because of language, country/region or specific business reasons. You can generate your own seed (User Specified) file and specify it using this technique. The generator will randomly source a row from the file for each row in the table which is qualified by the Condition clause. ### Notes on Seed files * Seed files are generally saved as .txt files. * The Encoding must be explicitly specified. The rule default is UTF-8. * An empty row in the seed file will randomly populate the column in focus with an empty value. ### Value from multi-column seed file Generate data across multiple columns that should be consistent based upon a multi column seed file. For example; replace a city, postal code and street based on correlated seed file containing addresses. You can map multiple columns of the table to the specified seed file. This file shouldn't contain headers and must be a comma separated (.CSV) file. ### Generator expression Combine different generators together with static text. Each generator can be referenced with the $ symbol. Entering the $ symbol in the expression editor will open a list of available generators. Pick one from the list; dependent upon the generator you will be presented with options to specify the generator parameters. **Examples:** |---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Email address | $random-regex(regex="\[A-Z\]{1}").$random-last-name("US")@$weighted-random-generator(values="[gmail.com](http://gmail.com/),[outlook.com](http://hotmail.com/) ",weights="30,70") This will generate: [P.Sabella@gmail.com](mailto:P.Sabella@gmail.com), [O.Veach@hotmail.com](mailto:O.Veach@hotmail.com), [R.Levy@hotmail.com](mailto:R.Levy@hotmail.com), etc | #### Syntax overview | Generator | Generator expression syntax | |------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------| | Random Date/Time | $datetime-generator(min="1970-01-01 00:00:01",max="2038-01-19 03:14:07",format="yyyy-MM-dd HH:mm:ss") | | Random decimal number | $random-decimal(min="0.0",max="1000",precision="0") | | Random whole number | $random-integer(min="0",max="1000") | | Random string | $random-string(min="5",max="20") | | Sequential number | $sequential-number-generator(start="0",step="1") | | Brand | $random-brand() | | Company | $random-company("NL","DE") | | Female first name | $random-first-name-female("IT","DA","DE") | | Male first name | $random-first-name-male("ES") | | Last name | $random-last-name("IT","DE","NL") | | 2 letter country code | $country-code-2() | | 3 letter county code | $country-code-3() | | City | $random-city("NL","DE","US") | | Country | $random-country("DE","FR","NL","US") | | Street | $random-street("DE","NL","US") | | A-Number/GBA Number | $gba-generator() | | BSN (Dutch Social Security Number) | $bsn-generator() | | Color | $random-color() | | Color code | $random-color-code() | | Credit Card account number | $credit-card-number(issuers="Maestro") | | Currency code | $random-currency-code() | | Currency symbol | $random-currency-symbol() | | Genre | $random-genre() | | IBAN | $iban-bic(countries="NL") | | Job / Profession | $random-job("US") | | SSN (US Social Security Number) | $ssn-generator() | | User Agent | $random-user-agent() | | Regular Expression | $random-regex(regex="\[A-Z\]{1}") | | Value from seed file | $random-seed-value(file="C:/file.csv",charset="UTF-8") **Or, alternatively:** $random-seed-value(file="C:\\\\file.csv",charset="UTF-8") | | Weighted list | $weighted-random-generator(values="F,M",weights="50,50") | ### Regular expression Generates values based upon a regular expression. The syntax for the regular expression used in Privacy is specific to the package we use, so please refer to the specifications [here](https://www.brics.dk/automaton/doc/dk/brics/automaton/RegExp.html). **Examples:** | Function name | Regular Expression | |-------------------------------------------------------|----------------------------------------------------------------------------------------------------------------| | **Address Type** Pick a random value from a list | Billing\|Delivery\|Home\|Office\|Primary\|Shipping | | **Dutch Postal Code** 3067ZG, 9372ED, 8423BE | \[1-9\]\[0-9\]{3}\[A-Z\]{2} | | **US Zip Code** 48357/4986 , 19414, 21237-6415 | \[0-9\]{5}(\[- /\]?\[0-9\]{4})? | | **IP address** 250.171.5.8, 4.8.5.180, 91.252.251.253 | ((25\[0-5\]\|(2\[0-4\]\|1{0,1}\[0-9\]){0,1}\[0-9\])\\.){3,3}(25\[0-5\]\|(2\[0-4\]\|1{0,1}\[0-9\]){0,1}\[0-9\]) | ### **Weighted list** Generates values based on user specified, comma separated, values. The weights parameter is a comma separated list which gives each value a proportional weighting. **Examples:** Values: Male, Female Weights: 60, 40 In this case 60% of the values that are generated are Male and 40% of the generated values are Female. ### Value from list Note: This feature can use AI to generate data. Please review our [AI Disclaimer](https://docs.datprof.com/privacy/latest/ai-integration-disclaimer) Use this generator when you want to supply a predefined set of values that the masking process will randomly choose from. This is useful when the valid replacement values are limited, controlled, or should belong to a specific domain such as product categories, department names, status codes, or organizational units. You can define the list manually within the properties box or use the help of AI to generate a list of values. Each time the generator will randomly select one value from the list and apply it to the column. The file **must contain one value per line**. Each line represents a possible replacement value. Example list file: Finance Operations Sales Human Resources Logistics Generate your own list file with the help of the AI Assistant. Before you can use it, navigate to **Help → AI Assistant Settings** and enter a valid OpenAI API key to get started. ![DATPROF Privacy AI Value From List.png](https://docs.datprof.com/__attachments/a_cb41e3cbc1ca8798bd83335a6d037ede4798b180b046e97a1e0b4dca25684e1c/DATPROF%20Privacy%20AI%20Value%20From%20List.png?cb=b40b7487bcb8e2cd7d4745d2b86ca10b) ## Generator settings ### Post SQL Expressions You can further modify data on a generator set by filling in the Post SQL text box on any generator function. This expression will trigger after a field has been generated, and can be used to further customize the generated data. You can also reference different fields in the table and use them in the expression here. In terms of functionality, the syntax is identical to regular SQL expressions. ![image-20221004-083343.png](https://docs.datprof.com/__attachments/a_35713adec4fca15b1b815ed6d07268a37f57014a47e2403727cea953b161e32c/image-20221004-083343.png?cb=d1a1c4540753c69b33c2ae6c7e345cc3) ### Generator Seed If you generate data and need the data to be identical upon later execution, you can specify a seed value. Doing this will ensure that whenever you run this generator set again, your data will be identical. This can be useful if changing data can impact testing in a negative way. However, if you intend to execute a template multiple times on the same dataset, duplicate data will be generated. For more information on this general principle, please refer to the **Deterministic Masking**portion of the Privacy documentation. ### Null Values % Using this setting you can seed a percentage of NULL values into your resulting dataset. Having NULL values can be an important part of testing, as missing data occurs regularly in production databases. However, if you specify a field as having a NOT NULL constraint in the database, attempting to seed NULL values in the resulting dataset will cause a run to fail. --- version: "4.23" language: "en" --- # Data Masking In the masking chapter we'll go over the Masking tab in Privacy, and the various masking functions available to the user. In order to configure any functions in a project, meta data has to be imported after project creation. ## Schema / Table overview On the left-hand side of the Privacy interface the user can consult the metadata overview. This is a collection of tables and schemas imported into the project divided by per schema. To switch between the currently depicted schema content the user can click on the drop-down letterbox at the top of the table overview window containing the name of the currently active schema. ![Schema and Table overview.png](https://docs.datprof.com/__attachments/a_fb267a0906d15b743558828112b028de75edd229904ed32ec32e703e90cd2d60/Schema%20and%20Table%20overview.png?cb=a0cad04ca747613b25c7fefe3188dde4) ### Searching tables In order to search for tables, the user can enter a search query at the top of the schema overview. As the search result gets filtered down results that do not comply with the query will be hidden from the overview, leaving the user with only matching tables. On top of this, the drop-down box containing the name of the schema also indicates how many matches are found using the currently active search query. If the user opens this drop-down box, it will indicate the amount of matches per schema. If a matching table is located in a different schema, the result will not be shown automatically to the user. ![image-20230808-075102.png](https://docs.datprof.com/__attachments/a_9e47c65133cacd12c94df8a23ffa6b3d713ec15cd4317fb8361c30511a85a20c/image-20230808-075102.png?cb=10d9689f786103bf08c32d489e96cff3) The user can supply wildcard characters in the search query as well, in order to find tables using partial matches. **Example:** Imagine we want to find all tables containing the word "supplier". Supplying "supplier%" as our search query would make sure all tables starting with the word supplier with any characters after it are matched, such as**SUPPLIER_CONTACTS** , **SUPPLIER_PRODUCT** , and **SUPPLIERS** , but not **DAIRY_SUPPLIERS**. ### Right-click context menu The user can right-click on any table in the table overview to do one of four actions: * Count the number of rows in one or more table(s), for which an active database connection is required. The result is stored as a value next to the name of any given table. * Clear any previously generated table row counts. * Delete the table(s). This merely deletes the table from the Privacy template and does not affect the data model itself. Any existing functions and configurations on a table are simultaneously removed. * Sort the displayed tables in one of four ways: table name (ascending), table name (descending), row count (ascending), and row count (descending). You can select multiple columns using the standard Windows \ or \ options to select a range or selection of columns. ### Additional Options As of now there are two additional options available below the table overview: * Hiding all tables without functions. * Hiding all tables with zero rows. This option only affects tables of which the row count is known, so a row count is required beforehand. ## Column overview In the column overview on the right-hand side of the masking tab the user can access available information about all imported columns in a table. Depending on the chosen database type, slightly different notation for metadata traits can be displayed. ![image-20230808-081619.png](https://docs.datprof.com/__attachments/a_c8d523674fa590f43afa13135f3a88209a6c1698411351470e74c463dfce8bc6/image-20230808-081619.png?cb=0fcfbff7e0415c38c03f99ec7841f823) ### Searching Columns Searching columns in the column overview works largely in the same manner as the searching tables, which is highlighted in the **Searching tables** section of this page. One key difference is that instead of hiding any results that do not match the search string, the column overview highlights any matching columns in green, unless the **Hide other columns**button on the right side of the search bar is ticked. ### Right-click context menu The user can right-click on any column in the table overview to do one of three actions: * Add a masking function. This option allows the user to select a valid masking function and create it on the select column(s), opening the function's configuration menu. Certain functions can be applied to multiple columns at once, such as the **Blank** function. * Toggle primary key. This option indicates a column as being a primary key in Privacy. This is useful is a primary key is not correctly imported due to database configuration. * Toggle translation key. This option indicates a selected column as a translation key. This functionality is explained later in the **Saving a translation table**chapter of this page. ## Data masking functions The following data masking functions are available in DATPROF Privacy: * **Blank** * **Scramble** * **Shuffle** * **First day in same month/year** * **Value lookup** * **Random Lookup** * **Custom Expression** * **Generate (containing multiple generation types)** * **JSON Functions** Masking functions are all data type dependant, and are automatically enabled or disabled depending on which column is selected. The Datatype and Function matrix at the end of this manual lists, per database, the functions available for the different datatypes. ### Blank This function will NULL the selected column(s). This function can't be used on columns which are specified as NOT NULL or columns upon which a Primary Key (PK) is defined. **Examples** | **Original value** | **New Value** | |--------------------|---------------| | Abcdefg | NULL | | 123456 | NULL | ### Scramble This scramble function ensures that for numeric data types all characters are replaced with the number '1 ', and alphanumeric data types all characters are replaced by the symbol 'x' or 'X', depending on whether the original character was capitalized or not. The length of any supplied cell is maintained. **Examples** | **Original value** | **New Value** | |-------------------------------------------------|-------------------------------------------------| | MindBox | XxxxXxx | | [email@provider.com](mailto:email@provider.com) | [xxxxx@xxxxxxxx.xxx](mailto:xxxxx@xxxxxxxx.xxx) | | 0612345678 | 1111111111 | | José Nemeç | Xxxé Xxxxç | For MySQL version 5.x databases the scramble function is simplified due to limitations on database functions. All alphanumeric characters are instead replaced by "XXX". Characters with diacritic marks are left unchanged ### Shuffle The shuffle function ensures that all values in the selected column(s) are randomly mixed, like shuffling a deck of cards. The data distribution remains the same, which will ensure that the same values in the former situation, also obtain the same new values. A normal shuffle is a shuffle without selecting the group check-boxes. Example of the Function Editor with a normal shuffle: Shuffle LAST_NAMES. ![image2019-2-21_14-34-30.png](https://docs.datprof.com/__attachments/a_0af4531c68b7b33e90934c530c248e4be9ecf4b2290515b085dfbda3c7e3c3ab/image2019-2-21_14-34-30.png?cb=a02c64dcb65ec6dc2905acbca4a7c988) If the function is applied to more than one column at the same time, a *multi column shuffle* is applied. The values of the selected columns remain together and will be shuffled as a whole. In this way, for example, the gender and a first name are held together. The option *'Exclude null values from shuffling'* ensures that NULL values will not be considered for shuffling, retaining their place in their specific row(s). The random order on a low number of records may result in original values. Consider this: the random order of 1,2,3 may well be 2,1,3, keeping "3" the original value. **Examples** | **Original situation** || **New situation** || | FIRST_NAME | LAST_NAME | FIRST_NAME | LAST_NAME | |------------|------------|------------|------------| | Steve | **Jobs** | Steve | **Gates** | | Bill | **Gates** | Bill | **Jobs** | | Blaise | **Pascal** | Blaise | **Page** | | Larry | **Page** | Larry | **Pascal** | | Melinda | **Gates** | Melinda | **Jobs** | A shuffle is a random function on the unique distinct values of the set. See "Gates" in the example above. Both Melinda Gates and Bill Gates were renamed to have Jobs as their surname, because they shared the same original surname. ### Group Shuffle The Group shuffle is a variation of the shuffle. In the application you add this function as a normal function and also check the Group check-boxes. You may select one or more columns as a group. Within this group the unique values of the selected columns are shuffled. NULL or Empty values in the Group column are considered as a group. By using a group shuffle fewer values are used to shuffle because only unique values within in a group are used to shuffle. For this reason the "group" column(s) should result in a reasonable number of values to shuffle. Example of the Function Editor with a group shuffle: Shuffle FIRST_NAMES grouped by GENDER. ![image2019-2-21_14-32-20.png](https://docs.datprof.com/__attachments/a_c71fa6afefd58597ce3cbf3f4c22cc5df545b1b08eeabc8454bc48beeee209ee/image2019-2-21_14-32-20.png?cb=ae253decdfcc8307dd33f8aa08344049) **Examples** | **Original situation** || **New situation** || | FIRST_NAME | GENDER | FIRST_NAME | GENDER | |-------------|--------|-------------|--------| | **Steve** | M | **Kelly** | M | | **Kelly** | M | **Steve** | M | | **Blaise** | | **Blaise** | | | **Kelly** | F | **Melissa** | F | | **Melissa** | F | **Kelly** | F | | **Kelly** | M | **Steve** | M | | **Melissa** | F | **Kelly** | F | | **Maxima** | | **Maxima** | | Above we see three Gender groups: M, F and Empty. Kelly exists as Male and Female. Blaise and Maxima have no gender. The group shuffle using FIRST_NAME as column and GENDER as Group results in shuffling three groups where the empty group, having only two records, does not change. It's important to make sure that the field you use to group your result is: * Complete. Having NULL values in your dataset can result in unmasked fields * Consistent. If you group by gender like in the above example and have a small percentage of fields with for instance "V" instead of "F", you can introduce fields that are shuffled in a tiny group, which can cause issues. ### Fixed day in same month/year This feature can be used for date fields only. Here the user has the ability to change the existing date to a fixed day in the same month. Or to a fixed day in the first month of the same year. With this change, in most cases the new values remain functionally viable. ![Function Editor Fixed Day.png](https://docs.datprof.com/__attachments/a_f310f95f42c88f077ce375995674b9b02a526bb139a7bac548f3b86b9c31f70d/Function%20Editor%20Fixed%20Day.png?cb=753cef83377f9c953f617111716a55e8) **Examples** | **Fixed day: 13** | **Original value** | **New value** | |---------------------------------------|--------------------|---------------| | Fixed day in same month | 23-04-2018 | 13-04-2018 | | Fixed day in first month of same year | 16-09-2020 | 13-01-2020 | ### Custom Expression **Note:** This feature ***can*** use AI to help with your custom expressions. Please review our [AI Disclaimer](https://docs.datprof.com/privacy/latest/about-datprof-privacy/ai-integration-disclaimer) The '*Custom Expression'* is a generic function where any and all database platform functions can be used to manipulate the data in the selected column. **Examples** | **Function** | **Original value** | **New Value** | |---------------------------------------------------------------------------------------------------------------|--------------------|---------------| | substr(\,1,3) | Datprof | Dat | | 'company' | Datprof | company | | translate(\,'aeiouyAEIOUY','\*\*\*\*\*\*\*\*\*\*\*\*') | Datprof | D\*tpr\*f | | \ / 4 | 10 | 2.5 | | \ - 2 | 21-05-2015 | 19-05-2015 | | regexp_replace('\','(\[\[:digit:\]\^\]{1})(\[\[:digit:\]\^\]{1})(\[\[:digit:\]\^\]{1})','\\3\\1\\2') | 789 | 978 | To verify the syntax of the expression it can be tested using the **Test** button which will send the defined expression to the database for verification. The verification shows if the expression is valid or not with a message next to the **Test** button, per the panel below, if invalid. A valid expression will return '*Query validated'* Testing an expression requires an active database connection. Starting with Privacy version 4.21.0, you can now use AI to assist in creating custom expressions. Always review and validate the AI‑generated output before using it. ![DATPROF Privacy AI custom expression query creator.png](https://docs.datprof.com/__attachments/a_55a686ac18539b48abd94674678f0f5fd43300b4944de21b55b7df43bf5bb32e/DATPROF%20Privacy%20AI%20custom%20expression%20query%20creator.png?cb=24792f2b5307ae2a034932ad98a9d119) ### Value Lookup With this function the replacement value will be obtained from a**lookup table** or **translation table**. This translation table may well be the result of a previously executed Privacy function where, for instance, consistent masking was implemented using the translation table methodology. ![image2018-5-8_15-2-36.png](https://docs.datprof.com/__attachments/a_73d5aeecfc0cd8eabf91e158a1965fcdd66d4c86af672b7239bdddf227a6e614/image2018-5-8_15-2-36.png?cb=c02f459937902d4fad7e2412cf405c4a) The field ***Name*** can be used to enter a meaningful name. The field ***Description*** allows you to enter a description of the rule which will be displayed in the *comments report.* The field ***Lookup Table*** is a combo-box which allows the user to either select a table from a list or specify a table name manually. This lookup table contains both the columns to search for and the columns containing data to replace the original values with. If another Privacy function uses the Translation Table feature the result is saved as a *Translation table* and will be shown in the drop-down list. If the Lookup table is built outside a Privacy function it will not be present in the list and must be manually entered. A manually entered Lookup table should have a fully qualified name including the Schema name and/or Database name. i.e: \[db1\].dbo\].\[mytable\] or "SCHEMAOWNER1"."VL_NAMES" or "schemaowner2"."vl_names" **Using a view as a lookup table** A manually entered lookup table may well be a view created by you or your DBA. Depending on the complexity of the view and size of the dataset you may encounter a significant performance penalty. Consider creating a temporary table instead of using a view. Benchmarks show a performance boost up to 95% by using a table instead of a view. The search criteria can be entered under **Input mapping**. The **Input column** is a drop-down list where the search column can be selected. The **Lookup column** is a combo-box where, if the Lookup table is generated by a Privacy function, the fields to search for are listed for selection. With other tables the field name should be entered manually. You can enter SQL functions here to filter the provided field. I.E. if you have a composite key (8 digits for customer number + 3 digits for region code for instance), and you want to search based on the first 8 digits, you could do the following in an Oracle environment: **SUBSTR(COMPOSITE_KEY,1,8)** This can turn a number such as 12345678900 into 123456789. Under**Output mapping** it is possible to select the columns to replace the original values with replacement data from the Lookup table. The **Original column** is a drop-down list where the column name must be selected with the data to be replaced. The **Lookup column**is a combo-box where, if the Lookup table is generated by a Privacy function, the column in the Lookup table can be selected from a list to replace the Original data with. With other tables the column name should be entered manually. The **If value not found then** feature provides options to take should the Original column value not exist in the Lookup table. Options include retaining the original value, NULL'ing the item, specifying a Custom Expression or Scrambling (redacting) the existing value. ### Random Lookup Using this function, which is a variation of the *Value Lookup* , a new value will be randomly obtained from a *'Lookup'* or *'Translation'* table. ![image2018-5-8_15-4-11.png](https://docs.datprof.com/__attachments/a_76fe32038cd468e70057a6134e756b7c8fe9be700d30e5b7f6f6f83b63a97a8b/image2018-5-8_15-4-11.png?cb=1f4493744b24310eccae24563497096a) When using a *Random Lookup* an equivalent value in two or more tables will not be consistently replaced. As with the *Value Lookup*a Lookup table and a Lookup column can be chosen from a combo-box list or be entered manually. A manually entered Lookup table should have a fully qualified name including the Schema name and/or Database name. i.e: \[db1\].dbo\].\[mytable\] or "SCHEMAOWNER1"."VL_NAMES" or "schemaowner2"."vl_names" ### JSON Functions DATPROF Privacy can detect JSON and JSONB column structures and apply masking or generation functions to individual fields within those documents using JSONPath expressions. This section covers how to configure JSON masking rules, write correct JSONPath selectors, and understand the behaviour of attribute ordering. JSON/JSONB masking is currently available for PostgreSQL, Oracle, SQL Server, and DB2 LUW. Support for additional databases is planned for a future release. **Important note on attribute ordering:** When processing JSON, the order of attributes in each JSON object cannot be guaranteed. Attributes are written out alphabetically. While this does not affect how data reading software interprets the file, it may appear unusual when inspecting the file manually. #### JSONPath To mask fields in your database, you'll need to define them using [JSONPath](https://jsonpath.com/). JSONPath provides a structured way to locate and reference specific fields within your data. This flexibility allows you to mask fields at various levels of your data hierarchy. The JSONPath expression always starts with $ (the document root) and uses dot notation to navigate to the target field. In the example below, you can see a demo database structure. To mask the field **first_name**, you can use its full JSONPath: $.first_name. For more complex structures, such as arrays, you'll need to use the JSONPath to navigate to the nested fields. For instance, consider an array named **customerhistories** that contains multiple fields. To mask the **first_name** field within this array, the JSONPath would be $.customerhistories.first_name. This ensures that the masking process targets the correct field within the array. Examples: |-------------------|-----------------------------------------------------| | **JSONPath** | **Targets** | | $.first_name | The first_name value at the root of the JSON object | | $.last_name | The last_name value at the root of the JSON object | | $.email | The email value at the root of the JSON object | | $.address.street | The street field inside the nested address object | | $.address.city | The city field inside the nested address object | | $.address.country | The country code inside the nested address object | #### Value lookup Value Lookup replaces the content of a JSON field with a value drawn from a reference (lookup) table. The replacement is deterministic: the same source value always maps to the same masked value across multiple runs, preserving referential integrity. We'll use this sample data to demonstrate the Value lookup function for JSON: {"id": 1, "age": 21, "tags": ["admin", "premium"], "location": "New York", "last_name": "Smith", "first_name": "Elias", "preferences": {"theme": "dark", "language": "en", "notifications": true}} {"id": 2, "age": 22, "tags": ["user"], "location": "London", "last_name": "Jones", "first_name": "Tarah", "preferences": {"theme": "light", "language": "es", "notifications": true}} {"id": 3, "age": 23, "tags": ["user"], "location": "Madrid", "last_name": "Diaz", "first_name": "Erica", "preferences": {"theme": "dark", "language": "fr", "notifications": false}} {"id": 4, "age": 24, "tags": ["user"], "location": "Berlin", "last_name": "Brown", "first_name": "Tad", "preferences": {"theme": "light", "language": "de", "notifications": true}} {"id": 5, "age": 25, "tags": ["user", "beta-tester"], "location": "Paris", "last_name": "Wilson", "first_name": "Francis", "preferences": {"theme": "dark", "language": "ja", "notifications": true}} {"id": 6, "age": 26, "tags": ["user"], "location": "Tokyo", "last_name": "Taylor", "first_name": "Maryalice", "preferences": {"theme": "light", "language": "en", "notifications": false}} {"id": 7, "age": 27, "tags": ["user"], "location": "Sydney", "last_name": "Anderson", "first_name": "Aron", "preferences": {"theme": "dark", "language": "es", "notifications": true}} {"id": 8, "age": 28, "tags": ["user"], "location": "Toronto", "last_name": "Thomas", "first_name": "Delila", "preferences": {"theme": "light", "language": "fr", "notifications": true}} {"id": 9, "age": 29, "tags": ["user"], "location": "Dubai", "last_name": "Jackson", "first_name": "Andy", "preferences": {"theme": "dark", "language": "de", "notifications": false}} {"id": 10, "age": 30, "tags": ["user", "beta-tester"], "location": "Rome", "last_name": "White", "first_name": "Beatrice", "preferences": {"theme": "light", "language": "ja", "notifications": true}} The goal is to mask the first_name with a lookup table "lookup" we've created already. This lookup table has the following data: 1 Alice 2 Bob 3 Carla 4 David 5 Eva 6 Frank 7 Grace 8 Hank 9 Iris 10 Jake To configure the Value lookup function, you must specify the input mapping and output mapping: ![DATPROF Privacy Masking Json value lookup settings.png](https://docs.datprof.com/__attachments/a_3fb75c1f420ae390b773d94f52e85728b1f6ef9e7c2d9063b9070eef4bcafcc3/DATPROF%20Privacy%20Masking%20Json%20value%20lookup%20settings.png?cb=2f215797c0d197fe4f1d4efb9f9498c1) Both input and output mapping require a JSON Path and a Lookup column. ##### Input mapping | **What** | **Explanation** | **Example** | |---------------|-------------------------------------------------------------------------------------|-------------| | JSON Path | requires the JSON path of the target field you want to match with the Lookup column | $.id | | Lookup column | requires the lookup column of which you want to match with the JSON value | id | | **What** | **Explanation** | **Example** | |---------------|----------------------------------------|--------------| | JSON Path | The JSON value you'd like to replace | $.first_name | | Lookup column | The column you'd like to use as values | firstname | Result: {"id": 1, "age": 21, "tags": ["admin", "premium"], "location": "New York", "last_name": "Smith", "first_name": "Alice", "preferences": {"theme": "dark", "language": "en", "notifications": true}} {"id": 2, "age": 22, "tags": ["user"], "location": "London", "last_name": "Jones", "first_name": "Bob", "preferences": {"theme": "light", "language": "es", "notifications": true}} {"id": 3, "age": 23, "tags": ["user"], "location": "Madrid", "last_name": "Diaz", "first_name": "Carla", "preferences": {"theme": "dark", "language": "fr", "notifications": false}} {"id": 4, "age": 24, "tags": ["user"], "location": "Berlin", "last_name": "Brown", "first_name": "David", "preferences": {"theme": "light", "language": "de", "notifications": true}} {"id": 5, "age": 25, "tags": ["user", "beta-tester"], "location": "Paris", "last_name": "Wilson", "first_name": "Eva", "preferences": {"theme": "dark", "language": "ja", "notifications": true}} {"id": 6, "age": 26, "tags": ["user"], "location": "Tokyo", "last_name": "Taylor", "first_name": "Frank", "preferences": {"theme": "light", "language": "en", "notifications": false}} {"id": 7, "age": 27, "tags": ["user"], "location": "Sydney", "last_name": "Anderson", "first_name": "Grace", "preferences": {"theme": "dark", "language": "es", "notifications": true}} {"id": 8, "age": 28, "tags": ["user"], "location": "Toronto", "last_name": "Thomas", "first_name": "Hank", "preferences": {"theme": "light", "language": "fr", "notifications": true}} {"id": 9, "age": 29, "tags": ["user"], "location": "Dubai", "last_name": "Jackson", "first_name": "Iris", "preferences": {"theme": "dark", "language": "de", "notifications": false}} {"id": 10, "age": 30, "tags": ["user", "beta-tester"], "location": "Rome", "last_name": "White", "first_name": "Jake", "preferences": {"theme": "light", "language": "ja", "notifications": true}} #### Generate The generate function creates brand-new synthetic values for a JSON field. Unlike the Value lookup function, it does not require a lookup table but makes use of the existing generators already built into Privacy. We'll again use this sample data and will mask the first_name inside the "profile" column. {"id": 1, "age": 21, "tags": ["admin", "premium"], "location": "New York", "last_name": "Smith", "first_name": "Elias", "preferences": {"theme": "dark", "language": "en", "notifications": true}} {"id": 2, "age": 22, "tags": ["user"], "location": "London", "last_name": "Jones", "first_name": "Tarah", "preferences": {"theme": "light", "language": "es", "notifications": true}} {"id": 3, "age": 23, "tags": ["user"], "location": "Madrid", "last_name": "Diaz", "first_name": "Erica", "preferences": {"theme": "dark", "language": "fr", "notifications": false}} {"id": 4, "age": 24, "tags": ["user"], "location": "Berlin", "last_name": "Brown", "first_name": "Tad", "preferences": {"theme": "light", "language": "de", "notifications": true}} {"id": 5, "age": 25, "tags": ["user", "beta-tester"], "location": "Paris", "last_name": "Wilson", "first_name": "Francis", "preferences": {"theme": "dark", "language": "ja", "notifications": true}} {"id": 6, "age": 26, "tags": ["user"], "location": "Tokyo", "last_name": "Taylor", "first_name": "Maryalice", "preferences": {"theme": "light", "language": "en", "notifications": false}} {"id": 7, "age": 27, "tags": ["user"], "location": "Sydney", "last_name": "Anderson", "first_name": "Aron", "preferences": {"theme": "dark", "language": "es", "notifications": true}} {"id": 8, "age": 28, "tags": ["user"], "location": "Toronto", "last_name": "Thomas", "first_name": "Delila", "preferences": {"theme": "light", "language": "fr", "notifications": true}} {"id": 9, "age": 29, "tags": ["user"], "location": "Dubai", "last_name": "Jackson", "first_name": "Andy", "preferences": {"theme": "dark", "language": "de", "notifications": false}} {"id": 10, "age": 30, "tags": ["user", "beta-tester"], "location": "Rome", "last_name": "White", "first_name": "Beatrice", "preferences": {"theme": "light", "language": "ja", "notifications": true}} ![DATPROF Privacy Masking Json firstname generator settings.png](https://docs.datprof.com/__attachments/a_66484b8fed1f65bd72bf8918c81247433054a0ec36494beab710863db944f08f/DATPROF%20Privacy%20Masking%20Json%20firstname%20generator%20settings.png?cb=addc3b841897c683a0504f9df44b41f8) Result: {"id": 1, "age": 21, "tags": ["admin", "premium"], "location": "New York", "last_name": "Smith", "first_name": "Rosana", "preferences": {"theme": "dark", "language": "en", "notifications": true}} {"id": 2, "age": 22, "tags": ["user"], "location": "London", "last_name": "Jones", "first_name": "Colby", "preferences": {"theme": "light", "language": "es", "notifications": true}} {"id": 3, "age": 23, "tags": ["user"], "location": "Madrid", "last_name": "Diaz", "first_name": "Joey", "preferences": {"theme": "dark", "language": "fr", "notifications": false}} {"id": 4, "age": 24, "tags": ["user"], "location": "Berlin", "last_name": "Brown", "first_name": "Marielle", "preferences": {"theme": "light", "language": "de", "notifications": true}} {"id": 5, "age": 25, "tags": ["user", "beta-tester"], "location": "Paris", "last_name": "Wilson", "first_name": "Gabrielle", "preferences": {"theme": "dark", "language": "ja", "notifications": true}} {"id": 6, "age": 26, "tags": ["user"], "location": "Tokyo", "last_name": "Taylor", "first_name": "Melia", "preferences": {"theme": "light", "language": "en", "notifications": false}} {"id": 7, "age": 27, "tags": ["user"], "location": "Sydney", "last_name": "Anderson", "first_name": "Roxie", "preferences": {"theme": "dark", "language": "es", "notifications": true}} {"id": 8, "age": 28, "tags": ["user"], "location": "Toronto", "last_name": "Thomas", "first_name": "Francis", "preferences": {"theme": "light", "language": "fr", "notifications": true}} {"id": 9, "age": 29, "tags": ["user"], "location": "Dubai", "last_name": "Jackson", "first_name": "Raleigh", "preferences": {"theme": "dark", "language": "de", "notifications": false}} {"id": 10, "age": 30, "tags": ["user", "beta-tester"], "location": "Rome", "last_name": "White", "first_name": "Patricia", "preferences": {"theme": "light", "language": "ja", "notifications": true}} ### Generation functions Generation functions are handled in the [**Generation**](https://docs.datprof.com/privacy/latest/using-datprof-privacy/generation)chapter of the documentation, as they function identically. ## Cross-Database Deterministic Masking Cross-database deterministic masking ensures that the same input value always results in the same masked value, even across different databases. This is achieved using a Global Translation Database. All values are hashed. **How it works:** A deterministic generator stores mappings (original → masked) in a translation table. When the same value appears again, in any database, the stored mapping is reused. This guarantees consistent masking across systems. ### Configuration First, in Project settings, configure a Global Translation Database connection. This database stores the translation tables used by deterministic generators. ![Global Translation Database.png](https://docs.datprof.com/__attachments/a_88d29244537d07916df195d35181d81b845d40b396d891af97fb4b7a81baf518/Global%20Translation%20Database.png?cb=c16e68d79ed3e8f5a8a8527d0290ed25) After configuring a Global Translation Database, deterministic masking can be used for unique generators such as IBAN or BSN. ![Global Translation Database Example IBAN.png](https://docs.datprof.com/__attachments/a_cbb8a2953591d27df515a260ad4de119d99d8a578312ebb128c801b516fef64d/Global%20Translation%20Database%20Example%20IBAN.png?cb=26060281468d3c458087542d0382206d) In the Function Editor, select a generator (for example IBAN) and set the Generation mode to Deterministic. Then define the Schema and Table, which determine where the translation table is stored and used for value mapping. When deterministic mode is enabled, new input values are stored in the translation table, while existing values are automatically reused. This ensures that identical input values always result in the same masked output. * The Global Translation Database can be a different database type than the source database and must be accessible during masking execution. * The same input value always results in the same masked value. This works across multiple databases and environments and ensures consistent masking and referential integrity. * For deterministic masking to work, the global salt has to be the same across projects. ## Adding data masking functions To configure a new masking function, the user must first select the column (or multiple columns) upon which the function is to be executed. Functions can be added with either the *Add Function* button or a right click on the selected the column(s). ![Function Editor.png](https://docs.datprof.com/__attachments/a_a2205885908056d074399190444f1e36af5b70f7aae12eca3e3c4f46048a858e/Function%20Editor.png?cb=f43d83be77f64b785d5cf4062a6b103c) The subsequent dialogue box (called the *Function Editor* ) allows the user to select various options for the function. These options are explained in the following sections. If the *OK* button is clicked, the function is added to the selected column(s). By unchecking the *Enabled* checkbox, this function will be skipped. ### Conditional masking The user has the option to specify a condition which selects the rows against which the function will, or will not, be executed. A condition is defined in the *Condition* tab in the *Function Editor*. ![Function Editor Condition.png](https://docs.datprof.com/__attachments/a_219492700618959f80c18ecace9616bdc447ae10f8160ec040d53a2918104bd7/Function%20Editor%20Condition.png?cb=982020b6fe9f70596ae5942c90b10dad) The condition can be specified by defining a *'where clause*' as in an SQL query. If the "Exclude null values from masking" option is selected this will be joined with your where clause. If a where clause is not specified but the "Exclude null values from masking" is selected then only column content which is NOT NULL will be masked. The**Test**button can be used to determine whether the defined condition is valid and what the result (qualified rows) of the condition will be. **Microsoft SQL Server:** To refer to the current table you can use the "table name as alias" concept. This is required whenever the "where clause" lists two identical column names. An example of a operation on table "Table2" with identical field name "name" is: `where exists (select 1 from [Table1] a where a.name = [Table2].name)` In this example "a" is the alias for \[Table1\]. ### Saving a translation table You can choose to store the result of the data masking function in a translation table. This is particularly useful in implementing consistent data masking between tables. Translation tables store the old and the new value for each column value in the table. You can also use a*ValueLookup*function against a Translation table when masking other tables or applications, effecting consistent masking across the Estate. Saving a translation table is configured in the *Translation table* tab in the *Function Editor*. This dialog allows the user to indicate that the translation table must be created or used in subsequent runs if it already exists. The user must specify the database and schema in which the translation table is to be saved as well as a name for it. ![image2020-2-13_12-3-7.png](https://docs.datprof.com/__attachments/a_e9f3db893e5c3a8da54a2bfe7c5fb79e0b965d73b46bc6716ce9130826039753/image2020-2-13_12-3-7.png?cb=bacaf1f88aefc03b86402347d4a04b8a) The specified translation table will be listed as a selectable table in the definition of the *ValueLookup* Function. **Using Translation Tables** Because translation tables contain before and after images they can present a security vulnerability in that they can be used to reverse engineer the masking process. They are not truncated or dropped by default. Care must be taken in securing these tables from unwanted access or a post-run script executed to drop them after the masking run completes. **Best Practice:**Create any used translation tables is another database which is protected by restricted/authorized access. ### Dependencies In certain cases it may be necessary to influence the sequence in which the configured functions are executed. This sequence can be specified in the **Dependency Editor** . The Dependency Editor can be opened using the **Dependencies** **tab** in the **Function Editor** . Choose the **Edit**button to invoke the dialogue: ![Dependency Editor.png](https://docs.datprof.com/__attachments/a_667b6aacbc1d7d8f1915b89c2c013dde68dbcd549a3bc79cea5dcfcb063d4768/Dependency%20Editor.png?cb=453453b9a6016c01c58f168ae76a34e5) With the *Dependency Editor* the predecessors and successors for a function can be specified. By specifying predecessors, functions or scripts can be selected to run ++before++ the function in focus. By specifying successors, functions or scripts can be selected that should run ++after++ the selected function. Predecessors and Successors are invoked by simply clicking on the function listed in either aspect. **Sample Use Case** You have a *ValueLookup* function which is based upon a Translation table created in another function so the *ValueLookup must run after* the Translation table has been populated. A dependency will enforce this. ### References When using *the Shuffle* , *ValueLookup* and *Custom Expression* functions the Function Editor shows an additional *References* tab*.* It shows all the available *Foreign Keys* for the current field. By checking the checkbox **"Update referencing foreign keys"** all instances of the updated field in the database are consistently modified by following the relationships. ![image2018-5-8_15-26-20.png](https://docs.datprof.com/__attachments/a_06ccb59ac3342fc43bd01ee5be79054815c02ac5a35f3d40d1f328ea7e3d4621/image2018-5-8_15-26-20.png?cb=fe29bd0657994672df0ff201b76d88a3) This function is extremely powerful with complex and large data models. It is not necessary to specify a Translation Table when using this function. The Translation Table may still be required for other functions and programs. ## Foreign keys The *Foreign Keys* tab shows the current Foreign Keys as gathered by the **Import and Synch Metadata** options. Sometimes, however, data relationships are implied at the application level and not imposed at the database level. When masking (or sub-setting) a database it's often useful to have these relationships noted and, potentially, acted upon. The Foreign Keys context allows you to create, edit or delete what is, effectively, a Virtual Foreign Key which expresses the application implied relationships. As well as listing the name of the table and the related columns the source of the Foreign Key is also displayed. There are four possibilities: ![Foreign Keys.png](https://docs.datprof.com/__attachments/a_2af97f40a61ab2477c095f7dd18ec549700f6ea1b78ac8d2fa1e4182c96bf74a/Foreign%20Keys.png?cb=f4328f10f0fd565b09f9c3d1cbbc2ed3) **ODBC** *:* The Foreign Key is imported from an ODBC connection. **DME** *:* The Foreign Key is imported using a DME file. **Manual**: The Foreign Key is manually created in DATPROF Privacy. **Unknown** *:* The Foreign Key shows no indication of how it has been imported. This is likely the case with Foreign Keys created in an earlier version of DATPROF Privacy. The distinction in the source of the Foreign Key is important to the synchronization process. By clicking on **'Add foreign key'** the *Constraint Editor*will be opened and a new virtual Foreign key can be defined. ![Constraint Editor.png](https://docs.datprof.com/__attachments/a_e1582e2709d8404bac5f5f40189d11b7e6846039203ba9f11d117ca54214e81a/Constraint%20Editor.png?cb=4a40f0089b2acf9db6bfb1515a24901c) Here,a parent schema and table can be selected. A new Foreign Key can be defined by selecting the current and related columns and by selecting the parent table. When saving a new Foreign Key it will be flagged as "manually" added. **About adding a virtual Foreign Key** * Adding a Foreign Key does NOT create a constraint in the database. It is exclusively used as a reference within Privacy. * The Foreign Key Condition option is currently ignored by Privacy. A manually added Foreign Key has an impact on the synchronization wizard. The wizard considers the database as the primary source of truth, so any manually added Foreign Keys will be flagged to be removed from the metadata during a synchronize step. Therefore, it's wise to distinguish any virtual Foreign Keys from existing Foreign Keys in the database. This way, you can avoid losing your virtual Foreign Keys. ## Advanced settings ### Oracle #### SplitLimit You can specify a Split Limit for every table. Selecting the tab Advanced settings for a table shows you this screen. ![image2018-10-25_15-13-31.png](https://docs.datprof.com/__attachments/a_1b1ae27ab8cfa9b3796531eaece347f7b49fbe485d02ee789faddba6166ae67b/image2018-10-25_15-13-31.png?cb=945d058d888ab4352e7bccf9e746df6a) Specifying the SplitLimit has an effect on the final insert back into the table in focus. SplitLimit is synonymous with "Commit Interval" in that it groups and commits the transactions into blocks of rows specified by the SplitLimit. This will reduce the space required in the Oracle Temporary tablespace but does not apply to the DATPROF temporary tables which are created without logging. The default value for SplitLimit = 0. All data is inserted and committed in one transaction. Specifying the SplitLimit will impact performance. The value used as SplitLimit is determined by your system resources, the row-count of your table and the size of the row. #### TempStatisticsPercentage By specifying a percentage \> 0 Oracle statistics will be gathered prior to the merge of the masked data into the resulting table. This percentage is used as the "**estimate_percent"** in the Oracle query. With large datasets this method gives a significant performance boost. For smaller datasets the time spent in gathering the statistics might exceed the time spent to merge the data, outweighing the benefits. This setting defaults to 0, implying that no statistics will be gathered. #### TempStatisticsMethodOpt The value is used for the "**method_opt**" parameter of the DBMS_STATS.GATHER_TABLE_STATS procedure. When left blank (default) the parameter will be set to "FOR COLUMNS DPV_RID SIZE AUTO". Any other value will be passed as-is to the "method_opt" parameter. For example, to gather statistics for all columns TempStatisticsMethodOpt could be set to "FOR ALL COLUMNS SIZE AUTO". #### Custom_DPV_RID The first step in masking a table is to copy the table to a temporary "Snapshot table" where a DPV_RID column is added to the table. This column will hold a Unique numeric key value. The parameter *Custom_DPV_RID* allows you to specify the method to fill this column. By default this setting is empty and the following expressions are used: * For Oracle 12 this column is filled with the outcome of *rownum*. * For all other Oracle versions this column is filled with the outcome of: *ROW_NUMBER() OVER (order by 1).* The reason the parameters differ is to optimize performance. On certain occasions faster solutions are available. You could use your own expression or you might specify an existing Primary key or a sequence. This is where the setting *Custom_DPV_RID* can be used. An expression returning a numeric value is required, for example a sequence: *dpv_rid_sequence.nextval*. Note: Normally you do not need to fill this setting. ### DB2 LUW #### SplitLimit Specifying the SplitLimit has an effect on the final insert back into the table in focus. SplitLimit is synonymous with "Commit Interval" in that it groups and commits the transactions into blocks of rows specified by the SplitLimit, thus preventing issues with the temporary tablespace. In LUW the SplitLimit applies to both the temporary tables and the final insert into the original table. #### TempStatisticsPercentage By specifying a percentage \> 0 DB2 statistics will be gathered prior to the merge of the masked data into the resulting table. This percentage is used as the percentage of the table rows using the TABLESAMPLE BERNOULLI method. With large datasets this method gives a significant performance boost. (See the IBM DB2 LUW documentation about this). This setting defaults to 0 and may be between 0 and 100. You can also use a precision of up to 2 decimal places. (ie: 0.05). ### TempTablesTableSpace When creating temporary tables the default tablespace is used. When specifying a value for this setting that tablespace will be used when temporary tables are created during the execution of the privacy function on the target table. ### TempTableIndexesTableSpace When creating temporary tables the default tablespace is used for indexes on a temporary table. When specifying a value for this setting that tablespace will be used for indexes on these temporary tables. ### Databricks Specifying JDBC properties allows fine-tuning of the connection behavior between DATPROF Runtime and a Databricks SQL Warehouse or cluster. These parameters control authentication, transport mode, encryption, and performance aspects of data transfer. ![DATPROF Privacy Databricks JDBC Properties.png](https://docs.datprof.com/__attachments/a_982f3fd0f585655db4187071df0c17acf6f9e0241cfb1b2d823e7bd62c8d5df0/DATPROF%20Privacy%20Databricks%20JDBC%20Properties.png?cb=8ef7e74266b65fb9b0ef4911317699bb) #### transportMode Defines the communication protocol used by the JDBC connection. When set to HTTP, the driver communicates with Databricks using an HTTP-based Thrift transport, which is required for Databricks SQL endpoints. Using binary mode is only applicable to on-premises Spark or Hive environments. #### SSL Enables or disables SSL encryption for the connection. Setting ssl=1 ensures that all data transmitted between the client and Databricks is encrypted. This setting should always remain enabled to meet Databricks' security requirements. #### authMech Determines the authentication mechanism used for the connection. Default is set to 3, the driver uses token-based authentication, which is required for connecting with a Databricks personal access token. Other mechanisms include: 0: No authentication 1: Kerberos 2: Username/Password 3: Token #### useThriftClient Controls whether the JDBC driver uses the Thrift binary client. Setting useThriftClient=0 disables the binary Thrift client, ensuring that HTTP transport is used for Databricks connections. This value must be 0 when connecting to a Databricks SQL Warehouse. #### enableBatchedInserts When enabled (1), multiple insert statements are grouped and executed as a single batch operation. This significantly reduces network overhead and can improve performance during large data insert operations. Setting this property to 0 disables batching, executing each insert individually. While this may reduce performance, it ensures more predictable behavior in certain scenarios, for example, when debugging insert errors, when strict row ordering is required, or when the target system or driver does not fully support batch operations. In most cases, batching should remain enabled to maximize data load efficiency. ## Execution of PowerShell scripts The OS Call scripts that can be created in DATPROF Privacy are either executed as "`cmd /c myscript`" on Windows systems or as "`/bin/sh -c myscript`" on Linux systems. To execute a PowerShell script it must be saved in a separate file which can then be used from the OS Call script: For Windows systems: Powershell.exe -executionpolicy bypass -File "C:/path/to/mypowershellscript.ps1" Note that PowerShell's execution policy on a Windows system defaults to "Restricted" and this prevents PowerShell scripts from running. By using the "bypass" policy the script can be executed. In this cmd script a PowerShell script can be called in the following way: Powershell.exe -executionpolicy bypass -File "C:/path/to/mypowershellscript.ps1" For Linux systems: pwsh "/path/to/mypowershellscript.ps1" --- version: "4.23" language: "en" --- # Getting Started The **Project Settings** screen is the starting point for any DATPROF Privacy project. It enables you to select a connection to a target database and to import or synchronize metadata. ![DATPROF Privacy Project Overview.png](https://docs.datprof.com/__attachments/a_16dde02ca0eadc1584f6ef4c9c4e2bdb5bd18f7ea4c9f2d5dc832488a309929d/DATPROF%20Privacy%20Project%20Overview.png?cb=edc0e7bd0545316c90cdb34147ba3b3f) ## Connection Editor ![Screenshot 2026-02-06 111235.png](https://docs.datprof.com/__attachments/a_5e7b00802b35ab039fcc3dab750080d4311b89611ab2d037ca8e5fde7ee778cd/Screenshot%202026-02-06%20111235.png?cb=ad4412feb8346a2777d4d4b7cefba061) In order to mask (parts of) a database, a connection to the database is required. This connection is used for the importing and synchronizing the metadata and will be used to execute the masking functions on that database. A database connection can be specified, selected and saved in the *Connection Editor* . The left side of the screen displays the saved connections. The right side of the screen shows the connection details of the connection that is used for the current project, this connection is known as *Project Connection* . A selected connection becomes the actual *Project Connection* by pressing **"Use this connection"**. **Managing your sensitive data when contacting DATPROF** It's entirely possible that when depending on our staff for support of our systems, the topic of transferring your project file to DATPROF comes up. Luckily, due to the way we design our software we don't handle the data within your database, and a project doesn't contain any sensitive data about your datasets. However, because it's possible to store and save the connection data to databases in our software a project file can contain privacy or security risk bearing connection data. Therefore, if you would like us to look at one of your project from a support perspective we strongly urge you to delete any existing connection data in the **Connection Editor**. As this is information we have no desire to have within our system, and could contain sensitive information, we will prune this information in any event this is attached erroneously to protect our data privacy conformity. Nevertheless, it falls to the submitter of any ticket to be wary of privacy sensitive data and remove it where possible. When defining a connection, the following parameters are required: ### Microsoft SQL Server ![Screenshot 2026-02-06 111414.png](https://docs.datprof.com/__attachments/a_30eaf89779d11152c58da7df075e31e3f12f581b7f692cc32ea6c14b2538ee4e/Screenshot%202026-02-06%20111414.png?cb=62be7d3158b62b13f561a01e49fc0534) #### Basic Connection Details * **Database type:** Microsoft SQL Server * **Host:** Host name or IP-address of the database server * **Port:** TCP port on which the database server is listening * **Instance:** Instance name of the database server * **Database:** Name of the target database #### Authentication Types SQL Server supports many types of authentication. In order to support as many as possible, the user can select which authentication type to use in this drop-down box. Authentication types can vary from simple password authentication to more complex Azure specific authentication types. Some authentication types require additional **JDBC-** or ado.netconnection properties, but even when not required the user may supply extra properties to account for database specific configurations. * **SQL Server Authentication** * **Description:** A simple username \& password authentication against the database. * **Extra properties required:** None aside from default. * **Windows Authentication** * **Description:** Windows authentication checks the computer name of the client connecting to the database for validation. * **Extra properties required:** None aside from default. * **Microsoft Entra Managed Identity** * **Description:** Microsoft Entra Managed Identity works by checking a specific client ID against the database for access, and is used to connect to SQL Server on the Azure cloud. For more information please refer to the Microsoft documentation: * **Extra properties required:**None aside from default. * **Microsoft Entra Service Principle** * **Description:** Microsoft Entra Service Principle works by checking a **service principal ID** and **service principal Secret** to connect a user to the database. For more information please refer to the Microsoft documentation: * **Extra properties required:** None aside from default. * **Microsoft Entra Password** * **Description:**Similar to the SQL Server Authentication, but using login details configured in an Entra ID environment. Uses a simple username + password combination. * **Extra properties required:** None aside from default. #### JDBC \& [http://ADO.NET](http://ado.net/) Properties In order to handle specific database configurations, it's possible to supply extra JDBC or [http://ADO.NET](http://ado.net/) connection properties. These are properties which are always passed on when connecting to the database, and are configured in a **Property + Value**pair. By default, Privacy contains a few JDBC properties, used to connect to SQL Server databases. Unless specifically needed, we don't recommend removing these. ![image-20240812-115744.png](https://docs.datprof.com/__attachments/a_39911679c4f034cd3ea81531dde7b57538da59c81c466e3d5cf70bc180478b1b/image-20240812-115744.png?cb=0cdefe2eeb26711d188e1b3a41b052ea) ##### JDBC Properties Overview **The above link refers to Microsoft documentation, which is subject to change.** ##### [http://ADO.NET](http://ado.net/) Properties Overview **The above link refers to Microsoft documentation, which is subject to change.** ### Oracle ![image-20240812-141516.png](https://docs.datprof.com/__attachments/a_cfad20153950ae2076ccfc414117c58bfbe674da460aff7de0bf9938cddee432/image-20240812-141516.png?cb=ff34c83cb14c7a21c7bee8b57e99eaf0) * **Database type:** Direct for connection through the interface, or ConnectionString to use a connection string. * **Username:** Oracle Database Username. * **Password:** Oracle Database Password. * **Host:** Hostname or IP-address of the database server. * **Port:** TCP port on which the database server is listening. * **SID / Service:** Unique database identification, which type you need to supply is dependant on database configuration. #### Connecting via a connection string ![image-20240812-141545.png](https://docs.datprof.com/__attachments/a_59cbb13bb2c9647ae877ce793263f5ee677763c363ffda376def1b8cfa38d903/image-20240812-141545.png?cb=44d04378495856ea08ac5c1b1f217251) With this option an Oracle Tnsname entry can be provided to connect to the database. In order to do this, a valid Tnsnames.ora must exist. You can also specify a complete Oracle Connection description which allows you to use all the features of Oracle Connection Manager. Example: *(Description=(ADDRESS=(protocol = TCP)(host = datprof-server)(port = 1521))(CONNECT_DATA = (SERVICE_NAME=datprofora1.localdomain)))* PrivacyConsole.exe can use an existing ConnectionString option but you cannot specify it on the command line. ### DB2 for Linux, Unix and Windows ![image-20240812-120544.png](https://docs.datprof.com/__attachments/a_9bb88aee0ae6e49d538d45de5d713ef3672b1ecd451135868637eac2dd19074c/image-20240812-120544.png?cb=656e1592138251545b0580d087f95f84) * **Database type:**IBM DB2 for Linux, Unix and Windows * **Host:** Hostname or IP-address of the database server * **Port:** TCP port on which the database server is listening * **Database:** Name of the target database. ### DB2 for i ![image-20240812-120632.png](https://docs.datprof.com/__attachments/a_4574926905772d4598e31e6cd70e2c0db9796df8799cd2b244129b117cf06440/image-20240812-120632.png?cb=cbcda97ceb4874e7dd46a1944bde3726) * **Database type:** IBM DB2 for i * **Host:**Hostname or IP-address of the database server * **Database:** Name of the target database * **Username \& Password:** credentials to access the target database ### DB2 for z/OS ![image-20240812-141137.png](https://docs.datprof.com/__attachments/a_27979bfa6c0d898005ee573890d739c966c608728d8e1f90336b061f3d412591/image-20240812-141137.png?cb=57e2d03e3ecfae01742c36d1622e49f9) * **Database type:** IBM DB2 for z/OS * **Username \& Password:**credentials to access the target database * **Host:**Hostname or IP-address of the database server * **Port:**Port over which to connect to the database * **Database:** Name of the target database #### Connecting through the connect descriptor When connecting to a DB2 z/OS database, it's possible for the user to supply a JDBC connect string instead of supplying individual values. In some situations, this is required (such as when the mainframe database requires a certificate to connect to it). For a complete overview of the syntax for DB2 z/OS connect strings, please refer to the following page of the IBM documentation: ![image-20240812-141203.png](https://docs.datprof.com/__attachments/a_619f3d95ab29bd5b7dae3ba4cc24af4c35cbe92d40806a3a3be1ce6122daf18c/image-20240812-141203.png?cb=7ed4ed7f4f800928cad4edcc85d47b13) **JDBC connect string example:** Syntax: jdbc:db2://\:\/\:sslConnection=true;sslTrustStoreLocation=\;sslTrustStorePassword=\; Example: jdbc:db2://My-Favourite-Database:8102/PRODUCTION:sslConnection=true;sslTrustStoreLocation=C:\\users\\me\\certificates\\db2Zstorecert.jks;sslTrustStorePassword=changeit; ### PostgreSQL ![Screenshot 2026-02-06 114151.png](https://docs.datprof.com/__attachments/a_db32bae1cd944ed00c58c18ac98fbef095e97b2c81c02cafbdcf4cd337e9d3e2/Screenshot%202026-02-06%20114151.png?cb=7efff4dc41933db4d0c04ae807fe7a71) * **Database type:** PostgreSQL * **Username \& Password:** credentials as used in the database * **Host:** Hostname or IP adress * **Port:**Port number of the database server * **Database:**Name of the database to obtain metadata from and execute the run against ### MySQL / MariaDB ![image-20240812-120859.png](https://docs.datprof.com/__attachments/a_4ba891b7a7ae8df46624ce8e6f0074066985dc027466802aa477d2edf35b5b5b/image-20240812-120859.png?cb=718a522e278b9fef2ca396539b1c1502) * **Host:**Hostname or IP address * **Port:**Port number of the database server * **Database:**Name of the database to obtain metadata from * **Username:**Database user * **Password:**Password for database user ### Azure SQL Currently, in order to connect to an Azure SQL database the user should use the "MS SQL Server" project connection option. In future releases, this is subject to change. For information about JDBC properties, please refer to the **SQL Server**chapter of this page. ![Screenshot 2026-02-06 114636.png](https://docs.datprof.com/__attachments/a_af21ad4eceefa0989bc99c7ba1eb5e8cc93aed936aff9fb4ca74a48590735e77/Screenshot%202026-02-06%20114636.png?cb=62f2806362dd0a6b7dc625d414a5e395) * **Database type:** Microsoft SQL Server * **Host:**Hostname or IP-address of the database server * **Port:**TCP port on which the database server is listening * **Instance:** Instance name of the database server. With a default Azure SQL deployment, this can be NULL. * **Database:** Name of the target database ### Databricks ![DATPROF Privacy Databricks Connection Editor Small.png](https://docs.datprof.com/__attachments/a_97355500d06c1c76f2d4dd51e52bf1a5aa9e0ac977cf001938dab55ab344f999/DATPROF%20Privacy%20Databricks%20Connection%20Editor%20Small.png?cb=e86592f48da573c6ef69a7ca9187392a) * **Database type:** Databricks * **Host:** Hostname or IP address * **Port:**Port number of the database server * **Catalog:** Name of the catalog * **HTTP path:**Enter the unique Databricks endpoint used to connect to your SQL warehouse or cluster. * **Username:**Always token, Databricks uses a Personal Access Token. * **Password:** Enter the Personal access token from Databricks ### Custom Database (JDBC) Connection Before you can use this feature, contact your sales or solution engineer. Opening the Custom Database Type Manager: 1. Open your DATPROF Privacy project. 2. In the top menu, click File. 3. Select Custom database type manager. ![Custom database type manager.png](https://docs.datprof.com/__attachments/a_6ca77590e4aa2727b7e723adffcdb64ff30bce7444b0ec32f77f00885fe55b28/Custom%20database%20type%20manager.png?cb=f7807c90d43ed5b07d0412510baed79d) ![Screenshot 2026-06-09 082101.png](https://docs.datprof.com/__attachments/a_668832e286d8b5fc6d27fb9106d20c1d956a220cf2afa37a9fc7d7a7783b134f/Screenshot%202026-06-09%20082101.png?cb=1b3ae033e2d1d93d68e8e546e812afa8) #### Creating a New Custom Database Type To create a new custom database type: 1. In the Custom database type manager, click New.... 2. The Add a new custom database type window will open. ![New Custom database type.png](https://docs.datprof.com/__attachments/a_4ae7fccabc816a57a0d5e6dda7450840122238419cd8f28103adb10738a8bc75/New%20Custom%20database%20type.png?cb=32b297f5a105ed2c75c4b34bd9ec2d9d) In this window, you must define: * **Name:**Enter a descriptive name for the custom database type. * **Base type:**Select the database platform that will act as the template. You can choose from the following base types: * Oracle * Microsoft SQL Server * PostgreSQL * MySQL / MariaDB * Databricks * IBM DB2 for i * IBM DB2 for Linux, Unix and Windows * IBM DB2 for z/OS 1. Click OK to create the new database type. 2. The new database type will now appear in the Custom database type manager, where further configuration can be applied. * The selected base type determines default behavior, data type mappings, and query compatibility. * Always choose the base type that most closely matches the target database system. #### Drivers To use a custom database type, the correct JDBC driver (.jar file) must be available to DATPROF Privacy. Download the required JDBC driver and place it in the "**Drivers"** directory of the DATPROF Privacy installation. It is recommended to store all JDBC drivers in this folder to ensure they are centrally managed and accessible. Example location: C:\\Program Files\\DATPROF Privacy\\4.23.0\\Drivers #### Adjusting Queries and Templates for Custom Databases After configuring the JDBC driver, additional adjustments may be required to ensure DATPROF Privacy functions correctly with the custom database. This is because query syntax and supported features depend on the database platform. ##### Privacy Queries ![Screenshot 2026-06-09 084755.png](https://docs.datprof.com/__attachments/a_4c7e328c3610fd39a6f6ddb03bf9ef60328e13829fa61575c813ff4b3dc597a6/Screenshot%202026-06-09%20084755.png?cb=d91b17bfbe35b14744ce0dee30ead7f6) The Privacy queries tab contains SQL queries used by DATPROF Privacy for internal operations. You may need to modify these queries if: * The database does not support specific SQL syntax * Keywords differ (e.g. `SELECT INTO`, `LIMIT`, `TOP`) * Object naming or schema handling behaves differently ##### Privacy Templates ![Screenshot 2026-06-09 084805.png](https://docs.datprof.com/__attachments/a_9c01e5f426ccf68a0071e8e2196852ccb9fa0c9ac9981bc13f7a5c0ff4f41922/Screenshot%202026-06-09%20084805.png?cb=ac7bd46613b212848ccfa2314d4191c8) The Privacy templates tab defines SQL-based templates used during execution (e.g. indexing, anonymization, reporting). You may need to modify these templates if: * The database does not support certain features (e.g. clustered indexes) * SQL syntax differs * Specific commands are not available in the target system ##### Database Type Data ![Screenshot 2026-06-09 084815.png](https://docs.datprof.com/__attachments/a_0f97c5455f6be3a665b7cfbd78afbd14300372ace73b759c8d80ca6e07926059/Screenshot%202026-06-09%20084815.png?cb=12e1ff09906aaaa0f8a4988cb9aade9e) The Database type data tab defines how data types behave and which operations are supported. You may need to modify this section if: * Data types behave differently in the target database * Length limits differ * Certain operations (e.g. scrambling, lookup) are unsupported #### Best Practice * Start with the default configuration from the selected base type * Only adjust queries/templates when errors occur * Test incrementally ### Test, Save, New Using the **Test** button the selected connection can be tested. This option will try to connect to the database using the specified connection details. The result is shown to the user. The *Save password* option ensures that the login information is stored and does not need to be re-entered in future sessions. The **Save** button ensures that the specified *Project Connection* is stored in the list of saved connections. All connections are stored in the file "*Privacy_connections.lst* " in "*\\\AppData\\Local\\\\\privacy\\*" The **New**button will clear the input fields allowing the user to define a new connection. You can also modify an existing entry and save it as a new entry. Whenever a connection is saved in your project, new connections are restricted to the Database type you saved. ## Importing meta data Through the button *Import metadata* (or using the menu option *Project → Import and synchronize → Import metadata wizard*), a wizard for importing meta data of the specified database is started. Importing the metadata through this wizard consists of three steps: 1. Selecting the schemas and tables 2. Selecting the relations (foreign keys) 3. Importing the specified metadata ### Selecting the schemas and tables During this first step, the user can select the tables to be imported. The screen consists of two parts. ![image2018-5-8_13-41-28.png](https://docs.datprof.com/__attachments/a_cd04543218da2ecb8aaa7b9797550ef1a90862cce6137d40cc5afa2011a8bab6/image2018-5-8_13-41-28.png?cb=3704d5c37e51127912b8828e8a2f3b03) The left part shows the metadata available in the database. In the drop-down box (top-left) the schema containing the tables can be selected. In the list below the user can select one or more tables which will be imported. The right part shows the tables selected to be imported. With the *single* *arrow* , the selected tables are added to the list of tables to import. The *double arrow* will add all tables from the selected schema into the list of tables to import. The *Next* button moves the wizard to the next step. ### Selecting relations (foreign keys) ![image2018-5-8_13-42-20.png](https://docs.datprof.com/__attachments/a_a31259424bbace23473783d8b29a3c06a5a01a8acf47f1011bb4fbbcbb24cdb2/image2018-5-8_13-42-20.png?cb=77c0cee9281b8cacf009430cf39c719b) This next dialog allows the user to select the foreign key relations (FK's) to import. This dialog has the same format as the previous dialog. The list of available FK's are derived from the selected tables. Again, the user can select one or more FK's and add them to the list of FK's to import. Normally you would select them all. The **Next** button moves to the next step. ### Importing the specified metadata ![DATPROF Privacy Import ODBC Source Wizard.png](https://docs.datprof.com/__attachments/a_062e6659ee507826315675ae8550cb22631623e873a1eeaff82cd970c34ce297/DATPROF%20Privacy%20Import%20ODBC%20Source%20Wizard.png?cb=2ca7b3dd17fdf8a0cd06d26defc08794) The last dialog of the wizard allows the user to begin the import of the selected metadata (tables and FK's). By clicking on **Start**, the application will retrieve the selected metadata from the database. The progress of the import can be monitored with the *Progress Bar*. Pressing **Logging** will unfold a dialog that prints log lines. This logging will help to resolve any errors that may occur during the import. ## Synchronizing metadata The *Metadata Synchronization Wizard* helps the user to process any changes to the data model of the target database, ensuring that masking can be executed against such changes. Synchronization can take place through a DME file or an ODBC database connection (*Project Connection)*. * The information in the database is the primary source. * Manually added Primary Keys using "Toggle Primary Key" are removed. * Manually added Translation Keys using "Toggle Translation Key" are still available.^(since v3.8.3)^ * The Synchronization wizard does not distinguish between self-defined Foreign Keys and Foreign Keys removed from the database. They are both marked for deletion in the wizard. * A Field order change will trigger the synchronization process. ![image2018-5-8_14-0-42.png](https://docs.datprof.com/__attachments/a_d5798abe94744138e03c80bc85575663a7d14d9c8e83dead4dfafd3dad1a89a8/image2018-5-8_14-0-42.png?cb=c9332788ea23207e747376c9e35dbf32) In this step, using an ODBC connection is the regular way of importing/synchronizing meta data. Functionality of DME files is handled elsewhere in this documentation. ![image2018-5-8_14-0-55.png](https://docs.datprof.com/__attachments/a_c641038bac92a281c292e83cbd9ec91c67ea1284f4eb9132b2c8223811cb3e9b/image2018-5-8_14-0-55.png?cb=d69190d5083804d857c047afd8949a97) The following screen shows a summary of the differences between the metadata from the target database and the metadata already imported into the application. If you decide to exclude certain metadata from an initial import, they will show up here too. The system cannot differentiate between "new" differences, and ones that have already been acknowledged by the user and accepted during an initial import. This synchronisation simply compares the metadata present in the project to what a new request for metadata to the database returns. ![image2018-5-8_13-59-52.png](https://docs.datprof.com/__attachments/a_ef365eaca506fa750aea32081e09d7cfe3678f39ed7e613be789fff4542eaf27/image2018-5-8_13-59-52.png?cb=9407d862954cecad2fd09ee4c135e055) Changes may include tables, columns and FK's. These can be added, changed or deleted. By clicking **Next**, a dialog will be opened for each type of change, where the user can decide whether to process the change in the project. The last screen allows the user to process the selected changes in the project. The progress of the synchronization can be monitored with the *Progress Bar*. ![image2018-5-8_14-3-49.png](https://docs.datprof.com/__attachments/a_3b1fcb5ba68d37ba5260d3a5debe82f3be94eabbf5fac2bedee356095d759f69/image2018-5-8_14-3-49.png?cb=07daf27d9a709afb2b5c8fb8aeefe698) ![image2018-5-8_14-3-36.png](https://docs.datprof.com/__attachments/a_5e9ca818847a7f90a3367f148b6c57112313ffb867bc81c0cfaf808d0124445a/image2018-5-8_14-3-36.png?cb=63961c8eb10d6c75b90523174dad4742) To show the log file made during import, press **Logging**. This logging also provides input for resolving any errors that occurred during synchronization. --- version: "1.6" language: "en" --- # DATPROF Analyze Manual ![analyze_dark_full@8x.png](https://docs.datprof.com/__attachments/a_def4462030c1eb31e5f085d7b099865f53daa7551adaa5ee6f423d5779f0d5c4/analyze_dark_full@8x.png?cb=753eb0a2152fe8368a6636ba6ca62376) DATPROF Analyze is a powerful data analysis tool designed to provide users with deep insights into large, structured databases. With advanced profiling capabilities, Analyze helps users assess data quality, identify specific patterns or data types, and detect statistical outliers, ultimately improving data consistency. ## In this manual This manual provides a comprehensive guide to the features, functionality, and options available to you as an Analyze developer. It is designed to equip you with all the tools needed to maximize your use of the software. However, please note that this manual is not intended as an FAQ or a knowledge base for known issues. For troubleshooting and specific issues, we offer a dedicated knowledge base on our documentation site. To navigate the manual, you can use the left-hand index to browse articles or the search bar to find specific topics of interest. If you find that any information is missing or unclear, we encourage you to contact us with your feedback. At DATPROF, we are committed to creating high-quality resources for our partners and are continually working to expand and improve the content to enhance its usefulness and clarity. This manual is continuously updated, and it is possible to select the version-appropriate material of the manual at the top-right of this page. --- version: "1.6" language: "en" --- # About DATPROF Analyze Sometimes, when working with large databases it can be difficult to get an accurate picture of what critical information lives within your tables. Often, privacy sensitive information is stored (accidentally or intentionally) in large text fields or in sections of the database where that information does not belong according to the database architecture design, having seeped in slowly over time. In order to find this information, explore your database, and get an idea of what data is stored in it, DATPROF Analyze gives you the ability to construct profiles which comb through your data to find specific patterns, and to collect and consult general data quality information. ## Who is the intended user for Analyze? Most often, the intended users for Analyze are professionals who have a basic understanding of SQL databases and need to gain extra insights into their data. When combing through data, the user requires a keen knowledge of the established requirements set to (test) data within their own organisation. An example of a use case would be for a Data Officer to consult a test data database to search for privacy sensitive data. Within the organisation it has been established that Dutch citizenship ID's must be scrubbed, and through masking of the database this has been done in the designated fields for such information. However, upon building an Analyze profile that matches with valid citizenship ID's, the data officer discovers that this data is also stored in a number of large text columns, presumably added through incorrect entry work. This knowledge allows the data officer to adjust any masking efforts, and ensure that (in general terms) no structural misplacement of sensitive data happens. --- version: "1.6" language: "en" --- # Installation To begin the installation process, execute the **AnalyzeSetup-\.exe** executable. Once initiated, a series of screens will appear. This process is generally straightforward, and with basic knowledge of software installation, it should be self-explanatory. However, the following steps provide detailed guidance on which options to select during specific dialog screens. ![DATPROF Analyze Setup 1.png](https://docs.datprof.com/__attachments/a_8f7128100c171402139700c89a377a507bfc2ed26234b9bb77111e618f733e8d/DATPROF%20Analyze%20Setup%201.png?cb=02941da335d4ef5d64c01306a9fdd816) Check and accept the license agreement to continue. ![DATPROF Analyze Setup 2.png](https://docs.datprof.com/__attachments/a_0108ffba064038c416e250490733fbe57f673ce59b0d9fce721b7ded2c529946/DATPROF%20Analyze%20Setup%202.png?cb=556f92b2116789d52b308ec25ae17b35) Select the location where you'd like to install DATPROF Analyze. ![DATPROF Analyze Setup 3.png](https://docs.datprof.com/__attachments/a_7b74be613cfd14c1c8c20091915555e9dd2ecaacaee349cec6962748442287b3/DATPROF%20Analyze%20Setup%203.png?cb=5cca0c44d42462f8236eba94d585e68b) Either tick the "Create a desktop shortcut" box to create an executable shortcut on your desktop, or leave it blank to skip this step. ![DATPROF Analyze Setup 4.png](https://docs.datprof.com/__attachments/a_0e3e367d689c7f547907c08226a7c7d24edaafb4630f8f95ff08ee2b76c3c6ab/DATPROF%20Analyze%20Setup%204.png?cb=3d1c8bc71be5dfa4462d51f38a7554c2) Review the previously specified settings, then click Install to begin installing DATPROF Analyze. ![DATPROF Analyze Setup 5.png](https://docs.datprof.com/__attachments/a_3a932d162dcb5cd79b48463ca9af97388a4f277e2497ed4db9691f071ce2792b/DATPROF%20Analyze%20Setup%205.png?cb=5c080ea79d21c5b140c53fd998df1ffc) The setup is now installing Analyze. ![DATPROF Analyze Setup 6.png](https://docs.datprof.com/__attachments/a_2be8311d3da7718f9c07cb1db8e446af48c8ff5137e90ba408f8048ab8ccf85b/DATPROF%20Analyze%20Setup%206.png?cb=1782ab538431f1dfd41b8d14a970ae29) Setup has finished installing, press **Finish**to launch DATPROF Analyze. ![DATPROF Analyze License Key.png](https://docs.datprof.com/__attachments/a_9154607705a4f6c8c77272ae231f4d42f89bef6545ea5e8ff9597d526392162e/DATPROF%20Analyze%20License%20Key.png?cb=fb03a7d0879312341439a0778d7bad92) After launching DATPROF Analyze, you will need to enter your license key to active the software. If you do not have access to your license key or experience any issues with the provided key, please submit a support ticket at . --- version: "1.6" language: "en" --- # Release Information | **Version** | **1.6.0** | |-----------------|----------------------------------------------------------------------------------------------| | **Description** | * Upgraded to .NET Framework 8 * Fixed crash when importing columns containing single quotes | | **Date** | 2026-02-27 | | **Version** | **1.5.0** | |-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | We're excited to announce the latest **DATPROF Analyze** release, now featuring full support for **Databricks** and **IBM DB2 z/OS**! With this update, you can now run your data profiling and statistics analysis directly on these platforms, giving you deeper insights and improved data understanding where your data lives. **Gain a comprehensive overview of your datasets with:** * **Counts** and **distinct counts** * **Null** , **positive** , and **negative** value detection * **Ordering** , **min/max** values * **Most frequent** and **least frequent** value identification DATPROF Analyze now makes it easier to define and validate your databricks and mainframe data with flexible profiling options. You can build custom profiles based on **lists of expected values** , use **"contains" patterns** to identify key strings , or apply **regular expressions (regex)** to match against the most frequent values. These profiling options make it effortless to detect possible privacy sensitive data across Databricks and IBM DB2 z/OS environments. | | **Date** | 2025-11-07 | | **Version** | **1.4.1** | |-----------------|----------------------------------------------------------------------| | **Description** | **Bugfixes:** Fixed connection issues with SQL Server and DB2 on LUW | | **Date** | 2025-07-16 | | **Version** | **1.4.0** | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | **Features:** * Added option to export profiling results to be used in DATPROF Privacy * Added label \& filter system to classify tables * Added option give stored database connections a name **Bugfixes:** * Fixed issue with opening a .dpa file from the filesystem | | **Date** | 2025-03-27 | | **Version** | **1.3.4** | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | * Resolved an issue where the length was not being calculated correctly on DB2 on LUW/iSeries. * Fixed an issue with incorrect project association, resulting in the wrong icon being displayed. | | **Date** | 2023-03-02 | | **Version** | 1.3.3 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | * Fixed issue with custom connection strings (Oracle) * Fixed issue with application name inside title bar * Fixed issue with crash when opening the About dialog | | **Date** | 2022-12-12 | | **Version** | 1.3.2 | |-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | The 1.3.2 version of Analyze introduced 4 bugfixes, primarily focused on resolving known issues in MySQL and DB2 for LUW | | | * Fixed issue on timestamps with only zero's (MySQL) * Fixed issue with FLOAT columns containing null values (MySQL) * Fixed sync meta data issue showing no changes in certain circumstances * Fixed issues while analyzing VARCHAR/CHARACTER columns (DB2 for LUW \& DB2 for iSeries) | | **Date** | 2022-10-18 | | **Version** | 1.3.1 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | The 1.3.1 version of Analyze introduced a few bugfixes. * Added missing datatypes for PostgreSQL * Added missing datatypes for SQL Server * Fixed crash when analyzing unknown datatypes | | **Date** | 2022-05-10 | | **Version** | 1.3.0 | |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | * Added support for IBM DB2 for iSeries * Added support for PostgreSQL * Added support for MySQL/MariaDB * Upgraded visualization dependencies | | **Date** | 2022-03-30 | | **Version** | 1.2.0 | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Description** | * Added foreign key tab to visualize incoming and outgoing foreign key relations * Added sort by option in table list * Added new logo's and icons * Help menu item is now refering to online documentation | | **Date** | 2019-04-26 | | **Version** | 1.1.0 | |-----------------|--------------------------------------------------------| | **Description** | * Added support for IBM DB2 for Linux, Unix \& Windows | | **Date** | 2019-01-01 | | **Version** | 1.0.7 Public Beta | |-----------------|-----------------------------------| | **Description** | Added: MS SQL Server 2016 Support | | **Date** | 2017-09-29 | | **Version** | 1.0.6 Public Beta | |-----------------|-------------------| | **Description** | Internal release | | **Date** | 2017-09-22 | | **Version** | 1.0.5 Public Beta | |-----------------|-------------------------------------------------------------------------| | **Description** | Fixed issue with duplicates functionality Status changed to Public Beta | | **Date** | 2016-12-09 | | **Version** | 1.0.3 Internal Beta | |-----------------|------------------------------------------------------------| | **Description** | **Important:**This is an internal and invite-only release. | | **Date** | 2016-03-18 | --- version: "1.6" language: "en" --- # Requirements ## Operating System **DATPROF Privacy requires one of the following:** * Microsoft Windows 7 or later * Microsoft Windows Server 2012 or later Additionally, the .NET Framework 4.8 must be installed, and the user installing the software must have the necessary permissions to install applications. ## Hardware * **CPU**: x86 1 GHz or more * **Memory**: At least 4 GB RAM * **Disk Space**: At least 1 GB of free disk space ## Supported Databases The databases listed in the table below are supported. However, as software evolves, certain database versions that have reached their end-of-life may no longer be compatible with future versions of Analyze. DATPROF follows the lifecycle policies recommended by database vendors to maintain compatibility and reliability. | **Database Type** | **Supported Versions** | |--------------------------|-------------------------------------------------------------------------------------------------------------| | **Oracle** | Version 11.2 and later | | **Microsoft SQL Server** | Version 2008 (not for Runtime) Version 2012 Version 2014 Version 2016^\*^ Version 2017^\*^ Version 2019^\*^ | | **DB2 LUW** | 10.5 and later | | **DB2 for i** | 7.2 7.3 | | **DB2 for z/OS** | 13.0 and later | | **PostgreSQL** | 9.5 and later | | **MySQL** | 8.0 | | **MariaDB** | 10.4 | ## Additional Database Requirements | **Database Archetype** | **Additional Requirements** | |--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Oracle** | * Select permissions on the tables in the schemas to analyze | | **SQL Server** | * Datareader permissions in the database to analyze | | **DB2 LUW / DB2 For iSeries** | * To use DATPROF Analyze on DB2 for Linux, Unix, Windows (LUW) the following IBM software is required on the client PC: The **64 bit** version of ***IBM DB2 Data server driver package.*** * After installation extend the PATH environment variable with the bin folder of the installation. (usually: c:\\program files\\IBM\\IBM DATA SERVER DRIVER\\bin ) * Select permissions on the tables in the schema to analyze | | **DB2 for z/OS** | * Select permissions on system catalog tables (e.g. `SYSIBM.SYSTABLES`, `SYSIBM.SYSCOLUMNS`, `SYSIBM.SYSINDEXES`) * Select permissions on all tables in the schemas to analyze | | **PostgreSQL, MySQL, MariaDB** | * Data reading / select permissions on all imported tables for the user account used to connect through Analyze to the database. | --- version: "1.6" language: "en" --- # Analyze Best Practices Throughout this manual, we've covered the various functionalities of Analyze and how they can assist end-users in identifying potential issues with their datasets. To make the most effective use of the software, we recommend following a general framework of steps. A solid approach can be outlined as follows (which we will discuss in more detail below): * Understanding your source database * Identifying what your organization considers data quality issues * Compiling your findings * Converting findings into actionable insights ## Understanding your source database When working with data, it's crucial to understand how your source database is configured. This doesn't mean knowing the exact data content, but rather understanding the overall design of the database and the purpose behind its tables. For example, when working with legacy databases, some tables may be outdated or irrelevant to the current business processes. While these could be considered data issues, it's likely that they are already known within the organization. Focusing on identifying issues with archived tables can hinder the analysis of more critical, active tables. Spending some time at the start of a project to identify which tables and schemas are relevant can help complete the project more quickly and ensure the findings are actionable. It's also important to avoid analyzing the entire organization's data at once. Instead, break the project into manageable chunks, based on factors such as: * Integrating the Analyze step into projects that involve duplicating specific data can lead to smaller, more focused data requirements. * For larger projects, it's important to break the source database into logical units. For example, if an Analyze project is designed to identify issues with accounts payable or financial data, it becomes easier to engage relevant stakeholders within the organization. These stakeholders can help provide data privacy officers with the necessary information and input on their requirements. ## Identifying what your organization considers data quality issues When using Analyze, the goal is to identify potential issues with the data. However, many organizations may not have a clear understanding of which issues are most important to them. While GDPR compliance is often cited as a key concern, which is a great starting point for establishing concrete requirements, there may be other data elements specific to the organization that are undesirable. These additional issues can also be detected using Analyze. ## Compiling your findings Once you've identified data quality issues using Analyze, the next step is to compile your findings in a clear and actionable format. This process involves organizing the issues into categories based on their severity, impact, and relevance to the organization's goals. Start by documenting the specific problems detected, such as missing data, inconsistencies, or non-compliance with regulations like GDPR. It's also helpful to categorize the issues by data source, schema, or table for easier reference. ## Converting findings into actionable insights After analyzing the source database, defining a manageable project, and compiling a solid list of requirements, the next step is to develop profiles and run the Analyze project. However, the job doesn't end with execution. Once the analysis is complete, you can generate an overview of the findings by selecting **Project → Generate Analyze Report**. This will create a file that outlines any potential issues detected during the analysis. With this data in hand, it's important to create a plan of action to address the identified issues. For example, in the previous scenario where vehicle model names were found in the description field, the next step would be to assess the description field. A specification must then be developed to determine which data is relevant for the duplicate dataset, and which data should be scrubbed or removed. --- version: "1.6" language: "en" --- # Tool Overview This chapter covers the global functionality available in the title and menu bar of DATPROF Analyze. Subsequent chapters will dive deeper into the core functionality and detailed settings of the tool. In general, the analysis process includes the following steps: * **Creating a new project** * **Providing a connection to the database** * **Importing metadata from the database** * **Configuring the Analyze profiles** * **Running Analyze statistics** * **Analyzing the database and interpreting used profiles** These steps form the foundation for using DATPROF Analyze effectively. ## Getting Started: Create or Open a Project When DATPROF Analyze is launched, a dialog appears, allowing the user to choose between creating a new project or opening an existing one. The user is presented with three options: ![DATPROF Analyze Getting Started.png](https://docs.datprof.com/__attachments/a_59effde1c71ef492157d5364e32541392727d293fde192f8ba7041278885682d/DATPROF%20Analyze%20Getting%20Started.png?cb=5f07fd060c759b4b68bf423cd2d71195) * **New project**: This option prompts the user to enter a project name and select a location to save the project. Both fields are prefilled with default values. Upon confirmation, an empty project is created, and the Settings window is displayed. * **Recently opened projects**: The right side of the dialog lists recently opened projects. Clicking on any project from this list opens the last saved version. * **Open existing project**: This option displays a dialog box for selecting and opening a previously saved project. ## Menu Overview The menu provides access to various features and functions. It includes the following four options: ![DATPROF Analyze Menu Bar.png](https://docs.datprof.com/__attachments/a_1c15dfec766f317d2661d4a777ddbf544f608c3e5d6d3394abb8c174c8649590/DATPROF%20Analyze%20Menu%20Bar.png?cb=fa632b8a3a9ea2710f682cf2fe994d73) * **File** * **Project** * **Visualize** * **Help** ### File The **File** menu provides options for managing and organizing your projects: ![DATPROF Analyze Menu File.png](https://docs.datprof.com/__attachments/a_c9b463141129dae9f2f62a66733aa40a14b311700cdbebd534b33286d1595a30/DATPROF%20Analyze%20Menu%20File.png?cb=8837f2331ee9db76bdf7a29352d11304) * **New Project:** Opens a dialog box to create a new project. * **Open existing project:** Opens a dialog box to load an existing project. * **Save**: Saves changes made to the current project. * **Save as**: Saves the current project as a new project with a different name or location. * **Close project**: Closes the current project and returns to the opening dialog. * **Exit**: Closes the DATPROF Analyze application. ### Project The **Project** menu offers several features for managing metadata, generating reports, and viewing profiling data: ![DATPROF Analyze Project Menu.png](https://docs.datprof.com/__attachments/a_05462b3da4d711829912f26644c998279d7e4ea63f3b6a6458309abf3424dd21/DATPROF%20Analyze%20Project%20Menu.png?cb=56f4e591353c6f5e96b688eaf0c3ec79) * **Import and synchronize**: Opens a dialog to import metadata or use a DME file along with the synchronization metadata wizard. * **Export Metadata to DME File**: Opens a dialog to export the data model (partially or entirely) to a generic DATPROF file format. This format allows the exchange of data models and metadata within the DATPROF product suite, including DATPROF Subset, DATPROF Privacy, and DATPROF Analyze. * To export, navigate to **Project → Export Metadata to DME File**, and select the tables and foreign keys to export. ![DATPROF Analyze Export meta data to DME-file.PNG](https://docs.datprof.com/__attachments/a_6863a2e66d9eaa4a2ca5ceab6e8cb894fbe4faf25d8629a24fedc1eaf65fbccf/DATPROF%20Analyze%20Export%20meta%20data%20to%20DME-file.PNG?cb=df1a617ce093713d8d147da7513e9a33) Using the menu option **Project → Import and Synchronize metadata from DME file**, the user can import a data model from a DME file into this project. * **Generate analyze report:**Collects all statistics and profiling data and generates an HTML report. * **Profile Overview:**Opens a view displaying all tables and columns, along with their respective analyzed profiles. * **Export profiles:** This option allows you to export profiles, enabling you to easily import them into other DATPROF software in the future. ### Visualize The **Visualize** menu offers an option to view and interact with the data model in a visual format: ![DATPROF Analyze Menu Visualize.png](https://docs.datprof.com/__attachments/a_f516f780ed02c3b1d4126f1f3d8412450567fa02db45f0078a1a2d6b8e69aa23/DATPROF%20Analyze%20Menu%20Visualize.png?cb=49432bd2352f8490bfc6b519ca0a058f) * **Visualize data model**: Displays a visual representation of the imported data model, including all user-defined relationships. The diagram shows all imported entities, along with any custom-added relations. Additionally, the window provides options to save or print (parts of) the displayed diagram. *Note: Before visualizing the data model, the metadata must be imported via the database connection.* ![DATPROF Analyze Visualization.png](https://docs.datprof.com/__attachments/a_c2511586fc1107059c65c1f82852db493aa2a99e949ac84e67a39b12a91a5bb6/DATPROF%20Analyze%20Visualization.png?cb=b5212700362c411ccaeaccf020ddeefd) ### Help The **Help** menu provides access to resources for assistance, licensing, and product information: ![DATPROF Analyze Help Menu.png](https://docs.datprof.com/__attachments/a_523ee16c02b6b78454ce735e2ab7ecea8965f210db0d45bab301ed32f516950b/DATPROF%20Analyze%20Help%20Menu.png?cb=258b91af2f340411b10e0de4cc297884) * **Help** : Opens this user manual for DATPROF Analyze. You can access it online at [this link](https://docs.datprof.com/analyze/latest). * **Enter licence key**: Opens a dialog where you can enter a purchased license key to activate DATPROF Analyze. * **About DATPROF Analyze**: Opens the About screen, displaying version information and license details for the DATPROF Analyze application. * **Debug:** Opens a Developer Logging with a list of threads. ![DATPROF Analyze About.png](https://docs.datprof.com/__attachments/a_db1d6ac7176467c31bac638ca60db27496ea2dd5b25625c1983999f256173f43/DATPROF%20Analyze%20About.png?cb=ac9f8a85b5a6fb4c7ea81a380daa3a3f) --- version: "1.6" language: "en" --- # Using DATPROF Analyze Discovering and understanding data sources is often a challenging task, especially when working with complex applications. Gaining insights into data quality and identifying privacy-sensitive information can be time-consuming and resource-intensive. DATPROF Analyze simplifies this process by helping users profile and understand their data sources quickly and efficiently. With powerful analysis tools, it enables users to gain valuable insights into their data, facilitating better decision-making and enhanced data management practices. This guide covers the key topics needed to get started: * **Getting Started** Set up a connection and define profiles to start analyzing your data. * **Analyze** In this section of the application, users can define analysis functions for the columns within the imported metadata and perform data analysis directly on the database. --- version: "1.6" language: "en" --- # Analyze Once all the connection data has been configured and profiles have been defined, we can move on to the main functionality of Analyze; the **Analyze**tab. In this part of the application the user can specify analyze functions on the columns of the imported metadata and Analyze the data in the database. On the left side of the window, you will see the imported tables or views. The right side displays the details and statistics for the selected table or view. ![DATPROF Analyze Profiling Overview.png](https://docs.datprof.com/__attachments/a_aa845b5d86e6aadd3aea6fc3205bc8d71bc9cb2ed48e4fdfe0efcba40066feed/DATPROF%20Analyze%20Profiling%20Overview.png?cb=56f3dd56f21fd7eb8a19ba3db9a50c92) To analyze one or more tables or views, click the **Run Analyze Statistics** button located at the bottom left of the window. ![DATPROF Analyze 2.png](https://docs.datprof.com/__attachments/a_bdc4e84997e853fa04113d768e215c9be5471c1898af8022f5a88a89a38340a6/DATPROF%20Analyze%202.png?cb=9ff68266591f9966909c3b320a1b2bf5) There are three icons giving you 3 options. From left to right: * ![image2018-6-28_10-3-28.png](https://docs.datprof.com/__attachments/a_a7f0c2c762ce4abba57dd5c98a39a6a3a9cdebdf22e5c2718ed21f565733e4d2/image2018-6-28_10-3-28.png?cb=3705f4026e530c674c144bdb14c65e44) **Run Statistics**: The Analyze run will start. * ![image2018-6-28_10-3-2.png](https://docs.datprof.com/__attachments/a_2070371e9c6ec133e49af926ed3aa566a6c8082a96be200665fa86af6aaae583/image2018-6-28_10-3-2.png?cb=56c657b67c76bc50f176a48833562417) **Pause Statistics**: The run will pause, a pop-up will appear. To continue press "OK". * ![image2018-6-28_10-4-2.png](https://docs.datprof.com/__attachments/a_f469e03ab27451e4305409a2dd2e6d229b1f83a5973ef43158659bfd9e633c8f/image2018-6-28_10-4-2.png?cb=91d6420b192d5ef5931fd77b30fb3a4d) **Abort Statistics**: The analyse run will abort. To run statistics a connection to the database is required. **Saving your project** After collecting statistics for your tables, it is recommended to save your project. Since gathering statistics can be time-consuming, saving the project allows you to skip this step when reopening it, unless you choose to collect statistics again. ## Tables and views On the left side of the screen the imported tables and views are shown. ![DATPROF Analyze Right Click Menu.png](https://docs.datprof.com/__attachments/a_fbf0a123e8f9ecd96f4f432f5c66b92be83ed841fa0524167d3d696c5daa3862/DATPROF%20Analyze%20Right%20Click%20Menu.png?cb=3013ac55f39cd41863019d9b08e4c6a7) From top to bottom: * **Search Box:** Allows you to filter the tables or views displayed.You can use `%` as a wildcard. * **Schema Drop-down Menu:**Contains all available schemas, shown only if more than one schema is imported. * **Tables and Views:**These are displayed with the following icons: * ![Analyze Tables icon.png](https://docs.datprof.com/__attachments/a_d103e27f7a02d5704feb6dd8b49a485435a223cdae5db7f3b0b5b890bd2516b5/Analyze%20Tables%20icon.png?cb=ffde4200e6df0d509bfbdc9c3b53ae88) for tables * ![Analyze Views icon.png](https://docs.datprof.com/__attachments/a_903bc9a2f34695bb05ecdbed0dd10c485e60ee1ad0d8b2e00ac93f2f4595522e/Analyze%20Views%20icon.png?cb=03cc4e6bb58374bc7efe2ed7c2ae87c8) for views You can select one or more tables at the same time to perform actions on them collectively. Right-click on a table or view to open a menu with the following options: * **Count Rows:**The row count will appear next to the table or view name. * **Clear Row Count:**Clears the row count for the selected tables or views. * **Clear Profiles:**Clears the matching profiles for the selected tables or views. * **Sort By:**Sort the tables by object name, object type, or row count. ## Analyze Functions The right side of the screen features four distinct tabs: **Statistics** , **Profiling** , **Data** , and **Foreign Keys**. These tabs represent the core functionalities of the Analyze tool, each designed to provide valuable insights into the database you're analyzing. * **Statistics:** Provides statistical data for the selected tables or views, analyzing only those that you choose. * **Profiling:**Offers advanced capabilities that enable users to assess data quality, identify specific patterns or data types, and detect statistical outliers. Profiling can help uncover inconsistencies and anomalies in the data, ultimately enhancing data consistency and reliability. * **Data:** Provides a deeper dive into the actual content of your database, enabling detailed exploration and analysis of records. The screen features a SQL window where you can enter a WHERE clause to filter the data in the selected table. * **Foreign Keys:** Presents a clear view of relationships between tables, assisting you in understanding how data is interconnected across the database. Together, these tabs provide a comprehensive suite of tools that empower you to effectively analyze your database. ![DATPROF Analyze 4.png](https://docs.datprof.com/__attachments/a_97c098197b5a1bec9d10b4156839130c5e54f2979c3f9c24f3fdf26005c1cca8/DATPROF%20Analyze%204.png?cb=03d4c4841f4ed95b4c3e7e302d59744c) ### Statistics The **Statistics** tab provides statistical data for selected tables or views. Only the tables or views you select will be analyzed. Certain column characteristics are displayed without requiring a full analysis. These are collected from the metadata imported during the **Import meta data** process and reflect the available data at that stage. ![DATPROF Analyze Statistics Overview Clean.png](https://docs.datprof.com/__attachments/a_26303a2db707a6c290f92f03766645089d1c3059e32443ebb3f9284c3487926b/DATPROF%20Analyze%20Statistics%20Overview%20Clean.png?cb=907a8eb6d8b9269440ea3a98c67c3065) After an Analyze run, statistics are gathered from the data in the columns and displayed. ![DATPROF Analyze Statistics Overview.png](https://docs.datprof.com/__attachments/a_4516d6986018ca915df31925e5f62490e8e2f12a5848296b63d93e9f7ec6969e/DATPROF%20Analyze%20Statistics%20Overview.png?cb=fdf334c247535377f26a666a0752cbe4) * **# Nulls**: The amount of NULL values in this column. * **# Distinct**: The number of unique values in this column. * **Min**: Show the lowest value in this column ("A" is before "a", "-1" is before "1"). * **Max**: Show the highest value in this column. * **# Shortest**: Show the shortest value in this column. For numeric also the shortest length is given where the the value 11 has the length 2 and the value -2 also has the length 2. * **# Longest**: Show the longest value in this column. For numeric also the shortest length is given where the the value 11 has the length 2 and the value -2 also has the length 2. * **Least frequent**: Show the least often occurring value in this column. * **Most frequent**: Show the most frequently occurring value in this column. #### Filtering To analyze the entire content of a table, simply select the table and click **Run Statistics**. If you want to generate statistics for a specific subset of data, you can apply a filter. In the central part of the window, navigate to the **Filter** tab, where a text box will appear. Here, you can enter an SQL `WHERE` clause to define the desired data subset for analysis. ![DATPROF Analyze Filter where clause.png](https://docs.datprof.com/__attachments/a_7b2e10ac10c6e698c9d511491c0a2c4acba19ecafdd7bae90b64fb9be0491c97/DATPROF%20Analyze%20Filter%20where%20clause.png?cb=0aacfbcedf947ad40158d0a9fb8765bf) * When running statistics with a filter applied, only the filtered data will be analyzed. * The resulting count is displayed in the **Summary** section under **Data Filing**. ![DATPROF Analyze Filter where clause Results.png](https://docs.datprof.com/__attachments/a_07c583fbd689e718a22c222939820539ff0b8639fa53398f121c7c9ec7333453/DATPROF%20Analyze%20Filter%20where%20clause%20Results.png?cb=b802771a6fa2c853e41bd1cfff864b25) * A filter can be given for every table or view. * Filters are saved in the project. * Filters are NOT removed when clearing Statistic data. * The Number of records in the left panel still shows the total number of unfiltered records in this table. #### Summary The summary tab shows either one or two diagrams depending on whether the table was filtered. The data filling diagram shows the filling of this column. The other diagram is only visible for Numeric columns. It shows the number of positive, negative and "0" records in this column. Below the diagram part you find three blocks of information: Ordering, Length and Frequency. For all three blocks you can see the most extreme results. The other tabs elaborate upon these results. Below is an example of an alphanumeric and a numeric column, both filtered with the condition `WHERE "LAST_NAME" LIKE 'A%'` (meaning the last name value must start with the letter A): ![DATPROF Analyze Data Filling where filter salary year.png](https://docs.datprof.com/__attachments/a_b446d96acadf8dada1cffbc9ae74855f09c70a5eaf31ddd9317015fb9c4370f0/DATPROF%20Analyze%20Data%20Filling%20where%20filter%20salary%20year.png?cb=95b65a684f000e412a9e936a4c4e182e) ![DATPROF Analyze Data Filling where filter last name.png](https://docs.datprof.com/__attachments/a_3bd129b09acf433c2f1a99d313bbeea56a7017b3f1a9c2180d3f7db7fd6d95bd/DATPROF%20Analyze%20Data%20Filling%20where%20filter%20last%20name.png?cb=eb250eb3918713864a9b8b9ec55f8639) #### Ordening ![DATPROF Analyze Tab Ordening.png](https://docs.datprof.com/__attachments/a_37316b15ebd98219cac02caf4c46eb48f566dd45b04b639eed828fee019373a5/DATPROF%20Analyze%20Tab%20Ordening.png?cb=00e48ab10d114842fdad5055e056a286) This tab displays two panels: one for the minimum values and one for the maximum values. It shows the values (sorted from low to high and vice versa), along with their length and the number of occurrences. Double-click on a record to view the corresponding result(s). You can modify the filter and re-run the query if you'd like to further refine the data you've found. ![DATPROF Analyze Filter Last Name and Gender.png](https://docs.datprof.com/__attachments/a_b787a80d9d6256e28f97c5a26c9c2430f0026c02f06b14bb1622b3f2de9e4390/DATPROF%20Analyze%20Filter%20Last%20Name%20and%20Gender.png?cb=c701c611487ae689c7eeb3000e80eadf) #### Length ![DATPROF Analyze Tab Length2.png](https://docs.datprof.com/__attachments/a_637091b606e0eb58ec29938786d0718012e5804fd770522b6ea75e084ad6098e/DATPROF%20Analyze%20Tab%20Length2.png?cb=298880ee0050d0ba16fc1637fcf6c012) This shows two panels. One for the shortest and one for the longest values. It shows also the number of occurrences. #### Frequency In the frequency tab you can see which values occur most often and least often within a given table. In the **Value** column the actual value is shown, and in the **Occurrences**column the amount of times that value was encountered is shown. ![DATPROF Analyze Tab Frequency.png](https://docs.datprof.com/__attachments/a_f057f9224d2ace16f0c1169ba44d0e2f33268e68cb1871e776326b2f620da164/DATPROF%20Analyze%20Tab%20Frequency.png?cb=1fbf194fbfdcccb38b0cef039596d15e) ### Profiling The Profiling tab gives an oversight of which tables conform to the profiles we've configured earlier. In the example below, the columns **EMAIL** and **GENDER** have matching profiles. This is entirely expected, as we can reasonably assume that a column named "EMAIL" will contain email addresses. However, when dealing with abstract column names like ZGKLZ125 or ZGKLA124, the use of profiles becomes particularly valuable. Profiles help remove the abstraction, offering a clear understanding of what data is actually contained within these columns. ![DATPROF Analyze 7.png](https://docs.datprof.com/__attachments/a_fb9261d40032c261242f4dd2ba347c1290495eddb043754a192811d2f043c4cd/DATPROF%20Analyze%207.png?cb=b95264e18b74c25d9981f38693ac032c) ### Data This screen features a SQL window where you can enter a **WHERE** clause to filter the data in the selected table. The results are displayed in a grid, and you can sort the grid by clicking on the column headers. By default, only 100 records are retrieved. Since all data is transferred to the client, increasing this value may lead to performance issues. ![DATPROF Analyze Data Overview Where Clause.png](https://docs.datprof.com/__attachments/a_2d92531f4ef27ffbe7cd0acd4c60b10d437be2e6f51fd2d3105afa7c38bc0432/DATPROF%20Analyze%20Data%20Overview%20Where%20Clause.png?cb=e6a6b96d8be57865a8d3d52f0c0b4477) ### Foreign Keys The **Foreign Keys** tab visualizes the existing foreign keys for the selected table. This information is readily available as it was retrieved through the **Import Metadata** and **Sync Metadata** options earlier. ![DATPROF Analyze Foreign Keys.png](https://docs.datprof.com/__attachments/a_0a31cdc947df9d2a31d8e78978a65ed59107510750d89658798334d2f257f67a/DATPROF%20Analyze%20Foreign%20Keys.png?cb=b8e23bce239fd5a6a73d7db207b8dd4f) --- version: "1.6" language: "en" --- # Getting Started When starting a new project, the **Settings** screen is where we configure the foundational details of the Analyze project, such as selecting the database to connect to and choosing the analysis profiles to apply. Additionally, the **Settings** screen allows you to import metadata from the source database and synchronize it if any changes have occurred since the project was created. ![DATPROF Analyze Project Settings.png](https://docs.datprof.com/__attachments/a_f772d401a87f58ae89833d24d8890873e44be66e577a5fa73c019d4343c32d38/DATPROF%20Analyze%20Project%20Settings.png?cb=c036910a9aa1ee6bf6544c1d817d917a) Since we need to know which database we're connecting to, the logical starting point is the **Connection Editor** . To access it, simply click **Edit Connection**. ## Connection Editor ![Connection Editor.PNG](https://docs.datprof.com/__attachments/a_2800a20654a0b364aa951d76a881562406ff8c5510a6209fa097ecc70eb0eb93/Connection%20Editor.PNG?cb=e6c970c091cfcd6a98dc7bab84961b7c) To analyze a database or specific parts of it, a connection to the database is required. This connection facilitates importing and synchronizing metadata and is essential for executing Analyze functions on the database. Datprof Analyze supports the following types of databases: * Oracle Database * Microsoft SQL Server * DB2 for Linux, Unix \& Windows * DB2 for iSeries * MySQL/MariaDB The **Connection Editor** allows you to define, select, and save database connections. On the right-hand side of this screen, the **Project Connection** section provides an overview of the database connection details used for the current project. **Managing your sensitive data when contacting DATPROF** It's entirely possible that when depending on our staff for support of our systems, the topic of transferring your project file to DATPROF comes up. Luckily, due to the way we design our software we don't handle the data within your database, and a project doesn't contain any sensitive data about your datasets. However, because it's possible to store and save the connection data to databases in our software a project file can contain privacy or security risk bearing connection data. Therefore, if you would like us to look at one of your project from a support perspective we strongly urge you to delete any existing connection data in the **Connection Editor**. As this is information we have no desire to have within our system, and could contain sensitive information, we will prune this information in any event this is attached erroneously to protect our data privacy conformity. Nevertheless, it falls to the submitter of any ticket to be wary of privacy sensitive data and remove it where possible. ### Project Connection Different database types require specific variables to establish a connection, so the user must select the desired database type from the dropdown menu at the top of this section. Below is a brief overview of the required connection data for each database type. Once you've reviewed and confirmed your connection details, you can click "Use This Connection" to apply the current configuration and close the dialog, or click "Cancel" to discard any changes. #### **Test** After specifying the connection parameters, clicking this button will attempt to establish a connection to the database. If successful, a notification titled **"Connection Established"** will appear. If the connection test fails, Analyze will provide a detailed error message explaining what went wrong. #### **Save** Once the connection parameters have been specified, clicking this button will add the defined connection details to the **Saved Connections** section of the **Connection Editor** . This feature is particularly useful for users who frequently work with the same database across multiple projects. After saving, selecting an entry from the **Saved Connections** will automatically fill in the connection data in the **Project Connection** tab. You can then simply test the connection and use it for your project. It's important to note that the password is not saved by default, so the user will need to enter the password each time Analyze is started. However, this can be changed by ticking the **"Save password"** option. **Where are my saved connections stored?** Once a connection is saved, it is written to a text file named **"Analyze_connections.lst"** in the same local environment where Analyze is installed. Typically, this file is located in the following directory: "*\\\AppData\\Local\\DATPROF"* You can easily share this file with collaborators or replace it as needed, which can speed up the process of connecting to the same database, especially when multiple databases are in use. #### **New** Clicking this button will clear any connection data currently entered in the **Project Connection** tab, allowing you to start fresh and enter new connection details without having to manually remove each field. ### Oracle ![Analyze Project Connection Oracle.png](https://docs.datprof.com/__attachments/a_1f7c184085ead692bfd7db6cef305499d9dee797864d87a1b95fd605aa555ddc/Analyze%20Project%20Connection%20Oracle.png?cb=35176f159c1b4e46ced8eb778fd529c8) * **Username**: The user name to login. * **Password:**The password. * **Type: Direct** allows the user to enter the individual data to connect to a database. **Connect Descriptor**allows the user to paste a connection string to be used to validate the connection instead. * **Host**: Hostname or IP-address of the database server. * **Port**: TCP Port. * **SID / Service**: Unique database identification. ### Microsoft SQL Server ![Analyze Project Connection Microsoft SQL Server.png](https://docs.datprof.com/__attachments/a_da843ecd35e7753f683552621a7a1ab96ef7d72a4d368532de014da942a0cda6/Analyze%20Project%20Connection%20Microsoft%20SQL%20Server.png?cb=d2fb5ca0999c4725079f4a8d6ce13e1b) * **Host**: Hostname or IP-address of the database server. * **Port** : Port of the SQL Server database. By default this is 1433. When empty, the default port will be used. * **Instance**: If a named instance is used, fill in this field with the name of the instance. This property is optional, then the default instance on the host is used. * **Database**: Name of the database * **Use Windows Authentication**: Use your windows authentication to log on to the SQL Server database. When Windows Authentication is left unchecked: * **Username**: The user name to login. * **Password:**The password. ### PostgreSQL ![Analyze Project Connection PostgreSQL.png](https://docs.datprof.com/__attachments/a_2583673833c9f57d47a497e1bcc53d87690537d9b4864180cf5ab7106348138c/Analyze%20Project%20Connection%20PostgreSQL.png?cb=12cf5c3c22b1215f9d39059f9778c200) * **Username:**the username used on the postgreSQL database for authentication. * **Password:**the password used on the postgreSQL database for authentication. * **Host:**the hostname or IP address of the postgreSQL database. * **Port:**the port on which the database is available. By default for PostgreSQL databases this is port 5432. * **Database:**the name of database which you'd like to use in this project. This should be identical to the name in the actual database. ### DB2 for Linux, Unix and Windows ![Analyze Project Connection IBM DB2 for Linux Unix and Windows.png](https://docs.datprof.com/__attachments/a_48f81fc8a82506ed15e66ed7e43f88daba9c8390220e4ffdace969b359f450c6/Analyze%20Project%20Connection%20IBM%20DB2%20for%20Linux%20Unix%20and%20Windows.png?cb=a1d07ca74de0cad47c76a60d1bbf0e6d) * **Host:** Hostname or IP-address of the database server. * **Port:** TCP port on which the database server is listening. * **Database:** Name of the target database. * **Username**: The user name to login. * **Password:**The password. ### MySQL / MariaDB ![Analyze Project Connection MySQL and MariaDB.png](https://docs.datprof.com/__attachments/a_a517c9c90909385eb05fb0977a451aee9ae38e1b4d8bcd41824e44eff214e7d7/Analyze%20Project%20Connection%20MySQL%20and%20MariaDB.png?cb=501d5b79ea4c020cff2c0a457f95fb90) * **Username:**the username used on the MySQL/MariaDB database for authentication. * **Password:**the password used on the MySQL/MariaDB database for authentication. * **Host:**the hostname or IP address of the MySQL/MariaDB database. * **Port:** the port on which the database is available. * **Database:**the name of database which you'd like to use in this project. This should be identical to the name in the actual database. ## Configuration, Profiles and Comments Now that we've gone over most of the basic functionality, let's take a look at the **Configuration** tab, the **Profiles** tab, and **Comments**. ### Configuration ![image15.png](https://docs.datprof.com/__attachments/a_a715ca7b2f4a48c11b22ed317661d255dd560d70ae42faeb6ce341eb77e3e9ba/image15.png?cb=3d5b956bb7b21613a200dcf5b42ea101) Analyze attempts to maximize performance by running multiple parallel processes. This is a way to execute multiple different processes at the same time by opening multiple connections to the database. This is a value you can tweak yourself. The more parallel processes, the faster Analyze will run. Excessively high numbers of parallel processes can cause some unwanted behavior during execution of templates depending on the specifications of your local database and server hardware. By default this is set at 8. For most use-cases, this is plenty. If you experience issues during deployment (Deadlocks are a good example), a good first step can be to turn down the number of parallel processes to see if this impacts the deployment of a template. **Setting maximum parallel processes to 1** If you set the maximum allowed parallel processes to 1 the performance of any project will tank considerably, as this is an incredibly inefficient way of performing tasks. Therefore, we don't advise ever doing this aside for if you need to test certain performance related issues. ### Profiles Profiles are expressions which allow you to identify your data. What this means in layman's terms is that you create a function which checks a certain condition, and if this condition is true it will mark any location in the database (per table column) where this condition is true. This may sound daunting at first, but once you understand the core reasoning behind these profiles, these become self-explanatory. Profiling is primarily a tool made to identify the location of (sensitive) data, and not meant to address data quality issues. This is an important distinction to make. Using profiles allows us to see where potential problems lay in our data. After this, it becomes easier to resolve such issues manually through SQL commands or other means, or report the findings to others. As such, profiles are also an easy way to visualize to a layman which database columns have potential issues. In order to start executing our profiles we need to know (roughly) what our database looks like. This can be done by **Gathering statistics.**This is explained in the chapters to come. Once this is done, we can start taking a look at how to configure our profiles. Below is an image of what a typical profile configuration might look like. ![image2019-2-5_13-51-0.png](https://docs.datprof.com/__attachments/a_c485885b7246ed3d0361f074f60ce4d60cbad9fb5e9dd17272b5886da1d54ed0/image2019-2-5_13-51-0.png?cb=586488fc9822ccf669a22a20111a7b11) Let's take a look at the various fields and how to enter them to get the most out of their usage. #### **Profile name** This is the actual name given to a profile. This is also the text that will later be displayed on any column that fulfills this profile's criteria. As such, it's important to give profiles a short, descriptive name that explains what a fulfilled condition means on a column. ### Examples of profile names Let's imagine we're trying to build a profile that checks whether a column contains IBAN (European bank account) numbers. An example of a bad name would be something like **"Regex1"** or **"Profile1"** , as this name showing up on a specific table doesn't actually tell the user anything. A good name would something along the lines of **"IBAN_PRESENT"**. This conveys to the user in a glance what's wrong with a marked column without having to check the profiles tab. #### **Type** There are three types of profiles present within Analyze, each described briefly below. ##### Values The type **Values** checks whether one of the given values is exactly present in the database. An important distinction here is that this only returns as **True**when the exact value is found in a column. This is mostly used for columns where you know exactly what can be entered, or have data that is recurring. Examples of this would be gender, multiple-choice data fields, etc etc. ### Examples of values If we configure our profile to check for value **spaghetti** , it will go through any column in search of rows that contain **spaghetti** , and only **spaghetti** . If a column contains the defined value plus additional information, it does not match. For instance, if this profile would check a column with content "**spaghetti bolognese** ", it would return as **False.** #### Contains The type **Contains**checks whether (part) of the definition is present in any column. This does not require an exact match. If you're looking for a specific value which always starts with a specific string of text or number series, this can help you sort through large text fields quickly. ### Examples of contains If we configure our profile to check for contains **spaghetti** , it will go through any column in search of rows that contain **spaghetti** anywhere in the field, even if other data is also present in the field. If a column contains the defined value plus additional information, it will still match. For instance, if this profile would check a column with content **"spaghetti bolognese"** , it would return as **True**. #### Regex Regex (Regular Expression) is a way to search within text strings, and is a widely used standard which can be consulted on . Using regular expression allows a user to define very specific functions which check for certain patterns within columns. #### **Definition** A definition is either one of multiple values to check against, or a regular expression function. For **Values** and **Contains**, when supplying multiple strings to check against, it is important to realize that Analyze expects a few things: * Commas between values to be checked for. * No spaces between values or commas. In order to use multiple values follow this example → **Value1,Value2,Value3,Value4** * No spaces at the end of a collection of values. For Regex type profiles this is where you paste your regular expression. #### **Percentage** A function has to be supplied a percentage. This percentage is the amount of positive returns within a specific column before the profile is applied to it. The usage for this is to determine how much of a specific occurrence is considered significant. For instance; it can be reasonably expected that for certain profiles, it's not interesting to mark a column as faulty if only 2% of columns contain matching values. **Examples of percentages** Imagine we define a profile which checks whether a column contains "**spaghetti**". A configuration of that would look like the following: ![image2023-1-17_11-25-28.png](https://docs.datprof.com/__attachments/a_ad2c659b4471958233989d91a1ddb29f2a1ed24c1122c893005be204c4ff0f42/image2023-1-17_11-25-28.png?cb=54914d38cff0d6e5eeb206852ba6e51a) If we run profiling on a database, it will check all columns whether at least 40% of all rows of that column contain "spaghetti" anywhere in it. If this is the case, a profile will be applied to that column. If a column named **Italian_Recipes** contains 1000 rows, and 350 of them contain the word "spaghetti", then the profile is not applied. This is because 350/1000 rows = 35%, which is less than the 40% we specified. If the same column contained 400 rows that did contain "spaghetti", it would apply the profile to this column. If the same columns contained 500, 800, or even all rows, it would also apply the profile. #### **Description** The description field allows you to enter a description for what your profile is meant to do or other information regarding it. This is a simple text field. ### Comment Here you can enter your comments to keep track of the work you have done on any given project. ![image17.png](https://docs.datprof.com/__attachments/a_8555167f50e65254ecdce397d90268616e4e60d74ae0db9e39645bc6b6e1315d/image17.png?cb=f152baf5ca7b026d5cd6ab5348dc496b) ## Importing Metadata Using the button ***Import metadata*** (or using the top-left menu option ***Project → Import and synchronize → Import metadata wizard***), a wizard for importing meta data of the specified database is started. Importing the metadata through this wizard consists of three steps: 1. Selecting which tables and to import (and from which schema) 2. Selecting which foreign keys to import; 3. Starting the metadata import. ### Selecting the schemas and tables During this first step, the user can select the tables to be imported. The screen consists of two halves. ![Import ODBC wizard 1.PNG](https://docs.datprof.com/__attachments/a_fde94614fb899469149695b4a3901b39e9241c4da949675491f4a0a514321984/Import%20ODBC%20wizard%201.PNG?cb=b099efeea7824aae4ce972cdf9b0f602) The left part shows the metadata available in the database. In the drop-down box (top left) the scheme containing the tables can be selected. In the list below the user can select one or more tables which will be imported. The right part shows the selected tables to imported. With the **single arrow ( \> )** , the selected tables are added to the list of tables to import. The **double arrow ( \>\> )** will add all tables from the selected schema in the list of tables to import. The**Next**button moves the wizard to the next step, selecting all tables moved to the right-hand side as tables to be imported. ### Selecting relations (foreign keys) ![Import ODBC wizard 2.PNG](https://docs.datprof.com/__attachments/a_915279f41ff238ec8b725ea1ad96ac742e599e59fa94e0964a42d4f8cfe5d382/Import%20ODBC%20wizard%202.PNG?cb=2c41cac0aeb4d4936ae2ef787445ea15) The next dialog allows the user to select the foreign key relations (FKs) to import. This dialog has the same format as the previous dialog. The list of available FK's are derived from the selected tables. Again, the user can select one or more FK's and add them to the list FK's to import. Normally, it's desirable to import all foreign key relations. The **Next**button moves to the next step. ### Importing the specified metadata ![Import ODBC wizard 3.PNG](https://docs.datprof.com/__attachments/a_21fc05910bd5ad86f8ef5eadd2f69f9b7018c1ad39d7be879845413d23ef2554/Import%20ODBC%20wizard%203.PNG?cb=4953e83b17cd2dd43ee4a782b7752bba) The last dialog of the wizard allows the user to actually import the selected metadata (tables and FKs). By clicking on **Start**, the application will retrieve the selected metadata from the database. The progress of the import can be monitored with the Progress Bar. By clicking on **Logging**, the application shows the logging during import. This logging also provides input for resolving any errors that occurred during the import. ## Synchronizing metadata The***Metadata Synchronization Wizard*** helps the user to process any changes to the data model of the target database, ensuring that anonymization can be executed on a changed data model. Synchronization can take place through a DME file or an ODBC database connection (Project Connection*)*. In the example below we use the ODBC Database connection. ![ODB Sync 1.PNG](https://docs.datprof.com/__attachments/a_05202fee4b26b2277a86d6583e721c3c83df4d0ff7eb9d4115bb277a70bec3b9/ODB%20Sync%201.PNG?cb=5d4a6c8867bb08f8c1d53d9311ff82dc) ![ODB Sync 2.PNG](https://docs.datprof.com/__attachments/a_35439c7c1b7e587527609ca2cb654110d05cbcba90897d81f3e0317776bfea0d/ODB%20Sync%202.PNG?cb=4269088a381cc063866d559d1e1b3183) The following screen shows a summary of the differences between the metadata from the target database and the metadata already imported into the application. ![ODB Sync 3.PNG](https://docs.datprof.com/__attachments/a_acafc955bb2bd0eee5b7bf0241e45ab28dcc905d5870da19a448fa699b6a22f9/ODB%20Sync%203.PNG?cb=0e902bb996be94687c95b3938f3248a1) Changes may include tables, columns and Foreign Keys. In time they can be added, changed or deleted. The *Synchronization wizard* does not distinguish between self-defined Foreign Keys and removed Foreign Keys from the database. They are both marked for deletion in the wizard. By clicking **next**, a dialog will be opened for each type of change, where the user can decide whether he wants to process the change in his project. ![ODB Sync 4.PNG](https://docs.datprof.com/__attachments/a_cbcfcffa893de6566d3c6d94e1e3e2d681d9f863e01d7511af4f22cfa4d66374/ODB%20Sync%204.PNG?cb=cc8df2f40367af52bc3a39f9d7f1c566) These dialogs have the same format as the dialogs for importing metadata. The last screen allows the user to process the selected changes in his project. The progress of the synchronization can be monitored with the Progress Bar. ![ODB Sync 5.PNG](https://docs.datprof.com/__attachments/a_e89eab19bb8697d3464f05a5935fdcfe4d589c31a6f3869bcaa66dd123ad124a/ODB%20Sync%205.PNG?cb=767e33cb8230e818a14351f6071f04b7) --- language: "en" --- # DATPROF Privacy Training ![privacy_dark_full@8x.png](https://docs.datprof.com/__attachments/a_91c4672385e2b702bda173d2833c3a6f6264231ff50b9a3d7b34fac5acfa69ce/privacy_dark_full@8x.png?cb=0b28090c42f8c874a0e71f7ca94de249) Tip This training is designed for people who prefer **learning by doing.** We really advice you to follow this training to get a complete overview of the functionality of DATPROF Privacy and how to mask advanced and complex databases. This training does not replace any of the [documentation](https://docs.datprof.com/privacy/latest), but the training is complementary. The training is divided in separates sections. You can find and navigate directly to the sections with the menu on the left side. Also a complete overview of this training can be found on the right side. To complete this training successfully basic knowledge of databases and SQL is required. To learn more about SQL and Databases you can review the content of this online [Oracle Tutorial](https://www.oracletutorial.com/). --- language: "en" --- # Create your first masking template Now let's start building your first masking template. When you start DATPROF Privacy for the first time it asked you for a valid licence key. After entering a valid license key, you create a new project or open one of your recently opened projects. When creating a new project, you must choose a project name and a location on your hard disk where you want to save your template. A folder will be created with a \.dpv file inside and two extra folders. The scripts folder will be used to save all additional scripts*.*The deployment folder is used to save logging and audit reports of each masking run that you execute using DATPROF Privacy. When you generate your masking template as a Runtime application another folder will appear called **dpfgen.**This folder will contain the generated Runtime application. Now lets create your first project! * [ ] Click "**New project"**to create your first masking template * [ ] Enter a project name * [ ] Use the default location to save template, with Browse... button you can change the location. * [ ] Click **OK**to create and save your masking project template If you get stuck, you can also watch an short video clip how to create a new project. Make the video fullscreen to get a better view. *** ** * ** *** * [Connect to the Gasware Database](https://docs.datprof.com/privacy-training/create-your-first-masking-template/connect-to-the-gasware-database.md) * [Importing meta data](https://docs.datprof.com/privacy-training/create-your-first-masking-template/importing-meta-data.md) --- language: "en" --- # Connect to the Gasware Database When you created a new project, the first thing you will see is the **Project settings**view. In this section you manage the database connection. Also you can import and synchronize the meta data from database. Now we will connect to the Gasware Training Database. * [ ] Click **Edit connection**to open up the connection editor. * [ ] Create a new connection with the following informationfound in the bottom left in the Gasware application. ![Screenshot 2026-02-06 133317.png](https://docs.datprof.com/__attachments/a_e17c9dca50e4f3643d2545cc29af6fb790aec310f97719e2215ded59f97a980c/Screenshot%202026-02-06%20133317.png?cb=e45be564b423c0181082c86b80c027fe) * [ ] Click the**Test** button * [ ] If you get the "*Connection attempt was successful"* message, click **Save.** * [ ] If you are getting a database error, please contact [support@datprof.com](mailto:support@datprof.com) * [ ] Click**Use this connection** to select this connection and close the connection editor. **Warning** The chosen connection is not only used to import the meta data, but also to execute and deploy the template. So for future usage, make sure you don't use your connection to your production database! If you get stuck, you can also watch an short video clip how to connect to the Gasware Database. The connection details may differ from your situation. Make the video fullscreen to get a better view. [https://www.youtube.com/watch?v=E4JxyqY0PnA\&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR\&index=4\&t=0s](https://www.youtube.com/watch?v=E4JxyqY0PnA&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR&index=4&t=0s) --- language: "en" --- # Importing meta data So you have successfully connected DATPROF Privacy to the Gasware Database. Now it is time to import the meta data from the database. Importing meta data will save the structure of your database in your template file. The tables, attributes, data types and foreign key relationships are imported using the **Import meta data**wizard. The actual data inside your database is not imported, only the structure of your data model! DATPROF Privacy support importing meta data from different schemas. The Gasware Database contains only one schema per user. * [ ] Click the **Import meta data**button to open up the wizard * [ ] Wait while the meta data is loaded... * [ ] Import the meta data of all the tables, by clicking the ![image2019-8-1_15-40-35.png](https://docs.datprof.com/__attachments/a_64e0a039f1e1c639a1b02875ef61862da2ef3244cb0d2fc25f63fc41c716381a/image2019-8-1_15-40-35.png?cb=f8ccd71e885c49dfbbdc1ba4e031d511) button. All tables will be moved to the right side * [ ] Click **Next** to proceed to the next step * [ ] Import the meta data of all the foreign keys, by clicking the ![image2019-8-1_15-40-35.png](https://docs.datprof.com/__attachments/a_64e0a039f1e1c639a1b02875ef61862da2ef3244cb0d2fc25f63fc41c716381a/image2019-8-1_15-40-35.png?cb=f8ccd71e885c49dfbbdc1ba4e031d511) button. All foreign keys will be moved to the right side * [ ] Click **Next** to proceed to the next step * [ ] Click **Start**to actually import the meta data from the database into your template * [ ] Click **Finish**to return to the Project settings view Cool! You've imported the Gasware Training database model! If you get stuck, you can also watch this short video clip on how to import meta data from the Gasware Database. The connection details may differ from your situation. Make the video full screen to get a better view. --- language: "en" --- # Deployment You've finished up your first masking template, but nothing has been executed on database. We've only configured the masking template. There are two options to execute your masking template on the database. Directly from DATPROF Privacy or using the DATPROF Runtime platform. In this training we will run it directly from Privacy. We also learn you how to generate a Runtime application, but this training does not cover the installation and executing of the template within the Runtime platform. * [Execute using Privacy](https://docs.datprof.com/privacy-training/deployment/execute-using-privacy.md) * [Generate a Runtime application](https://docs.datprof.com/privacy-training/deployment/generate-a-runtime-application.md) * [Logging \& Audit Reports](https://docs.datprof.com/privacy-training/deployment/logging-audit-reports.md) * [Synchronizing meta data](https://docs.datprof.com/privacy-training/deployment/synchronizing-meta-data.md) --- language: "en" --- # Execute using Privacy It's time to execute your template on the database! In the menu bar next to the Project Settings and Development, there is **Deployment** section. Lets take a closer look to the Deployment view. On the top we have three buttons to start the masking process, to pause a run and to abort a run. Below the three buttons we have a progress bar that gives an indication of how far the masking process is. Below the progress bar there is a monitor that contains information about the different actions that are currently being executed. Below the monitor we have some advanced options. DATPROF Privacy will bypass triggers, constraints and indexes on the database by dropping or disabling them. After the run has finished all the triggers, constraints and indexes will be restored. Warning If you abort a run it could be that some constraints, triggers and indexes are not correctly restored. Always try to let DATPROF Privacy finish the run to prevent missing indexes, constraints or disabled triggers. At the beginning of a run a RestoreDDL.sql script is generated containing all bypassed objects that you can use to restore your database in case of an aborted run. ![image2019-8-6_11-1-43.png](https://docs.datprof.com/__attachments/a_5fe5786bcf736b63aec88deec47dbadf0a897684ff0b7a0c0f5dc8ade609923d/image2019-8-6_11-1-43.png?cb=76a1667e3da63eba06ca6296eab1946f) Now lets execute your template on your training database * [ ] Swith from **Masking** to **Deployment** * [ ] Click **Start**to start your masking run * [ ] A confirmation popup will ask you if this is the database that you want mask, Click **Yes**to proceed Now you have to sit and wait for DATPROF Privacy to execute all the different actions on the database. There could be two scenario's! Everything went perfect and after a couple minutes your first masking run finished correctly. Or if there is a mistake in your template an error will occur during the run. In that case the **Errors...**button will be enabled and the module that contains the error will shown in red. ## Error handling If there are errors, you can do the following * [ ] Click the **Errors...**button in the top right corner * [ ] Select the first error of the list * [ ] You will see the database error of the module that went into error, there are a couple of common errors * * Development errors: You have made a mistake in the configuration of masking function, for example in the *Condition, Lookup Table, Input or Output mapping, Expression* * Permission errors: You don't have the right permissions on your database to complete the function. * Database errors: These errors occur if your database is not configured correctly for the task. Not enough tempspace for example Most errors can be fixed during the run. For development errors, return to the development area and find and repair your mistake. For Database or Permission error, you may need to ask a database administrator to help you out. Then return to the **Errors** window. Select the error en choose **Retry** as *Action* and click **Apply** to retry the module. If you cannot fix this issue during the run, choose **Skip** en click **Apply**to continue the run. If you got stuck, just take a look at the following video clip --- language: "en" --- # Generate a Runtime application When your masking template is completely finished, you may want to automate the execution of your template on multiple databases. DATPROF Runtime is a central Test Data Management portal in which you can automate, schedule and monitor the execution of test data templates like masking, subsetting and generating data. DATPROF Runtime is a web application that you install on premise on your server. You login with your browser and configure your different test data environments. Within those environment you can upload and install a generated template (Runtime application). This training will not go into the Runtime details, but only will teach you how to generate a Runtime application. * [ ] To generate a Runtime application click on **Project→Generate for Runtime....** * [ ] Click**Generate** to start the generation * [ ] After generation a new explorer window is opened and there you have a ZIP file containing all the logic that Runtime needs to execute the masking process! In this short video clip you can also see how you can generate a Runtime application --- language: "en" --- # Logging & Audit Reports Each masking run produces logging and after a successful run also HTML Audit report. You can find these within your project folder. * [ ] Go to the location of your masking template (hint: the location of your template is shown in the title bar of DATPROF Privacy) * [ ] Go to the **deployments** folder * [ ] Each deployment has its own time stamp, open the folder with the time of your run Inside this folder you will find 3 of 4 files depended if the run was successful * **Auditlog.html**: This shareable HTML reports contains all functional and technical information of your masking run. It also contains the added descriptions and comments you have added inside your template * **Deploymentlog.log**: This is the main log file with all the different actions and errors * **deployment.sec:**This is encrypted debug logging with all the generated queries. The deployment.sec can be used to solve support questions. * **RestoreDDL.sql:**This SQL file contains all DDL scripts to add or enable the constraints, indexes and triggers. If you got stuck, just watch this short video clip --- language: "en" --- # Synchronizing meta data When you are developing a template, you will import meta data from database. After a while, your database may have changed. New tables could be added, tables could be changed, etc. Luckily you don't have to start all over after your database model has changed. DATPROF Privacy offers an easy to use Synchronization Wizard, to import the changes from your database. If your changes contains new privacy sensitive information you have you add additional functions, but most of the time you can sync and run your template again. So how do you sync your template * [ ] Go to the **Project settings** * [ ] Click the **Sync meta data** button next to the *Import meta data* button * [ ] You can choose if you want to sync with the database or with a DME (txt) file. Select **Using an ODBC connection** * [ ] Your current connection is selected, press **Next** * [ ] The meta data is now compared with the database, first is an summery of the changes. * [ ] Click **Next**to continue * [ ] For each change you can decide if you want to import them or not. **Move** the items to the right to sync them. * [ ] Click **Next**to continue * [ ] In the final step, click **Start**to synchronize your meta data * [ ] After the synchronization, click **Finish** to close the Sync wizard If you got stuck, just watch the following video --- language: "en" --- # Developing your generation template Now let's start building your first generation template. When you start DATPROF Privacy with the intention to use the generation module you need a custom license key. Let's add that licence key: * [ ] Click "**Help" → "Add licence key..."**to add a new licence key * [ ] Enter the new key and click on**"Ok"** * [ ] **Restart**DATPROF Privacy * [ ] You have now a **fourth** option at the **top**of the Privacy application * [ ] Click on **"Generation"**to start building your generation template * [Create a generation set](https://docs.datprof.com/privacy-training/developing-your-generation-template/create-a-generation-set.md) * [Define generators](https://docs.datprof.com/privacy-training/developing-your-generation-template/define-generators.md) * [Create a second generation set](https://docs.datprof.com/privacy-training/developing-your-generation-template/create-a-second-generation-set.md) * [Define generators for Orders](https://docs.datprof.com/privacy-training/developing-your-generation-template/define-generators-for-orders.md) * [Create a dependency](https://docs.datprof.com/privacy-training/developing-your-generation-template/create-a-dependency.md) * [Execute the Generation template](https://docs.datprof.com/privacy-training/developing-your-generation-template/execute-the-generation-template.md) --- language: "en" --- # Create a dependency We are now going to create a dependency between both generation sets. With this we enforce that the foreign key values are first created in the CUSTOMERS table before the lookup can be performed from the ORDERS table. * [ ] Select the **Generate_Customers**generation set * [ ] Click on **Edit dependencies...** at the bottom right of your screen * [ ] Click on **Generate_Orders**from the list of successors * [ ] Click on **Ok** --- language: "en" --- # Create a generation set We start by creating a generation set. Complete the following steps to create a generation set: * [ ] Click on **'Add generation set'** A new set generation set appears * [ ] Modify the generation set name to **'Generate_Customers'** * [ ] Add the following description **'Generate Customers for my test goal'** * [ ] Select the table **'CUSTOMERS'** You can now expand the generation set on the left side of your screen. * [ ] Click **three** times on the **plus**that appears * [ ] Click on the **checkbox** for the table **CUSTOMERS** * [ ] Adjust the number of rows to generate to **1000** * [Create a generation set](https://docs.datprof.com/privacy-training/developing-your-generation-template/create-a-generation-set.md) * [Define generators](https://docs.datprof.com/privacy-training/developing-your-generation-template/define-generators.md) * [Create a second generation set](https://docs.datprof.com/privacy-training/developing-your-generation-template/create-a-second-generation-set.md) * [Define generators for Orders](https://docs.datprof.com/privacy-training/developing-your-generation-template/define-generators-for-orders.md) * [Create a dependency](https://docs.datprof.com/privacy-training/developing-your-generation-template/create-a-dependency.md) * [Execute the Generation template](https://docs.datprof.com/privacy-training/developing-your-generation-template/execute-the-generation-template.md) --- language: "en" --- # Create a second generation set We are now going to create a second generation set that has a relationship with the previously created generation set * [ ] Click on **'Add generation set'** A new set generation set appears * [ ] Modify the generation set name to **'Generate_Orders'** * [ ] Add the following description **'Generate Orders for my test goal'** * [ ] Select the table **'ORDERS'** You can now expand the generation set on the left side of your screen. * [ ] Click **three** times on the **plus**that appears * [ ] Click on the **checkbox** for the table **ORDERS** * [ ] Adjust the number of rows to generate to **1000** --- language: "en" --- # Define generators We are now going to determine which generators we will use to create test data. * [ ] Select the **CUSTOMER_ID** column * [ ] Choose the following generator: **'Sequential number'** * [ ] Enter the following start value: **1000000** After setting up this generator, you will see the warning icon change into a key. Key fields should always be given a generator to ensure the referential integrity of the database. * [ ] Select the **LAST_NAME** column * [ ] Choose the following generator: **'Last name'** * [ ] Select the lanuages: **NL** and **US** * [ ] Enter the following Seed: **'DATPROFGenerationSeed'** We are now going to generate surnames in the same way but in capital letters. * [ ] Select the **LAST_NAME_UPPER** column * [ ] Choose the following generator: **'Last name'** * [ ] Select the lanuages: **NL** and **US** * [ ] Enter the following Seed: **'DATPROFGenerationSeed'** * [ ] Enter the following Post SQL statement : **'upper(LAST_NAME_UPPER )'** Let's do one more: * [ ] Select the **GENDER** column * [ ] Choose the following generator: **'Weighted list'** * [ ] Modify the values to **Female** and **Male** * [ ] Make it **60** percent women and **40** percent men * [ ] Drag the Null values bar to **10 percent** --- language: "en" --- # Define generators for Orders We are now going to determine which generators we will use to create test data. * [ ] Select the **ORDER_ID** column * [ ] Choose the following generator: **'Sequential number'** * [ ] Enter the following start value: **10000000** After setting up this generator, you will see the warning icon change into a key. Key fields should always be given a generator to ensure the referential integrity of the database. * [ ] Select the **CUSTOMER_ID** column * [ ] Choose the following generator: **'Foreign key'** This generator creates a loop to the previously created generator in the CUSTOMERS table We are now going to generate dates * [ ] Select the '**PLAN_DATE'** column * [ ] Choose the following generator: **'Random date/time'** * [ ] Enter the following min value: **'2001-01-01 00:00:00'** * [ ] Enter the current date and time as max vallue --- language: "en" --- # Execute the Generation template We have created a basic template in which 1000 rows are added in different tables. Let's run this template * [ ] Swith from **Generation** to **Deployment** * [ ] Click **Start**to start your generation run * [ ] Click on **Generate**in the confirmation pop-up --- language: "en" --- # Developing your masking template In this part of training we will add different kinds of masking functions. DATPROF Privacy is more of a masking framework than a library of 1000 masking functions. The different masking functions can be applied in the Development view of DATPROF Privacy. * [ ] Now switch from the **Project settings** view to the **Masking** viewto take a closer look. On the left side you will see all the imported tables per schema. You can easily search for specific tables in the search function above schema selector. Use the % sign as a wildcard. If you have imported more than one schema, you can select other schema's from the dropdown list. The tables of the selected schema are directly listed below the schema selector. Below the Tables list, the translation tables are listed. Currently this list is empty because there is no function created yet with a translation table. On the right side you have your selected table view. In this view you have different tabs to see the *Columns* and *Foreign Keys* of the selected table. In the *Advanced settings*some extra properties can be tweaked for specific database types. Down below you see the configured functions. Currently this list is empty because we didn't add any function yet. To filter out all tables that have one or more functions use the checkbox **Hide tables without functions**. To add masking functions select a table and one or more columns and click **Add function**or right click the selected columns and use the context menu to add a function. In the following chapters we will add different functions to the Gasware Database. ![image2019-8-2_9-15-18.png](https://docs.datprof.com/__attachments/a_9fbd1b2fff1e742009625b8adcecbb026666ba9ce2113282401bcdbf36d2c1f0/image2019-8-2_9-15-18.png?cb=581deb9ff54631f59c3808e324044161) If you're got stuck, you can also watch this short video to show you all about browsing the imported meta data! Make the video fullscreen to get a better view. [https://www.youtube.com/watch?v=ZhYNhQcIWYg\&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR\&index=6\&t=0s](https://www.youtube.com/watch?v=ZhYNhQcIWYg&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR&index=6&t=0s) *** ** * ** *** * [Remove all comments fields (Blank)](https://docs.datprof.com/privacy-training/developing-your-masking-template/remove-all-comments-fields-blank.md) * [Switch first names (Shuffle)](https://docs.datprof.com/privacy-training/developing-your-masking-template/switch-first-names-shuffle.md) * [Mask names conditionally (Scramble)](https://docs.datprof.com/privacy-training/developing-your-masking-template/mask-names-conditionally-scramble.md) * [Change the birthday (Fixed Day in same Month/Year)](https://docs.datprof.com/privacy-training/developing-your-masking-template/change-the-birthday-first-day-in-same-month-year.md) * [Replace suppliers (Random Lookup)](https://docs.datprof.com/privacy-training/developing-your-masking-template/replace-suppliers-random-lookup.md) * [Creating Translation Tables (Shuffle)](https://docs.datprof.com/privacy-training/developing-your-masking-template/creating-translation-tables-shuffle.md) * [Using Translation tables (Value Lookup)](https://docs.datprof.com/privacy-training/developing-your-masking-template/using-translation-tables-value-lookup.md) * [Add dependencies between functions](https://docs.datprof.com/privacy-training/developing-your-masking-template/add-dependencies-between-functions.md) * [Using database expressions (Expression)](https://docs.datprof.com/privacy-training/developing-your-masking-template/using-database-expressions-expression.md) * [Creating new data (Generate)](https://docs.datprof.com/privacy-training/developing-your-masking-template/creating-new-data-generate.md) * [Using Determinstic Masking (Generate)](https://docs.datprof.com/privacy-training/developing-your-masking-template/using-determinstic-masking-generate.md) * [Adding additional SQL Scripts](https://docs.datprof.com/privacy-training/developing-your-masking-template/adding-additional-sql-scripts.md) --- language: "en" --- # Add dependencies between functions In the previous chapter we've configured a *Value Lookup* function that uses a translation table that would be created during the *Shuffle* on*the LAST_NAME* columns. If we would execute this template in the current state there are no hard dependencies between our functions. This could result in that the *Value Lookup* is executed before the *Shuffle.*During the execution of the Value Lookup you will get errors because the lookup table (Translation Table TT_LASTNAMES) has not been created yet. To prevent this error, we have to make sure that the Value Lookup will be executed after the Shuffle! DATPROF Privacy enables users to create dependencies between functions and scripts. Now let's add an dependency between the shuffle and the value lookup! * [ ] Select the *CUSTOMER_HISTORY*table * [ ] Double click on the ***Value Lookup*** function that we've created before The function editor will open, lets add the dependency * [ ] Click on the **Dependencies**tab * [ ] Currently the list is empty, because there are no dependencies * [ ] Click the **Edit**button to open up the dependency editor The dependency editor consist of three parts * The list of predecessors. You can select one or more functions that must be executed before the current function * The list of successors. You can select one or more functions that must be executed after the current function * A graphical representation of the added dependencies * [ ] In the predecessor list, select the *Shuffle* on the*CUSTOMER (LAST_NAME, LAST_NAME_UPPER)* * [ ] Check in the graph below if both function are in the correct order (f*irst the Shuffle than the Value Lookup)* * [ ] Click **OK**to add the dependency and close the dependency editor * [ ] The dependency has now been added to the dependencies of the Value Lookup as well as the Shuffle, click **OK**to save the changes en close the function editor Great, now you know how add dependencies between functions! * [ ] To get a complete overview of your functions and the dependencies go to the ***Visualize*** menu in the upper menu bar and click the **Visualize Process Model...** This will visualize all functions and scripts with their dependencies. You may notice some dotted line arrows. These are the system dependencies. DATPROF Privacy knows which functions can be technically executed in parallel. These functions are combined in groups. The normal line is your user defined dependency! If you got stuck, review this short clip how to add dependencies between functions. --- language: "en" --- # Adding additional SQL Scripts Remember that we've created a additional table within the database for the random lookup? Sometimes you want to do some extra work. Creating lookup tables, combining translation tables, creating extra masking function. We can do that off course manually with some database tool, but DATPROF Privacy also offers features to integrate those custom scripts into your template. We will add an additional script in the Script Manager that will create the synthetic lookup table. To open the Script Manager, click in the top menu bar **Project → Additional scripts...** The script manager consists of different sections. On the left side you have the list of scripts. On the right side you see the content of the script. Each script can be Enabled/Disabled from execution. When disabled, the script will not be executed. You can choose if the script you've created is a SQL script or Operating System call. In the dependency settings, you can specify when the script must be executed. There are 5 options: * Before disable process: this will execute the script at the complete beginning of the process * Before main process: this will execute the script after all constraints, triggers and indexes are bypassed, but before any of the configured masking functions are executed. * After main process: this will execute the script directly after all masking functions, but before the constraints, triggers and indexes are restored * After enable process: this will execute the script after the constraints, triggers and indexes are restored. * Edit dependencies: you can manually specify which functions or scripts are the predecessors or successors of this script. ![image2019-8-6_10-1-4.png](https://docs.datprof.com/__attachments/a_93c4f79979e79dbbe2cd655c4a189054288d59f1a3ec4ceb4b09158106a6e476/image2019-8-6_10-1-4.png?cb=246414a99b84d20dd35466b89f83a684) We will add an script that will create or replace our lookup table from the template * [ ] Open the script manager by clicking in the upper menu on **Project → Additional scripts...** * [ ] Click on **Add script**to create a new script * [ ] Double click on the newly created script and change the name to **CRE_LOOKUP** * [ ] For the script content you**can copy paste / or write**the following PL/pgSQL script. This will drop the lookup table if it already exists and than creates it. DO $$ BEGIN IF EXISTS ( SELECT 1 FROM information_schema.tables WHERE table_schema = 'public' AND table_name = 'DPF_ORGANIZATIONS' ) THEN DROP TABLE "DPF_ORGANIZATIONS"; END IF; CREATE TABLE "DPF_ORGANIZATIONS" ( value VARCHAR(200) ); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Akorn'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Allegheny Technologies'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('AutoNation'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Bed Bath Beyond'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Boyd Gaming'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Commercial Metals Company'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('CNO Financial Group'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Chesapeake Energy'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Charles River Laboratories'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Mercury General Corp'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('Maximus Inc'); INSERT INTO "DPF_ORGANIZATIONS" VALUES ('NetScout Systems'); END $$; * [ ] Make sure the *Script type* is set to **SQL** * [ ] Click on**Edit dependencies** and add a**successor dependency**to the Random Lookup on SUPPLIER.NAME (The script must run first!) * [ ] Click **OK** in the dependency editor and**Close** in the Script manager. If you got stuck in adding the script, this short clip will show you how to use the script manager. [https://www.youtube.com/watch?v=2RE9B7rseYQ\&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR\&index=17\&t=0s](https://www.youtube.com/watch?v=2RE9B7rseYQ&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR&index=17&t=0s) --- language: "en" --- # Change the birthday (Fixed Day in same Month/Year) Someones date of birth, especially when combined with other data, is really privacy sensitive data and can be easily used to re-identify someone from 'anonymised' data set. That's why masking the date of birth is really important. Often the age of someone is also an important attribute which relates to functional and technical requirements. Just generating a random date is not always possible. When changing someones age, related data can become inconsistent. What we can do is change the dates to the first day of the same month of the same year. This makes re-identification much harder. To go even further you can even change the date of birth to the 1st of January of the same year. We will add the first day in month function to the Gasware Database * [ ] Select the *CUSTOMER*table (if not selected already) * [ ] Right click the *DATE_OF_BIRTH* column and click **Add Function → Fixed Day in same Month/Year** The function editor will open. You have two options * [ ] Change the existing date to the first day of the same month and year * [ ] Change the existing date to the first of January of the same year The first option(default) is good for now! * [ ] Click **OK**to add the function and close the function editor! Awesome! You are getting there! If you got stuck, just watch the short video clip of adding the First Day function, click play and make the video full screen for a better experience. --- language: "en" --- # Creating new data (Generate) A safe method to mask data is generate new data. With the **Generate**option data will be replaced with data based on seed files or logic. To add the **Generate**function follow the following steps * [ ] Select the *LOCATIONS*table from the table list * [ ] Right click STREET column * [ ] From the context menu choose **Add function → Generate** The function editor will open and you have different options. We open the selection drop-down menu (current filled in: 'Random date/time') and choose: 'Street' * [ ] Add as a description: **Generates US streets** * [ ] Use the drop-down menu 'Countries'to select **US**streets * [ ] Select the: **Replace the same original values with the same generated value**Checkbox below in the window. This checkbox will replace all identical values with the same value * [ ] Click**OK**to add this function and close the function editor. Now we want to complete the masking by also replacing the cities with generated names: * [ ] Right click CITY column * [ ] From the context menu choose **Add function → Generate** The function editor will open and you have different options. We open the selection drop-down menu (current filled in: 'Random date/time') and choose: 'City' * [ ] Add as a description: **Generates US cities** * [ ] Use the drop-down menu 'City's'to select **US**cities * [ ] Select the: **Deterministic**radio option and define a salt key. * [ ] Click**OK**to add this function and close the function editor Well done! You've configured your first synthetic masking function. The function is added to the function list. You can **double click** or right click and choose **Edit function...** to open up the function editor again. --- language: "en" --- # Creating Translation Tables (Shuffle) Sometimes when you shuffle or randomly lookup data, you may need to the masked data also in another table or database. DATPROF Privacy offers the option to create a (temporary) translation table that contains the primary key, original value and masked value. This translation can than be used within other functions to create a consistent masked data set. For each masking function that you configure you have the option to save a translation table. **Tip** Translation tables contains privacy sensitive data, so make sure you remove them afterwards or store them in a save location! In the Gasware Training database customer data is stored in the CUSTOMER and CUSTOMER_HISTORY table. We want to shuffle the last_names of the *Civilians* and save the translation for further use. * [ ] Select the *CUSTOMER*table (if not selected already) * [ ] Select the *LAST_NAME* and *LAST_NAME_UPPER columns* * [ ] Right click the *selection* column and click **Add function → Shuffle...** The function editor will open and now we will configure the shuffle. First we will add the condition to this table. We placed a *Scramble*on all name columns for the military people, so we don't want those last name to be shuffled * [ ] Click the **Condition** tab * [ ] Enter the following condition text and test it. where "TYPE" = 'Civilian' * [ ] Click the **Translation table** tab * [ ] Click the **Save original values in translation table**checkbox * [ ] For this training we will **select the same schema** as the user we are connecting with. (Normally you will use a separate secured schema for all translation tables) * [ ] Enter as table name **TT_LASTNAMES** * [ ] Click**OK**to add this function and close the function editor Awesome, you've configured a masking function that will shuffle last_names and save the translation for further use. You see in the translation table list (below the *Hide tables without function checkbox*) a new translation table. If you select the translation table you can see the structure of it. The primary key, old and new value will be added to this table. We can use that in a later stadium to mask other last names consistently within the customer history table. If you got stuck in creating a translation table, take a look at the following video clip. [https://www.youtube.com/watch?v=35Mn4-gmyRM\&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR\&index=10](https://www.youtube.com/watch?v=35Mn4-gmyRM&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR&index=10) --- language: "en" --- # Mask names conditionally (Scramble) In some occasions you want to apply a masking function only on a specific part of the rows within your table. Within DATPROF Privacy you can add a condition to a function to only mask specific rows that meets the condition. In the Gasware Database there are two types of customers, the largest group are Civilians and the second group are Military personal. We will use the **Scramble**function to mask all name fields of the military customers within the Gasware Database. The scramble function replaces all alphanumeric characters (A-Z) with an 'X' and for numeric datatypes all numbers with an '1'. All special characters are staying intact and also the casing of the values will stay the same. Conditions are SQL snippets that will be added to the generated query. All conditions must start with the SQL [WHERE](https://www.w3schools.com/sql/sql_where.asp)clause. This condition must be valid SQL specific for the database you are developing for. To validate a condition, click the Testbutton in the condition editor to count the amount and the percentage of the affected rows. ## Scramble Results | **Original** | **Masked** | |-----------------|---------------| | John Edwards | Xxxx Xxxxxxx | | Chris@Evans.com | Xxxx@Xxxx.xxx | | 393.293 | 111.111 | Now lets add a conditional Scramble function * [ ] Select the *CUSTOMER*table (if not selected already) * [ ] Select all name columns *(FIRST_NAME, LAST_NAME, LAST_NAME_UPPER, PARTNER_NAME* and*FULL_LAST_NAME*) * [ ] Right click the selection and click **Add function → Scramble...** The function editor will open. All the name columns are already checked. If you have missed one, you can always add them from here. Now lets the condition to this function. * [ ] Click the **Condition**tab next to the current tab General * [ ] Enter the following condition in the editorwhere "TYPE" = 'Military' * [ ] Click the **Test**button to validate how many rows will be affected * [ ] Click the **OK**button to add the function and close the function editor Great! You've configured your first conditional function. If you're got stuck, you can also watch this short video to show you how it is done. Make the video fullscreen to get a better view. [https://www.youtube.com/watch?v=Ku20MWuT3Ms\&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR\&index=7](https://www.youtube.com/watch?v=Ku20MWuT3Ms&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR&index=7) --- language: "en" --- # Remove all comments fields (Blank) One of the safest data masking techniques is to complete remove all the data from a column. This will also cause that these values cannot be used for any development or test purposes. The **Blank** function will replace all the values of a column with *null.* When a column is mandatory the **Blank**function cannot be applied. Also Primary Key columns cannot be blanked. To add the **Blank**function follow the following steps * [ ] Select the *CUSTOMERS_HISTORY* table from the table list * [ ] Right click REL_COMMENTS column * [ ] From the context menu choose **Add function → Blank** The function editor will open and you have different options. For now we will only add a description for this function and leave the rest of options. * [ ] Add as a description: **Removes all comments from customer history** * [ ] Click**OK**to add this function and close the function editor. Awesome! You've configured your first masking function. The function is added to the function list. You can **double click** or right click and choose **Edit function...** to open up the function editor again. If you're got stuck, you can also watch this short video to show you all about browsing the imported meta data! Make the video fullscreen to get a better view. --- language: "en" --- # Replace suppliers (Random Lookup) Sometimes shuffling or scrambling is not good enough and you want to replace all existing values with other data. DATPROF Privacy offers to randomly select a value from a lookup table. You can create your own lookup table and fill it with synthetic data and use that as a lookup table. In the current Gasware training database there is no lookup table with synthethic test data available, but with an easy script we can generate our own. ## Create new Lookup Table Connect to your database using your favourite database editor and run the following script to add a new lookup table with some 'synthetic test data'. You can change the names of the organisations if you like. We will use this table later on with the random lookup. create table "DPF_ORGANIZATIONS" (value varchar(200)); insert into "DPF_ORGANIZATIONS" values ('Aarons Inc.'); insert into "DPF_ORGANIZATIONS" values ('AECOM'); insert into "DPF_ORGANIZATIONS" values ('Chesapeake Energy'); insert into "DPF_ORGANIZATIONS" values ('Clean Harbors Inc'); insert into "DPF_ORGANIZATIONS" values ('Cinemark Holdings Inc'); insert into "DPF_ORGANIZATIONS" values ('Ingredion Inc'); insert into "DPF_ORGANIZATIONS" values ('Owens-Illinois'); insert into "DPF_ORGANIZATIONS" values ('Transocean'); insert into "DPF_ORGANIZATIONS" values ('Urban Outfitters'); insert into "DPF_ORGANIZATIONS" values ('Werner Enterprises Inc'); insert into "DPF_ORGANIZATIONS" values ('Aqua America'); insert into "DPF_ORGANIZATIONS" values ('Weight Watchers'); insert into "DPF_ORGANIZATIONS" values ('Aqua America'); Make sure that the table is created and commited. Now we can proceed to use this table within our template. If you don't know how to run this script in your database editor, just skip this page and continue with the training. ## Masking suppliers We will change the name of the suppliers. * [ ] Select the *SUPPLIER*table (if not selected already) * [ ] Right click the *NAME* column and click **Add function → Random lookup...** The function editor will open in which we can fill in the Random lookup details. The random lookup function consists of a lookup table and mapping the original columns to the columns in the lookup table. Because the lookup table could be temporary table that is not part of your imported tables, you have to fill in the details yourself. Lookup tables can also be within another schema, in that case you have to make sure that the user that you will use to execute the masking template has at least read (select) privilege to find the lookup table in the other schema. In that case you can prefix * [ ] For the **Lookup table** enter *DPF_ORGANIZATIONS* * [ ] Double click the empty O*riginal column* and select***NAME*** * [ ] Now double click the empty Lookup column and enter: **VALUE**(this is the column of the lookup table, see the above script) * [ ] Click **OK**to add the Random lookup function and close the function editor Great! This one was a bit harder, but you are making progress. If you got stuck, check the following video. Make it full screen for a better viewing experience [https://www.youtube.com/watch?v=JdmUc6l1pDk\&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR\&index=10\&t=0s](https://www.youtube.com/watch?v=JdmUc6l1pDk&list=PL5ZDGm7-9phIBeNMcEljY5aJ3WLoj2QzR&index=10&t=0s) [Next Page](https://docs.datprof.com/llms-full.txt/1)