Azure Data Factory

https://learn.microsoft.com/en-us/azure/data-factory/
Admin
Azure Data Factory is Azure's cloud ETL service for scale-out serverless data integration and...
Tokens:1,478,737
Snippets:8,922
Trust Score:10
Update:1 day ago
Show doc for...
Tokens:
Raw
### startsWith example 1

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example checking if 'hello world' starts with 'hello'. Returns true.

```text
startsWith('hello world', 'hello')
```

--------------------------------

### Example Certutil -verify -urlfetch command

Source: https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide

Example usage of the Certutil command with actual file paths. This is a concrete example from the guide.

```cmd
Certutil -verify -urlfetch c:\users\test\desktop\servercert02.cer > c:\users\test\desktop\Certinfo.txt

```

--------------------------------

### startsWith example 2

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example checking if 'hello world' starts with 'greetings'. Returns false.

```text
startsWith('hello world', 'greetings')
```

--------------------------------

### PowerShell: Set up express custom setup with license key

Source: https://learn.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup

Creates a ComponentSetup object with an express custom setup and license key, adds it to an array, and applies it to the Azure-SSIS IR using Set-AzDataFactoryV2IntegrationRuntime. Then starts the IR with Start-AzDataFactoryV2IntegrationRuntime.

```powershell
$setup = New-Object Microsoft.Azure.Management.DataFactory.Models.ComponentSetup($ExpressCustomSetup, $licenseKey)
        }    
        # Create an array of one or more express custom setups
        $setups = New-Object System.Collections.ArrayList
        $setups.Add($setup)
    
        Set-AzDataFactoryV2IntegrationRuntime -ResourceGroupName $ResourceGroupName `
            -DataFactoryName $DataFactoryName `
            -Name $AzureSSISName `
            -ExpressCustomSetup $setups
    }
    Start-AzDataFactoryV2IntegrationRuntime -ResourceGroupName $ResourceGroupName `
        -DataFactoryName $DataFactoryName `
        -Name $AzureSSISName `
        -Force
```

--------------------------------

### Install self-hosted integration runtime without firewall configuration

Source: https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime

Use this msiexec command to install the self-hosted integration runtime without configuring the firewall, which is useful when a partner or other firewall causes setup issues.

```bash
msiexec /q /i IntegrationRuntime.msi NOFIREWALL=1
```

--------------------------------

### Example: Create MyTrustStoreFile with password

Source: https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-rds-for-oracle

Example command that creates a PKCS-12 truststore file named MyTrustStoreFile with password 'ThePWD' from cert.txt.

```bash
openssl pkcs12 -in cert.txt -out MyTrustStoreFile -passout pass:ThePWD -nokeys -export  
```

--------------------------------

### Create and start a schedule trigger with .NET SDK

Source: https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-schedule-trigger

Add this code to the main method of a .NET SDK quickstart sample. It creates a schedule trigger named 'MyTrigger' that runs every 15 minutes for one hour, associates it with the 'Adfv2QuickStartPipeline' pipeline, and starts it. Requires the quickstart setup.

```csharp
            // Create the trigger
            Console.WriteLine("Creating the trigger");

            // Set the start time to the current UTC time
            DateTime startTime = DateTime.UtcNow;

            // Specify values for the inputPath and outputPath parameters
            Dictionary<string, object> pipelineParameters = new Dictionary<string, object>();
            pipelineParameters.Add("inputPath", "adftutorial/input");
            pipelineParameters.Add("outputPath", "adftutorial/output");

            // Create a schedule trigger
            string triggerName = "MyTrigger";
            ScheduleTrigger myTrigger = new ScheduleTrigger()
            {
                Pipelines = new List<TriggerPipelineReference>()
                {
                    // Associate the Adfv2QuickStartPipeline pipeline with the trigger
                    new TriggerPipelineReference()
                    {
                        PipelineReference = new PipelineReference(pipelineName),
                        Parameters = pipelineParameters,
                    }
                },
                Recurrence = new ScheduleTriggerRecurrence()
                {
                    // Set the start time to the current UTC time and the end time to one hour after the start time
                    StartTime = startTime,
                    TimeZone = "UTC",
                    EndTime = startTime.AddHours(1),
                    Frequency = RecurrenceFrequency.Minute,
                    Interval = 15,
                }
            };

            // Now, create the trigger by invoking the CreateOrUpdate method
            TriggerResource triggerResource = new TriggerResource()
            {
                Properties = myTrigger
            };
            client.Triggers.CreateOrUpdate(resourceGroup, dataFactoryName, triggerName, triggerResource);

            // Start the trigger
            Console.WriteLine("Starting the trigger");
            client.Triggers.Start(resourceGroup, dataFactoryName, triggerName);
```

--------------------------------

### first examples

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Two examples showing first with a string and an array. Returns 'h' and 0 respectively.

```text
first('hello')
first(createArray(0, 1, 2))
```

--------------------------------

### dmgcmd -sus -StartUpgradeService

Source: https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime

Starts the self-hosted integration runtime upgrade service.

```APIDOC
## dmgcmd -sus -StartUpgradeService

### Description
Starts the self-hosted integration runtime upgrade service.

### Method
Command-line action

### Endpoint
`dmgcmd -sus`

### Parameters
No arguments.

### Request Example
```
dmgcmd -sus
```

### Response
No response body. The action completes with a success or failure message.
```

--------------------------------

### Create input.txt file content

Source: https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-python

Sample content for the input.txt file used in the tutorial. Save this text as input.txt on your disk.

```plaintext
John|Doe
Jane|Doe
```

--------------------------------

### Response for GET diagnostic settings (example 2)

Source: https://learn.microsoft.com/en-us/azure/data-factory/monitor-logs-rest

Another example 200 OK response for the GET diagnostic settings request, showing actual resource names in the properties. Includes the same log categories and retention policies.

```json
{
    "id": "/subscriptions/<subID>/resourcegroups/adf/providers/microsoft.datafactory/factories/shloadobetest2/providers/microsoft.insights/diagnosticSettings/service",
    "type": null,
    "name": "service",
    "location": null,
    "kind": null,
    "tags": null,
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/shloprivate/providers/Microsoft.Storage/storageAccounts/azmonlogs",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/shloprivate/providers/Microsoft.EventHub/namespaces/shloeventhub/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/ADF/providers/Microsoft.OperationalInsights/workspaces/mihaipie",
        "eventHubAuthorizationRuleId": null,
        "eventHubName": null,
        "metrics": [],
        "logs": [
            {
                "category": "PipelineRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "TriggerRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "ActivityRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            }
        ]
    },
    "identity": null
}
```

--------------------------------

### Response for GET diagnostic settings (example 1)

Source: https://learn.microsoft.com/en-us/azure/data-factory/monitor-logs-rest

Example 200 OK response for the GET diagnostic settings request. Shows the diagnostic settings configuration with storageAccountId, serviceBusRuleId, workspaceId, and logs for PipelineRuns, TriggerRuns, and ActivityRuns.

```json
{
    "id": "/subscriptions/<subID>/resourcegroups/adf/providers/microsoft.datafactory/factories/shloadobetest2/providers/microsoft.insights/diagnosticSettings/service",
    "type": null,
    "name": "service",
    "location": null,
    "kind": null,
    "tags": null,
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>//providers/Microsoft.Storage/storageAccounts/<storageAccountName>",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>//providers/Microsoft.EventHub/namespaces/<eventHubName>/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>//providers/Microsoft.OperationalInsights/workspaces/<LogAnalyticsName>",
        "eventHubAuthorizationRuleId": null,
        "eventHubName": null,
        "metrics": [],
        "logs": [
            {
                "category": "PipelineRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "TriggerRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "ActivityRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            }
        ]
    },
    "identity": null
}
```

--------------------------------

### take examples

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Gets the specified number of items from the front of a string and an array. The first returns 'abc', the second returns [0, 1, 2].

```text
take('abcde', 3)
```

```text
take(createArray(0, 1, 2, 3, 4), 3)
```

--------------------------------

### startOfMonth example

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example call to startOfMonth with timestamp '2018-03-15T13:30:30Z'. Returns the start of the month: "2018-03-01T00:00:00.0000000Z".

```text
startOfMonth('2018-03-15T13:30:30Z')
```

--------------------------------

### startOfHour example

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example call to startOfHour with timestamp '2018-03-15T13:30:30Z'. Returns the start of the hour: "2018-03-15T13:00:00.0000000Z".

```text
startOfHour('2018-03-15T13:30:30Z')
```

--------------------------------

### Configure Kerberos realm and KDC server with Ksetup

Source: https://learn.microsoft.com/en-us/azure/data-factory/connector-hdfs

Run Ksetup commands to set the Kerberos realm and add a KDC server on the self-hosted integration runtime machine. Replace REALM.COM and your_kdc_server_address with actual values, then restart the machine.

```bash
C:> Ksetup /setdomain REALM.COM
C:> Ksetup /addkdc REALM.COM <your_kdc_server_address>
```

--------------------------------

### startOfDay function example

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example call startOfDay('2018-03-15T13:30:30Z') that returns the start of the day, "2018-03-15T00:00:00.0000000Z".

```plaintext
startOfDay('2018-03-15T13:30:30Z')
```

--------------------------------

### min function examples

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Gets the lowest value from a set of numbers and from an array created with createArray. Returns 1 for both examples.

```plaintext
min(1, 2, 3)
min(createArray(1, 2, 3))
```

--------------------------------

### max function examples

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Gets the highest value from a set of numbers and from an array created with createArray. Returns 3 for both examples.

```plaintext
max(1, 2, 3)
max(createArray(1, 2, 3))
```

--------------------------------

### add function example

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example of adding 1 and 1.5, returning 2.5.

```text
add(1, 1.5)
```

--------------------------------

### guid function example with format 'P'

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Generates a GUID enclosed in parentheses using the 'P' format specifier. Returns a string like "(c2ecc88d-88c8-4096-912c-d6f2e2b138ce)".

```text
guid('P')
```

--------------------------------

### dmgcmd -s -Start

Source: https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime

Starts the self-hosted integration runtime host service.

```APIDOC
## dmgcmd -s -Start

### Description
Starts the self-hosted integration runtime host service.

### Method
Command-line action

### Endpoint
`dmgcmd -s`

### Parameters
No arguments.

### Request Example
```
dmgcmd -s
```

### Response
No response body. The action completes with a success or failure message.
```

--------------------------------

### range function example

Source: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Example call range(1, 4) that creates an integer array starting from 1 with 4 integers, returning [1, 2, 3, 4].

```plaintext
range(1, 4)
```
Tokens:
Raw