Friday, April 22, 2022

Deletion of logs from $logs container from ADLS gen2 storage account using Azure Data Factory (ADF)

Background:

Many times, we have a requirement to delete the older logs from $logs as the Retention days is not supported at the moment for ADLS gen2 storage account

 

Please refer to article https://docs.microsoft.com/en-us/azure/storage/common/manage-storage-analytics-logs?tabs=azure-portal#modify-log-data-retention-period for more information

 

Deletion of logs from the $logs can be achieved from the following options

  • Azure Storage explorer
  • REST API
  • SDK’s
  • PowerShell
  • Azure Data Factory

This blog demonstrates how we can use ADF for deleting logs from $logs

 

Step 1:

Create an azure data factory resource from azure portal.  If you are new to ADF, please refer this link on how to create one:  https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal

 

Step 2:

Open Azure data factory studio in the overview

 

Deeksha_S_A_0-1650611779053.png

 

Step 3:

On the data factory portal click on the pen item -> + > Create pipeline 

 

Deeksha_S_A_1-1650611779062.png

 

Step 4:

Click on the Pipeline that has been created -> Under Activities -> General-> select ‘Delete’ and drag it to the pipeline area.

 

Deeksha_S_A_2-1650611779070.png

Step 5:

Under Source -> Select New for Datasource =-> choose Azure Data Lake Storage Gen2

Deeksha_S_A_3-1650611779077.png

 

Step 6:

In the New linked service, you can use the name that’s by default or customize as per requirement 

If the account selection method is azure subscription you can choose the storage account, else you can enter manually too.

Then click on ‘Create’.

 

Deeksha_S_A_4-1650611779082.png

 

Step 7:

In the file path: enter the container as $logs and folder name(according to your requirement)

 

Deeksha_S_A_5-1650611779084.png

Step 8: 

Coming back to the pipeline, you can set the below properties as per the image :

 

 

Deeksha_S_A_6-1650611779087.png

Step 9 :

Click on Debug, to run the pipeline:

Deeksha_S_A_7-1650611779091.png

 

Note :  Can make use of  Add Trigger to  Trigger the pipeline manually

https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal#trigger-the-pipeline-manually

 

Also you can make use of Add Trigger to Trigger the pipeline on a schedule

https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal#trigger-the-pipeline-on-a-schedule

 

Posted at https://sl.advdat.com/3MtlIbvhttps://sl.advdat.com/3MtlIbv