Friday, April 22, 2022

Deletion of logs from $logs container from ADLS gen2 storage account using Azure Data Factory (ADF)


Many times, we have a requirement to delete the older logs from $logs as the Retention days is not supported at the moment for ADLS gen2 storage account


Please refer to article for more information


Deletion of logs from the $logs can be achieved from the following options

  • Azure Storage explorer
  • SDK’s
  • PowerShell
  • Azure Data Factory

This blog demonstrates how we can use ADF for deleting logs from $logs


Step 1:

Create an azure data factory resource from azure portal.  If you are new to ADF, please refer this link on how to create one:


Step 2:

Open Azure data factory studio in the overview




Step 3:

On the data factory portal click on the pen item -> + > Create pipeline 




Step 4:

Click on the Pipeline that has been created -> Under Activities -> General-> select ‘Delete’ and drag it to the pipeline area.



Step 5:

Under Source -> Select New for Datasource =-> choose Azure Data Lake Storage Gen2



Step 6:

In the New linked service, you can use the name that’s by default or customize as per requirement 

If the account selection method is azure subscription you can choose the storage account, else you can enter manually too.

Then click on ‘Create’.




Step 7:

In the file path: enter the container as $logs and folder name(according to your requirement)



Step 8: 

Coming back to the pipeline, you can set the below properties as per the image :




Step 9 :

Click on Debug, to run the pipeline:



Note :  Can make use of  Add Trigger to  Trigger the pipeline manually


Also you can make use of Add Trigger to Trigger the pipeline on a schedule


Posted at