Tuesday, September 21, 2021

Manual Compaction of Service Fabric Actor Replicas

The auto-compaction is not enabled by default . If you see the settings CompactionThresholdInMB mentioned at Deleting Actor is not freeing up the Disk space - Microsoft Tech Community

 

  • CompactionThresholdInMB (type: Integer, default value: 0, valid range: >= 0): Compaction is performed only when physical DB size >= CompactionThresholdInMB. Auto-compaction can be disabled through setting CompactionThresholdInMB to a value <= 0.

However to enable the auto-compaction, the prior requirement is to have disk space, that is adequate disk space should be present on the node to enable the auto-compaction i.e. if you bloated size of the replica is showing 100 GB then you will need twice the amount of disk space to be free for auto-compaction to be enabled, that is additional 100 GB should be free. 

 

if you don't have adequate space and DB are bloated or you don't want to enable auto-compaction then you may consider performing manual compaction of the replicas/DB. 

 

In this we perform the logical build to compact . The logical build replica doesn’t have the physical size it only contains the actual data, once secondary logical build is completed, it is swapped with primary and then other replica is built from new primary and other bloated replica are discarded/ dropped. 

 

Below are steps to perform manual compaction. 

 

IMPORTANT NOTE : Never kill the primary. 

  • Find node name of primary using Service Fabric Desktop Explorer.
  • RDP into the primary node.
  • Change settings:
    • Go to W:\<Primary_Node_Name>\Fabric\
    • Open Fabric.Package.Current.xml file
    • Open settings.xml under Fabric.Config.CONFIG_FOLDER_VALUE/settings.xml under Fabric folder.
  • Update settings:
    • Similar to EseStoreSettings add or find a section with ReplicatorSettings 
      <ConfigPackage Name="Fabric.Config" Version="{{CONFIG_FOLDER_VALUE}}" />
    <Section Name="ReplicatedStore">
        <Parameter Name="EnableFileStreamFullCopy" Value="false" />
    </Section>
  • Edit Fabric.Package.Current.xml to add a space.
    • Editing triggers the dynamic update on the property.
  • Start a new replica build :
    • If there is an existing inbuild/down/bad state replica, remove it using Remove-ServiceFabricReplica**. This should trigger new replica build by logical way.
    • If there are targetReplicaSetSize > minReplicaSetSize number of replicas, and you have targetrelicasetsize good replicas, then remove one of the active secondary. This should trigger a replica build by logical way.
    • Otherwise, increase targetreplicasetsize by 1 using Update-ServiceFabricService**
  • Confirm logical build is going on :
    • Using SFX, find the new node where new secondary replica is building.
    • RDP to new secondary replica and find new replica folder like here on this machine.
    • You should see that db file size is growing. This is logical copy way.

Important

  • While replica build is going on, Revert the settings in first settings.xml and then remove space from Fabric.Package.Current.xml so that we don't impact other replicas.

  • Keep checking SFX in 5 min interval when replica build is done.

  • Once you have an active secondary with compacted or logically build db, then just follow steps (2) and (3) to move primary and create new replicas using compacted/logically build primary.

  • ReplicaFolder:

    • Under W:\Apps\ folder, search for "P_<Partition_Id>" in explorer.
    • Open P_<Partition_Id> folder location : W:\Apps&lt;AppName>\work\P_<Parition_Id>.
    • Make sure there is R_<ReplicaId> folder here. Lets call this folder ReplicaFolder

  1. Steps to make new compacted secondary as Primary:

    • When compaction is done, compacted replica will come up and join the replica set size as new secondary. Either standby or active.
    • If compacted secondary is standby,
      • Then Primary must already be building a new secondary which would be inbuild.
        • Remove the inbuild replica using Remove-ServiceFabricReplica** cmdlet. This will force SF to make the compacted standby replica an active secondary.
      • If not, then remove any active secondary using Remove-ServiceFabricReplica ** cmdlet. Make sure there are target-set - 1 ready replicas including primary before this.
    • Once compacted secondary becomes active,
      • Call Move-ServiceFabricPrimaryReplica that will make compacted secondary as new primary.

  2. Steps to build new compacted secondaries from compacted primary:

    • Call Remove-ServiceFaricReplica on down/inbuild/active secondary
    • SF will automatically trigger a new replica build.
    • Unless this new replica is up and you have checked by looking at corresponding replica_folder, don't perform Remove-ServiceFabricReplica** on other secondaries
    • Once new compacted secondary is up, follow similar steps for other secondary.
    • Move-ServiceFabricPrimaryReplica -PartitionId <PARTITION_ID> -NodeName "<SECONDARY_REPLICA_NODE_NAME>" -ServiceName <SERVICE_NAME>
    • Remove-ServiceFabricReplica -NodeName "<SECONDARY_REPLICA_TO_REMOVE_NODE>" -PartitionId <PARTITION_ID> -ReplicaOrInstanceId <SECONDARY_REPLICA_ID>
Posted at https://sl.advdat.com/3icwczl
Labels: ,