1. Windows News: Windows 2022 General Availability, Zonal Shared Disk for Windows ASCS and Azure Files SMB for ASCS Released
Windows Server 2022 has been officially released by Microsoft. The testing and integration of Windows 2022 for SAP Applications has already begun. There will be announcements in this blog as soon as SAP announces support for SAP applications on Windows Server 2022.
The official Microsoft announcement about Windows Server 2022 can be found here: https://aka.ms/WS2022GABlog
More information https://cloudblogs.microsoft.com/windowsserver/
Windows Server 2022 features of interest
Hotpatch for Windows Server Azure Edition (preview) | Microsoft Docs
Automanage for Windows Server Services (preview) | Microsoft Docs
TLS 1.3 Enabled by default in Windows 2022
What's new in Windows Server 2022 | Microsoft Docs
Windows Server Summit | Microsoft
Reminder: Windows 2012 and Windows 2012 R2 approach End of Life
Azure Zonal Shared Managed Disks is now Generally Available and supported for the SAP ASCS Windows Cluster and for other scenarios such as Interface Directories. Testing with high availability configuration for other standalone engines such as MaxDB and Content Server will happen later
Azure Files SMB with full Active Directory integration is now officially supported for the ASCS sapmnt, Interface and Transport directories for Windows guest OS. Documentation can be found here Azure VMs HA for SAP NW on Windows with Azure Files (SMB) - Azure Virtual Machines | Microsoft Docs
2. Spotlight on Running SAP on Oracle on Azure
Oracle Database is the platform used to run many of the world’s largest and most complex SAP systems.
In the last two years increasing numbers of large multinational corporations have selected Azure as the cloud provider to host their SAP landscape. Microsoft and Oracle have developed some guidance for running SAP on Oracle on Azure. A full blog dedicated to “SAP on Oracle on Azure” will follow later with many more details.
This section provides an update on guidance as of September 2021
Top 10 recommendations for SAP on Oracle on Azure customers
- Use ASM + ASMLib for Premium Storage or Ultra Disk. Do not use conventional LVM and XFS, this is much more complex and has lower performance than ASM
- Use dNFS and NFS 4.1 if using Oracle on Azure NetApp (check this blog for updates on dNFS + NetApp). The February 2022 Oracle SBP contains an important fix for dNFS. Without the fix in February 2022 Oracle SBP only NFS 3.x can be used
- Always use Oracle Compression. Azure has fast powerful CPU and Oracle systems will be faster when the database is compressed
- Always use the latest available Oracle Linux and Oracle DB version + SAP Bundle Patches. OL 8.4 and Oracle 19.12 at the time of writing
- Use large M-series VMs and configure a large SGA. SGA sizes measured in Terabytes are deployed already. Use Huge Pages for the SGA and evaluate performance with _enable_NUMA_support=TRUE and _px_numa_support_enabled=TRUE. Verify NUMA topology with numactl utility
- There is no requirement for Mirror Logging on Azure deployments with ASM. Select EXTERNAL redundancy and follow SAP Note 888626 - Redo log layout for high-end systems - SAP ONE Support Launchpad
- Use Azure Backup for Oracle – test backup and restore procedures and compare to RMAN and other solutions
- Use Oracle Real Application Testing (RAT) to validate performance switching from on-prem to Azure or to test SGA or NUMA configuration changes
- Test network and disk performance between Oracle DataGuard servers and/or Azure Availability Zones with oratcptest.jar
- Configure an ASM sector size of 4k in the ASM utility – this mitigates the possibility of sector misalignment on 512e disks
Oracle Linux 8.4
Oracle solutions are fully supported on Oracle Linux.
Announcing the release of Oracle Linux 8 Update 4
Announcing the Unbreakable Enterprise Kernel Release 6 for Oracle Linux | Oracle Linux Blog
Oracle Linux 8.4 supports NCONNECT for customers not running dNFS
The recommended solution for Oracle is to use dNFS. There will be further posts in this blog regarding Oracle dNFS + NetApp NFS 4.1 requirements shortly.
Oracle Linux provides a very useful GUI management utility
Oracle web console Oracle Linux: Install Cockpit Web Console on Oracle Linux
Upstream Cockpit Project — Cockpit Project (cockpit-project.org)
Oracle Linux has a new package management tool – DNF
Oracle Linux 8: Package Management made easy with free videos | Oracle Linux Blog
Oracle® Linux 8 Managing Software on Oracle Linux - Chapter 1 Yum DNF
Very Large Oracle SGA Configurations
SAP on Oracle on Azure customers have single databases larger than 100TB in size. These customers configure SGA sizes in excess of 3TB. Modern versions of Oracle handle very large SGA sizes very well. Customers are encouraged to review their AWR reports and increase the SGA size to reduce read IO. Multiple customers have demonstrated dramatic performance improvements and large reduction in read IO throughput and IOPS by configuring very large SGA.
As general guidance Linux Huge Pages should be configured to approximately 70% of the VM RAM size. The SGA size can be set to fractionally less than this value. A very approximate example would be a m192ms VM with 4TB of RAM would have Huge Pages set approximately 3TB. The SGA can be set to a value a little less such as 2.95TB.
Large SAP customers running on High Memory Azure VMs greatly benefit from HugePages
NUMA systems vm.min_free_kbytes should be set to 524288 * <# of NUMA nodes>. See Oracle Linux : Recommended Value of vm.min_free_kbytes Kernel Tuning Parameter (Doc ID 2501269.1)
Memory and NUMA configurations can be tested and benchmarked with a very useful tool - Oracle Real Application Testing (RAT)
Oracle Real Application Testing: What Is It and How Do You Use It? (aemcorp.com)
Azure Backup for Oracle Databases – including ASM Support
Azure Backup now fully supports Oracle including Snapshot Backups on Oracle
Long-Term Support Release 19c
Oracle 19c Patch 12 is the recommended release as of August 2021. The latest SBP should be used as per SAP Note 2799920 - Patches for 19c: Database
Optimizing Oracle DataGuard Network Traffic
The links below contain information on testing and optimizing the network traffic between DataGuard nodes.
oratcptest accurately simulates the network and disk overhead of Oracle DataGuard. It is recommended to use this tool to test prior to activating DataGuard on Azure.
oratcptest https://www.thegeekdiary.com/oracle-database-measuring-network-capacity-using-oratcptest/
How to Optimize a Data Guard Configuration – The Geek Diary
Assessing and Tuning Network Performance for Data Guard and RMAN (Doc ID 2064368.1)
Example #1: (Full sync mode Max Availability)
To use, start the program on the receiving VM (the HA/standby VM which are running the redo logs on write accelerated disks).
/oracle/PRD/19.0.0/jdk/jre/bin/java -jar /oracle/PRD/oratcptest.jar -server -port=8000 -sockbuf=1048576 -file=/oracle/nocachingdisk/oratcp.tmp
To trigger the test, run the same program from the sender VM (Active DB VM).
/oracle/PRD/19.0.0/jdk/jre/bin/java -jar /oracle/PRD/oratcptest.jar dboraclep40 -port=8000 -mode=sync -write -duration=160s -interval=20s -length=64k -sockbuf=1048576
Oracle ASM Disk Sector Size Setting for all Azure Disks
Set the ASM Sector Size to 4K for all deployment on Azure. This is a fully supported configuration as documented in this link Creating Disk Groups (oracle.com)
Setting the ASM Sector Size to 4K for both 512e and 4K Native disks mitigates against a possible sector misalignment issue that may occur on Azure Premium disk.
Oracle DataGuard configurations must have “symmetrical” storage configurations, meaning it is not supported to run the Primary Oracle VM on UltraDisk and Oracle DataGuard replicas on Premium.
Do not create partitions for ASM. Present the entire physical disk and let ASM manage all disk addressing and geometry.
It is strongly recommended to use ASMLib rather than udev.
A detailed discussion of 4K support in Oracle can be found here Deep Dive: Oracle with 4k Sectors | flashdba
Additional Links
Non-SAP specific guidance for migrating on-premises Oracle databases to Azure VMs
Design and implement an Oracle database on Azure - Azure Virtual Machines | Microsoft Docs
2799900 - Central Technical Note for Oracle Database 19c - SAP ONE Support Launchpad
Thanks to Phil Tan for contributing to SAP on Oracle on Azure.
3. Azure Sentinel for SAP
The Azure platform includes a Security Information and Event Management (SIEM) and Security Orchestration Automation Response (SOAR) solution called Azure Sentinel
Sentinel has a connector for SAP applications allowing full stack monitoring, correlation and alerting.
A step-by-step procedure for deploying Sentinel for SAP Deploy SAP continuous threat monitoring | Microsoft Docs
More information can be found here #47 - The One with the Analyzing Security Threats in SAP (Yoav Daniely & Koby Mymon) | SAP on Azure
Additional videos here:
SAP on Azure Video Podcast - Episode #33 - The One with Azure Runbooks - YouTube
- Blog : Protecting SAP applications with the new Azure Sentinel SAP threat monitoring solution
- Deployment guide : Deploy SAP continuous threat monitoring with Azure Sentinel
- Kusto overview : Getting started with Kusto
- SAP Workbooks, rules and watchlists: Azure Sentinel SAP solution: security content reference
- SAP solution logs : Azure Sentinel SAP solution logs reference
- Custom analytics rules : Create custom analytics rules to detect threats
- Customer story : ASOS centralizes security operations, tackles cyberthreats with Azure Sentinel
Many more SAP on Azure videos can be found here on this YouTube channel https://youtube.com/SAPonAzure
4. Using Proximity Placement Groups with SAP Applications
SAP Note 2931465 - When to use Proximity Placement Groups on Azure to Reduce Network Latency – 3 Tier NetWeaver or S/4HANA architecture has been updated with new recommendations
Proximity Placement Groups should not be used as a “default” for SAP on Azure deployments. Instead it is recommended to run the ABAPMeter tool in SE38 (/SSA/CAT). If the result in columns Access DB and E.Access DB in ABAPMeter tool is consistently and significantly higher than 100 then it is recommended to:
- Verify if the DBMS server and SAP Application Servers are in the same Availability Zone and the same Virtual Network. Higher latency is normal for cross zone deployments. PPG will not improve cross-zonal latency significantly in most of the cases
- Ensure Accelerated Networking is used for both SAP Application Servers and Database Server
- Ensure the Standard Load Balancer is used and not the Basic Load Balancer (if DBMS solution uses ILB for High Availability)
- Deallocate and then Start the Azure VMs. All VMs for a specific SID should be Deallocated at the same time including SAP Application Servers, Database Servers, ASCS Servers and any DBMS High Availability secondary servers. This procedure will often improve network latency automatically by allowing the Azure fabric to optimally locate VMs
If all the above steps have been tried and there has been no improvement in Network Latency and this is adversely impacting performance then Proximity Placement Groups can be considered
PPG must never be used to reduce application level latencies between ECC and BW or ECC and PO.
2879613 - ABAPMETER in NetWeaver AS ABAP
Azure proximity placement groups for SAP applications - Azure Virtual Machines | Microsoft Docs
5. Azure Backup for SQL Server – Exciting New Features
In response to requests from multiple customers Azure Backup for SQL Server how supports Snapshot Backups and Restores without recovery. This allows customers to initialize or “seed” large multi-terabytes AlwaysOn deployments within and between zones in minutes as opposed to many hours for conventional Backup and Restore methods.
There is now a preview of snapshot-based backups for large SQL DBs in Azure VM. This is a key enhancement to our in-market stream-based backup solution (launched in 2019), that is used by thousands of Azure customers already. While the streaming backups scale up well for up to 4-6 TB databases, enterprises are looking for even more performant backups and restores for large DBs. Azure Backup has come up with a unique snapshot-based backup solution that combines the goodness of snapshots for better RTO and the benefits from frequent log backups for low RPO:
- Minimal impact on the source server: Takes only a few seconds to get an application consistent snapshot and there is no continuous resource sharing with the source server.
- Cost-effective: Incremental snapshots lead to low storage costs.
- Better RTO: Restores are much faster since the recovery is triggered from snapshots retained in the customer subscription itself.
- 15-min RPO: As in the case of stream-based backup, log backups continue to be streamed every 15 minutes and can be applied on top of the restore snapshot at the time of recovery.
If this interests you, we would like to invite you to try out the (private) preview. Attached is the onboarding guide and a demo video to get you started. For any queries, please write to AskAzureBackupTeam@microsoft.com.
6. Updates for SQL Server and SAP ASE Customers
The latest ODBC drivers for SQL Server support new TCP socket tuning options allowing an administrator to increase the connection timeout limits. These parameters may be useful on public cloud or VMWare environments. These parameters permit increasing the maximum blackout interval before the TCP session between the SAP application server and the Database server is reset. Slightly increasing these timeout intervals might permit VMWare vMotion to occur with no impact to long running batch jobs or user sessions.
ODBC Driver 17.8 for SQL Server Released - Microsoft Tech Community
Download ODBC Driver for SQL Server - ODBC Driver for SQL Server | Microsoft Docs
The two connection options are KeepAlive and KeepAliveInterval
These need to be set in the SAP default profile dbs/mss/conn_opts and Windows Environment MSSQL_CONNOPTS
For example add this syntax to default.pfl and Windows Environment
dbs/mss/conn_opts = KeepAlive=30;KeepAliveInterval=12;
2020535 - Profile parameter conn_opts
1248222 - ODBC DBSL profile parameters and connect options
SAP ASE database now supports Floating IP and Azure Standard Load Balancer. High Availability is now possible on Azure for non-NetWeaver applications that do not have client aware database connections.
In addition NetWeaver applications can either use a Floating IP or Client Aware parameters
More information can be found in these SAP Notes:
3086679 - SYB: Fault Manager: floating IP address on Azure
1959660 - SYB: Database Fault Management
2906375 - SYB: Database Fault Management: testing the Floating IP
7. Important Change for Customers with Standard Load Balancer & Hana or SQL Server High Availability
Over the last year several customers have observed ABAP shortdump DBSQL_SQL_ERROR or similar on their NetWeaver systems. The shortdump text would state that the database connection had been terminated after a TCPIP Reset. The message “TCP Reset 10054” would frequently be contained in the shortdump text.
The problem has been isolated to the Azure Standard Load Balancer and guidance issued on how to resolve this problem. The problem is fully mitigated by switching from explicit port rules to “HA Ports” on the Standard Load Balancer.
All SAP on Azure customers running DBMS High Availability solutions with an Azure Internal Load Balancer should check that the Standard Load Balancer SKU is in use and that HA Ports is selected.
3083711 - Azure - ST22 shows DBSQL_SQL_ERROR
Azure Customers - ABAP dump DBSQL_SQL_ERROR with error message TCP RESET 10054 on Azure ILB
Overview of Azure load-balancing options - Azure Application Architecture Guide | Microsoft Docs
8. Updates for Azure Site Recovery & Azure Backup
New features and capabilities for Azure Backup for SQL Server and Hana can be found here Azure Backup (@AzureBackup) / Twitter
This link will also contain updates regarding all Azure Backup features: Azure updates. It is recommended to cross check the Hana Backup support matrix and SQL Server support matrix regularly.
- ASR now has a Preview for high churn rate applications. Normally most SAP NetWeaver components do not require high churn but some standalone engines may need this https://aka.ms/ASRHighChurnPreview and https://aka.ms/ASRHighChurnSignup
- ASR transcontinental replication has recently been released. More information can be found here
- ASR currently does not support Azure Shared Disks. More information will be posted in this blog in 2022 about Shared Disk support
- ASR support for Oracle Linux 8.4 is being worked on
- Zone to zone DR is now supported for VMs which are in a Proximity Placement Group pinned to an Availability Set
9. DB2 on Linux on Azure NetApp Files is Now Supported
Azure NetApp Files now supports Hana, Oracle and DB2 databases accessing data and log files on NFS shares. It is recommended to follow the guidance in this link when mounting NFS for DB2 Linux NFS mount options best practices for Azure NetApp Files | Microsoft Docs
The official documentation for IBM DB2 on Azure NetApp Files is here
Azure NetApp Files fully supports Kerberos Encryption Performance impact of Kerberos on Azure NetApp Files NFSv4.1 volumes | Microsoft Docs
More information in this Podcast #53 - The one with encryption and Azure NetApp Files (Ralf Klahr) | SAP on Azure Video Podcast - YouTube
10. Network Settings & Tuning for SAP on Azure
Running SAP applications may benefit from specific network tunings when running on Azure public cloud.
A more detailed blog dealing with network settings, performance tuning, packet capture and troubleshooting will be released separately.
The summary below highlights the Top Ten Network recommendations for SAP on Azure:
- Upgrade from the Basis Azure Load Balancer to the Standard Load Balancer. Standard sku supports Accelerated Networking and other features
- Use “HA Ports” for the DBMS and ASCS Highly Available systems using Standard Load Balancer. HA Ports is the recommended configuration method in all cases
- Follow the SAP on Windows HA guide and adjust TCP/IP KeepAliveTime and KeepAliveInterval
- Follow the SAP on Linux HA guide and adjust 1410736 - TCP/IP: setting keepalive interval
- On Windows implement 1431619 - Disable NetBIOS over TCP/IP in Windows environments and 2672632 - Windows naming resolution is slow
- On SQL Server use the most recent ODBC driver – Keep Alive options can be set on ODBS 17.8 or higher
- Set net.ipv4.tcp_timestamps = 0 on Hana systems 2382421 - Optimizing the Network Configuration on HANA- and OS-Level
- When using Windows Application servers to connect to Hana DB it is recommended to use Suse 12.4 or higher or RHEL 8.x or higher 2789262 - Connection problems between Windows hosts and HANA database on Linux hosts - SAP ONE Support Launchpad
- Take great care when implementing Security tools such as Windows Defender, CrowdStrike or other Endpoint Protection agents as these agents may block network connections between SAP servers, prevent the SAPStartService from starting or may lock open Transport files or Job logs. These tools should never be deployed without involvement of the SAP Basis team and without testing in non-production systems
- On Suse Linux and especially on Scale Out scenarios on very large VMs like Mv2 the HCMT internode network topology test may fail if the deamon irqbalance is not stopped. systemctl stop irqbalance / systemctl disable irqbalance. Run this command to set the queue count to approximately 16 ethtool -L eth<X> combined 16
After initial installation and every 2-3 months run /SSA/CAT -> ABAP Meter and check the columns DB Access and E. DB Access. Cross check 2931465 - When to use Proximity Placement Groups on Azure to Reduce Network Latency – 3 Tier NetWeaver or S/4HANA architecture
If you are running SAP scenarios where you use an Azure Load Balancer, you should not use TCP time stamps on the VMs behind the load balancer: High availability of SAP HANA on Azure VMs on SLES - Azure Virtual Machines | Microsoft Docs
Network problems frequently require network packet captures. The recommended tools are tcpdump, WireShark and NetMon. Care must be taken when running these tools on busy SAP systems as the volume of network traffic on a DBMS server may be extremely high.
The following SAP Note explains how to debug KeepAlive timeouts with Packet Capture tools such as tcpdump or WireShark 2963141 - ENQUEUE: frequent work process reconnect attempts to the enqueue server
11. New Documentation for Business Objects and SAP IQ on Azure
SAP BusinessObjects and SAP IQ customers can review recently released documentation for Azure deployments
This SAP Note details which SAP BO solutions are supported on Azure 2145537 - Support of SAP BusinessObjects BI platform on Microsoft Azure
The two links below contain guidance for deployment of SAP BO on Windows and Linux
The link below is documentation for SAP IQ Near Line Storage solution for SAP BW
Implement SAP NLS/SDA with SAP IQ on Azure - Azure Virtual Machines | Microsoft Docs
The SAP IQ NLS solution for SAP BW is supported on Azure Managed Disks (Premium Storage recommended), Azure Shared Disks and Azure NetApp Files Deploy SAP IQ-NLS HA Solution using Azure NetApp Files on SUSE Linux Enterprise Server. - Microsoft Tech Community
12. Moving Microsoft’s Own Internal SAP Landscape to Azure
Microsoft has a large internal SAP deployment running on Windows/SQL Server and Suse/Hana 2.0.
The case study below summarizes the lessons learned and best practices for moving a large complex SAP landscape to Azure public cloud.
Diagram below shows Microsoft's own internal SAP landscape used to run Microsoft worldwide.
Interesting SAP OSS Notes and Links
SAP on Azure YouTube Channel https://youtube.com/SAPonAzure
How to replace an on-premises file server with Azure file shares - YouTube
We recently published a blog announcing the private preview for the Configuration and SAP Installation within our SAP Deployment Automation Framework
TR-4891: SAP HANA disaster recovery with Azure NetApp Files
What is vm.min_free_kbytes and how to tune it? (linuxhint.com)
2791572 - Performance Degradation Because of Missing VDSO Support in Azure VM
2513384 - SUSE Linux Enterprise Server for SAP Applications on Azure
2243692 - Linux on Microsoft Azure (IaaS) VM: SAP license issues
2764907 - Tomcat session failover using Azure ILB fails
3058953 - The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
2352117 - MSSQL: DBSL Trace and Profiling
3076635 - SAP installation on Azure with Netapp fails with permission denied - Netweaver
3075607 - Windows Print Spooler Remote Code Execution Vulnerability (PrintNightmare)
3071026 - List of Blogs for information about SAP Systems on Windows Server
3068557 - List of Blogs for information about SAP Systems on Microsoft Failover Cluster
2881788 - End of SAP Business Suite 7 mainstream maintenance
1928533 - SAP Applications on Azure: Supported Products and Azure VM types
2999679 - Indexserver Crash in DataRecovery::LoggerImpl::disableLogRetentionForSegments()
2969063 - Microcode Validation Failed in HCMT on Azure
In Preview: Intel Dv5/Ev5 Azure VMs based on the latest 3rd Generation Intel Xeon Platinum 8370C (Ice Lake) processors with hyper-threading enabled will be certified with both NetWeaver AnyDB and for Hana. An announcement will be made in this blog when these VMs are certified by SAP.
Azure ANF announcements What's new in Azure NetApp Files | Microsoft Docs
Azure Region Map Explorer Azure Global Infrastructure (microsoft.com)
3rd party content in this blog is used under “fair use” copyright exception for the purpose of promoting scholarship, discussion, research, learning and education
Posted at https://sl.advdat.com/31pbZkg
