Wednesday, September 29, 2021

Querying WHOIS/Registration Data Access Protocol (RDAP) with Azure Sentinel and Azure Functions

With the amazing increase in domains and top-level domains (TLD's) on the Internet, it's difficult to know just where our users are going. Newly registered domains, domain generation algorithms, and typo-squatting are all tactics used by adversaries to compromise users. Recently I was talking with a customer about Azure Sentinel and they had a question about if and how they could raise an alert when a user received an email from a newly registered domain (by their definition this was any domain that had been registered in the last thirty days).  While we don't have a built-in feature for this in Sentinel, it is possible to extend Sentinel to include this type of functionality. This blog post is about one way that such an extension could be created. 

 

Domain registration history

 

First off, we need to understand domain registrations in general. To be usable on the Internet, all domain names must be registered so they can be propagated throughout the global DNS world. This information is created with a domain registrar or their resellers who are accredited by the Internet Corporation for Assigned Names and Numbers (ICANN), a not-for-profit public-benefit corporation who defines the policies and rules around domain registration. A simplified process flow is that when a domain is requested, the registrar will check if the domain name is available for registration and if so, will then create a "WHOIS" record with the domain name registrant's information.  WHOIS is a protocol defined by the Internet Engineering Taskforce (IETF) in RFC3912. WHOIS is a TCP based connection using port 43 and responds in a human readable format. Each registrar maintains their WHOIS infrastructure, and you often must know just which registrar is authoritative for a particular domain.  To keep the telephone book analogy going it's kind of like knowing that a person named "Matt Egen" exists somewhere in a telephone book in the world, but without knowing exactly where he is, you'd have to check all the telephone books around the world to find him. While this is fine for looking up occasional data (and back in 1985 there were a lot fewer domains (as well as much fewer Top-Level Domains (TLD) like .com, .net, .org, etc.)), it’s rather difficult to automate the process as the data isn't designed to be read by a machine.  To counter this and account for not only the growing domain count but the invention of new "Generic Top Level Domains" (gTLD) like .store, .app, etc, ICANN and the Internet Engineering Task Force (IETF) came up with a new protocol called the Registration Data Access Protocol (RDAP).  RDAP is a REST API with the same information as the traditional WHOS service, except its data is returned in a standardized JSON format. This makes it rather straightforward in parsing the returned data (although it still maintains a problem in finding the correct RDAP server / source to begin with, but we can deal with that) and automating the process.

 

Extending Azure Sentinel with Azure Functions

 

Azure Sentinel offers us several tools we can use to automate tasks.  One method is to use Playbooks which are based on Azure Logic Apps and these provide an outstanding solution for creating a visual flow in your automation process.  We could have used one here (in fact, in the v1 of this solution I did exactly that), however, there is another method we can use as well:  Azure Functions. Azure Functions is a cloud service available on-demand that provides all the continually updated infrastructure and resources needed to run your applications. You focus on the pieces of code that matter most to you, and Functions handles the rest. Functions provides serverless compute for Azure. You can use Functions to build web APIs, respond to database changes, process IoT streams, manage message queues, and more.  In this case, we’re going to use an Azure Function with a timer trigger to handle our RDAP query and run it on a regular schedule.

 

Architecture and process flow

 

The example follows a straightforward flow:

 

  1. On a regular schedule the Azure Function will trigger.
  2. The Function will query the Azure Sentinel instance and call a Function which will get new domain names.  We’re using a Function in Sentinel because we want it to be flexible and maintainable outside of the Azure Function.  This means we can modify it whenever we have / want a new source of domain names.  In this example we’re using the domains returned from the DeviceEvents table which comes from the Microsoft Defender 365 data connector.  However, by using a Function it could be from any or multiple source(s) so long as it’s returned in the expected format.  By using a Function and leveraging joins, we can handle all the sources we want.
  3. Using the returned domains, we will then call the RDAP “Bootstrap” service.  This is a special list of all of the gTLDs and their requisite RDAP server endpoints.  Think of this as a phone book of phone books.
  4. After getting the correct RDAP server, we then query it and get the results in JSON notation.
  5. We take the relevant information (in this example this is just the registration date) and then call back into our Azure Sentinel instance to store the data in a custom log table.
  6. Finally there is an Analytic Rule which runs against this custom log and if it finds a domain that is younger than the set criteria, it raises an alert.

 

Required Information for this example

 

Since we’re going to be accessing data in an Azure Sentinel instance we need some information to enable that.

 

Creating the Sentinel Function

 

To retrieve the domain names that we want to resolve, we’re using a Sentinel Function. While it has a similar name to an Azure Function, it’s different. A Sentinel Function is written in the Kusto Query Language (KQL) and is a query that you save and then call later using an alias. You can use a Function in place of a table and use it like any other table. I’m not going to go too deep into the use cases or creation of Functions, but for our case we’re using it as a convenient tool so that we can maintain the query outside of our Azure Function and tune / adjust it as needed for different environments. If you like to learn more about Sentinel Functions, you can read about them here:  Functions in Azure Monitor log queries - Azure Monitor | Microsoft Docs.  For this use case, we’re going to show a straightforward example that calls into the DeviceNetworkEvents table, cleans up some of the data, checks to make sure it’s not in an exclusion list and finally that it’s not been already looked up in the last 90 days.

 

// Function Name: GetDomainsForRDAP
// ExcludedDomains is a dynamic list of domains and TLDs to not bother searching for
// either because we already trust them, or perhaps we know they don’t have an RDAP server implementation.
let ExcludedDomains = dynamic(["cn","io", "ms", "microsoft.com","somerandomsender.com"]);
// Query the DeviceNetworkEvents table for the last 1 hour
DeviceNetworkEvents
| where TimeGenerated >= ago(1h)
| where isnotempty(RemoteUrl) //only return records that have a RemoteUrl value
// A little cleanup just in case
| extend parsedDomain = case(RemoteUrl contains "//", parse_url(RemoteUrl).Host, RemoteUrl) // handle scenarios where the RemoteUrl includes protocol data (e.g. http/s, etc.)
| extend cleanDomain = split(parsedDomain,"/")[0] // throw away anything after the last “/” character
| extend splitDomain = split(cleanDomain,".") //split the domain name on the “.”
| extend Domain = tolower(strcat(splitDomain[array_length(splitDomain)-2],".",splitDomain[array_length(splitDomain)-1])) // recombine just the last two parts of the domain (the TLD and gTLD)
| extend TLD = splitDomain[array_length(splitDomain)-1]  // grab the gTLD so we can see if it’s in exclusion list along with the domain
| where TLD !in(ExcludedDomains)
| where Domain !in(ExcludedDomains)
| summarize DistinctDomain = dcount(Domain) by Domain  //De-duplicate the list
| project Domain // return just the domain
// Now join the results above to our table of already resolved domains.  We don’t want to waste cycles querying for things we already know about
//| join kind=leftanti (ResolvedDomains_CL
//| where TimeGenerated >= ago(90d)) on $left.Domain == $right.domainName_s //Uncomment these lines after the FIRST run of the Azure Function.

 

One thing you may notice in the above Sentinel Function: the last two lines are commented out.  This is because until the Azure Function runs the first time, the “ResolvedDomains_CL” custom log table doesn’t exist and this Sentinel Function will fail.  After successfully running the Azure Function one time, you should then uncomment the last two lines. 

 

The Azure Function

 

Now that we have the Sentinel Function out of the way, let’s talk about the Azure Function. As noted before, while having a similar name to a Sentinel Function, Azure Functions are completely different.  Azure Functions is a cloud service available on-demand that provides all the continually updated infrastructure and resources needed to run your applications. You focus on the pieces of code that matter most to you, and Functions handles the rest. Azure Functions can be written in an array of different stacks including .NET, Node.Js, Python, Java, and even PowerShell Core and can be hosted on either Windows or Linux infrastructure. In this case we’re using .NET as the stack, the language is C#, and the infrastructure is Windows.

 

To read data from Azure Sentinel, we need to create Azure AD application credentials with permission to the Azure Sentinel instance

 

Creating an Azure AD Application with read permissions to Azure Sentinel

 

This blog post is already getting a little long so rather than rewrite the steps to create an Azure AD Application, I’m just going to provide a link to my peer Rin Ure’s great blog post on the API’s and creating credentials: Access Azure Sentinel Log Analytics via API (Part 1) - Microsoft Tech Community  The specific permission that we want to make sure we gran to the applications is covered in the linked article under the “Give the AAD Application permissions to your (Sentinel) Log Analytics Workspace” section.  After following the steps in that article, we will have two of the settings that we will need for the RDAP Query engine:  the Client ID and the Client Secret.  We will be using these later when we configure the Azure Function.

 

Ok, so now we have what we need to read the data, but what about writing back our results? To write data to the Azure Sentinel instance, we need the Workspace ID and either the Primary or Secondary key for the workspace.

 

Getting the Workspace ID and Key for Azure Sentinel

 

Azure Sentinel uses Log Analytics as the underlying data store.  To write data to the Log Analytics workspace, we need the workspace ID and Key and can access these very simply in Azure Sentinel. In Sentinel, go to Settings...

 

 

MattEgen_0-1632321618709.png

 

Then, select “Workspace Settings” from the top of the resulting page...

 

MattEgen_1-1632321618715.png

 

And finally, select “Agents Management”

MattEgen_2-1632321618719.png

 

This will take you to a screen that will show you the Workspace ID and two keys, a primary and secondary, that can be used to send data to the workspace...

 

MattEgen_3-1632321618721.png

 

We can use either the primary or the secondary, it doesn’t matter which one we choose just remember to copy and save the Workspace ID and one of the keys as we’re going to need them in the next section.

 

Now that we have the values we need to access Azure Sentinel we are going use then in our Azure Function by storing them in Application Settings.

 

Azure Function – Application Settings

 

As an Azure Function, the example can be configured via . Application settings are encrypted at rest and transmitted over an encrypted channel. Application Settings are exposed as environment variables for access by the application at runtime. This allows us to store keys and values in the Azure Function without having to store them in code. There are two advantages to this: 1) we’re not storing secrets in the actual code itself and 2) we can change them if we need to later.

 

For this Function we use the following Application Settings:

 

"SharedKey": "[LogAnalytics WorkSpace Primary or Secondary Key]",
"WorkspaceID": "[LogAnalytics Workspace ID]",
"LogName": "[The name of the custom log to store results. Recommend:ResolvedDomains]",
"tenant_id": "[AzureAD TenantID]",
"client_id": "[AzureAD Application Client ID]",
"client_secret": "[AzureAD Application Client Secret]",
"grant_type": "[Grant Type for Bearer Token]",
"resource": "[Resource URL for Bearer Token]",
“query_string”,”[The Sentinel function name to call]”

 

These values are used in the C# code that does the actual work in the Azure Function and will be populated from the Application Settings into the code at runtime. This makes it very easy for us to change settings without having to rewrite / redeploy code.  After deploying the application to an Azure Function, we configure the Application Settings on the Configuration blade...

 

MattEgen_4-1632321618734.png

 

 

For each of the settings, simply click the “New Application Setting” button and enter the name and value of the setting...

 

MattEgen_5-1632321618737.png

 

 

Now let’s look at the code and how we can deploy it to an Azure Function.

 

RDAP Query Engine C# Code

 

I’m not going to go through every line in the code, but instead leave that as an exercise for the reader.  Keep in mind this is an example and as such could probably be improved.   The code is hosted on GitHub here: Azure-Sentinel/Tools/RDAP/RDAPQuery at master · Azure/Azure-Sentinel (github.com). However, I’m going to cover the main function and the overall structure.

 

CheckDomains

MattEgen_6-1632321618745.png

 

 

CheckDomains is the initialization point for the code and is called by the Azure Function framework based on a Timer trigger (in fact CheckDomains is an alias for the interal “Run” function that Azure Functions is calling). The timer is set to fire at a set period of time (default every five minutes) and in turn makes all of the other calls to get authentication, retrieve data from Sentinel, query RDAP, and finally write the results back to Sentinel. Let’s map out this function...

[create a visio of the function?]

 

 “QueryData”

MattEgen_7-1632321618756.png

 

 

One of the first things that CheckDomains does is call the QueryData function which is responsible for calling into Azure Sentinel and retrieving any new domains to lookup. It takes one parameter which is the actual query to execute. Since querying the data in Azure Sentinel requires us to authenticate, it first retrieves an OAUTH bearer token (via a call to “GetBearerToken()” and then uses it in the subsequent call to the Log Analytics API.  If we receive a successful status code from the call, we then deserialize the results into a QueryResults object and return it back to the CheckDomains function.

 

After calling QueryData, we run a ForEach() loop over the results, do a little cleanup (by splitting out the top-level domain (.com, .net, etc.) with a split() function) and then call BootStrapTLD.

  

BootStrapTLD()

MattEgen_8-1632321618765.png

 

 

BootStrapTLD takes the passed in top-level domain (TLD) and uses it to call the IANA bootstrap URL at https://data.iana.org/rdap/dns.json (this is hardcoded as it’s supposed to never change) which is a JSON file that has all the different TLDs mapped out with the appropriate RDAP server URL that we can call to get the detailed information about the domain.  We make a call to the JSON file and then deserialize it into a Services object. Since we deserialized the entire list, we then run a foreach loop over the returned values to check for a match with the TLD we’re searching for. When we have a match we then break out of the ForEach and return the value back to CheckDomains()

Now that we have the RDAP server that is responsible for the TLD, it’s time to call it and get the information we want.

 

QueryRDAP

MattEgen_9-1632321618772.png

 

 

This function is very straightforward as it simply calls the passed in URL, deserializes the results (if any) into an RDAPResponse object and then returns that back to CheckDomains.  CheckDomains then parses the returned object to find the “events” node and specifically one with an eventType of “registration”. If we find one, we create a new “RDAPUpdate” object which just holds the domain name we looked up along with the registration date that was returned. This object is then passed to the WriteData function which will store it into Sentinel / Log Analytics.

 

WriteData()

MattEgen_10-1632321618775.png

 

 

WriteData is possibly the simplest function in all of this code as it just takes the passed object, converts it into a JSON string, builds a signature (using the Workspace ID and keys from earlier) and then calls PostData which does the actual write to Sentinel / Log Analytics.

 

PostData()

MattEgen_11-1632321618780.png

 

And finally, PostData() calls the Log Analytics API and commits our data. 

 

So…I’ve got a bunch of domain names and their registration dates, now what?

 

Going back to our original need (alert on domains that are younger than 30 days), we can write a very in the Logs blade of Sentinel to search for these:

 

ResolvedDomains_CL
| where TimeGenerated >= ago(1h)
| where registrationDate_t >= ago(30d)

And to automate this query, we could convert it into an Analytics rule to generate an Incident for an analyst to review by selecting the “New Alert” drop down and choosing “Create Azure Sentinel Alert”

 

MattEgen_12-1632321618785.png

 

Another use case could be to create an enrichment query to add registration data during an investigation.  For example, create a join() between a domain source table and the ResolvedDomains_CL table to add in the registration date for any domains seen, and then add that data to an analytic using the new Custom Details feature.

 

Next steps / further improvements

 

One thing I noticed in creating this example was that not every top-level domain has activated an RDAP server yet. Notably, a number of country TLDs are still using the traditional WHOIS infrastructure (this is why I added the ability to exclude domains and TLD’s in the GetDomainsForRDAP Sentinel Function).  As a next step for this project, I am going to look to add traditional WHOS queries (via a TCP connection to port 43) in cases where RDAP cannot find a domain / receives an error. Also, the code currently doesn’t handle raw IP addresses (either IPv4 or IPv6) and instead just does a lookup and fails. I’m looking at modifying the code to support RDAP queries for IP addresses as well, but since it’s an IP address it doesn’t have a “registration date” per se. Would love some feedback on what you think would be useful information from an IP address.  Look for this update soon.

 

Until next time, happy hunting!

 

 

 

 

 

 

 

 

 

Posted at https://sl.advdat.com/3oknsek