LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. To learn more about LightIngest, see Use LightIngest to ingest data into Azure Data Explorer.
LightIngest can be particularly useful to load historical data from an existing storage system to Azure Data Explorer. While you can build your own command using the list of Command-line arguments, this article shows you how to auto-generate this command through an ingestion wizard. In addition to creating the command, you can use this process to create a new table, and create schema mapping. This tool infers schema mapping from your data set.
This article shows you how to create a new table, create schema mapping, and generate a LightIngest command for one-time ingestion using the LightIngest tool.
- Open Web Explorer
- Select ingest new data or Ingest new data from container at Data Page
- Select Cluster, database and table (new or existing)
- Source tab - for more than 5,000 blobs select Historical data
- Select a source or enter SAS URL to source
- Select "Advanced parameters" to support:
- Creation Time
- blob name pattern
- Tag
- limited amount of files
- Don't wait for the ingestion to be completed (recommended for big amount of data )
- Review and adjust the auto-suggested schema structure
-
In the Data ingestion completed window, all three steps will be marked with green check marks.
Copy the generated LightIngest command by clicking on the copy icon to the top right of the command box.
- To complete the ingestion process, you must run LightIngest using this copied command.
read more on generate lightingest command
