![]() These machine-learning models can be consumed from Azure Cognitive Services or custom ML models from Azure ML. These ML models can be used to enrich your datasets and generate further business insights. As part of your data transformations, you can invoke machine-training models from your SQL pools using standard T-SQL or Spark notebooks.Use either data flows, SQL serverless queries, or Spark notebooks to validate, transform, and move the datasets from the Raw layer, through the Enriched layer and into your Curated layer in your data lake. You can save the data in delimited text format or compressed as Parquet files. Within the Raw data lake layer, organize your data lake following the best practices around which layers to create, what folder structures to use in each layer and what files format to use for each analytics scenario.įrom the Azure Synapse pipeline, use a Copy data activity to stage the data copied from the relational databases into the raw layer of your Azure Data Lake Store Gen 2 data lake. Pipelines can be triggered based on a pre-defined schedule, in response to an event, or can be explicitly called via REST APIs. Use Azure Synapse pipelines to pull data from a wide variety of databases, both on-premises and in the cloud.Structured and unstructured data stored in your Synapse workspace can also be used to build knowledge mining solutions and use AI to uncover valuable business insights across different document types and formats including from Office documents, PDFs, images, audio, forms, and web pages. Data consumers have the freedom to choose what data format they want to use and also what compute engine is best to process the shared datasets. Business analysts use Power BI reports and dashboards to analyze data and derive business insights.ĭata can also be securely shared to other business units or external trusted partners using Azure Data Share. Power BI models implement a semantic model to simplify the analysis of business data and relationships. Load relevant data from the Azure Synapse SQL pool or data lake into Power BI datasets for data visualization and exploration. If you are using Spark notebooks, the resulting datasets can be persisted either in your data lake or data warehouse (SQL pool). The resulting datasets from your SQL Serverless queries can be persisted in your data lake.You can access the selected Dataverse tables and then combine datasets from your near real-time business applications data with data from your data lake or from your data warehouse. When using Azure Synapse Link for Dataverse, use either a SQL Serverless query or a Spark Pool notebook. You can access the Azure Cosmos DB analytical store and then combine datasets from your near real-time operational data with data from your data lake or from your data warehouse. When using Azure Synapse Link for Azure Cosmos DB, use either a SQL Serverless query or a Spark Pool notebook. Azure data services, cloud native HTAP with Azure Cosmos DB and Dataverse ProcessĪzure Synapse Link for Azure Cosmos DB and Azure Synapse Link for Dataverse enable you to run near real-time analytics over operational and business application data, by using the analytics engines that are available from your Azure Synapse workspace: SQL Serverless and Spark Pools. Please refer to the Data lake zones and containers documentation for a full review of Azure Data Lake layers and containers and their uses. Data then moves to the Curated layer, which is where consumer-ready data is maintained. ![]() In the next stage of the lifecycle, data moves to the Enriched layer where data is cleaned, filtered, and possibly transformed.As the name implies, data in this layer is in raw, unfiltered, and unpurified form. The Raw layer is the landing area for data coming in from source systems.Azure Data Lake is organized by different layers and containers as follows: ![]() In the following sections, Azure Data Lake is used as the home for data throughout the various stages of the data lifecycle.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |