What is Big Data?<\/h2>\n\n- Definition: A term for data sets that are so large or complex<\/u><\/em><\/strong> that traditional data processing application s\/w if inadequate to deal with it. Nishant stresses \u201ccomplex\u201d.<\/li>\n
- Challenges: Capturing (velocity) data, data storage, data analysis, search, sharing, visualization, querying, updating, and information privacy. So \u2026 there\u2019s a few challenges
![\"Smile\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/wlEmoticon-smile.png\")
<\/li>\n<\/ul>\nThe Azure Data Landscape<\/h2>\n
This slide is referred to for quite a while:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb.png\" "\\"image\\"")
<\/a><\/p>\nData Ingestion<\/h2>\n
He starts with the left-top corner:<\/p>\n
\n- \nAzure Data Factory<\/div>\n<\/li>\n
- \nAzure Import\/Export Service<\/div>\n<\/li>\n
- \nAzure CLI<\/div>\n<\/li>\n
- \nAzure SDK<\/div>\n<\/li>\n<\/ul>\n
The first problem we have is data ingestion into the cloud or any system. How do you manage that? Azure can manage ingestion of data.<\/p>\n
Azure Data Factory is a scheduling, orchestration, and ingestion service. It allows us to create sophisticated data pipelines from the ingestion of the data through to processing, through to storing, through to making it available to end users to access. It does not have compute power of it\u2019s own; it taps into other Azure services to deliver any required compute.<\/p>\n
The Azure Import\/Export service can help bring incremental data on board. You can also use it to bulk load on Azure. If you have terabytes of data to upload, bandwidth might not be enough. You can securely courier data via disk to an Azure region.<\/p>\n
The Azure CLI is designed for bulk uploads to happen in parallel. The SDKs can be put into your code, so you can generate the data in your application in the cloud, instead of uploading to the cloud.<\/p>\n
Operational Database Services<\/h2>\n\n- Azure SQL DB<\/li>\n
- Azure Cosmos DB<\/li>\n<\/ul>\n
The SQL database offers SQL Server, MySQL, and PostgreSQL.<\/p>\n
Cosmos DB is the more interesting one \u2013 it\u2019s NoSQL and offers global storage. It also supports 4 programming models: Mongo, Gremlin\/Graph, SQL (DocumentDB), and Table. You have flexibility to bring in data in its native form, and data can be accessed in an operational environment. Cosmos DB has plugs into other aspects of Azure that make it more than just an operational database such as, Azure Functions or Spark (HDInsight).<\/p>\n
Analytical Data Warehouse<\/h2>\n\n- Azure SQL Data Warehouse<\/li>\n<\/ul>\n
When you want to do reporting and dashboards from data in operational databases then you will need an analytical data warehouse that aggregates data from many sources.<\/p>\n
Traits of Azure SQL Data Warehouse:<\/p>\n
\n- Can grow, shrink, and pause in seconds \u2013 up to 1 Petabyte<\/li>\n
- Fill enterprise-class SQL Server \u2013 means you can migrate databases and bring your scripts with you. Independent scale of compute and storage in seconds<\/li>\n
- Seamless integration with Power BI, Azure Machine Learning, HDInsight, and Azure Data Factory<\/li>\n<\/ul>\n
NoSQL Data<\/h2>\n\n- Azure Blob storage<\/li>\n
- Azure Data Lake Store<\/li>\n<\/ul>\n
When your data doesn\u2019t fit into the rows and columns structure of a traditional database then this is when you need specialized big data storages \u2013 capacity, unstructured sorting\/reading.<\/p>\n
Unstructured Data Compute Engines<\/h2>\n\n- Azure Data Lake Analytics<\/li>\n
- Azure HDInsight (Spark \/ Hadoop): managed clusters of Hadoop and Spark with enterprise-level SLAs with lower TCO than on-premises deployment.<\/li>\n<\/ul>\n
When you get data into a big unstructured stores such as Blob or Data Lake then you need specialized compute engines for the complexity and volume of the data. This compute must be capable of scaling out because you cannot wait hours\/days\/months to analyse the data.<\/p>\n
Ingest Streaming Data<\/h2>\n\n- Azure IoT Hub<\/li>\n
- Azure Event Hubs<\/li>\n
- Kafka on Azure HDInsight<\/li>\n<\/ul>\n
How do you ingest this real-time data as it is generated? You can tap into event generators (e.g. devices) and buffer up data for your processing engines.<\/p>\n
Stream Processing Engines<\/h2>\n\n- Azure Stream Analytics<\/li>\n
- Storm and Spark streaming on Azure HDInsight<\/li>\n<\/ul>\n
These systems allow you to process streaming data on the fly. You have a choice of \u201ceasy\u201d or \u201copen source extensibility\u201d with either of these solutions.<\/p>\n
Reporting and Modelling<\/h2>\n\n- Azure Analysis Services<\/li>\n
- Power BI<\/li>\n<\/ul>\n
You have cleansed and curated the data, but what do you do with it? Now you want some insights from it. Reporting & modelling is the first level of these insights.<\/p>\n
Advanced Analytics<\/h2>\n\n- Azure Machine Learning<\/li>\n
- ML Server (R)<\/li>\n<\/ul>\n
The basics of reporting and modelling are not new. Now we are getting into advanced analytics. Using data from these AI systems we can predict outcomes or prescribe recommended actions.<\/p>\n
Deep Learning<\/h2>\n\n- \nCognitive Services<\/div>\n<\/li>\n
- \nBot Service<\/div>\n<\/li>\n<\/ul>\n
Taking advanced analytics to a further level by using these toolkits.<\/p>\n
Tracking Data<\/h2>\n\n- \nAzure Search<\/div>\n<\/li>\n
- \nAzure Data Catalog<\/div>\n<\/li>\n<\/ul>\n
When you have such a large data estate you need ways to track what you have, and to be able to search it.<\/p>\n
The Azure Platform<\/h2>\n\n- \nExpressRoute<\/div>\n<\/li>\n
- \nAzure AD<\/div>\n<\/li>\n
- \nNetwork Security Groups<\/div>\n<\/li>\n
- \nAzure Key Management Service<\/div>\n<\/li>\n
- \nOperations Management Suite<\/div>\n<\/li>\n
- \nAzure Functions (serverless compute)<\/div>\n<\/li>\n
- \nVisual Studio<\/div>\n<\/li>\n<\/ul>\n
You need a platform with enterprise capabilities in the best ways possible in a compliant manner.<\/p>\n
Big Data Services<\/h2>\n
Nishant says that that darker shaded services are the ones usually being talked about when they talk about Big Data:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-1.png\" "\\"image\\"")
<\/a><\/p>\nTo understand what all these services are doing as a whole, and why Microsoft has gotten into Big Data, we have to step all the way back. There are 3 high-level trends that are a kind of an industrial revolution, making data a commodity:<\/p>\n
\n- \nCloud<\/div>\n<\/li>\n
- \nData<\/div>\n<\/li>\n
- \nAI<\/div>\n<\/li>\n<\/ul>\n
We are on the cusp of an era where every action produces data.<\/p>\n
The Modern Data Estate<\/h2>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-2.png\" "\\"image\\"")
<\/a><\/p>\nThere are 2 principles:<\/p>\n
\n- \nData on-premises and<\/em><\/div>\n<\/li>\n
- \nData in the cloud<\/div>\n<\/li>\n<\/ul>\n
Few organizations are just 1 or the other; most span both locations. Data warehouses aggregate operational databases. Data Lakes store the data used for AI, and will be used to answer the questions that we don\u2019t even know of today.<\/p>\n
We need three capabilities for this AI functionality:<\/p>\n
\n- \nThe ability to reason over this data from anywhere<\/div>\n<\/li>\n
- \nYou have the flexibility to choose \u2013 MS (simplicity & ease of use), open source (wider choice), programming models, etc.<\/div>\n<\/li>\n
- \nSecurity & privacy, e.g. GDPR<\/div>\n<\/li>\n<\/ul>\n
Microsoft has offerings for both on-premises and in Azure, spanning MS code and open source, with AI built-in as a feature.<\/p>\n
Evolution of the Data Warehouse<\/h2>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-3.png\" "\\"image\\"")
<\/a><\/p>\nThere are 3 core scenarios that use Big Data:<\/p>\n
\n- \nModern DW: Modernizing the old concept of a DW to consume data from lots of sources, including complexity (big data)<\/div>\n<\/li>\n
- \nAdvanced Analytics: Make predictions from data using Deep Learning (AI)<\/div>\n<\/li>\n
- \nIoT: Get real time insights from data produced by devices<\/div>\n<\/li>\n<\/ul>\n
Implementing Big Data & Data Warehousing in Azure<\/h2>\n
Here is a traditional DW<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-4.png\" "\\"image\\"")
<\/a><\/p>\nData from operational databases are fed into a single DW. Some analysis is done and information is reported\/visualized for users.<\/p>\n
SQL Server Integration services, a part of the Azure Data Factory, can allow you to consume data from your multiple operational assets and aggregate them as a DW.<\/p>\n
Azure Analysis Services allows you yo build tabular models for your BI needs, and Power BI can be used to report and visualize those models.<\/p>\n
If you have existing huge repositories of data that you want to bring into a DW then you can use:<\/p>\n
\n- Azure CLI<\/li>\n
- Azure Data Factory<\/li>\n
- BCP Command Line Utility<\/li>\n
- SQL Server Integration Services<\/li>\n<\/ul>\n
This traditional model breaks when some of your data is unstructured. For example:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-5.png\" "\\"image\\"")
<\/a><\/p>\nStructured operational data is coming in from Azure SQL DB as before.<\/p>\n
Log files and media files are coming into blob storage as unstructured data – the structure of queries is unknown and the capacity is enormous.\u00a0 That unstructured data breaks your old system but you still need to ingest it because you know that there are insights in it.<\/p>\n
Today, you might only know some questions that you\u2019d like to ask of the unstructured data. But later on, you might have more queries that you\u2019d like to create. The vast scale of economy of Azure storage makes this feasible.<\/p>\n
ExpressRoute will be used to ingest data from an enterprise if:<\/p>\n
\n- \nYou have security\/compliance concerns<\/div>\n<\/li>\n
- \nThere is simply too much data for normal Internet connections<\/div>\n<\/li>\n<\/ul>\n
Back to the previous unstructured data scenario. If you are curating the data so it is filtered\/clean\/useful, then you can use Polybase to ingest it into the DW. Normally, that task of cleaning\/filtering\/curating is too huge for you to do on the fly.<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-6.png\" "\\"image\\"")
<\/a><\/p>\nHDInsight can tap into the unstructured blob storage to clean\/curate\/process it before it is ingested into the DW.<\/p>\n
What does HDInsight allow you to do? You forget that the data was structured\/unstructured\/semi-structured. You forget the complexity of the analytical queries that you want to write. You forget the kinds of questions you would like to ask of the data. HDInsight allows you to add structure to the data using some of it\u2019s tools. Once the data is structured, you can import it into the DW using Polybase.<\/p>\n
Another option is to use Azure Functions instead of HDInsight:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-7.png\" "\\"image\\"")
<\/a><\/p>\nThis serverless option can suit if the required manipulation of the unstructured data is very simple. This cannot be sophisticated \u2013 why re-invent the wheel of HDInsight?<\/p>\n
Back to HDInsight:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-8.png\" "\\"image\\"")
<\/a><\/p>\nAnalytical dashboards can tap into some of the compute engines directly, e.g. tap into raw data to identify a trend or do ad-hoc analytics using queries\/dashboards.<\/p>\n
Facilitating Advanced Analytics<\/h2>\n
So you\u2019ve got a modern DW that aggregates structured and unstructured data. You can write queries to look for information \u2013 but we want deeper insights.<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-9.png\" "\\"image\\"")
<\/a><\/p>\nThe compute engines (HDIsnight) enable you to use advanced analytics. Machine Learning can only be as good as the quality and quantity of data that you provide to it \u2013 the compute engine\u2019s job. The more data machine learning has to learn from, the more accurate the analysis will be. If the data is clean, then garbage results won\u2019t be produced. To do this with TBs or PBs of data, you will need the scale-out compute engine (HDInsight) \u2013 a VM just cannot do this.<\/p>\n
Some organizations are so large or so specialized that they need even better engines to work with:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-10.png\" "\\"image\\"")
<\/a><\/p>\nAzure Data Lake store replaces blob storage for greater scales. Azure Data Lake Analytics replaces HDInsight offers a developer-friendly T-SQL-like & C# environment. You can also write Python R models. Azure Data Lake Analytics is serverless \u2013 there are no clusters as there are in HDInsight. You can focus on your service instead of being distracted by monitoring.<\/p>\n
Note that HDInsights works with interactive queries against streaming data. Azure Data Lake is based on batch jobs.<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-11.png\" "\\"image\\"")
<\/a><\/p>\nYou have the flexibility of choice for your big data compute engines:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-12.png\" "\\"image\\"")
<\/a><\/p>\nReturning to the HDInsight scenario:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-13.png\" "\\"image\\"")
<\/a><\/p>\nHDInsight, via Spark, can integrate with Cosmos DB. Data can be stored in Cosmos DB for users to consume. Also, data that users are generating and storing in Cosmos DB can be consumed by HDInsight for processing by advanced analytics, with learnings being stored back in Cosmos DB.<\/p>\n
Demo<\/h2>\n
He opens an app on an iPhone. It\u2019s a shoe sales app. The service is (in theory) using social media, fashion trends, weather, customer location, and more to make a prediction about what shoes the customer wants. Those shoes are presented to the customer, with the hope that this will simplify the shopping experience and lead to a sale on this app. When you pick a shoe style, the app predicts your favourite colour. If you view a shoe, but don\u2019t buy it., the app can automatically entice you with promotional offers \u2013 stock levels can be queried to see what kind of promotion is suitable \u2013 e.g. try shift less popular stock by giving you a discount to do an in-store pickup where stock levels are too high and it would cost the company money to ship stock back to the warehouse. The customer might also be tempted to buy some more stuff when in the shop.<\/p>\n
He then switches to the dashboard that the marketing manager of the shoe sales company would use. There\u2019s lots of data visualization from the Modern DW, combining structured and unstructured data \u2013 the latter can come from social media sentiment, geo locations, etc. This sentiment can be tied to product category sales\/profits. Machine learning can use the data to recommend promotional campaigns. In this demo, choosing one of these campaigns triggers a workflow in Dynamics to launch the campaign.<\/p>\n
Here\u2019s the solution architecture:<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-14.png\" "\\"image\\"")
<\/a><\/p>\nThere are 3 data sources:<\/p>\n
\n- \nUnstructured data from monitoring the social and app environments \u2013 Azure Data Factory<\/div>\n<\/li>\n
- \nStructured data from CRM (I think) \u2013 Azure Data Factory<\/div>\n<\/li>\n
- \nProduct & customer profile data from Cosmos DB (Service Fabric in front of it servicing the mobile apps).<\/div>\n<\/li>\n<\/ul>\n
HDInsight is consuming that data and applying machine learning using R Server. Data is being written back out to:<\/p>\n
\n- \nBlob storage<\/div>\n<\/li>\n
- \nCosmos DB \u2013 Spark integration<\/div>\n<\/li>\n<\/ul>\n
The DW consumes two data sources:<\/p>\n
\n- \nThe data produced by HDInsight from blob storage<\/div>\n<\/li>\n
- \nThe transactional data from the sales transactions (Azure SQL DB)<\/div>\n<\/li>\n<\/ul>\n
Azure Analysis Services then provides the ability to consume the information in the DW for the Marketing Manager.<\/p>\n
Enabling Real-Time Processing<\/h2>\n
This is when we start getting in IoT data, e.g. sensors \u2013 another source of unstructured data that can come in big and fast. We need to capture the data, analyse it, derive insights, and potentially do machine learning analysis to take actions on those insights.<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-15.png\" "\\"image\\"")
<\/a><\/p>\nEvent hubs can ingest this data and forward it to HDIngsights \u2013 stream analysis can be done using Spark Streaming or Storm. Data can be analysed by Machine Learning and reported in real-time to users.<\/p>\n
So the IoT data is:<\/p>\n
\n- \nFed into HDInsights for structuring<\/div>\n<\/li>\n
- \nFed into Machine Learning for live reporting<\/div>\n<\/li>\n
- \nStored in Blob Storage.<\/div>\n<\/li>\n
- \nConsumed by the DW using Polybase for BI<\/div>\n<\/li>\n<\/ul>\n
There are alternatives to this IOT design.<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-16.png\" "\\"image\\"")
<\/a><\/p>\nYou should use Azure IoT Hub if you want:<\/p>\n
\n- \nDevice registration policies<\/div>\n<\/li>\n
- \nMetadata about your devices to be stored<\/div>\n<\/li>\n<\/ul>\n
If you have some custom operations to perform, Azure HDInsight (Kafka) can scale up from millions of events per second. It can apply some custom logic that cannot be done by Event Hub or IoT Hub.<\/p>\n
We also have flexibility of choice when it comes to processing.<\/p>\n
![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image_thumb-17.png\" "\\"image\\"")
<\/a><\/p>\nAzure Stream Analytics gives you ease-of-use versus HDInsight. Instead of monitoring the health & performance of compute clusters, you can use Stream Analytics.<\/p>\n
The Azure Platform<\/h2>\n
The platform of Azure wraps this package up:<\/p>\n
\n- \nExpressRoute: Private SLA networking<\/div>\n<\/li>\n
- \nAzure Data Factory: Orchestration of the data processing, not just ingestion.<\/div>\n<\/li>\n
- \nAzure Key Vault: Securely storing secrets<\/div>\n<\/li>\n
- \nOperations Management Suite: Monitoring & alerting<\/div>\n<\/li>\n<\/ul>\n
And now that your mind is warped, I\u2019ll leave it there I thought it was an excellent overview session.<\/p>\n","protected":false},"excerpt":{"rendered":"
These are my notes from the recording of this Ignite 2017 session, BRK2293. Speaker: Nishant Thacker, Technical Product Manager \u2013 Big Data This Level-200 overview session is a tour of big data in Azure, it explains why the services were created, and what is their purpose. It is a foundation for the rest of the … Continue reading “Understanding Big Data on Azure–Structured, Unstructured, and Streaming”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":20724,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[14],"tags":[170,224,218,222,176,177,219,203,201,223,221,220],"class_list":["post-20722","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-eventnotes","tag-azure","tag-data-warehouse","tag-dig-data","tag-event-hub","tag-eventnotes","tag-events","tag-hdinsight","tag-ignite","tag-iot","tag-iot-hub","tag-machine-learning","tag-stream-analytics"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/Azure-HDInsight.png","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/20722","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=20722"}],"version-history":[{"count":2,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/20722\/revisions"}],"predecessor-version":[{"id":20725,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/20722\/revisions\/20725"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/media\/20724"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=20722"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=20722"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=20722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
<\/li>\n<\/ul>\nThe Azure Data Landscape<\/h2>\n
This slide is referred to for quite a while:<\/p>\n
He starts with the left-top corner:<\/p>\n The first problem we have is data ingestion into the cloud or any system. How do you manage that? Azure can manage ingestion of data.<\/p>\n Azure Data Factory is a scheduling, orchestration, and ingestion service. It allows us to create sophisticated data pipelines from the ingestion of the data through to processing, through to storing, through to making it available to end users to access. It does not have compute power of it\u2019s own; it taps into other Azure services to deliver any required compute.<\/p>\n The Azure Import\/Export service can help bring incremental data on board. You can also use it to bulk load on Azure. If you have terabytes of data to upload, bandwidth might not be enough. You can securely courier data via disk to an Azure region.<\/p>\n The Azure CLI is designed for bulk uploads to happen in parallel. The SDKs can be put into your code, so you can generate the data in your application in the cloud, instead of uploading to the cloud.<\/p>\n The SQL database offers SQL Server, MySQL, and PostgreSQL.<\/p>\n Cosmos DB is the more interesting one \u2013 it\u2019s NoSQL and offers global storage. It also supports 4 programming models: Mongo, Gremlin\/Graph, SQL (DocumentDB), and Table. You have flexibility to bring in data in its native form, and data can be accessed in an operational environment. Cosmos DB has plugs into other aspects of Azure that make it more than just an operational database such as, Azure Functions or Spark (HDInsight).<\/p>\n When you want to do reporting and dashboards from data in operational databases then you will need an analytical data warehouse that aggregates data from many sources.<\/p>\n Traits of Azure SQL Data Warehouse:<\/p>\n When your data doesn\u2019t fit into the rows and columns structure of a traditional database then this is when you need specialized big data storages \u2013 capacity, unstructured sorting\/reading.<\/p>\n When you get data into a big unstructured stores such as Blob or Data Lake then you need specialized compute engines for the complexity and volume of the data. This compute must be capable of scaling out because you cannot wait hours\/days\/months to analyse the data.<\/p>\n How do you ingest this real-time data as it is generated? You can tap into event generators (e.g. devices) and buffer up data for your processing engines.<\/p>\n These systems allow you to process streaming data on the fly. You have a choice of \u201ceasy\u201d or \u201copen source extensibility\u201d with either of these solutions.<\/p>\n You have cleansed and curated the data, but what do you do with it? Now you want some insights from it. Reporting & modelling is the first level of these insights.<\/p>\n The basics of reporting and modelling are not new. Now we are getting into advanced analytics. Using data from these AI systems we can predict outcomes or prescribe recommended actions.<\/p>\n Taking advanced analytics to a further level by using these toolkits.<\/p>\n When you have such a large data estate you need ways to track what you have, and to be able to search it.<\/p>\n You need a platform with enterprise capabilities in the best ways possible in a compliant manner.<\/p>\n Nishant says that that darker shaded services are the ones usually being talked about when they talk about Big Data:<\/p>\n To understand what all these services are doing as a whole, and why Microsoft has gotten into Big Data, we have to step all the way back. There are 3 high-level trends that are a kind of an industrial revolution, making data a commodity:<\/p>\n We are on the cusp of an era where every action produces data.<\/p>\n There are 2 principles:<\/p>\n Few organizations are just 1 or the other; most span both locations. Data warehouses aggregate operational databases. Data Lakes store the data used for AI, and will be used to answer the questions that we don\u2019t even know of today.<\/p>\n We need three capabilities for this AI functionality:<\/p>\n Microsoft has offerings for both on-premises and in Azure, spanning MS code and open source, with AI built-in as a feature.<\/p>\n There are 3 core scenarios that use Big Data:<\/p>\n Here is a traditional DW<\/p>\n Data from operational databases are fed into a single DW. Some analysis is done and information is reported\/visualized for users.<\/p>\n SQL Server Integration services, a part of the Azure Data Factory, can allow you to consume data from your multiple operational assets and aggregate them as a DW.<\/p>\n Azure Analysis Services allows you yo build tabular models for your BI needs, and Power BI can be used to report and visualize those models.<\/p>\n If you have existing huge repositories of data that you want to bring into a DW then you can use:<\/p>\n This traditional model breaks when some of your data is unstructured. For example:<\/p>\n Structured operational data is coming in from Azure SQL DB as before.<\/p>\n Log files and media files are coming into blob storage as unstructured data – the structure of queries is unknown and the capacity is enormous.\u00a0 That unstructured data breaks your old system but you still need to ingest it because you know that there are insights in it.<\/p>\n Today, you might only know some questions that you\u2019d like to ask of the unstructured data. But later on, you might have more queries that you\u2019d like to create. The vast scale of economy of Azure storage makes this feasible.<\/p>\n ExpressRoute will be used to ingest data from an enterprise if:<\/p>\n Back to the previous unstructured data scenario. If you are curating the data so it is filtered\/clean\/useful, then you can use Polybase to ingest it into the DW. Normally, that task of cleaning\/filtering\/curating is too huge for you to do on the fly.<\/p>\n HDInsight can tap into the unstructured blob storage to clean\/curate\/process it before it is ingested into the DW.<\/p>\n What does HDInsight allow you to do? You forget that the data was structured\/unstructured\/semi-structured. You forget the complexity of the analytical queries that you want to write. You forget the kinds of questions you would like to ask of the data. HDInsight allows you to add structure to the data using some of it\u2019s tools. Once the data is structured, you can import it into the DW using Polybase.<\/p>\n Another option is to use Azure Functions instead of HDInsight:<\/p>\n This serverless option can suit if the required manipulation of the unstructured data is very simple. This cannot be sophisticated \u2013 why re-invent the wheel of HDInsight?<\/p>\n Back to HDInsight:<\/p>\n Analytical dashboards can tap into some of the compute engines directly, e.g. tap into raw data to identify a trend or do ad-hoc analytics using queries\/dashboards.<\/p>\n So you\u2019ve got a modern DW that aggregates structured and unstructured data. You can write queries to look for information \u2013 but we want deeper insights.<\/p>\n The compute engines (HDIsnight) enable you to use advanced analytics. Machine Learning can only be as good as the quality and quantity of data that you provide to it \u2013 the compute engine\u2019s job. The more data machine learning has to learn from, the more accurate the analysis will be. If the data is clean, then garbage results won\u2019t be produced. To do this with TBs or PBs of data, you will need the scale-out compute engine (HDInsight) \u2013 a VM just cannot do this.<\/p>\n Some organizations are so large or so specialized that they need even better engines to work with:<\/p>\n Azure Data Lake store replaces blob storage for greater scales. Azure Data Lake Analytics replaces HDInsight offers a developer-friendly T-SQL-like & C# environment. You can also write Python R models. Azure Data Lake Analytics is serverless \u2013 there are no clusters as there are in HDInsight. You can focus on your service instead of being distracted by monitoring.<\/p>\n Note that HDInsights works with interactive queries against streaming data. Azure Data Lake is based on batch jobs.<\/p>\n You have the flexibility of choice for your big data compute engines:<\/p>\n Returning to the HDInsight scenario:<\/p>\n HDInsight, via Spark, can integrate with Cosmos DB. Data can be stored in Cosmos DB for users to consume. Also, data that users are generating and storing in Cosmos DB can be consumed by HDInsight for processing by advanced analytics, with learnings being stored back in Cosmos DB.<\/p>\n He opens an app on an iPhone. It\u2019s a shoe sales app. The service is (in theory) using social media, fashion trends, weather, customer location, and more to make a prediction about what shoes the customer wants. Those shoes are presented to the customer, with the hope that this will simplify the shopping experience and lead to a sale on this app. When you pick a shoe style, the app predicts your favourite colour. If you view a shoe, but don\u2019t buy it., the app can automatically entice you with promotional offers \u2013 stock levels can be queried to see what kind of promotion is suitable \u2013 e.g. try shift less popular stock by giving you a discount to do an in-store pickup where stock levels are too high and it would cost the company money to ship stock back to the warehouse. The customer might also be tempted to buy some more stuff when in the shop.<\/p>\n He then switches to the dashboard that the marketing manager of the shoe sales company would use. There\u2019s lots of data visualization from the Modern DW, combining structured and unstructured data \u2013 the latter can come from social media sentiment, geo locations, etc. This sentiment can be tied to product category sales\/profits. Machine learning can use the data to recommend promotional campaigns. In this demo, choosing one of these campaigns triggers a workflow in Dynamics to launch the campaign.<\/p>\n Here\u2019s the solution architecture:<\/p>\n There are 3 data sources:<\/p>\n HDInsight is consuming that data and applying machine learning using R Server. Data is being written back out to:<\/p>\n The DW consumes two data sources:<\/p>\n Azure Analysis Services then provides the ability to consume the information in the DW for the Marketing Manager.<\/p>\n This is when we start getting in IoT data, e.g. sensors \u2013 another source of unstructured data that can come in big and fast. We need to capture the data, analyse it, derive insights, and potentially do machine learning analysis to take actions on those insights.<\/p>\n Event hubs can ingest this data and forward it to HDIngsights \u2013 stream analysis can be done using Spark Streaming or Storm. Data can be analysed by Machine Learning and reported in real-time to users.<\/p>\n So the IoT data is:<\/p>\n There are alternatives to this IOT design.<\/p>\n You should use Azure IoT Hub if you want:<\/p>\n If you have some custom operations to perform, Azure HDInsight (Kafka) can scale up from millions of events per second. It can apply some custom logic that cannot be done by Event Hub or IoT Hub.<\/p>\n We also have flexibility of choice when it comes to processing.<\/p>\n Azure Stream Analytics gives you ease-of-use versus HDInsight. Instead of monitoring the health & performance of compute clusters, you can use Stream Analytics.<\/p>\n The platform of Azure wraps this package up:<\/p>\n And now that your mind is warped, I\u2019ll leave it there These are my notes from the recording of this Ignite 2017 session, BRK2293. Speaker: Nishant Thacker, Technical Product Manager \u2013 Big Data This Level-200 overview session is a tour of big data in Azure, it explains why the services were created, and what is their purpose. It is a foundation for the rest of the … Continue reading “Understanding Big Data on Azure–Structured, Unstructured, and Streaming”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":20724,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[14],"tags":[170,224,218,222,176,177,219,203,201,223,221,220],"class_list":["post-20722","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-eventnotes","tag-azure","tag-data-warehouse","tag-dig-data","tag-event-hub","tag-eventnotes","tag-events","tag-hdinsight","tag-ignite","tag-iot","tag-iot-hub","tag-machine-learning","tag-stream-analytics"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/Azure-HDInsight.png","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/20722","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=20722"}],"version-history":[{"count":2,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/20722\/revisions"}],"predecessor-version":[{"id":20725,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/20722\/revisions\/20725"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/media\/20724"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=20722"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=20722"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=20722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"image\"](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2017\/10\/image.png\" "\\"image\\"")
<\/a><\/p>\nData Ingestion<\/h2>\n
\n
Operational Database Services<\/h2>\n
\n
Analytical Data Warehouse<\/h2>\n
\n
\n
NoSQL Data<\/h2>\n
\n
Unstructured Data Compute Engines<\/h2>\n
\n
Ingest Streaming Data<\/h2>\n
\n
Stream Processing Engines<\/h2>\n
\n
Reporting and Modelling<\/h2>\n
\n
Advanced Analytics<\/h2>\n
\n
Deep Learning<\/h2>\n
\n
Tracking Data<\/h2>\n
\n
The Azure Platform<\/h2>\n
\n
Big Data Services<\/h2>\n
<\/a><\/p>\n\n
The Modern Data Estate<\/h2>\n
<\/a><\/p>\n\n
\n
Evolution of the Data Warehouse<\/h2>\n
<\/a><\/p>\n\n
Implementing Big Data & Data Warehousing in Azure<\/h2>\n
<\/a><\/p>\n\n
<\/a><\/p>\n\n
<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\nFacilitating Advanced Analytics<\/h2>\n
<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\nDemo<\/h2>\n
<\/a><\/p>\n\n
\n
\n
Enabling Real-Time Processing<\/h2>\n
<\/a><\/p>\n\n
<\/a><\/p>\n\n
<\/a><\/p>\nThe Azure Platform<\/h2>\n
\n
I thought it was an excellent overview session.<\/p>\n","protected":false},"excerpt":{"rendered":"