![]() ![]() I recommend creating some PowerShell to handle this deployment from Visual Studio. dll files into our Azure Blob Storage container. Next, we build our console application and from the Visual Studio project ‘bin’ folder we copy the. ![]() Deploy an Azure Blob Storage account and create a container where our compiled code can live in the form of a master copy.Finally, we have a Program class where everything comes together. For the file cleaning example above let’s say we have an Azure Data Lake file handler class, some file cleaner class with methods for different cleaning operations and an authentication wrapper class for Azure Key Vault, I’ll talk more about key vault later. Within the project we write our custom code and classes for whatever we need.Just something that compiles to an executable. We start with a Visual Studio solution containing a C# console application project.In the image above moving from left to right this is what needs to happens and what we need to under to create our own Custom Activity: Maybe even save it to view on a second monitor while we go through it. Let’s start with the high level image I use in my community talks and add some narrative for all the arrows and services! Please do click and enlarge this image. Custom Activity – long running (if needed), ability to scale out automatically and scale up compute nodes if needed.Azure Function – quick execution, no scale out abilities.Next, you may think why do we need a Custom Activity for the data preparation example and not handle this with some custom code wrapped up in an Azure Function instead? Well, an Azure Function can only run for 10 minutes and won’t scale out in the same way our custom activity will. We need an ADF Custom Activity to perform an operation within our orchestration pipeline that isn’t yet supported nativity by our ADF Activities. We can write some C# to do anything! Or, to answer the question another way. Of course, we can use a Custom Activity for 9000 other reasons as well. ![]() Why do we need an ADF Custom Activity? To prepare our data using transformation tools that don’t require a redefined data structure. These line feeds, as we know, break our CSV structure meaning schema-on-read transformation tools struggle to read the file. My go-to example here is a CSV file, text qualified, but it contains an attribute of free text that some lovely frontend user has added carriage returns to when typing in notes or whatever. ![]() Typically in our data platform, data analytics or data science solution an element of data cleaning and preparation is required. If not, things can be switched around as you prefer and I’ll highlight alternatives as we go through.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |