Often, when a new connector source type is added, they may not have all the features available you want. For a while Snowflake, you couldn’t use dataset and linked service parameters. In fact, I started this blog a year ago, and Snowflake didn’t have…but by the time I got around to writing it, they had added.

However, recently I was working on implementing Mongo as a data source, and ran into the same issue. (So this time I am blogging before it changes. )

Linked Service Parameters

For many ADF linked service connectors, when you add, you can go down to the bottom of the fly out tab and you’ll see Parameters. You name the parameters, then can reference them in other places within the linked service setup. You’ll see a little “Add Dynamic Content” underneath text box, and when you use them the box will turn blue, indicating it is indeed dynamic.

So in ideal workflow you’d have linked service parameters, that would show up in the dataset, and then you’d create dataset parameters and fill in the linked service parameters with them.

Then in the ADF activity, such as a copy activity, you’d see the dataset parameters. Then to go one step farther, you’d create pipeline parameters so you could make everything dynamic at run time. And all those dynamic elements would be obvious, or explicit, in UI.

Linked Service WITHOUT Parameters

So above is the ideal scenario, where you can have this nice little chain of parameters. 

But what if the Linked Service doesn’t have parameters. Can I make it dynamic or am I stuck with the purgatory of creating a ton of individual static linked service?

Luckily, we can use a lot of the same practices we have above and kinda hack our way through, but with a little less guidance from the UI.

Mongo right now happens to be a good example. When I open the linked service, there is not even a place for parameters (I’d like to rant about how they should have a template by now, but let’s set that aside). So this what I am shown.

Even though I don’t have parameters, I can manually add the same pipeline parameters I used earlier and put it in the text box. However, notice that there is no “dynamic content” pop up and it didn’t turn blue. And I have to include these nice curly brackets, so ADF knows to consider it a parameter.

So it works, but I won’t see it indicated to me in the dataset, and since not in dataset, it won’t show up in pipeline authoring (since remember that is coming from the dataset, not the linked service itself).

Pros and Cons

So we see that we can accomplish our goal, to make reusable artifacts in data factory that can be made dynamic, by passing in parameters from as high a level as possible, ideally at run time in pipeline.

However, those dynamic elements won’t be obvious. Meaning there is, for example, no way to preview data or test, because by definition I only have pipeline parameters, when running a pipeline. So if I want to test, I will need to manually change the linked service to a static value and preview, or run a pipeline, not exactly ideal. Also, when collaborating, the other users of your “implicit” linked service parameter won’t know to have that parameter in their pipeline, and even if they do, they have to make sure it is the same name, and is even case sensitive. So probably a handy idea to document it somewhere.

Ultimately, the positives of having this re-usable element rather than miasma of linked services, outweighs the negatives, but just keep them in mind.

That night’s talk wasn’t recorded, but found a few previous versions on YouTube. I’ll apologize ahead of time if my paraphrasing of his content is not near as elegant as it was in person.

When he was talking about “Outside In” design he was meaning that often back-end developers go off and polish this absolutely beautiful model of everything they think the software should be. Meanwhile, the front end developers and BA’s are talking to users about what the app actually needs to do. And when the two sides are silo’d you have this disconnect.

In the discussion with some of those there that night, he mentioned that everything has to be within constraints, and often one of the constraints is time. If we look at waterfall versus agile, you can’t take forever to have an MVP product, you have to produce something and iterate on it. He said that represented the contention between Excellence and Pragmatism. (I noticed one his books even has “Pragmatism” in the title.)

Excellence versus Pragmatism

…That really stuck with me.

It made me recall having a former boss give me this notebook that in big letters said “Let Me Overthink This.” I’m not sure if he was saying my overthinking was a good or bad thing, but I’ll own it. I do over-think and over-engineer.

I’ll spin my wheels on the edge case, that one account that has all this convoluted ETL logic when then other 400 don’t, and they represent less than 1% of the total volume of data. That same manager would often say “is the juice worth the squeeze.” And to that I probably am like until it is perfect, it is worth it. But we know that isn’t true, you make the perfect the enemy of the good.

So yes, you should try to be a craftsman (person?) and hone your skills and evolve and improve your designs, but don’t do it at the expense of getting something (usable) delivered.

But for a lot of folks, the cloud is still an abstract concept and unless you have a job where you are forced into the cloud, you may be unsure where to start to learn.

Last week, I wrote about using vendor certifications as a way to help build a learn path, whether you take the tests or not. So let’s start there.

Beginner Cloud Certs

All three major cloud vendors, Amazon Web Services (AWS), Google Cloud (GCP), and Microsoft’s Azure have a beginner certification. They each call them something different  [ AWS: Cloud Practioner; Azure: Azure Fundamentals (AZ-900); GCP: Cloud Digital Leader ] but they cover a lot of the same territory.

Prior to even diving in, you should consider which one to start with. Any of the three will introduce you to basic cloud concepts, why you should move to the cloud, selling points, etc. To differentiate, you should consider which is popular in your market, or what kind of job you want, or which cloud vendor your company talking about, or is using. For high level differences, AWS tends to be more developer first, Azure tends to grow out of Microsoft uses in other areas of a business, say Office, SQL Server, Teams, etc.; and Google is strong in data and ML, and for not being one of the other two .

What To Study

All three have study guides and on-demand learning to help get you started. Pay attention to those, and realize they can change over time, so if you started months ago, they may have added new technologies they now think are important.

Start with what are the advantages of the cloud – cost, scalability, disaster recovery, security, and resource allocation, i.e. who is responsible for managing what. Then how are the cloud resources physically maintained, where are the data centers (the cloud may seem ephemeral, but they are real places in the real world), who manages the power, the networking, the servers, all the physical things that make our modern digital world work. Each cloud provider might have a different term for their regions and zones, so learn those, and how they are connected, provisioned, and what their latency/uptime is within the region. Also, when it comes to networking, learn how they connect to your on premise resources, those are different services for each, and how they globally scale.

Shared Responsibility Model – What is the cloud providers responsibility, things like power to data center, physical security, server hardware. And what is the customer’s responsibility, like application code, data, etc.

Identify which service falls into each service type: Infrastructure as a Service (IaaS), like virtual machines; Platform as a Service (PaaS), like managed database services; or Software as a Service (SaaS), full managed software, like Office 365, GMail. Also, learn the service names, all three have pretty analogous product offerings, like each has a data warehouse service, AWS: Redshift, GCP: BigQuery, and Azure: Synapse. You won’t need to know what each does in depth, but you need to learn a one line definition for each. That being said it wasn’t a huge part of any of the exams, but you don’t know which they will ask. And AWS has way more services than the others, and GCP, well a lot of theirs are pretty intuitive, like Storage is “Google Cloud Storage” which is similar to AWS S3. (Oh, and on storage at least have a basic understanding of tiers and retention and use cases.)

Lastly, understand how each does billing and account management. Remember this is an intro to cloud services, so a lot of folks may just be coming in, provisioning services, and next thing they know get a huge bill. Or a company many have many departments that spun up shadow IT infrastructure, you need to understand, at a high level, how to bring those under one account and manage and monitor them.

Where are They Different

Primarily, all three are asking questions around the same topics, and largely geared towards helping people sell cloud migration or for those new to the cloud. However, each one emphasized some components more than others. They may word their questions differently too, so may ask the same kind of information, but in a different way. Azure asks Lowest Administrative Effort for “easiest”, and GCP has a phrase for cheapest which you will see a few times, while AWS may say which will help with the Cost Optimization WAF Pillar, which also just means cheapest. These are based on my tests, so your mileage may vary, you should still pay close attention to the study guide.

AWS

AWS Cloud Practioner was the most recent one for me, and also the service I felt I needed to study the most for, so the differences stood out more.

Well Architected Framework, Well Architected Framework, Well Architected Framework – Learn It, Live It, Know It

AWS thinks their WAF is really core to building on their cloud, so know the pillars of the framework. Which service would you choose if you more interested in Cost Optimization than Reliability, or which pillar does cloud elasticity apply to.

I had several questions on using spot instances to keep prices low. All three have spot instances, i.e. instances that the cloud provider can reclaim quickly, and for which you are willing to be booted off of, in exchange for a lower rate, but AWS asked me three questions on it.

AWS Organizations. Billing and management is key for each, but the mechanism differs, so learn about Organizations and Master Payer Accounts, and how that can get your enterprise price breaks.

Horizontal versus vertical scaling, and which one EC2 auto-scaling uses (horizontal, i.e. you add more instances, not bigger instances in vertical).

Learn which services AWS considers serverless, it surprised me in some cases, and they might ask if you want to make sure is a serverless instance type which do you pick (often this is Lambda).

Monitoring, knowing Cloud Trail versus Cloud Watch is important, and well, that one was hard for me.

Azure

Azure asked more service definition questions than the other two on my tests. Luckily, you don’t have to learn icons, that would be a mess (however, makes for a fun trivia game if you want for your office team). But they do want you to be able to tell them which container service is which for example, like if it says Kubernetes, you know AKS.

Azure has a concept of resource group, the others don’t, so that is important to understand. It is a logical grouping for management, but has no physical requirements, meaning if you are deploying a certain app you could keep all the resource in it, and apply permissions to it, but the services themselves could be of all types spread around the world. (This is probably one of the things I miss the most when in GCP and AWS, they manage this differently.)

Azure was probably the most straight forward one, and easiest for me, but not sure if that is because I have more Azure experience. Azure is also offering free practice test for some of their certs now, so check those out.

Google Cloud

GCP had more questions around why you should go to the cloud, what are the key advantages, what is CapEx versus OpEx, etc. This is an important topic to understand regardless of which cloud vendor you use, just GCP emphasized more. In addition, they asked more questions around the shared responsibility model, like who is responsible for paying the security guard at the data center, or making sure there is redundant power and networking (GCP is).

GCP has more data and ML service offering questions, not surprising for the company that gave the world TensorFlow.

Happy Learning

Besides using the vendors’ sites for the study guides, learning paths, and free practice questions, there are a few other resources. ExamTopics is where people have put sample questions for a variety of tests, but, the caveat is, sometimes you have to read the answer discussions, the one marked right, may not always be right, what you get for crowdsourcing. However, understanding how the questions are worded is very helpful. ACloudGuru and PluralSight are both good, paid sites. PluralSight owns both but they have different content and different costs. Both offer a short free trial so try before you buy. If you think you will be doing more than one test or want a career doing more they could be a good investment.

One last tip, if you take the test, you may not know every answer, but start by eliminating the ones you know it can’t be, if AWS wants something serverless you know it won’t be EC2, and then narrow your choices of possible correct answers. If you have to guess, better to do so when you have a 50/50 chance than a 1 in 4 or 1 in 5 chance.

Good luck, whether you take the tests or not, learning about the cloud is hugely important for anyone in (or wanting to get into) technology in our current world.

After I started as Slalom, I was talking to someone about what a bad student I was in college, but how I loved to learn, that I would love to “get paid to learn.” And the person said, “well you kinda do.” That really stuck with me. The thing about technology that I enjoy is there is always something new to learn, some new product, or way to solve a problem.

In past roles, I’ve wanted to learn, but the company’s cultures didn’t often align with that goal. And learning for yourself can be hard. At Slalom, they have this expression about having a “Learner Mindset,” and when I interviewed I was asked about how comfortable was I in Google and AWS, versus Azure where I spent most of my focus. I said I had some AWS experience, but always wanted to learn more. I felt like it was a challenge, and so I did my best in last year to live up, and have now passed 10 certs in a variety of technologies.

Where Do I Start?

I think the hardest part if you are in technology is to figure out where to start. That was why I stumbled so many times before (look back at my blog on learning Python, I must have taken those classes four+ times). I needed structure.

A couple of caveats to start. I am definitely not advocating for taking cert tests just to take them, and don’t take them without understanding the topics just thinking that will pop for an employer, it will cause you both pain.

However, once you have a subject you are interested in, use a certification for that topic as syllabus to help you structure your learning. They tell what they think is relevant to their technology, their ecosystem, and for job roles. Many of the providers have free trainings on their recommend learnings, some have classes to help get you started or tailored to a specific technology or certification. Use those as a starting place.

Next, time box your learning goal. I often said I wanted to learn X, but never put a date on that goal. If you taking a test, that is an easy goal, schedule the test based on how long you think you need to study, and try to stick to it. If you aren’t taking the test, still set a firm-ish goal date, don’t be unrealistic, but also not so far out that there is not a bit of pressure to learn.

Additionally, put some structure around when you learn. I put a calendar reminder for every day for an hour. I may not always study at that time, but it is always there to keep me on task. If I want to study something in particular, I may change to “Azure study time” to help reinforce this. (Remember when I said I was a bad student, if only I had done this in college.)

Another thing I find helpful is to sign up for free sessions on the topic. Maybe Snowflake has a webinar on new python capability. It encourages you to set aside time to focus on that topic.

The last thing I’d do is use the syllabus you came up with as a starting point, not an end. If you find something that you think sounds interesting, dig into it. You can get on most cloud vendors and get free credit time, go build something that you learned (read how I did that with AWS Lightsail). Reach out to people in that community (I would have used to have said on Twitter, but sadly, that’s diminishing, but there are still good places to go, LinkedIn, Reddit, the vendors). If you are in a town that has Meetups, go on there and look for ones in that area, or there are tons of virtual ones.

Last thing…everyone learns differently, so learn yourself and tailor you methodology to yourself, and give yourself credit for starting the journey.

Let me know if you have suggestions that work for you.

Some tech vendor learning sites:

• AWS
• Azure / Microsoft
• Google Cloud
• DataBricks
• Snowflake

I have taken several certifications over last year since I started at Slalom, the most recent being the AWS Cloud Practioner one (basically their beginner fundamentals one). In reviewing the myriad services, one caught my attention, Lightsail makes it easy to create simple preconfigured web app stacks, in this case WordPress, but it has a lot of others. But, I’m usually the Azure guy, surely they have something similar and I can just put in an Azure resource group.

It took some googling but there is a handy post out there about using pre-configured app services in Azure for this purpose. And it has a direct link to the quick start. So decided would give it a try, and while I was at it, spin up an AWS Lighsail instance at the same time to compare.

Well, the Azure experience wasn’t that great…and AWS came out way ahead on this one.

Azure WordPress

So the Azure post tells where to get started and it has a wizard that walks you through it. You have to go through the same steps for creating about any Azure resource, including sizing for app service and database (but you aren’t shown pricing), along with tags etc. That’s pretty standard, but maybe not newbie user friendly.

The real issue for me with Azure was that if I were in the portal, and I said, “hey, I want a WordPress site”, is there a way to get to this wizard? Well, yes and no…you get a ton of options from the marketplace, but not this one exactly. So if I didn’t come in from the blog post, I don’t even know this option exists. I get 14 options if I search for “WordPress,” even if I guessed that I needed to pick the “App Service” one, how do I know which to pick? I guess if I pick the one that says “Microsoft” I finally get back to the same place, after another click. Is this a frictionless experience?

AWS LightSail

Let’s compare that experience to LightSail. You have to know to go to Lightsail, so I admit that might be the one thing you need to know, but from then on it is SUPER easy.

You pick the app you want, you fill in a few other settings, and then the best part, you get a simple price picker for which size instance you wanted and some of them even have free months to start, in my case I picked the $3/mo one. (Compare that to I was paying GoDaddy $14/mo for domain hosting and I still don’t know what my Azure costs were going to be).

Then to make it even better, once I hit create instance, it ran through creating it and getting me to it, super quickly. And the instance itself was much more responsive than the Azure one, maybe that is because AWS has a larger CDN/caching network, or their small instance better, I don’t know, but as a user it was just a better experience from one end to the other.

I didn’t move the DNS at first (which I should note it is nice that AWS has Route53 that can be a domain registrar to get even farther from GoDaddy, Azure doesn’t have that as a service), I could run all three sites simultaneously. The Azure one gave me an AzureWebSites.com second level domain I had picked during creation, on AWS I started out just using an IP to view it. I found an awesome backup plugin, UpDraftPlus, that made it super easy to copy over all the files from the original one to my two test mirrors. For fun, I themed each differently, just to see them looking a bit apart.

Today, I switched the name servers over to AWS and it is live there, and I am very happy. So for the normally all Azure guy, this one is where I picked AWS.

Jean Joseph, Ed Pollack, and all the organizers did a great job.

I took some pics, but none of which were of the audience for my session, but it was 15-20 folks with lots of good questions. I was rusted being live and in person, and it was a new topic, so was great to get that first time and didn’t bomb.

Presentation Slides

Looking forward to next week headed back to Baton Rouge.

I randomly found Girl Code in my library app (free audio book is such a nice community thing, hope we don’t lose libraries) and seemed like it was perfect to get something with a different perspective.

The book was short, but was exactly what I needed. These two young women, Andrea Gonzalez and Sophie Houser, found programming, and as the cover says accidentally went viral, but ended up getting so much more out of it.

At a macro level the book was about how much we need to expand opportunities for everyone to learn to become a developer. How much we should make sure we are encouraging young women to be software engineers (or any kind of engineer), something that raising a five year old girl hits close to home. (Girls who Code and similar initiatives are laudable.) How much our lives are now surrounded by code, as Sophie and Andrea do a good job informing us. I did also learn a lot about Menstruation Taboo, that I had no idea about.

At a more micro level, I enjoyed them talking about the joy of learning to code. It made me think of those times when I’ve solved a problem, maybe I feel like I have spend hours spinning my wheels, only to have a revelation. Also, they talk about the power of collaboration, from working with each other, to pairing with other programmers when they built the mobile version.

That was the pure, inspiring thing that was exactly what I needed. I think when you work in it every day, what ever programming you do (and I don’t claim to be a developer), you can forget that thrill.

The roundtable was on Job Hunting and Finding a Mentor. I certainly don’t think I am an expert, but by happenstance, I just got a new job (my fifth since 2017, don’t judge), and also trained and mentored at previous gig, so hopefully I had some helpful advice. At the end of the night I realized I could boil down my advice to three tenets.

#1 – Lean Into Your Experiences

What makes you is what you are selling in a job interview. It is also fundamental to what you may be looking to do in your next role.

Make sure that you feature your background, regardless of where you coming from; tech, not tech, life… and weave that into your story and make it part of your narrative. Don’t shy away from that, use it.

As a data example…say I used to work at Taco Bell and I saw every day the long drive thru times, how could I consume that data, how could I visualize it, how could Taco Bell make more money.

When thinking about that next job or next role, consider your background too. Maybe your background is more creative, so you want to find something in tech, but appeals to that, or maybe you have a math or engineering background, help that guide you.

#2 – Always Be Learning

Where I just started at Slalom, I had a recent seminar on Knower versus Learner Mindset, and that really struck me.

The reason I love tech, the reason I get up in the morning, is to constantly be learning. The more I know, the more I know I am not an expert. Expert level knowledge is a nirvana, but every day I can work towards it. Don’t be afraid to ask others, maybe you have used this tech and they haven’t or vice versa, maybe someone else knows this business domain and you want to learn it. Ask, listen, and grow.

#3 – SQL

The last one is very concrete and close to my heart.

SQL is core to data technologies. Every day there is a new stack, a new vendor, but SQL has been here for 40+ years (Wikipedia says 48) and is core to so much in the data space. Learning SQL is beneficial to whatever you data role you take.

However, maybe I don’t just mean the Structured Query Language in all it’s flavors…maybe I am lumping in there, understanding databases and relational algebra at some core level. Understand how to break apart a problem, how to optimize and improve your code or analysis.

I’d love to hear how other people think about job hunting, career growth, finding a mentor, being a mentor. And all the reason SQL is old tech and Python is King.

My last posts were somewhat (very?) self serving as I went through trying to learn more software engineering practices and specifically Python for my job with a web app startup. So, I wanted to circle back with how it was going learning.

I spent the last 8 months embedded with a software engineering team with some data scientists, most of both groups could write Python. As I’ve said before, I had taken numerous EdX and DataCamp sessions on Python, but nothing had stuck. So there was a big learning curve going from all SQL and GUI ETL workload to using Python as core language.

The biggest takeaways from the experience for me are:

• There is no substitute for working in a language or technology. When you do it for your own education or personal projects, it just doesn’t stick like it does when you are “forced” to do it. It really focuses your mind.
• Social Learning is a big help. Having a team that you can go and ask (bug?) with questions when you get stuck (or your StackOverflow searching doesn’t help) is invaluable. The fact that I had smart people who were patient, and often told me more about the context or tips and tricks was a huge help.
• Data Engineering Education still isn’t great. DE’s habitat a world in the in between. We aren’t software engineers with all the mature practices from there, but we also aren’t solely living in a data analyst/scientist world, we are a hybrid. And education for the hybrid really isn’t the best, there is lots of good app dev and data science training, but not the middle (getting better though).

Since I think the training out there not great, I did want to share how I think I learned.

I did do some formal training in DataCamp (still more geared towards data scientists) and on MS Learn (which I am a big fan of, even if more toward specific Azure tech, but still helped). Beyond just bugging team members, the biggest helpful resources I found were RealPython.com and StackOverflow. (The caveat with S/O is often you fall down a rabbit hole of competing package methods, but it’s always educational.)

Being honest, a few months ago I thought I’d be writing that I felt like I had not learned much at all, but the past few weeks it has finally come together. I don’t know that it would be my automatic go to for all DE projects, but definitely if the need lives more on the programming/automation side I can see the big benefits.

Of the many of the posts I see, seem to stress the core idea of if you don’t know Python are you even a data engineer? For many people in the data space, they start their career learning Python, often before they learn SQL. For me, that is hard to wrap my head around.

I’ll admit my bias.

I have taken numerous Python classes; Coursera, EdX, DataCamp…but much like the ad I keep seeing for FreeMote,

I learned it and promptly forget 90% of it. I was writing SQL all day long, and never using Python in my day to day, so it just atrophied. Every time I came to learn again, I had retained a little (10% ???), but generally started from scratch.

My block might be, going back to last post, I think of Python as a general purpose programming language, and I am not a coder, but a data guy, and SQL, is my lingua franca.

Or maybe the problem is the way it is positioned and taught.

Each of the classes above were for Data Scientists, and there are many others that are for Software Engineers. There are few out there that are specifically for Data Engineers. This one seems like a BIG gap to me. Tons of training on how to build a web site, or design a BI report, or do regression and machine learning algorithms, but little for data engineering as a pure skill.

Will knowing Python help me as a data engineer, absolutely.

Many jobs will require it or at least like to see it. Moreover, if you are in a company where they use Spark for big data, then mostly likely you will use some Python. Alternatively, if you are working closely with data scientists it will help to understand their language to better interoperate. Ultimately, the more languages and programming concepts you know, the more successful you will be when tomorrow a new language or methodology appears.

That said….I think YOU can be a Data Engineer and not know Python, and certainly not know it well, you just may not be as successful as you want to be. However, I don’t think you can and should be a data engineer and not know SQL.

Side note-- I am now learning Python again....maybe this time it will stick!