Cost of Not Having Continuous Integration & Deployment

No really, hear me out for a minute.

By now you’ve heard the terms CI & CD and that it’s the greatest concept since the espresso.  So as any natural and curious developer would, you began trying to understand what it is all about and how you can look like a superstar to your bosses and colleagues.  But this proved to be an extremely daunting task because you don’t control the product/project.

In reality it is suppose to be a daunting task otherwise it would be caked into a project for you.  Lets face it every project has separate and certain needs and trying to create a mold around that would only add to the complexity of the problems.

Nevertheless, I’m here to informed and cheer you on as I believe anyone can achieve CI & CD with a little perseverance!  The hardest part of this won’t be doing the work to get the bits flowing, but to convince you boss(es) and colleague(s) that this is a NEED not a nicety, so that they purchase the tools or allow for the time.  So I’d like to help you make the factual case so you can focus on what really matters.

I’ve previously gone through this process and was successful at the CI & CD and the selling to the “higher ups”.

Cost of Manual Steps

What do business people care about the most?  A good product? Yes.  Happy customers? Hopefully!  Enthusiastic employees?  Of course!

But at the end of a day, they care most about the bottom line of the business.  Therefore, we must make that our target when selling the process to the right people.  I’ll use a real world example that I’ve been apart rather than make something up.

Let’s say you are working at a company that has an established product and “process” that has been “working” for 10+ years.  Why would you ever want to change it?  Well when you start to break down this process you begin to understand that it doesn’t work as great as it could.  The “higher ups” believe the process hums along like a Hellcat Supercharged 6.2 Hemi, but its more like a MOPAR 2.2L I-4.  This is because there are a few select individuals that actually know how the 2.2L runs and if they are out or no longer with the company everyone is left scratching their heads.  If bad enough the company burns the entire team’s time just to get this process re/started.  But since “Frank” always needs his report, the select individuals continually added Gorilla-Tape to the 2.2L, which gets everyone over the hill (for now). 

Usually, this means that the team ignored Industry Standards and “just made it work”.  Which eludes that there the one or two individuals that have “tribal” knowledge on how it works.  Which then leads to an individual having to do 1-10 manual steps just to get the engine started, rather than turn a keep.  And if you can not get a hold of that individual on their vacation then your company cannot produce a product.

Starting to see the problem yet?

Not quite?

Yea but the cost is too much to swallow because we don’t know what it’ll take…

Okay, lets talk some numbers to put into perspective.  Lets, take Bob who is a Quality Assurance Senior and is currently the person in charge of testing and deploying the product out to the customers.  Bob makes 65K a year, which means he gets paid 31.25 (65000 / 2080) an hour.  A mediocre salary for what Bob actually endures on a day-to-day basis, but that’s not the point here.

Now the developers have notified Bob that they’re work is “checked in” and ready for testing.  So Bob begins the process of creating the necessary bits to test with; however, the first step takes about an hour.  Until now, no one has ever questioned “WHY?” it just how it was.  So now Bob waits an hour (31.25 paid) just for the bits to be ready.  Once that is completed, Bob has to then install the necessary bits on remote machines to simulate what is in production; however, Bob knows that this is a joke because the QA machine is 1/18 of what a production machine is capable of.  Because of this it takes Bob about a half hour to install the bits anyway (46.875 paid).  Mean while, the developer realized that something was left out by accident and informs Bob that he’ll need to do another “build”.  So Bob obliges and starts the process over again (93.75 paid).

Up to this point the company has just paid Bob ~100$ for waiting!

Lets stop and think about what this means.

  1. It takes roughly an hour and a half to go from developer to something testable (and that’s if the process doesn’t break down, which it will).
  2. Bob isn’t the only person waiting on a “build” to finish, so the additional analysts’ salaries should be included in the current paid amount.
  3. Obtaining multiple builds a day is extremely time-some and painful.
  4. The business spends a lot of money waiting.

Think about an entire year.  Let’s say a team does an average of three builds a day. That means that Bob is being paid 140.625 (46.875 * 3) a day to complete a “build”.

  • Weekly: 140.625 * 5 = 703.125
  • Monthly: 703.125 * 4 = 2,812.50
  • Yearly: 2812.50 * 12 = 33,750.00

    This is if everything goes perfect, which experience tells me with this kind of process it never will!

This is just one person’s cost to wait to be testing.  This doesn’t include the developer(s) who are waiting on feedback from the items Bob is trying to test.  Now think about scaling this current process out to a team double, triple, quadruple the size.  This is in-deterministic equation since there is no possibility to predict the future!

What Does This Mean?

It means that you need Continuous Integration & Continuous Deployment, unless your company likes throwing money in the wind!

Pretty accurate when you think about it isn’t it?  Why is this important for you the developer?  Well think about the money that their paying Bob to sit around and wait, if Bob could improve this process he has a card in his deck to catch some of the money on his next review!

Challenge those around you how important it is to “pull a lever” and have a testable product.  In the 21st Century there is no reason to be waiting for code to compile for an hour and installations to half that time, let alone having a human manually execute all these tasks!

Now there are a plethora of options when designing a Continuous Integration &| Continuous Delivery system.  Personally, I find it more beneficial to strive to use current Industry Standards for you frameworks.  This way individuals that come behind you in another 5 years won’t be cursing your name every 15 minutes.  Nevertheless, strive to have a prototype running when you go to pitch your plan to the necessary individuals.  People are naturally more visual and makes presenting your case easier, in addition remember talk NUMBERS, NUMBERS, NUMBERS.  If you can show that a process taking over an hour can be automated down to even 20 minutes, you can prove how much money the company will save and how much more work will get done throughout the year!


Till next time!

Automated Build Processes

Over the past few weeks I’ve been diving into automation in general and wanted to share some knowledge.  Soon I’ll be writing up an article that dives into versioning and the importance of combining these processes.  Anyways…

Tasks that are repeated continually should be easily automated to completion.  That’s a pretty hard statement to argue against.  Grab a cup of coffee and lets look at some pros and cons of automation, specifically how you build and deploy your products.


  • Audit trail
  • Removal of human error
  • Faster
  • Part of development process
  • No “tribal” knowledge needed


  • Extra development time

Now Pros are usually easier to come up with and I could continue the list on another five to ten bullets, but if you’ve made it this far you know the benefits of an automated build and release process.  One may argue that there is no accountability when automating something but I (and many others) argue back that is the furthest thing from the truth.  Having something laid out in source code is the greatest accountability one can reach.  So long as the changes are kept track of.

Types of Build Processes

Over the past few months, I’ve been exposed and developed multiple strategies for a build processes. Each one seemed to be written with different set of tools than the last because of that “one” thing that made the process unique.  And that is okay!  Not every application will be built the exact same way.

Lets take a mainstream product and break down a particular build process. The product has a client front end (mobile or web) and the front end clients talk to a web service that we’ll call the “back end”.  What you may not realize at first but all 3 of these “ends” may need different build processes because they are all targeting a different endpoint.  Mobile especially will have different endpoint if you’re targeting more than one platform.  Therefore, you maybe asking yourself: “Well how can I create a one and all build process?”.  You shouldn’t. Really, don’t try and make a overly complicated build process that is resilient to change because in 2, 5, or 10 years when you’re ready to make a change in the way your business works; you’ll be regretting the decision you or another developer made.

Within the past month or so, I’ve managed to create a flexible build process for two mobile apps, five different web services, and five or so command line utilities.  These build processes range from simple to complex (thanks to Apple).  However, all these build processes were created using PowerShell and no more than 25 lines.

PowerShell for building & automation

We all know PowerShell is great and the answer to most DevOps tasks.  In addition, it can be a great tool for building your build and deploy automation because it is so flexible and runs anywhere.  No really anywhere, even Linux (now).  At first, this journey may seem overwhelmingly difficult to tackle but the key is to take it one step at a time.  For example, if you find yourself right-clicking in Visual Studio to publish a web site to a folder on your machine; you could be saving so much time with automation.

Did you know you can use “MSBUILD” to build and execute that same publish steps with the following:

& 'C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe' "$pathToProject\ProjectFile.csproj" /p:Configuration=Release /p:DeployOnBuild=true /p:PublishProfile=Production /p:VisualStudioVersion=14.0

Really that’s it.  Since, publish profiles are part of a solution and project that means any changes to “Production” are caught in your source control management tool. This is where your accountability comes into play!  Next add some PowerShell to zip the folder up and drop it anywhere else would be as simple as:
Add-Type -Assembly ""
[io.compression.zipfile]::CreateFromDirectory($publishedLocation, $DeployDestination)

See how incredibly flexible that is?!  I bet you haven’t even taken a sip of coffee in a while!  Not only can you accomplish file copies and zips using PowerShell, but you can create a publish profile that will complete a Web Deploy for you! All you have to do is change the name of the PublishProfile you supply in you’re script (and maybe some credentials).


If you or your company is anything like me and mine, the developers don’t put anything into production.  Not only does that prevent tribal knowledge for leaking in but it also ensures documentation is correct and accurate.  However, distribution can easily be automated using PowerShell or the alike.  Once you’re able to do these two things, you’ve officially achieved the ability to have Continuous Integration and Continuous Delivery.  Then all you need is something like JENKINS, TeamCity, or any other flavor that comes around and you just need to drop your PowerShell scripts in place and you’ve successfully automated your build and distribution process.


Till next time!

MSBUILD vs. Web Deploy & Automation

So you’re here, probably on your third or fourth cup of coffee and struggling to automate the deployment of your web site(s). Don’t worry you’re not alone, I alone went threw an entire box of K-cups by the time I finally found the answer! You probably started out with using MSBUILD and the “DeployOnBuild=true” parameter for your publish profile and at first all was great! Your files were in the right location (maybe) and the site was up and running in IIS with no issues, minus the whole authentication hole you created (our secret). So you tried your mad Google skills and stumbled upon Web Deploy which looked extremely promising; but long story short, it fell short or your thinking there has to be a better way of doing this!?!?

And your right there has to be a better way! But first lets understand what MSBUILD and WebDeploy are…

Continue reading “MSBUILD vs. Web Deploy & Automation”