Got Logs? Get a PaperTrail: First thoughts

I stumbled upon Papertrail through a Twitter Ad (hey, those things work sometimes!) and figured that I should take a quick look. Given the amount of work I’ve been doing around compliance management and deployment of distributed systems, this seems like it may be an interesting fit. Luckily, they have a free tier as well which means it’s easy to kick the tires on it before diving in with a paid commitment.

The concept seems fairly easy:

The signup process was pretty seamless. I went to the pricing page to see what the plan levels are which also has the Free Plan – Sign Up button nicely planted center of screen:

What I really like about this product is the potential to go by data ingestion rather than endpoints for licensing. Scalability is a concern with pricing for me, so knowing that the amount of aggregate data drives the price was rather comforting to me.

The free tier gets a first month with lots of data followed by a 100 MB per month follow on limit. That’s probably not too difficult to cap out at, so you can easily see that people will be drawn to the 7$ first paid tier which ups the data to 1GB of storage and 1 year of retention. Clearly, at 7 days retention for the free tier, this is meant to just give you a taste and leave you looking for more if the usability is working for you.

First Steps and the User Experience

On completion of the first form, there is a confirmation email. You are also logged in immediately and ready to roll with the simple welcome screen:

Clicking the button to get started brings you to the instruction screen complete with my favorite (read: most despised) method of deploying which is pushing a script into a sudo bash pipe.

There is an option to run each script component which is much more preferred so you can see the details of what is happening.

Once you’ve done the initial setup process, you get a quick response showing you have active events being logged:

Basic logging is one thing for the system, so the next logical step is to up the game a bit and add some application level logging which is done using the remote-rsyslog2 collector. The docs and process to deploy are available inside the Papertrail site as well:

Now that I’ve got both by system and an application (I’ve picked the Apache error log as a source location) working, I’m redirected to see the live results in my Events screen (mildly censored to protect the innocent):

You can highlight some specific events and drill down into the different context views by highlighting and clicking anywhere in the events screen:

Searching the logs is pretty simple with a search bar that uses simple structured search commands to look for content. Searches are able to be saved and stored for reporting and repetitive use.

On the first pass, this looks like a great product and is especially important for you to think about as you look at how to aggregate logs for the purpose of search and retention for security and auditing.

The key will be making sure that you clearly define the firewall and VPC rules to ensure you have access to the remote server at Papertrail and then to make sure that you keep track of the data you need to retain. I’ve literally spent 15 minutes in the app and that was from first click to live viewing of system and application logs. All that and it’s free too.

There is a referral link which you can use here if you want to try it out.

Give it a try if you’re keen and let me know your experiences or other potential products that are freely available that could do the same thing. It’s always good to share our learnings with the community!

Setting up a Slack WebHook to Post Notifications to a Team Channel

If ChatOps is something you’ve been hearing a lot about, there is is a reason. Slack is fast becoming the de facto standard in what we are calling ChatOps. Before we go full out into making chatbots and such, the first cool use-case I explored is enabling notifications for different systems.

In order to do any notifications to Slack, you need to enable a WebHook. This is super easy but it made sense for me to give you the quick example so that you can see the flow yourself.

Setting up the Slack Webhook

First, we login to your Slack team in the web interface. From there we can open up the management view of the team to be able to get to the apps and integrations. Choose Additional Options under the settings icon:

You can also get there by using the droplets in left-hand pane and selecting Apps and Integrations from the menu:

Next, click the Manage button in the upper right portion of the screen near the team name:

Select Custom Integrations and then from there click the Incoming WebHooks option:

Choose the channel you want to post to and then click the Add Incoming WebHooks Integration button:

It’s really just that easy! You will see a results page with a bunch of documentation such as showing your WebHook URL:

Other parts of the documentation also show you how to configure some customizations and even an example cURL command to show how to do a post using the new WebHook integration:

If you go out to a command line where you have the cURL command available, you can run the example command and you should see the results right in your Slack UI:

There are many other customization options such as which avatar to use, and the specifics of the command text and such. You can get at the WebHook any time under the Incoming WebHooks area within the Slack admin UI:

Now all you have to do is configure whatever script or function you have that you want to send notifications to Slack with and you are off to the races.

Top vBlog voting is underway

The shear amount of blogs listed over on Eric Siebert’s vLaunchpad is simply amazing!  I don’t know how many there is listed there – but there is certainly a lot of scrolling that needs to happen in order to get to the bottom – it’s awesome to see just how much information is being shared

The post Top vBlog voting is underway appeared first on mwpreston.net.

Top vBlog Voting 2017 – Supporting Community Bloggers

Every year we are seeing more and more community contributors in the blogging ecosystem. My own work here at DiscoPosse.com and through my role at Turbonomic in the community has been so enjoyable to be a part of because of the support that I continue to receive from readers and peers in many tech communities.

Eric Siebert has been hosting the Top vBlog voting for years, and it has grown from a handful of participants to a veritable must-read list that covers every aspect of virtualization, networking, scripting, and more. This year I am honoured to be among the contributors listed and am also very proud to have Turbonomic sponsor the voting.

My blog is listed in the voting under my name (just search for DiscoPosse) and my podcast (GC ON-Demand) is also in the running for best podcast.

I would greatly appreciate a vote if you feel that I’m providing content that is valuable, and of course, please extend your votes to all of the great IT community who surrounds us all. For those who know the work that Angelo (@AngeloLuciani), Melissa (@vMiss33) and I do with Virtual Design Master, you will know that many of the participants are also in the voting.

Your support of our amazing blogger and podcast community is always appreciated.  Thank you!

Vote here for this year’s event: http://vsphere-land.com/news/voting-now-open-for-top-vblog-2017.html?utm_source=rss&utm_medium=rss

MSPOG – Accepting the Reality of Multiple Single Panes of Glass

You probably dread the phrase as much as I do. We hear it all the time on a sales call or a product demo: “this is the single pane of glass for you and your team”. The problem is that I’ve been working in the industry a long time and have been using a lot of single panes of glass…at the same time. Many of my presentations have been centered around the idea that we must embrace the right tool for the right task, and not try to force everything through one proverbial funnel because the reality is that we cannot do everything with any single product.

For this reason, it’s time to embrace MSPOG: Multiple Single Panes of Glass

Many Tools, Many Tasks, One Approach

Using a unified approach to something is far more important than the requirement to using a single product to do it. I’m not saying that you should just willy nilly glue together dozens of products and accept it. What I am saying is that we have to dig into the core requirements of any task that we performa and think about things in a very Theory of Constraints (ToC) way. Before we even dive into some use-cases, think about what we are taught as architects: use the requirements to define the conceptual, logical, and then physical solution. All the while, understanding and making our decisions based on risks and constraints.

If you have a process that requires two or three different processes within it, you may be able to use a single tool for those processes. What if one of the processes is best solved with a different tool? This becomes the question of the requirements. Is it a risk if we embrace a second tool? More importantly, is it a risk or a constraint to use a single tool? This is the big question we should be asking ourselves continuously.

Imagine a virtual machine lifecycle process. We need to spawn the VM from a template, give it a network address, deploy an application into it, and then make sure it is continuously managed by a patch management and configuration management system. I know that you’re already evaluating how we should do this at the physical level by saying “use Ansible!” or “use Puppet!” or “use vRealize Automation!”. Stop and think about what the process is from end-to-end.

Our constraints on this is that we are using a VMware vSphere 6.5 hypervisor, a Windows 2016 guest, and using NGINX and a Ruby on Rails application within the guest.

  1. Deploy a VM from template – You can do this with any number of tools. Choose one and think about how we move forward from here
  2. Define IP address – We can use vRO, vRA, Puppet, Chef, or any of a number of tools. You can also even do some rudimentary PowerCLI or other automation once the machine is up and running
  3. Deploy your application – App deployment can be done with something like Chef, Puppet, or Ansible, as well as the native vRO and vRA with some care and feeding
  4. Patch management – Now we get more narrow. Most likely, you are going to want to use SCCM for this one, so this is definitely bringing another pane of glass in
  5. Configuration management – Provided you use SCCM because of the Windows environment, you can use that as well for configuration management…but what about the nested applications and configurations such as websites and other deeper node-specific stuff. Argh!!!

Even if you came out of the bottom of those 5 steps with just two tools, I would be thinking you may need to reevaluate because you have have overshot on the capabilities of those two tools. It is easy to see that if we start narrowing to a single pane of glass approach, that we are now jamming square blocks into round holes just to satisfy our supposed need to use a single product.

What we do need to do look for the platforms within that subset of options that has the widest and deepest set of capabilities to ensure we aren’t stacking up too many products to achieve our overall goals.

The solution: Heads up Display for your Single Pane of Glass

Automate the background and display in the foreground. We need to think more about having the proverbial single pane of glass be a visible layer on top of the real-time activity that is happening. Make your toolkit a fully-featured solution together with focus on how you can do as much as possible within each product. Also, reevaluate regularly. I can’t even count how many times i’ve been caught out by using something a specific way, only to find out that in a later version that the functionality was extended and I was using a less-desirable method, or even a deprecated method.

There is a reason that we have a mainframe at the centre of many large infrastructure shops. You wouldn’t tell them to shed their mainframe just to deploy all their data on NoSQL, right? That would be lunacy. Let’s embrace our Multiple Single Panes of Glass and learn to create better summary screens to annotate the activity. This way we also train ourselves to automate under the covers and trust the underlayers.

I, for one, welcome our Multiple Single Panes of Glass.

 

Why I Aeropress Coffee but Automate Everything Else

Many of my presentations start with me quoting the Rule of Three. Then I tell you three things about myself:

  1. I’m lazy
  2. I despise inconsistency
  3. Did I mention I’m lazy?

The reason that these are important things to know is that being lazy is a fundamental reason why I have leapt into automation from early on in my career.

Being the Right Kind of Lazy

The word lazy can sound like a bad thing. In the case of automation, it is a good thing. Clarence Bleicher of Chrysler was once quoted in the early days of the company as saying:

“When I have a tough job in the plant and can’t find an easy way to do it,” Mr. Bleicher said, “I have a lazy man put on it. He’ll find an easy way to do it in 10 days. Then we adopt that method.”

That pretty much sums it up. Laziness in the sense of not wanting to do repetitive, mundane tasks is the kind of laziness that we are aspiring for here. Not just lay down and do nothing lazy, as tempting as that is.

There was a key moment that happened in my first year of my work. When I saw a way to make something faster, or more efficient by taking safe and appropriate shortcuts, I took them. When I made the leap into a technology career, it didn’t take long to find the shortcuts. That was the whole idea of technology after all!

I drive a Stick Shift and Aeropress my Coffee

The reason that I personally do a lot of automation is so that I can choose to take the additional time I have to put towards the removal of other technical debt or even just enjoying myself. Part of the fun dichotomy of many of the very pro-automation technologists is that a lot of us tend to also be huge coffee enthusiasts. I’m talking about the hand-grind, slow steep, Aeropress and nearly scientific recipe kind of coffee people. Shouldn’t a lazy, automation-oriented person try to eliminate the time being spend on that effort? Ahhhh…there is the interesting part.

Automation needs to be in service of the goal. The goal is quality. I could increase my output by putting a Keurgig or a Nespresso, or some kind of automated espresso machine. That rolls up to additional costs, and then the quality of the taste. My choice is to take the lower cost to get a handcrafted taste that I know I enjoy. I have also done the math and realize that it may amortize over the long run to buy the machine, but I can also use my Aeropress on road trips and such. That is the consistency target I choose.

My choice to drive a manual transmission was primarily about cost, and secondarily around the enjoyment of it. That is really all there is to it. The time/effort savings of having an automatic impacts my personal enjoyment of the experience which I don’t necessarily feel is worth it.

Knowing and Measuring your Goal

How we define quality is as important as how we achieve it. Without a tangible way to measure the results of automation and the net effect on quality, we can end up just acquiring more technical debt, or in spending time on tasks that don’t remove constraints at the right level. Just like with my Aeropress and my manual transmission, I have chosen my measurement of where I can achieve quality through automation to attack other constraints.

Personal coffee taste is somewhat intangible. The time you spend deploying servers to the cloud and running patch management routines and other repetitive tasks is very tangible. Between time and quality, the effort to automate many operational tasks pays off rather quickly. Having had a background in desktop support at the onset of my enterprise IT career, I quickly created scripts and processes to avoid doing those repetitive tasks. Put all of that on a server and then all I need to do manually is connect to the server. Voila!

Measure your quality in either time, cost, or sometimes in those intangible ways such as just personal enjoyment of the work. If you are spending hours in a week doing repetitive work, you could spend a little time automating it and then spending that time you gain back the following weeks into new work and more exciting tasks.

Measure always, and not just for you, but for your team and your organization. Then you can sit back and make a nice Aeropress coffee while you watch your automation work happening for you.