LISA the Third

$dayjob sent me to LISA again, for which I am quite grateful. It almost didn’t happen due to me moving to another business unit (read: promotion) a few weeks before the conference, but my new manager was able to pull some strings and make it happen (Thank you Brent!).

The Training

Like every year I showed up on the Monday, so I could get settled and do 1 day of training on the Tuesday and then 3 days of sessions.

The training on Tuesday was not a great selection in my opinion, there were too many full day training sessions which didn’t allow for a lot of choice. Of the remaining choices, one of the courses I had taken 2 years ago, so my choice was even further restricted. I ended up taking “Build a Sysadmin Sandbox” as the first course and wasn’t really getting anything out of it, so I bailed and went to puppetcamp which was also occurring on the same day. It was good course but the material wasn’t anything I didn’t already know.

Puppetcamp was very packed, but I was interested to see Garrett Honeycutt speak, as I do use a few of his modules and he is well known in the puppet community.

After that particular session I decided to just do the hallway track until my second training session.

The second training session was an introduction to Go for systems programming. I have heard a lot of hubub about Go, and was interested to see if the training/trainer could convince me that it was a good idea to stop using python and start using go. Long story short, I remain unconvinced that it would be a good idea to move from python to go for sysadmin type scripts. However if you need to worry about multithreading / multiprocessing then it would be a good idea to use go (go with go?). The material was good and the trainer did a good job; it’s just that according to me, go adds a bit of complexity that you don’t need to worry about in python. I do have a freakishly long python script that is multiprocess that would be fun to rewrite in go, but that project won’t see the light of day for years. :)

Tuesday night we headed out to the Taphouse with a fairly large (18ish) group to socialize and swap stories. It was pretty fun and right across the street from the conference hotel.

The Sessions

My favorite sessions were:

Brendan Gregg’s Linux Performance Analysis: New Tools and Old Secrets was fantastic. Only 8 people in the audience had heard of Ftrace, I was not one of them. I have already downloaded perf-tools and am likely to deploy this to all systems.

Brian Atkisson’s Open Source Identity Management in the Enterprise was a very good talk, with the right amount of complexity. This is the exact type of talk I’ve come to expect at LISA. Worth a watch if you are doing anything related to identity management.

I missed the first 10 minutes, but Ben Rockwood had a very good session called I am SysAdmin (And So Can You).

Thomas Uphill had a puppet session called Puppet for the Enterprise that I went to that was good, but either the crowd didn’t want to participate or the level of puppet knowledge in the crowd was very very new. I didn’t realize before attending that he wrote Mastering puppet. I read the book shortly after it came out and do recommend it. I had some conversations Thomas later at the EMP museum about workflows with large teams and he seams to think we are doing everything right but I have issues with the current workflow that I don’t like. I believe I have a solution but I’ll save that for another blog post.

Birds of a feather

The birds of a feather were quite good.

There was also 1 BOF on Thursday, which I ran. Didn’t take a picture of the board, as there was only one (Besides the Google BOF).

I stole the idea from LISA 2012, and hosted the BOF “I’ve made a huge mistake”. Due to scheduling I had to have this during the Google BOF, so I didn’t really think many people were going to show up. Much to my surprise, so many people showed up we had to move to a bigger room. I should have taken a picture but totally didn’t. We moved to a larger room and everyone shared stories of their epic failures. Some of the stories were really hilarious.

Other than my own BOF, I liked the RedHat and Puppet vendor BOFs. There was also a Infrastructure Testing BOF that had me take several notes.

Random

Every LISA that I have been to so far, there has been an event that “paid for the conference”. And sometimes you don’t know when that will occur. For example, last year the flamegraph talks paid for the conference, as I have used that knowledge to solve several problems that would have taken weeks of man hours if I had not. Then months later we were looking for a Sunray replacement, and I remembered a hallway track conversation that I had with Tobi Oetiker about what he was doing with Sunrays, and I was able to suggest that we test that product. No-one in my organization had heard of that product it wasn’t even in our eval radar. Adding that product to our eval and having us use it will have saved the company the price of the conference several times over.

This year, Brendan Gregg’s talk paid for the conference. I’m working on a few other ideas inspired by the conference that may pay for the conference several times over. I am always very invigorated after a good conference, I think I will be realizing additional value of the conference for quite some time.

I got to meet Dave Josephson from Librato who recently collaborated with me on graphios to add librato and statsd support. We had some excellent conversations and from those conversations I will be making some changes to what I am monitoring to try and preserve the interesting data without having to keep my data forever. Dave also did 2 talks at the Nagios world conference that mention graphios here and here. Not to mention he did two talks at LISA here and here. I caught the second talk at LISA and I quite enjoyed it.

Notes for future first timers

• Wear your badge from the beginning of the day until the end of the day. Otherwise people aren’t going to know that you are an attendee. Wearing the badge makes you approachable for hallway track conversations.
• If you get a table for breakfast or dinner in the hotel, get a larger table and tell the greeter to send other attendees your way. (You may want to check if someone else is already doing this!).
• Do a write up for your manager / coworkers when the whole thing is over. You need to pass on what you learned at the conference, and thank your manager for the opportunity of going. Your goal should be to prove that the company spent money wisely when they opted to send you there, and have justification for sending you again.
• Do 1 day of training, so you can get all the training material.
• If you aren’t sick of chatting with people, wear your LISA shirt at the airport. I had some excellent airport conversations, with other attendees because they recognized my shirt.

Random Notes I took

• Read book Inspired
• Read book Remote
• Read book The quantum thief
• Watch documentary Jiro Dreams of sushi
• Look into bats (bash automated test system)
• Look into nats.io apparently beats the pants off zeromq and rabbit written in go
• Look into graphite-ng
• Testkitchen apparently works for puppet
• Several people told me to use Cassandra instead of Opentsdb
• Have project managers do 15 minute hipchat meetings instead of 30 minute conference calls (This is amazing)
• Make sure the CA root is kept offline on a usb key or whatever (This is likely already done in my org).
• Watch British TV show A touch of cloth
• Check out paper on ntopng
• Lots of zookeeper haters / badmouthing
• Facebook’s Opencompute datacenter is 38% more energy efficient, at 24% less cost (energy cost). If this was the only benefit it would be worth looking into, but there is so much more.

Final Thoughts

I had a great time at LISA, and look forward to attending again next year.