AB Testing

BC Visits BU for TechConnect 2015!

techconnect-website

Big props to our friends at BU for putting on an awesome TechConnect Conference this year!  The theme was Digital Crossroads: Products and Ideas and featured an awesome array of speakers that really complemented each other and meshed with the conference theme.  It is a ton of work to pull something like this off so my hat is off to Jaymie and her team for a job well done!  A single post is not enough to do justice to the amazing amount I learned, but I’ll do my best to share a couple thoughts from each speaker I saw.

Bruce McCarthy-Effective Product Roadmapping

I have seen Bruce speak several times now at various Boston Product Management Association events and I am always struck by his enthusiasm and desire to help others.  It’s always a pleasure and I always learn something new.  In this case, Bruce emphasized the importance of your product road map telling a story about the value of the product.  He went into detail and shared a lot of knowledge but I will sum it all up by listing the Dirty (Bakers) Dozen common pitfalls of Product Roadmaps:

  1. Being Too Agile
  2. Planning Based on Your Gut
  3. Over or Under Estimating
  4. Not Having Strategic Goals
  5. Inside out Thinking
  6. Trying too Hard to Please
  7. Focusing on Features
  8. Not having a Buffer
  9. Playing Catch-Up (to competitors)
  10. Not Getting Buy-in
  11. Being too Secretive
  12. One Size Fits All
  13. Not having a Story

Want to hear more?  Check out Bruce at ProductPowers.com

Meghan Keaney Anderson-Product Launches

Like Bruce, I have met Meghan a few times before.  Not only is she Hubspot’s Director of Product Marketing, but she is also a BC Alum!  Meghan combined her deck on product launches with real examples from Hubspot to demonstrate her points in a highly effective and meaningful way.  She summarized the role of Product Marketing as coaxing the meaning out of the Product.  She shared with us her Golden Rule of Product Marketing: It’s never about the product, it’s about the person using the product.

Meghan

While I won’t share everything Meghan talked about, one of the biggest take home messages for me was how thorough and methodical an approach good product marketing requires.  Meghan follows a step by step process, ensuring that each necessary element is taken care of, tested, and resonates with customers.  She maintains a product marketing menu of all the different tasks, tactics, and channels they could use for a launch or activity and then selects which are the best for any given time.  Likewise, for a launch event, Meghan and her team actually have everything planned down to the minute or even the word of a keynote.  They maintain a command room at the event to facilitate a smooth launch!  All this effort and preparation can seem overwhelming, but it is crucial.  While there is certainly a lot there, according to Meghan, not all product marketing was created equal.  “The key is good positioning.  If you get that right, everything else is easy.”

C. Todd Lombardo-Ideas Lifecycle

C. Todd runs the InnoLoft at Constant Contact and had both an awesome presentation and a superbly aesthetically pleasing deck.  Most people are familiar with the concept of an MVP or Minimum Viable Product from the Lean Startup approach.  C. Todd suggested that before and MVP, you should pursue an MVC or Minimum Viable Concept.  Unlike an MVP, this doesnt even have to work, only to do a good enough job demonstrating the concept to get a sense of what it does.  He used IDEO’s Elmo Monster Maker prototype as an example. At this point you can test with 5 to 7 customers to get a sense of whether it is worth pursuing an MVP.  C. Todd suggested doing design sprints, or few day deep dives into a problem using the Define, Understand, Ideate, Build, Test cycle repeatedly.  He talked about the importance of being a “Designtist,” a creative designer that nonetheless approaches their design using the scientific method of testing hypotheses over and over to arrive at the right ends.  It’s a messy process and certainly not too straightforward.  C. Todd suggested that it is more like the design squiggle than a straight line.  “If it feels too linear, you are probably not doing it right!”

Ctodd

The other thing I really liked about C. Todd’s presentation was the concept of Job Stories.  We are all familiar with user stories and the amount of Jira tickets I have written stating “As a salesperson, I need to be able to…” could fill up more than a few Excel spreadsheets.  Job stories are different in that they lend context to the request and are thus more powerful.  They take the form “When (situation) happens, I want/need to (action), so I can (benefit).”  This format forces you to think about whether you are really addressing the pain point and makes it hard for anyone to misinterpret intent.

C. Todd left us with a few key take-aways that I will reproduce here:

  • Your opinion doesn’t matter
  • Talk to customers. Listen.
  • Check your assumptions. Again.
  • Think like a designtist.
  • It’s not failure if you learn.
  • F*** It, Test it.

Allen Murray-Agile Methodology

Allen is an SVP at Mobile Device Management company Apperian and his session was perhaps the most fun all day, not only because he is charismatic and approachable and I learned a ton about small aircraft flight, but also because we did the marshmallow challenge.  For those unfamiliar, it involves working in groups to form the tallest possible tower using spaghetti, tape, string, and a marshmallow on top.  MBAs typically perform terribly at this, while kindergartners do much better because the task lends itself well to trial and error and iteration and not advanced planning giving the time limits and lack of expertise in most participants (architects as the exception do quite well when planning).  Happily, we outperformed most MBAs and even kindergartners but learned that most of our assumptions were ill founded.  Allen used the activity as a jumping off point to talk about effective teams and Agile product development.

Marshmallow

Key Take-Aways from Allen’s Session:

  • Diversity in Teams matters, need leadership, facilitation, and specialization.
  • Incentives magnify outcomes.  With the right team, great things are possible but incentivizing a team without the right skills will only magnify the failure
  • The crowning moment in your professional career is when you see your customers truly satisfied with something you have given them.
  • If you can manage four things you will be successful using Agile:
    • Visibility
    • Adaptability
    • Business Value
    • Risk
  • Declaring work done is crucial to a successful scrum cycle.
  • You are not working on a technology, you are working on a business problem.
  • Must create dynamic on teams to want to improve.

Dan Cobb-Creating a Culture of Innovation

The final keynote of the day came from EMC SVP Dan Cobb who was slated to talk about creating a culture of innovation.  To be honest, to me his talk was a bit of a let down.  Dan spent the majority of the time educating the audience about EMC and essentially making a pitch for the company.  As a BC MBA, I am very familiar with EMC as they are a top recruiter on campus.  Nonetheless, Dan did hit on some points that I found very interesting.

Key Take-Aways:

  • Big companies must balance innovation and disruption
  • Innovate through internal R&D and aquisitions
  • When you aquihire you must empower the talent to stay on.  EMC does this by involving its acquisitions in future M&A decisions.
  • 3 Major Keys to Innovating Successfully:
    • Develop a brutally candid view of the world.
    • Align innovation agenda with strategy
    • Institutionalize feedback loops and then step back to listen, partner, engage, and measure.
  • EMC is “absolutely, positively, the most paranoid place in the world” and that is a good thing.
  • Innovation is a strategic means to an end.

David LoVerme is a 2nd Year MBA at Boston College and the President of the Grad Tech Club

Better late than never: the first BC Tech Club Ad Experiment Results

So we completed our first BC Tech Club online ad campaign last month. It’s a bit late, but I’d like to share with you the results of our little experiment.  As I detailed in my blog post of one month ago (link) we dipped our collective techy toe in to the digital waters with a series of Facebook ads with a goal of driving traffic to the Hire An Eagle page of the BC Tech Club website. To recap, below are the three ads we ran from October 8th, to November 8th. In total, between all three ads, we received 50 click-throughs to the Hire An Eagle page, and generated 2,144 impressions, as well as a paltry 9 likes to the BC Grad Club FB page as a result of seeing the ad. In other words, while only a handful of people may have visited the website, 2,144 unique pairs of eyeballs saw one of the three ads. Facebook counts impressions not just from our target audience segment, but also from target audience members’ extended web of connections. So when someone in our target segment likes our ad, that like makes the ad visible to all of that person’s friend via the news feed, dependant of course on varying news feed settings.

We thought it would be fun to experiment with Facebook’s A/B testing feature by offering all three ads with the same copy, same target audiences, and testing three different ad images. It was our postulation that the ad with the photo of Gasson Hall would be the most successful of the three. Since we were targeting specifically Boston College alums, it stands to reason that (hopefully) positive memories and associations of would draw them like a magnet to click on that picture.

Alaina1

We discovered however, that we’re terrible at prediction.

The most popular ad by far was the group photo ad, with 22 clicks and 1,734 impressions. It’s impossible to say for sure why this picture piqued the most interest among our target. My best guess is that the photo shows individuals clearly working in a business setting which best matches the subject matter of the ad. The Gasson photo on the other hand, while it might make alumni swell with Eagle pride, does not in any way reference recruitment or business. So the Gasson photo might not stand out amongst the other BC-related FB ads that our alumni audience is probably seeing.

And Now Onto January…

And with these lessons being learned, we’re ready to launch another Digital Marketing Experiment. For the month of January we are going to turn our ordinary ol’ BC Tech Club Twitter account into a PROMOTED Twitter account! The way it works is simple, we set the parameters for the segment of Twitter users that we want to target. These individuals then will see “BC Grad Tech Club” appear under the “Who To Follow” section on their homepage, and our account will show up higher in related searches. The targeting options for promoted accounts are relatively limited, presumably to ensure clients maintain a broad reach. We can segment by targeting twitter-users similar to a specific account’s followers, for example we might want users similar to those who follow @TechCrunch. We can also segment by interest category, and within the Technology & Computing category there are plenty of interests to choose from. And that’s basically it. It’s a very straightforward service to use, and clients are only charged per new follower so we are paying explicitly for desired outcomes, rather than for less meaningful interactions.

Alaina4

The goal here is audience building; we want to enhance discovery of the BC Grad Tech club among those individuals who would be interested in learning about our activities at Boston College, and who might want to hear what we have to say. We want to increase overall engagement with our Twitter account and increase our followers. Every new twitter follower is a new touchpoint for the BC Grad Tech Club, and a new potential networking contact for all our members.

– Alaina Tucker is a 2nd Year MBA at Boston College and the VP of Marketing for the Grad Tech Club