Mobile Development Story: 2 Engineers Tapping Into Massive Scale

by Charles McBrian, Sr. Manager, Mobile Engineering @WalmartLabs

As a mobile engineer and manager, I’m always on the lookout for engaging projects for our teams. Challenging projects that have a high signal-to-noise ratio. Lots of real work. Most importantly, I want projects that make a difference for our customers. Earlier this year, we found one of those projects: Walmart Android Photo (codename: Blixt). This is the story of how two engineers, leveraged the massive scale of Walmart, and made a huge impact. Planning Blixt Earlier this year, my partner on the business side came to me and asked if we could “sneak-in” photo support for the Walmart mobile app. At the time, a lot of our people were working on a mobile Pharmacy (a “mega-project” many of us were involved with as well). Finding volunteers was easy: photo was an interesting and relevant project. We put together a two-man team: a summer intern we turned into a node.js developer and a senior Android developer. Our first task was to align the team on what we were building. We started with “page one” specs that defined the basic spirit of what we wanted to make. We prototyped at the same time. We reviewed and iterated on each other’s specs, demo‘d prototypes, and set down a clear roadmap of what we could build, incorporating additional feedback from the larger team. From all of this, we created a pitch demo and a plan of what we should build. This was presented to our execs. The execs told us to go for it, but advised us to “stay small and scrappy.”

Stealth project. Define it. Chip away at it. Be scrappy. Business has your back. Engineering is in the driver’s seat. Own it end to end (client and services). What was there not to love?

Building Blixt

Every week, more than 140 million Americans visit Walmart stores or Walmart’s web site. APIs that connect to our stores and our customers create massive leverage for us in engineering. For Blixt, we tapped into a service that allowed us to print photos at more than 3,400 stores. Using the service required that we write some additional services code that would transiently store and print photos at the store. We called this Blixt Server. Blixt Server was written using the hapi framework in node.js. At its core, Blixt Server handles image uploading, printing, image cropping, and notifications. Uploads are stored in a replicated Riak database. Printing is handled through the previously mentioned store printing API. We’ve implemented server-side cropping/zooming using node.js GraphicsMagick / ImageMagick libraries. Notifications are done via Walmart e-mail services and Google Cloud Messaging (Android). We’ve got load balancers in front of a scalable cluster of VMs, a rolling-stage environment that allows us to rationally upgrade software, logging, and more. We started with the premise that we wanted to give users a way to choose which photos they liked and easily order prints from within the app—a workflow we dubbed “pick and print”. With that one directive, we gave our Android developer free reign to create the feature as he saw fit. When he was done, our UX folks were pleasantly surprised that the design was mostly there. It was truly amazing to see the improvements in our workflow and visual design once our designers got their hands on the feature.

Blixt Client on Android is made up of a few basic components: local photo gallery, cart, cropping UI, uploader, order placement, and notifications. It was critical that photo gallery performed well even with a high number of images on the device. We required the uploader to work in the background. The uploader also needed to be incredibly robust with built-in backoff and retry, as well as cached re-uploading. Specific print sizes could be specified for all images or a la carte on a per-image basis. Prints could be cropped and zoomed. Users could specify the store location at which to pick up their order. Users would receive native push notifications when orders were confirmed and ready for pick up in-store. All of this talked to the Blixt Services via RESTful interfaces that were jointly defined by our client and service engineers. I was happy to see tight collaboration on the API specification.

Shipping Blixt — It Ain’t Ready Til It’s Ready

One of the most and fun and unique things about working for Walmart is the connection of our mobile applications to physical stores. Even us techies get to go to stores, talk to store employees (Associates), and try stuff out. I remember how much fun it was to do our first in-store, end-to-end, smoke test. Watching the first prints get uploaded from an Android phone and printed through an in-store printer was a very gratifying experience for the team. We fixed a couple bugs and talked to the folks staffing the Photo Center about their process and what we could do to make things better. More importantly, we got to be customers for a day. Standing in the shoes of our store Associates and customers was a great empathy-building moment.

A few days before deploying to the Google Play store we ran one last round of usability testing. As a team, we had used the product a zillion times and, printed countless photos. Overall, we felt confident that we had built the right product. Good thing we ran that last usability test. Out of 12 subjects, 11 couldn’t figure out the “bulk order” screen, a fundamental part of the order flow. We asked our usability expert, the same woman who commissioned the tests, how bad it was. She showed us videos of the test subjects trying to get through the screen. Ugh. It was agonizing, especially given the timing.

After the findings, our small team huddled and worked through a range of options. We settled on a plan of action that would delay our ship date by a week. As someone who cares deeply about what we ship, I was happy was to see the team rally around the problem and fix it. So How Did We Do? Alight, enough cheerleading.

How did we actually perform? Performance for Android alone, as of September 9, 2014:

• Android is currently outperforming desktop web by 2x.

• Growth is purely organic. No marketing. Soft launch.

• Services uptime and client quality have been very good.

• Financial metrics (private) have far exceeded expectations.

With the addition of an iOS client (in development soon) and marketing support (fingers crossed), our numbers could soon rise to 6x to 10x of the already amazingly high numbers we are already seeing. When the holidays arrive (our busiest time), we’ve forecast 6x on top of that! Over the course of running the Blixt Service for the past couple months, the uptime has been really good. Bugs happen and in every case development, QA, and dev ops have done an amazing job at keeping the system up for the customers. Whenever you ship a new production service you have a period of time where you are watching it scale, and look to optimize the footprint. The initial demand for this feature has been 5x what we originally forecast. It’s been great to watch Blixt server evolve and scale to meet this unexpected challenge. With the growth, we once again found some interesting issues that we could quickly debug and fix. For this article I’ve focused mostly on engineering. Given the nature of the project, it was possible for the core engineering team to deliver the bulk of the work. That being said, the Blixt team is standing on other people’s shoulders, who are standing on the shoulders of our giant company. Mobile design, usability, quality, dev ops, and business folks gladly came in and contributed to this project. We have a truly amazing mobile team and we work for a truly amazing company. If it weren’t for the APIs that allow us to print to photo centers, there would be no Blixt. The fact that every week we have over 140 million customers, many of whom visit one of our 3400+ photo centers is just… mind boggling. For the Blixt team, Walmart really delivered on its promise of scale. Even a really small team, a core team of 2 developers, can have an enormous impact.

About the Author

“Software engineering and making great mobile products are my passions. It still feels like it’s early days for mobile. It feels like the best is yet to come.” Lives in Belmont with his two boys (10 and 12), fiancée, and dog (Augie). Loves music and art.