Spaghetti and marshmallows — I wouldn’t recommend it for dinner, but it did make for a winning tutorial on how to build a successful predictive model at the Casualty Actuarial Society Ratemaking, Product and Modeling Seminar and Workshops in Chicago in March.
These mismatched foods were two ingredients in a team exercise that about 30 people participated in as part of a session called, “Getting Impact from Predictive Analytics: You Have a Model. Now What?”
The lesson to be learned: How the right kind of teamwork can create successful predictive models.
The Experiment
The audience divided into about a half dozen groups of five or six people. The challenge: In 15 minutes, build the highest tower of spaghetti that is strong enough to support a single marshmallow.
Each team got one marshmallow, 20 strands of spaghetti, a roll of masking tape and twine. Timekeeper was David Wang, FCAS, a data solution consultant at Zurich North America.
We fiddled with the materials for a bit — none of us were architects. We knew the spaghetti would break easily, so we created stronger spaghetti poles by doubling them up.
We turned the roll of tape on its side to make a base. One team member tore off strips of tape first, so we could use them later to lash the spaghetti together.
We started with a tepee, leaning three spaghetti poles into each other and taped together where they met near the top. Then we made another tepee and turned it upside down. It nestled into the original tepee’s top. This gave us a two-story tepee, with the inverted tepee on top, its spaghetti poles extended wide.
Then we added a third story, this time a right-side up tepee. The apex of the third tepee created a tiny cradle to hold the marshmallow.
We knew the spaghetti would break easily, so we created stronger spaghetti poles by doubling them up.
But the marshmallow didn’t want to stay still. It bowed the weakest spaghetti poles, and, while it never broke the structure, it wasn’t too sturdy or too pretty. The Pritzker Committee won’t be calling any of us soon.
Still, our tower was 24 inches tall, pretty close to what most of the other teams constructed. The tallest was 33 inches.
Timekeeper Wang says most groups build towers around 20 inches tall. Success, he says, depends on understanding that the spaghetti is fragile and the marshmallow is surprisingly heavy.
Lessons Learned
The key takeaways from this challenge are:
- Many hands make light work. “If you put capable people together, you are much more likely to pool insights and arrive at a better solution,” Wang says.
- Sometimes you can have too many people. The largest group at our session — about eight people — seemed to succumb to that problem. In a real-life project, they may end up creating meetings to go over unimportant details “that make you go to sleep.”
- Successful groups collaborate. Teams should have diverse skill sets and be seeking a common goal, Wang says.
The best tower builders revealed that they quickly broke into separate duties. Two or three were builders. One person did nothing but cut tape. Another only cut string. “It was a group dynamic,” one of them said. “We worked pretty well together.”
Real-World Projects
Building a predictive model is a lot harder than constructing a spaghetti tower, but it can fall prey to the same pitfalls. Projects tend to have people with similar skills. They huddle in low-efficiency meetings. Their responsibilities tend to be ill-defined.
Too often, a Big Project has a Big Buildup, and then becomes a Big Failure. The other presenter, Jessica Leong, FCAS, lead data scientist at Zurich North America, laid out a typical predictive modeling project:
- Spend a few weeks scoping the project, talking to stakeholders to learn what needs to be done.
- Clean up the entire data set so no data issues emerge.
- Do a one-way analysis to find variables that could potentially drive the model.
- Build the model carefully.
- Roll out the model.
If the rollout is a failure, she says, it is often because some of the most important lessons were learned during the construction process — the equivalent of learning that the marshmallow is too heavy for the spaghetti poles.
Too often, a Big Project has a Big Buildup, and then becomes a Big Failure.
Recognizing that the first stab at success rarely succeeds, Leong said a better approach has emerged from Silicon Valley in recent years — the lean startup method, which goes something like this:
- Start with a quick and dirty business plan — maybe one page long.
- Create a minimum value product — one that covers some of the bases but has obvious shortcomings.
- Launch that product (at least internally).
- Get feedback and return to step 1.
The method recognizes that the job isn’t to build the most awesome predictive model. “The job is to solve business problems using predictive analytics,” Leong says.
