Recapping LFI Conf 2023

LFI is a community challenging conventional views and reshaping how the software industry thinks about incidents, software reliability, and the critical role people play in keeping their systems running. The Learning from Incidents in Software community recently gathered together in Denver over mid-February for LFIConf 23. The conference was a forum of top experts, researchers, and industry leaders to share stories of incidents, incident handling, and learnings. Software engineers who handle large-scale distributed software systems shared stories of how they are making learning from incidents and continuous improvement core tenets of their organizations’ approach to operating their services.The conference was powered by Jeli, in partnership with sponsors Indeed, Verica, Adaptive Capacity Labs, FusionAuth, Etsy, Great Circle, Honeycomb.io, Metrist, Nobl9, Sleuth, and Steadybit.

Jeli Founder & CEO, Nora Jones, kicked off the event with her opening keynote, during which she asked the audience to join her in an exercise comparing two different incident reviews conducted for the same incident. The first used a traditional template while the second was an in-depth review of transcripts that led to creating a data-rich timeline surfacing insights from contributors across the organization, leading to a much deeper understanding of the incident, what actually happened, and what needed to be addressed in order to prevent the same incident to reoccur. The second incident review revealed the stark difference in a basic incident review template in order to “close” the incident, versus leveraging the existing information available within your systems in order to approach incidents with a mindset of learning and continuous improvement. We saw the risk a basic incident review presents and the value in-depth reviews hold in building more resilient infrastructure, systems, and teams.

Throughout the two-day event, we heard from industry thought leaders as well as first-time speakers, together offering a multi-layered conference full of learnings from diverse backgrounds and experiences. Speakers shared their successes as well as failures– and sparked conversations around how to make the work of learning from incidents sustainable across organizations. We learned how Salesforce and Indeed learn from multi-party incidents, how Enova used learning from incidents in their DevOps transformation, and how individual contributors at Quizlet have been able to get involved in designing a process that works for folks at different parts of the organization. 

Creativity and ownership were major themes on both days. We learned how people were able to turn stagnant processes around, convince skeptical parties, and engage various stakeholders. Some of my personal favorite moments took place in the intentional hallway track and incident stories that expanded on the work we have done at separate CasesConf events. Attendees had a chance to share their personal experiences in a safe space; the result was a shared understanding of our work. Throughout the conference, attendees raved about the insights during David Leigh and Randy Horwitz of IBM’s talk on how they are transforming the practice of LFI in a 12,000 person organization.

On day two, John Allspaw from Adaptive Capacity Labs took the stage for the closing keynote and spoke on the progress of Learning From Incidents as a practice, highlighting the need of learning and understanding to reach the broadest possible audience. Allspaw then brought Jason Koppe of Indeed to the stage, who shared some of the success his team has accomplished with LFI, focusing on patience, investment, and strategic partnerships. Jason now leads a team of seven analysts focused solely on learning from incidents. It was a proud moment for me as a Jeli Bean, who has worked closely with Indeed’s LFI team along their journey. 

The event ended with Jeli Beans and LFI Conf co-chairs, Nora Jones and Dr. Laura Maguire, sharing some anecdotes from the start of the community in 2019, highlighting the importance of sharing our stories and finding common ground, as well as a touching anecdote on the lasting impact of the late Dr. Richard Cook who helped spark this movement. You can now view all the recordings from LFI Conf on YouTube!