Why this mattered

Leveraging User Research Insights to Influence Product Strategy

User ResearchService Design

Nov 21

In the late-2018 Pivotal Application Single Sign-On (SSO) team, composed of a Product Manager, Product Lead, Engineering Lead, Engineers, and a designer (me), started to explore opportunities in aim to improve application developers’ identity management experience while working on applications on a multi-cloud platform.

My role as the design lead

• Understanding the problem space.

• Collaborated on planning and leading exploratory and quantitative research.

• Co-established the product’s direction from understanding the user’s needs to the conceptualization of an alpha version release for a limited customer base.

• Facilitate cross-team discovery & validation research initiatives.

What is application Single Sign-On on Pivotal Cloud Foundry?

Pivotal’s flagship product, Pivotal Cloud Foundry (PCF), is a multi-cloud platform that abstracts away the process of setting up and managing an application runtime environment (infrastructure) so that developers can focus solely on their applications and associated data.

The Application Single Sign-On (SSO) service running on PCF lets users log in through a single sign-on service and access other applications (for e.g. my Twitter username is my identity to log in to twitter application) that are either hosted or protected by the service. This improves security and productivity since users do not have to log in to individual applications.

View fullsize

App SSO allows applications on the PCF platform to integrate securely with enterprise identity providers.

Current Landscape of SSO

SSO tile (software) only integrates with applications that run on the PCF platform. These applications need to be written using a standard set of languages for them to work. For every application which is using a different framework:

• Developers must research the best security practices and tailor an in-house solution for their specific tech stack.

• This not only leads to more engineering hours being spent but also more risks being taken to build and maintain a solution that lives within an application.

• Ideally, none of these issues should be a concern for application developers.

Product Strategy

Based on the prior knowledge formed by the customer conversations and market overview, the product team defined product strategy to provide speed and security by letting application developers focus on delivering business value while abstracting away authentication and authorization to the platform.

With these factors in mind and based on the past customer conversations and the existing SSO journey, we created a potential future customer journey to understand what pieces remain, what pieces change, and what pieces need research.

SSO proto user journey to identify gaps and opportunities.

That led to the problem statement—

How might we make SSO application language agnostic and improve the delegation/self-service of security policies so that more organizations can fit SSO tile with their enterprise working model with confidence?

Assumptions We Held

Before we could tackle building a better experience, we may have to shake up some of the assumptions holding us back. I led the workshop with stakeholders to generate assumptions and hypotheses with a prompt of

What do we believe to be true for the application developer experience when implementing SSO?

Assumptions session with stakeholders to surface the team’s collective thoughts.

Hypotheses mapped by risk and validation effort in 2x2.

Why did we decide to talk to our users?

To validate many of our hypotheses, we planned to do a round of user research to understand how identity security policies are managed at different companies.

High-level goals were to:

• Understand how SSO is adding value for our customers’ application developers when they build applications that require identity-based security. (Identity-based security allows organizations to grant access to specific users to a variety of digital services using the same credentials, ensuring the accurate match between what users are entitled to and what they actually receive, while also permitting other access constraints such as company, device, location and application type.)

• Identify what are known limitations and blockers of the existing SSO product for customers and potentially identify new ones

• Validate the overall product direction

As we had many questions to answer, we separated our research into two persona-based phases Operators/Architects and Application Developers over 5 weeks.

Why Platform Operator and Application Developer personas?

SSO has two primary enterprise personas that leverage its services. Operators configure secure enterprise Identity Provider configurations for Application Developers to consume.

Application developers utilize self-service capabilities to create application security policies and use low-touch integrations with Spring Boot (Java) or Steel Toe (.NET) frameworks to auto-configure their applications.

…and the research began

To get the team to align on the research effort I created a research plan to capture research objectives with clear milestones.

A research plan, with recruiting strategy, research timeline, and learning goals.

Expert Panels and Deep Collaboration

Shortly into the development of the research plan, I was concerned about all the unknowns. Like, are we validating/invalidating the right hypothesis? Are we asking the right questions? Are we reaching out to the right people? And some more.

We formed an expert panel to serve as customer proxies throughout the process. This panel consisted of platform architects, and product lead with two attributes: deep expertise in identity policies for applications, and a close connection with customers.

“Working with customers and prospects, you hear a lot of comments and pain points,” says Mark Fynes, customer representative and one of the subject matter experts on the panel.

The expert panelists played an integral role in product development. They participated in the creation of a research plan, interview practice runs, and with helping validate — and invalidate — assumptions about the domain and user behavior. The result, better guidance for the eventual success of the product.

Phase 1 — Platform Operators

For the next two weeks, we conducted exploratory research remotely and in-person with Platform Operators and Security Architects from 11 customers across different industries.

Method: We split our research conversations with customers into two sections:

• In the first part, we asked general questions about their roles, existing solutions, and usage of SSO in their organization.

• In the second part, we ran an experiment to understand their user journey across their current working model and ideal working model.

An experiment in the form of a storyboard to capture users’ current and ideal user flows.

How did we synthesize?

After each interview, I conducted debriefs with the attendees to captures our observations and takeaways. After every three meetings, I led mini synthesis sessions; this helped us to start seeing the patterns from early on. As we started seeing what details were missing, we tweaked our interview script for the rest of the interviews.

We organized data around themes to help us go from group knowledge to start building a shared understanding of our users’ needs. As a group, we identified high-level themes and insights, with each insight supported by data points and quotes.

Affinity map was an exercise where we mapped out insights, clustered them together, identified connections, and understanding why the problem existed.

Phase 1 Research Takeaways

• Insights from customers’ mental models helped guide our technical research direction for phase 2 with the application developers.

• We learned that we could have collected basic info before each interview session, which would have helped us in personalizing the script a bit.

Phase 2: Application Developers

Learning goals focused on application developers.

• What are application developers looking for when they build an application that requires identity-based security?

• Understand known limitations and blockers of the existing Pivotal SSO product while identifying any new pain points.

• Capture first reactions to the overall product direction.

We spent a little over 2 weeks to talk to Enterprise Application Developers from 18 customers (both SSO and non-SSO users) across different industries.

How did we gather common data and customize each customer conversation?

We collected initial closed-ended questions from our interviewees through a survey to help us align our experiment with their use cases.

We used Google Forms to collect data. 18 of 19 interview participants and one survey-only participant filled out the survey before their sessions.

How did we frame the user experience for our research?

Based on our understanding of the existing application developer user experience and the future envisioned developer user experience, we created a mock user experience journey.

Learning Goals

• Verify our assumptions about the Web App and API flows for existing and future developer experience around use cases that apply to application developers attaching end-user identity to their apps.

Method and participants

18 customers reviewed the prototype, one with extensive experience with SD-WAN and all others with some understanding.

The Study

We split our research conversations with customers into three sections:

• In the first part, we introduced the use cases of building a web application and building an API that requires end-user authentication.

• In the second part, we talked through the developer user journey using Spring Boot, an open-source Java-based framework, today. (We skipped this part if a user did not use SSO.)

• We gave participants a set of tasks to attach end-user identity to their apps to test the coherence of information, workflows, and relation between different parts of the application.

At the end of phase 2 research, we identified high-level themes and insights, with each insight supported by data points and quotes.

Influencing Product Strategy

Once we’re done with the synthesis and discussing the individual clusters and issues, I sat down with the product lead, product manager, and engineering lead and addressed the product strategy. I described the problems we observed, what we make of them as a team, and give recommended next steps for the product and the roadmap.

Based on the design recommendations, the product management decided to update the SSO roadmap to elaborate on the various milestones and MVPs for the alpha version of a product release with updated user flow.

Other recommendations turned into updating the information architecture of the SSO Application Dashboard as well as the internal technical work in the backlog.

Prioritization of epics influenced by the research insights in the product roadmap.

What lessons did I learn in this journey?

• Develop and maintain anti-goals. A nice counter to goals is “anti-goals.” What is something that is not a high priority that would distract the team today? It helped to align us to avoid any unnecessary or unwanted surprises down the road.

• Use visual storytelling! The experiment formatted as a storyboard was able to prompt a lot of insight from customers.

• Use surveys! By asking close-ended questions in a pre-meeting survey, it helped us narrow down what open-ended questions to ask during the research interview. We were also able to get feedback by using the survey as an alternative for those customers that were not able to commit to a research interview session.

• Adopt Lean UX! As we learned during the process, we mapped our findings and related tasks to existing roadmap milestones and topics.

• Make research a team event! Highly encouraged the development team to attend meetings and participate in research synthesis as it builds customer empathy and alignment towards solving customer problems.

• Complete the circle: It’s equally important to talk to people who are influencing the product, such as Security Admins, CIOs, Application Admins, etc. than users who are using the product daily. As many times influencers are the decision-makers

Tian Wang (Product Manager) and I co-presented on Application Identity specific use cases at Cloud Foundry Summit 2019. We shared our learnings with native application developers across different industries.

Edit Page

Previous

This led us to think about ways we can make the UI for Prompt Builder better for users.