Use Case: Forecasting Agile Projects with Agility GPT

Hey there,

Today's newsletter is a collaboration with Frank Schultheiss, author of "Agile Habits" and Director of Agile CoE at Capgemini Germany.

Together we have created Agility GPT - a custom GPT packed with 7,000 characters of instructions, 200 lines of Python code, and 10 knowledge files to help you make better forecasts for your agile projects.

So if you're a project manager or scrum master, read on! If not, you might still find this episode useful as proof that custom GPTs can be used for more than just "chat with your documents" and "AI girlfriend" apps.

Let's go!

Agile projects are a bit like the weather

If you want to forecast any work or a project which is organized in an agile way, it’s a bit like trying to predict the weather for a camping trip - important, but not always easy.

Agile projects change a lot (hence the name!), and figuring out how long it will take to develop a certain amount of stories, features, or even epics can be really tricky. Also, every team works differently, and things can change quickly. This makes it difficult to plan accurately.

Despite these challenges, being good at forecasting is very important. Especially in large organizations, where delays can cost millions every day, it is a must. It's also key to keeping your promises to (internal) customers about when you're going to get it done.

To make this a little more tangible, let's consider a concrete use case.

Situation and Problem Statement

Imagine you're responsible for an agile value stream improving an existing software platform. The product owner shows up in one of your meetings and says you " definitely " need to include this super cool feature he's heard about from a competitor (did I hear someone say AI?). At the same time, of course, it is critical to ship the next release as planned. So what do we do?

In agile value streams, we often use "story points" to estimate how much work a feature will take. Typically, the team comes up with a certain number of story points using a method like planning poker (the agile crew has even cooler words than the AI people).

Let's say the team agrees that this new feature will have about 500 story points in total.

You would assume that more story points (= more complexity and other dimensions) correlate with the time it takes to complete the amount of work (= cycle time). Unfortunately, this is rarely true in practice, and that's why forecasting is so hard.

To solve this, the solution would be to dig through historical data and perform a thorough analysis combined with a proven forecasting methodology to inform planning. However, because project managers rarely have the time and sometimes even the skills to do this, they often fall back on using the story points as a time estimate. Coupled with an over-reliance on past performance, this ultimately leads to poor forecasts.

That's where I teamed up with agile expert Frank Schultheiss to create the world's first Agile Forecasting GPT to help solve these problems.

And here's how it works:

Solution Overview

Agility GPT lives (currently) behind the ChatGPT interface which means you need a ChatGPT plus subscription to use is. But looking forward, it could be as simple as a dedicated chatbot in Microsoft Teams or even Jira.

Agility GPT works by combining three core components:

• Detailed instructions to guide the user through the forecasting process

• A custom Python script file that contains the forecasting logic

• Curated knowledge documents to provide more context

Here’s a high-level overview of the architecture:

There's absolutely no technical knowledge required to interact with the GPT! You just need to be familiar with agile project management.

Solution Breakdown

Let’s explore how this solution works by walking through the different stages.

Data Layer

To make it work, the GPT needs some data. Right now, the easiest way is to take an export from a project management tool like Jira and upload that data as a CSV or Excel file to the GPT. Mandatory columns are 'Issue Key' and 'Resolution Date', optional columns are 'Story Points', 'Start Progress', 'Issue Type', and others. Each row contains a single issue. This is the default export format in Jira. Here's what it might look like:

Looking ahead, this could be simplified by connecting directly to the Jira API.

When the data is uploaded, the GPT will perform some sanity checks and apply some pre-defined data preparation logic, such as removing duplicate Issue IDs and handling Issue IDs without a resolution date, etc.

Analysis Layer

Within the analysis layer, the GPT can use the custom Python code along with instructions and knowledge documents to guide the user through the forecasting process. This includes three options:

1. Extended Data Analysis

2. Throughput Based Forecast (issues per time unit)

3. Velocity Based Forecast (story points per time unit)

Option 1: Extended Data Analysis

In this option, the GPT analyzes the historical throughput and velocity of completed issues from the forecast dataset. This step allows the user to understand how throughput and velocity are correlated, if there are any historical outliers, and ultimately which metric - throughput (issues resolved per time unit) or velocity (story points resolved per time unit) - is more appropriate for running the forecast.

Here's what a Throughput vs. Velocity chart could look like:

Going further, Agility GPT calculates the variation for both metrics and informs the user which one is better suited for the forecasting task:

Another way to look at this is cycle time - the amount of time it takes to complete an issue. If story points were a good proxy for forecasting, there would be a high correlation between story points and cycle time. If they don't, the GPT will point that out.

After the extended data analysis (which takes about 5 minutes), the user will clearly understand whether to use velocity (story points) or throughput (e.g. user stories) for prediction.

The user can switch to prediction mode at any time. Let's switch gears and see how AgilityGPT performs prediction

Option 2/3: Forecasting

First, the GPT asks the user how many story points or issues they would like to predict the completion time for.

Let's take our 500 story points example above. The GPT will start with a static burn down chart forecast - familiar to every agile coach and project manager.

The burn down chart can be used to estimate how long it will take to complete the work based on average historical performance. Agility GPT does this and also explains the limitations of this approach:

Next, the GPT runs a Monte Carlo simulation that takes into account not only historical performance, but also some variation in the delivery pace and scope increase. As you can see, one burn down line now becomes 10,000 burn down lines, with a few 100 randomly plotted:

Suddenly 4 month completion time does not seem so likely!

Next, we can create a histogram of completion times, giving us a better estimate and probability of when the work will actually be done.

As we can see, with 85% probability, 500 story points will be completed in 6 months or less.

Understanding Percentiles

• Probability comparison: The 50th percentile, also known as the median, is like a coin flip in terms of probability, meaning there's an equal chance (50%) that the actual outcome will be either below or above this value. Obviously, planning based on the average cannot have an above-average probability of occurrence. That's why planning often fails.

• Usage: When more confidence is needed than a simple coin flip (50%), higher percentiles (like the 85th or 90th) are used to ensure a higher probability of the outcome falling below the forecasted time frame.

User layer

Since the user is interacting with a full-fledged LLM, the beauty is that they can ask any question they want about these plots!

For example, if you're struggling to interpret a particular plot or are confused about the relationship between cycle time and story points, you can simply ask the GPT and it will answer these questions based on curated expert knowledge and your own custom data:

To be clear, before the era of large language models, something like this was completely out of scope. But now it's within reach. While there's definitely room for improvement, the GPT already works quite well.

Conclusion

Agility GPT improves agile planning and forecasting by integrating AI-powered forecasting and easy-to-use interaction.

By empowering users with insights such as cycle time, throughput, story point reliability, and scope completion estimates, Agility GPT is a straightforward solution for agile coaches, scrum masters, and project managers striving for delivery excellence and efficiency.

I hope you enjoyed this deep dive into the world of agile forecasting. And don't forget:

Reply with "Agility GPT" if you want access to the GPT and I'll send you the link!

Can’t wait to hear your feedback!

See you next Friday,

Tobias