What’s the indicator that determines the software development team's efficiency? It could be the performance metrics of a team and each developer individually if we measure it accurately. And we should. Developer productivity impacts the quality of the created digital solution and the speed of its release to market. If you outsource IT tasks, it is also essential to control the success of the software development service provider. So what is developer productivity, and why does the IT community focus so much attention on this topic?
- Measuring developer productivity: reality or myth?
- Possible barriers in measuring the tech teams' productivity
- Why do you need a system of metrics to evaluate performance?
- How to choose the right KPIs?
- Metrics to determine developer effectiveness
- KPIs for measuring team productivity
- Methods for measuring developer effectiveness
- Well-known Companies’ Experience
The productivity of an IT team consists of three aspects:
1 — create high-quality software in the shortest possible time,
2 — strive for a zero-bug-product,
3 — be easily scalable and maintainable.
It’s simple to understand but seems impossible to measure, right?
Let’s consider the realism of getting an objective assessment of an IT engineer's efficiency. We will discuss what metrics should be used for this and what a company that pays attention to such painstaking work will achieve.
Measuring developer productivity: reality or myth?
In today's IT environment, there is a vigorous debate about how realistic it is to objectively assess this metric. Some experts believe it is impossible to do so because any metric allows only approximate results that are more likely to distort analysis than help it. Some experts assure there is no way to reasonably measure performance. Others have an opposite point of view.
According to the Software Developer Productivity Survey results, more than 89% of developers believe that there are few effective ways to measure productivity. Furthermore - 91% of specialists would rather measure the results of their own work… somehow.
Possible barriers in measuring the tech teams' productivity
The first issue task a company faces when measuring the productivity of IT professionals is the choice of metrics. It is definitely worth paying attention to. If this stage is behind, however, a manager may be faced with these three challenges:
1.Lack of objectivity. Some metrics, such as the number of written code lines, cannot objectively reflect the engineers’ productivity. After all, the amount of work done work does not always mean its high quality. 2.Not obvious connection between the indicator and the financial benefit. It is easiest to evaluate productivity when there is a clear relationship between the metric and the company's revenue growth. Obviously, the software fault-tolerance and zero-bug state will help save money on fixing them, increase user loyalty, and thereby increase the company's profit. But this is only the end result. The process that led to it usually depends on communication between employees. It does not directly affect finances. For this reason, assessing the quality of communication within the team is extremely difficult, but this indicator is also essential.
According to the results of a survey conducted by the consulting company Gartner, it is optimal that 56% of the indicators were related to the business results of the team. That is, they should reflect the increase in revenue, return on investment, and implementation of the company's business strategy.
3.Contradiction in some metrics. If you want the team to work quickly, be prepared the product quality may suffer. If you want the developers to minimize software defects, don't demand unrealistic deadlines. The desire to combine speed and quality is understandable, but it must be achievable. Listen to your CTO or tech partner’s warning if they say the task is not doable.
Knowing the possible difficulties in IT engineers' performance evaluation, you can prepare in advance to overcome them and create a quality digital solution within the shortest possible time.
Why do you need a system of metrics to evaluate performance?
Measuring developer productivity enables us to understand what and how affects the process of creating a software product. As a result of obtaining such information, it is possible to influence each of the metrics in order to increase team productivity, improve the quality of delivered software, and reduce the speed of its release. By paying due attention to detailed research about the team, the company's management gains the following benefits:
Effective workflow planning
A detailed analysis of the development team activities allows you to evaluate the project in terms of:
- finances: calculate how much each stage of development (design, development itself, testing, etc.) costs and optimize the budget;
- time: analyze how much time it takes to write the same amount of high-quality code using different tech stacks or methodologies, and implement the optimal time-saving variant;
- quality: check the quality of the digital solution and use new technologies to increase it.
An in-depth analysis of the team allows you to plan the implementation of the project, taking into account its features and complexities.
Increasing team productivity
Clearly defined metrics enable developers to understand what is expected of them and constantly work to achieve better results. And the manager can track team problems and take action to overcome challenges to improve performance.
Improving IT product
This advantage is a direct consequence of the previous two points. Effective workflow planning and increased productivity of your developers will help achieve any team's main goal – to release quality software within the budget and without extending deadlines.
How to choose the right KPIs?
There are some universal recommendations for choosing metrics to measure the programmers’ productivity. For example, some experts advise using the SMART method. It stands for:
- Specific: achieve certain indicators and measure them more easily if they are formulated clearly;
- Measurable: it is better to choose KPIs that you can objectively measure (e.g., time to complete a task, number of defects in the software, etc.);
- Achievable: don't demand that the team create a high-quality app in a week; it is better to set a realistic deadline, so then the quality will not falter;
- Realistic: it is important to pay attention to the feasibility of the indicator, not only to its achievability. The team may be able to meet the deadline, working 24 hours a day and 7 days a week. But is it realistic, and is it really necessary?
- Time-bound: it is necessary to set a clear individual deadline for each task on the project. Without this condition, it would be unclear if each team member is working productively to their full potential.
Once you have selected the indicators you are interested in, determine which are appropriate to measure the productivity of the whole team and which types will better reveal the performance of individual employees. There is an opinion that it is not reasonable to evaluate the work of each developer because the success of the project is influenced by the whole team. This opinion would be valid in an ideal world, but in reality, without evaluating every IT engineer, you can't understand what the team's problem is if some aspect doesn’t work.
Metrics to determine developer effectiveness
The most apparent metrics you can measure are:
- number of lines of code,
- working hours,
- number of bugs fixed
- number of tasks completed.
Though experts argue that relying only on these metrics is inefficient because of the following.
Lines of code
A lot of lines of code cannot guarantee the quality of the final product. On the contrary, haste in development most often leads to poor code quality and problems later on. Working hours
Often, the best programmers work faster, which is not reflected in the product quality. At the same time, the desire to increase the index may induce specialists to overwork. This can cause the opposite effect – a decrease in productivity due to professional burnout and mental fatigue.
Number of bugs fixed
We need to bet on quality but not on quantity. Using such an indicator can lead to ignoring complex defects, which take a long time to fix. Naturally, this will affect the overall quality of the code.
Completed tasks
All tasks on a project differ in their complexity, the effort involved, the degree of importance, and the time it takes to complete them. It would be unfair to say that a specialist who completed five simple tasks is more productive than an IT engineer who spent a whole day implementing one tricky function.
These four indicators are easy to measure, but they must be used in combination. Each of them separately will not bring the proper benefit to your team, and sometimes they can even be detrimental, forming a false impression about the programmers’ work.
KPIs for measuring team productivity
Each team has to choose metrics depending on the goals and specifics of their project. Below we present a list of metrics to be measured in general situations.
Code quality and bugs
According to statistics, developers spend, on average, 20% of their working time on fixing bugs. Accordingly, the more bugs in the source code, the less efficient the team is. This makes managers put the quality/defects metric at the top of the importance of measurement.
Team speed
In addition to the quality of the released product, it is important to measure the time it takes to develop it. This metric is widely used in teams that practice Agile methodology; thus, project managers evaluate the amount of work done during one sprint. The results are useful for planning further work and predicting its results.
Please note: It is not reasonable to compare the performance of several teams based on this metric because their tasks may differ in complexity, and the result of the comparison will be biased.
Burndown
This metric shows the progress of the Agile team during the sprint. Monitor the ratio of completed to remaining tasks with the Sprint Burndown Chart. This chart was created specifically for Scrum and is designed to clearly show the project's current status.
Cycle time
This metric demonstrates the degree of flexibility of your team and its willingness to change. In other words, the metric allows you to determine how much time it will take for developers to make changes to the project.
MTBF and MTTR: mean time between failures and mean recovery time
The metrics show how long the created software can work between failures, how often these failures occur, and how long the team will solve the problems.
We combined these metrics into one because today's experts believe it is impractical to use them separately. Moreover, measuring them together with other metrics is more appropriate to get a more accurate picture of the IT team's productivity.
Defects Opened and Closed Rate
We should monitor the number of defects reported on the project and those that have been closed. This will not only help in measuring developer performance but will also ensure that no bugs are overlooked. After all, fixing them after release will cost 100 times more than in the initial stages of development.
Frequency of deployment/release
The metric helps track how often the team deploys or releases code to the production environment. Teams can set up continuous deployment to improve results and achieve the metric more than once or even twice a day. This will be their distinct advantage as they can work on projects that require more frequent releases.
Throughput
This metric is a more detailed analog of the "Team Speed" indicator. It also shows the amount of work performed over a certain period, but in terms of specific functions, tasks, mistakes, etc. This KPI allows the project manager to understand where the time was wasted during the iteration and to plan subsequent work.
WIP – work-in-progress
Unlike throughput, which measures a team's past performance, this metric is used to measure its current performance.
Communication Index
This tool for measuring developer productivity helps the project manager determine how well the communication within the team works. This is especially relevant if third-party developers are working on an outsourcing model in the team, as well as for different teams that are involved in the same project. Here it is important to assess English language skills, cultural differences, time differences, and communication skills. If one of these parameters appears to be at a low level, when problems with the code arise, it can be assumed that this is the cause.
Security Responses
This indicator is used to evaluate the effectiveness of the development team in security matters because it determines the profitability of the IT project and the demand for the app in the market. The resilience of software to cyber threats is an important component of its quality. After all, according to statistics, the level of cybercrime has increased by 600% since the beginning of the global pandemic.
End-user satisfaction
You can use different channels to measure this indicator, including specialized tools (NPS, CES, and CSAT), as well as consumer feedback in app stores (for mobile development), review sites, and thematic forums. If the rating of a software product is high, one can conclude that the team works productively and brings benefits to the company.
In addition to these metrics, the project manager must also monitor the overall processes within the team:
- How involved are developers during retrospectives and other meetings?
- Do IT engineers report emerging problems?
- How high is the team's motivation level, and is there a tendency for burnout?
Such attention to detail will allow the manager to timely notice problems in the team and promptly eliminate them to achieve the project's ultimate goal.
Methods for measuring developer effectiveness
To measure employee productivity in IT companies, often resort to methods that have proven their effectiveness in practice. They allow selecting of the necessary metrics and their combinations, as well as correctly interpreting the obtained results. Among the most famous methods are SPACE and OKR, but many well-known companies also use other methods.
SPACE
This framework was created taking into account that it is impossible to measure developer productivity using a single metric. In order to obtain objective results, the developers of the platform suggest evaluating the work of IT engineers in a complex, namely:
- Satisfaction of specialists. Employees who feel morally satisfied with their work are more productive and less prone to professional burnout and fatigue. That's why SPACE creators suggest monitoring the emotional state of developers, their availability of necessary tools for work, and the level of each specialist's workload constantly.
- Performance of the team as a whole, not just individual workers. The platform's authors believe that it is complicated to evaluate the contribution of an individual programmer to a project. After all, a high speed of work may mean low quality of code, the product's value may be not only in the absence of defects, and many completed tasks may indicate their ease.
- Employee Activity. Although it is difficult to quantify the performance of IT engineers, some metrics are still measurable. They can provide important information in conjunction with other metrics. These can include the number of pull requests, assemblies in CI/CD, test coverage, defects detected, and more.
- Communication within the team. We have already talked about such a metric as the communication index. This metric is also measured in SPACE.
- Effectiveness and flow. The metric allows you to determine how ready the team is to complete a certain action, minimizing downtime and delays. To assess this metric, we measure the time of interruptions in work, the team's ability to bring what they have started to completion, etc.
Please note: You should not include too many metrics in your performance measurement list. This can give developers the impression of an impossible task, which can cause their productivity to decrease.
OKR
OKR stands for Objectives and Key Results and is a method widely used in project management to keep tight control of the fulfillment of assigned tasks. It is aimed at measuring the developers' productivity on the basis of a set of metrics that each organization can define for itself. It may be the frequency or the number of code checks, the number of commits, and others.
It is worth noting that the method does not always provide enough information to evaluate the whole team's effectiveness, but it allows you to compare several IT professionals.
Well-known Companies’ Experience
Each company can choose a tool to track developer productivity based on individual needs and goals. See which world-renowned companies don't neglect to measure the productivity of their IT engineers.
- Uber created Eng Metrics tool
Recently Uber introduced an IT productivity measurement tool called Eng Metrics Dashboard into their workflows. This dashboard helps measure productivity based on pull requests, code checks, and concentration time per engineer.
- Google implemented GSM framework
Google has created a special team of researchers to measure developer performance, which includes social scientists and software engineers. At the beginning of the work, they choose which metrics are appropriate to measure using the GSM framework: Goals/Signals/Metrics. That is, they determine:
- The goal they need to achieve.
- The signal that will show that the goal has been achieved.
- Metrics that will help achieve those goals.
- Samsung, eBay, Nestle, Spotify chose the Jellyfish tool
Yes, all these world-renowned companies use one tool in their project management: Jellyfish. The platform aims to simplify the digitization process for companies that have identified it as their priority development area. It offers productivity metrics (including DevOps Metrics), consulting services, and training programs to achieve business goals as quickly as possible.
Conclusion
There are different views on whether the productivity of IT engineers can be measured. Experts cannot come to a common ground regarding the practicality of measuring the effectiveness of individual developers. Although they all agree on one thing: team productivity is a value that must be measured in one way or another. Only by knowing the values of the main indices is it possible to influence them and change the quality and terms of the delivered software.