How can you jugde the effectiveness of an organization's requirement processes? What key performance indicators would you use?
For the purpose of this question, requirements processes are processes that produce or amend requirement specifications, or which render requirement specifications redundant.
Clarification added April 18, 2007:
It is generally accepted that incomplete or incorrect requirements lead to an increase in the cost of satisfying them, so I would like to focus on processes that are intended to deliver the requirements in time, rather than later.
As a general rule, the effectiveness of any process can be gauged by the value added as a proportion of the costs of the process. The major challenge in this case appears to be identifying how much value is added by the requirements, rather than how much it costs to add that value.
Clarification added June 28, 2007:
First of all, we need to be clear what we mean by the "requirement process". Some people advocate writing tests for requirements before trying to build anything. In part, the rationale for this is that trying to define the tests is an effective way of detecting defects in the requirements. You might then regard the test definition process as being partly a part of the requirement process. By extension, you might regard the whole life of the system, in development and in operation, as being, to a greater or lesser extent, part of the requirement process.
Let's confine our attention to a process that is primarily concerned with identifying and specifying requirements. What is the point of such a process? How does it add value for the system's stakeholders? What does it cost?
Cost is perhaps the easier question to tackle first. The principal costs of a requirement process are the manpower and other resources used by the process and the duration of the process itself. From a cost perspective, I would regard as "effective" a requirements process that uses less resources than it saves and saves more time than it takes. In theory, we should be able to come up with precise measurement of these costs, but we cannot, even in theory, measure the savings directly. Consequently, we cannot measure the effectiveness of the process without adjusting our definition of "effective".
Answers (9)
John L. E
♣ LION@LOCRIS.CO.UK ♣ 30K FCMI FIMIS FIBC MBCS CITP MIoD MCIPS ♣ Interim Director ♣ Change Management ♣ TOPLIN
Best Answers in: Using LinkedIn (4), Job Search (1), Professional Networking (1)
The quality of the people they hire!
Links:
Richard F. Z
Experienced Business & IT Executive
Best Answers in: Corporate Governance (1), Organizational Development (1)
Dear Alan
As a general remark: don't trust KPI's for the long term (but sometimes they are useful).
When we setup a requirements definition process we ask to calculate the bottom line impact vs. cost of implementation. Requirements without a comprehensible calculation or a reference to a set of requirements with comprehensible overall calculation are rejected. This initiated recurring process will increase dramatically the quality and the effectiveness of accepted changes planned and released into organization and IT.
Kind regards
Richard
Steven Miles [LION] S
Director at Grampian Outdoor Pursuits LTD
Best Answers in: Project Management (6), Organizational Development (3), Quality Management and Standards (2), Education and Schools (1), Occupational Training (1), Staffing and Recruiting (1), Criminal Law (1), Business Development (1), Lead Generation (1), Change Management (1), Planning (1), Inventory Management (1), Interface Design (1), Career Management (1), Ethics (1), Enterprise Software (1), Computers and Software (1), Computer Networking (1), Information Security (1), Telecommunications (1), Web Development (1), Using LinkedIn (1)
Delta between requirements definition at start of project and requirements definition at close of project.
David N
Technologist, Evangelist, Entrepreneur, Inventor ► Executive Consultant, CTO
Best Answers in: Using LinkedIn (229), Software Development (21), Computers and Software (20), Web Development (17), Customer Service (10), Staffing and Recruiting (10), Career Management (10), Small Business (7), Enterprise Software (7), Project Management (6), Quality Management and Standards (6), Starting Up (6), Accounting (5), Business Development (5), Organizational Development (5), E-Commerce (5), Computer Networking (5), Wireless (5), Business Analytics (4), Professional Networking (4), Certification and Licenses (3), Internationalization and Localization (3), Internet Marketing (3), Public Relations (3), Customer Relationship Management (3), Distribution (3), Market Research and Definition (3), Ethics (3), Telecommunications (3), Purchasing (2), Job Search (2), Mentoring (2), Mergers and Acquisitions (2), Compensation and Benefits (2), Corporate Law (2), Intellectual Property (2), Advertising (2), Writing and Editing (2), Corporate Governance (2), Planning (2), Non-profit Management (2), Manufacturing (2), Product Design (2), Franchising (2), Blogging (2), Regulation and Compliance (1), Travel Tools (1), Education and Schools (1), Freelancing and Contracting (1), Conference Venues (1), Budgeting (1), Venture Capital and Private Equity (1), Economics (1), Financial Regulation (1), Risk Management (1), Personnel Policies (1), Exporting/Importing (1), Treaties, Agreements and Organizations (1), Finance and Securities Law (1), Employment and Labor Law (1), Direct Marketing (1), Events Marketing (1), Graphic Design (1), Change Management (1), Labor Relations (1), Equity Markets (1), Inventory Management (1), Personal Taxes (1), Branding (1), Positioning (1), Biotech (1), Databases (1), Information Security (1), Information Storage (1)
In addition to using defined KPIs, watch for and identify any problems with the current processes...
- late delivery of products,
- budget over-runs,
- poor quality,
and/or staff problems...
- poor staff morale,
- reluctance of people to take responsibility,
- meeting-dominated process.
and/or problems in process understanding.
Chris Phillips-Maund (
[LION] 4.5K+ Management Consultant and owner FUnK IT Consultancy (TopLinked.com)
Best Answers in: Project Management (4), Internationalization and Localization (1), Change Management (1), Organizational Development (1), E-Commerce (1)
Using KPIs in this instance could become dangerous (as in most cases); trying to measure the value add over cost of the requirement process could trigger the workforce to produce less requirements - to raise their efficiency ratio. Unfortunatley this may have an adverse effect of higher costs through re-work later on in the project.
For me, you cannot measure the effectiveness of the requirements process in isolation you need to measure this with regard to the total project lifecycle i.e. initial costs of the requirements process plus any requirements rework at a later date. It is also very difficult in the later stages to capture rework on requirements as the project team are focusing on project delivery any requirement rework may get subsumed into test/delivery - discipline is important here; but also having the time a luxury we don't always have!
Are you asking regarding value-add regarding the requirements process or the requirements (I have assumed the former for the statement below).
Over time you will be able to derive a picture of value-add of the requirements process (this will not be feasible from a single project) but I suggest it will be more evident that if the requirements process is not effective you will be able to identify the additional cost associated with rework or uncertaintly (this will help you gauge value-add of an effective process, reduction in rework costs).
Regards
Chris
Peter J
Corporate Marketing & Sales Manager at London College of Management Studies
One measure of effectiveness is precisely how many changes are needed downstream after the initial requirements are produced, and the cost of those changes.
One set of variables involved tend to be with the client, and the culture they operate with. Some will say blow the cost, the end product has to be 100% right, others are more prepared to compromise. Collecting metrics can lead at least to clients understanding the need to comply with requirements discussions in the first place.
There is then further merit in monitoring issues arising and the impacts they have on requirements. Again, some issues can be dealt with, others less so. Again, metrics can show clients it is beneficial for them to stay involved in the process of managing requirements against impacting issues.
The final factor must lie with the quality of the initial requirements produced. How easy are they to understand and agree ? Have they got performance indicators in where third party suppliers are concerned ? Are they digestible for smaller suppliers ?
If using PRINCE2, has it been adapted successfully for the project in question ?
The above factors can all be assessed, listed, agreed and reviewed post-project through a standard issue log, which with the risk log often tells a true story for each project.
Hi Alan
If your key challenge is "identifying how much value is added by the requirements" - your priority should be to tie the requirements to the business case for the requirements process. (Don't have one? Start there). Trace the requirements process to the 'benefits' components. Do this by identifying how the benefits would be realised financially and non-financially. Work out what is 'valued' by your sponsors/stakeholders - design indicators based on this - i agree with Richard (earlier post) that dont trust KPI's alone as measure, rather the 'process of benefits realisation' is key. - allow atleast a couple of iterations if this is for large/public initiatives.
Cheers
Nitish
www.essencenetworks.com
Kevin B
CIO / CTO, ITIL and IT Governance Consultant @kevinbehr on Twitter
Best Answers in: Organizational Development (3), Computers and Software (2), Business Analytics (1), Change Management (1), Project Management (1), Career Management (1), Information Storage (1)
An effective requirements gathering and vetting process will cost less money than listening to flight recorders, piecing together the wreckage and trying to figure out what made the plane crash.
or my pithy phrase " Clear vision and controls are always cheaper than pathology and archeology"
The better understanding executive management has of IT as a constrained resource the more it will be certain that projects released in to the process actually diminish or eliminate real business constraints.
Measure twice cut once is ALWAYS less expensive. Test definition (how do we know that this project meets the business goals) up front allows defects (though, planning or execution) to be caught earlier in the project life cycle. The earlier they earlier they are caught the less expensive the repair costs will be. Not to mention less rework and resource thrashing will occur in the total life of the project.
Also determining what methodology of PM will be used to best suit the projects nature. Large complicated projects should use critical chain pm versus smaller low risk projects could use PMI BOK.
for a rear view look I use some simple metrics to judge the effectiveness of the process:
Projects into production on time
Percentage of projects delivered with initial QA acceptance
Percentage of projects delivered with initial production acceptance
Percentage of projects delivered with initial security acceptance
Change success rate of software and infrastructure changes implemented
Percentage of unauthorized changes due to project work
Percentage of unplanned work created by project implementation
Percentage of unplanned work create by projects in production (warranty period of production operation).
these help me judge the effectiveness and continuous learning progress of the requirements / testing process.
hope this helps,
kb
Andrew M
Requirements Architect, Idea Geek, Problem Solver and Transformation Agent thinking about new challenges
Best Answers in: Change Management (1), Organizational Development (1), Software Development (1)
I'll answer your first question and ignore your second as part of answering your first.
How would I judge the effectiveness of an organization's requirements process? Mine is more of an evaluative criteria for an effective requirements process.
Your requirements process should include the following:
* change is built into the process with lots of openness to change at the beginning getting tighter and tighter towards the end. It never completely closes, but the cost of change versus the cost of continuing on should be clear.
* you must start with a clear notion of the value you're looking for. Without this you have no idea how much value is enough. Scope, BTW, is not about how many requirements we deliver, it's about how much value we deliver.
* you must be able to steer the process and change direction often. Small milestones in requirements as well as all parts of the project are important.
* Separation of business requirements from any technical requirements. If you don't understand, from an abstract, technically-agnostic level, why the business needs a project (what value they're looking for to achieve what goals), and what steps they need to achieve these goals (what are the tasks, activities and events necessary to achieve these goals), then you'll never really know if your project is really delivering the right solution.
I could go on, but if you find these things alone, you've found a highly effective requirements process. It all comes down to delivering value. You have to know what value you're delivering, how much of it you've delivered, and what you will achieve for the "business" (or user or customer) in the end. Your requirements process should gather and communicate the requirements with these goals in mind.
Further bits of hard-won wisdom:
* It's all about the conversation. Written requirements are merely placeholders for the right conversation with the right person at the right time. They help drive that right conversation, and recored the results of it for common validation. But in the end, it's really about the converstiaon.
* Use professional business analysts who have the skills and experience. A skilled business analyst can be put into any situation and be successful if they have the right skills. You do not need to be a subject matter expert, and sometimes it's a handicap to be a SME because you get trapped into the same old way of thinking that caused the problem to begin with.
That should be enough to confuse anyone. Enjoy!
