What metrics do you use for technical writing?
I've found technical writing to be a challenging endeavor to quantify. When managing documentation functions, I've wanted to monitor/control quality and productivity. However, I quickly realized that the standard Page Count was a terrible metric: If writers are evaluated on the number of pages they produce, they'll give you as many pages as they can churn out, not as many as the deliverable needs in order to be optimally useful to readers.
I'm interested in what kinds of metrics others in the technical communication field have used, especially documentation managers, to monitor/control quality and productivity.
Good Answers (9)
Paul M. A.
Owner, PMA Technology Group
Best Answers in: Software Development (3), Staffing and Recruiting (2), Web Development (2), Government Policy (1), Work-life Balance (1), Advertising (1), Internet Marketing (1), Business Development (1), Writing and Editing (1), Planning (1), Product Design (1), Career Management (1), Green Business (1), Green Products (1), Using LinkedIn (1)
While not a titled documentation manager, I suspect that I have enough experience as a technical writer and editor of technical publications to justify offering some thoughts on the subject.
I would first agree that page count has no place in the discussion. There is frequently an inverse relationship between quantity and quality - at least when viewed in the context of two versions of the same document.
Beyond that, my suspicion is that while quality is easy enough to quantify (The document is either accurate, well written, and achieves the stated objective or it doesn't.), productivity can only be judged in comparative/relative terms (How does the writer's output compare to that of others producing similar materials).
The operative term is, "similar materials". Two different documents covering the same product, process. or procedure but targeted at different audiences can have significantly different time requirements even when both documents are created by the same author. Correspondingly, a document that requires only that the writer translate detailed data into readable, understandable text is quite different from one that requires considerable interpretation and research before the desired result can be achived.
At the end of the day, the so-called "metrics" become subjective determinations. How does the performance of different writers compare? Typically the star performer will meet the deadlines and get it right on a regular basis while others on the team might not be able to consistently deliver comparable quality and quantity.
At the end of the day, all writing, including technical writing, when done well is less of a science than an art and must of necessity be be judged in that context.
Keith O.
Principal Software Engineer at MasterPeace Solutions Ltd.
Best Answers in: Software Development (5), Green Business (2), Computers and Software (2), Information Security (2), Web Development (2), Starting Up (1), Energy and Development (1), Green Products (1), Blogging (1), Computer Networking (1)
A very challenging question. While I haven't done much documenting myself I have been in charge of documentation efforts and have had the same sort of interests. To me, the in the final analysis any document is only valuable if it answers the questions the user has so they are not required to make some sort of call to a support person. The ideal documentation essentially anticipates what the user wants to know and presents it as they seek it. Likely impossible in any real-world situation, particular with a static document, but I have read so many documents that were confusing and contradictory that I found them to be valueless and got my answers in other places.
Good luck and if you think you find something you think is valuable I would appreciate being told about it.
Laura L.
More learning. Less waste. (tm)
Best Answers in: Using LinkedIn (3), Education and Schools (2), Job Search (1), Occupational Training (1), Resume Writing (1), Accounting (1), Economics (1), Financial Regulation (1), Staffing and Recruiting (1), Sales Techniques (1), Business Analytics (1), Change Management (1), Software Development (1), Web Development (1)
I urge you to contact STC - the Society for Technical Communication.
Technical documentation is only good if its usable and addresses the needs of the reader. Clean, concise writing addressing specific user needs is usually at the crux of quality documentation.
Susan R.
Web Copywriter and Marketing Strategist at Content Wheel - Make your ideas roll
Best Answers in: Viral Marketing (1), Writing and Editing (1)
As an experienced director of technical publications, I can tell you Paul is on the right track. One of the most useful measures is one that compares the performance of different writers. I use Hours Per Page.
Picture for a moment a professional environment with good project management and a well-written style guide in place to consistently enforce and produce the company's Brand Voice. Imagine also that the Customer Support department keeps sufficient metrics before and after the implementation of this dev process and style guide to measure their impact. Now you have a quality standard and all you have left to measure is how productive your writers are.
Hours Per Page is useful in many situations, but here's one conversation I'll never forget from software tech writing's heyday.
The bean counter asked, "Why do you have all these senior contract writers at $65 per hour and only one junior at $35? You'd save us a lot of money if you reversed that."
"The senior writers produce a page in 3 to 3.5 hours. Junior writers take seven, plus an hour from me to edit and coach them. With our deadlines, we can't afford to do a lot of training."
That was the last I saw of that bean counter. I hope Hours Per Page brings you similar good fortune.
Dave G.
Documentation Consulting
Best Answers in: Writing and Editing (9), Professional Books and Resources (1), Starting Up (1)
I like Susan's "beancounter" analogy.
Here's my take on it:
JoAnn Hackos of the Society for Technical Communication presented wonderful seminars on how to plan, schedule, and budget for technical documentation.
Here are metrics remembered from those classes:
Typical Silicon Valley page of technical docs = ~7-8 hours/page
(this hour figure includes scoping, audience analysis, planning, scheduling, budgeting, research, rough draft, illustrations, polished draft, reviews, editing, production)
Factors affecting this metric are:
---complexity of content material (more time required for more complex, less time required for less complex)
---number/complexity of illustrations (more time required for many/complex illustrations, less time required if few/simple illustrations)
---availability of source materials and SMEs (less time required for easy and cooperative availability, more time required for hard-to-find/understand source material and uncooperative/unavailable SMEs)
---skill level/backgrounds of tech writers (less time required for very experienced tech writers, more time required for newbie tech writers)
---tools/applications/platforms used (less time if proper tools are readily available and doc professionals skilled with them, more time if improper tools are used and doc professionals are not skilled with them)
It used to be that tech docs were assembled by teams--SMEs, technical writer(s), technical illustrator, technical photographer, compositor/layout, editor, print-plant/website designer, etc.
Unfortunately, we've seen the demise of tech illustrator/photographers (everyone knows how to use Adobe Illustrator/Photoshop, right?), we don't need compositors/layout folks (everyone knows how to use InDesign/Pagemaker, right?), we've seen the demise of technical editors (everyone has "spellcheck", right?), and we're seeing the demise of web designers (everyone knows how to use DreamWeaver, FrontPage, and XML/HTML, right?).
All this is now being dumped on technical writers (for whom the original requirement was to be excellent writers and skilled with software apps such as MS-Word or FrameMaker).
Metrics? Developing technical illustrations requires its own set of metrics.
Editing technical docs requires its own set of metrics (2-5 pages/hour for highly technical content with more than just "proofreading" required, 8-10 pages per hour for less complex content and just proofreading for typos and usage problems). Website design also requires its own set of metrics (planning, scheduling, coding, linking, SEO, content development, content editing, testing, etc.).
Don't use "metrics" to evaluate technical writers unless you've taken into account all the factors mentioned above. A good writer may not look good if that same writer must deal with uncooperative SMEs, scanty resource material, non-working/inappropriate tools, and extremely complex content. A bad writer could look like a gem if SMEs basically write all the material, tools are all in place and foolproof as much as possible, a team of editors is willing to make the writer look good, and management doesn't care about the end product.
Measuring "page-count" is ridiculous. Writers can churn out pages and pages of drivel and gobbledegook to generate "pagecount". Editors can delete pages and pages of content to generate "pagecount".
A better measure is the readers' view of the documentation. Do they like it? Can they use it to solve their problems? Does the documentation reduce the number of calls to the tech support center? Does the documentation aid the tech support center in quickly resolving client problems? Does the documentation bring in more income? Those are the "good metrics."
Get a copy of Writers' Market 2010. There's a nice chart in there that shows rates for various types of writing/editing/doc services.
For good metrics, see the links provided below:
Links:
Julian M.
Technical author / documentation manager: Solvency II, banking, IT. Journalist, translator, novelist
Best Answers in: Writing and Editing (3), Antitrust Law (1), Business Development (1), Project Management (1), Blogging (1)
As you yourself (and several responders) have commented already, page count is a terrible way to measure productivity, especially as shorter often means better. For example, I spent several months *reducing* a user guide from 240 pages to 80, removing the repetition and waffle. Also, factors such as the sizes of the margins, illustrations and fonts all affect page length. More accurate than page count is word count: but since shorter often means better, word count isn't helpful either.
Also, I think that focusing on productivity in any form is to mistake efficiency for effectiveness. You can be very *efficient* at producing beautiful manuals with stunning illustrations and no grammatical errors: but if the user doesn't understand the product, the manual has failed.
The only useful matrices for documentation are those that measure comprehension. For example, to measure the effectiveness of software manuals you could measure the number of support calls caused by users misunderstanding the product, versus those caused by bugs, outages, etc. In fact, support calls are the best indicator of where a software manual has failed.
Paul C.
Senior Business Systems Analyst at Quest Diagnostics
Best Answers in: Government Policy (52), Government Services (4), Communication and Public Speaking (4), Using LinkedIn (3), Career Management (2), Ethics (2), Small Business (2), Education and Schools (1), Certification and Licenses (1), Job Search (1), Economics (1), Personnel Policies (1), Health Care (1), Work-life Balance (1), Treaties, Agreements and Organizations (1), Corporate Law (1), Tax Law (1), Internet Marketing (1), Business Development (1), Sales Techniques (1), Change Management (1), Equity Markets (1), Project Management (1), Supply Chain Management (1), Wealth Management (1), Starting Up (1), Energy and Development (1), Computers and Software (1), Web Development (1)
I'm a technical writer, and the only metrics which have ever been used to measure my 'performance' -- aside from assessing the quality of my work -- have been project-based.
Right now I'm working on a short-term project. I was hired with specific objectives: Create X process flows and Y procedures, due by Z. I meet with my manager once per week so that she may assess both the quality of my ongoing efforts, and whether I'll be able to finish the project on-time. Page-count and word-count are irrelevant; clarity, accuracy and speed are relevant.
Sometimes outside influences affect timelines for projects like this, rendering most metrics irrelevant. The information I need to gather in order to prepare these documents resides within the minds of my Subject Matter Experts (SME), and my own performance is directly affected by their availability, and the completeness of their information. One of the things my manager and I discuss during our weekly meetings is SME availability: If they are unavailable or uncooperative, I need my manager to step in and assist me in getting the information I need.
Likewise, my manager is able to assess, on an ongoing basis, the quality of the work I am producing. I'm able to adjust on-the-fly and make any needed modifications before the issue of missing a due date arises.
Hi Douglas - As a former documentation manager and tech writer, I've found the user's perceived usefulness of the documentation to be the biggest indicator of how effective it is. If a customer or tech support person is able to find the answer to his question quickly, he doesn't care how professional the documentation looks. This is why documentation written by a developer/engineer can sometimes prove to be more useful than that written by a professional tech writer, even though one document doesn't look as nice. Also, if a company's documentation is perceived to be not very useful by the users, and you make changes to improve the documentation's effectiveness, it will take work in convincing users to start referring to the documentation.
Measuring this effectiveness can be difficult and vary based on the environment. I think the key is to develop or improve the relationship between the people writing the documentation and those who use it (either tech support or client services - in the case of writing for end users). Any group who handles customer calls for help generally maintains some type of database to describe the customer's issue. Continually reviewing this information helps tech writers answer the question, "Is there something I could change in the manual so this call doesn't happen in the future?" This type of thinking will take you into a range of concerns including: size of manuals, binding type, format and layout, print vs. online, color vs. b/w, organization of manual, structure of entire document set, indexes, procedural vs. reference style, as well as fixing problems such as the manuals don't get to the people who need them.
Involving the team of tech writers (permanent and contractors) in reviewing customer calls and developing/improving the relationship between the tech writers and users gives them a stake in the process of answering the questions,
"How do we make more effective documentation?"
"Are there calls to customer service that could be avoided by changes to the documentation?"
"What changes should we make to the documentation?"
Customer surveys, peer reviews, and 360 reviews can also be useful in certain types of environments, but I've found developing that relationship with the users of the documentation is key to producing effective documentation.
Joy M.
pumps up profits fast
Best Answers in: Using LinkedIn (6), Staffing and Recruiting (2), Public Relations (2), Manufacturing (2), Professional Networking (2), Small Business (2), Job Search (1), Mentoring (1), Occupational Training (1), Foreign Investment (1), Personnel Policies (1), Events Marketing (1), Business Development (1), Search Marketing (1), Writing and Editing (1), Corporate Governance (1), Planning (1), Philanthropy (1), Inventory Management (1), Supply Chain Management (1), Branding (1), Market Research and Definition (1), Engineering (1), Interface Design (1), Career Management (1), Communication and Public Speaking (1), Ethics (1), Databases (1)
A reduction in calls to support groups.
More Answers (2)
Ashley G.
Publishing Director at Agency Editions, Inc.
Best Answers in: Writing and Editing (7), Event Marketing and Promotions (1), Conference Planning (1), Compensation and Benefits (1), Staffing and Recruiting (1), Events Marketing (1), Business Development (1), Communication and Public Speaking (1), Blogging (1), Computers and Software (1)
My key metric is clarity of thought reflected in the author's organization and mastery of composition. There are only four types of composition: exposition, description, argument and narrative. I can tell within a few pages (sometimes paragraphs) if a writer is succeeding or failing and can advise accordingly. Writers who write clearly usually write short, so I prefer low page counts as long as the topic is covered at the appropriate depth for the purpose. Another element is diction (word choice) and I always start with obtaining an agreement among developers and subject experts what vocabulary will be used in the material. If ten writers all start out understanding that buttons are clicked, not "clicked on" or selected, or pressed, or Heaven Forbid, "hit," work can progress quickly.
Assessing technical writing is not rocket science, although it is somewhat arty at times.
Hint: technical writing is more of an editorial function than a writing function.
