Readability Tests and the Contract Drafter

Earlier this year I redrafted a complex commercial agreement and sent it off to the client. I received in response a comment that I hadn’t expected at all—that the readability score for my draft was rather low.

This caused me to scratch my head—I’d never given a moment’s though to readability tests. So I did some rooting around online. (Here’s the relevant Wikipedia page.)

My client was referring to the Flesch Reading Ease test. It’s intended to indicate how difficult a piece of writing is to understand—the higher the score, the easier it is to read. Scores of 90-100 are considered easily understandable by an average 5th grader, whereas the Harvard Law Review apparently scores in the low 30s. Many government agencies require that certain documents meet or exceed a stated minimum score. For example, most states require that insurance forms score 40-50.

Microsoft Word will show readability statistics whenever you spell-check a document if you select the appropriate box (in “Options,” under the “Spelling & Grammar” tab). (Along with the Flesch Reading Ease score you’re also given the score under another test, the Flesch-Kincaid Grade Level test, which is only relevant in education.)

I duly checked the Flesch Reading Ease score for the draft I had sent my client, and it was in the mid 30s. I didn’t think that was any particular disgrace, but I needed some sort of frame of reference to compare it to.

For another client I had redrafted a “golden parachute” termination agreement. I dubbed it “RMA Widgets,” and it’s become something of a showpiece—I tend to offer it as an example of the glories that can be wrought when you redraft, MSCD-style, a mainstream corporate agreement. I checked the readability of the “before” and “after” versions of RMA Widgets—they were 14.5 (before) and 23.6 (after).

I had two reactions to these scores. On the one hand, I was gratified that my many hours of slaving over RMA Widgets had made it almost twice as easy to read. On the other hand, a score of 23.6 is nothing to crow about. But any improvements I might be able to make would, at this stage, be entirely marginal and would have next to no effect on the readability score. RMA Widgets is about as readable as I can make it.

I checked the scores for the “before” and “after” versions of another contract I redrafted, to the same effect—17.6 (before) and 25.3 (after). It would appear that mid 30s is, by my standards, as good as it gets.

Rather than concluding that I in fact stink at contract drafting, I’m inclined to attribute the mediocre “after” readability scores to an unavoidable feature of contracts—long sentences. I’m thinking in particular of sentences that are made up of enumerated clauses: they can go on and on, and would doubtless do a number on any Flesch Reading Ease score, but they nevertheless make any contract much easier to read, particularly when tabulated.

So now that I’ve been introduced to readability scores, do I think they serve any purpose in contract drafting? I suggest that if the Flesch Reading Ease score of any contract is in the teens, its likely that you’re dealing with a product of mainstream contract drafting, with all the deficiencies that that entails. Applying to it the recommendations in MSCD would doubtless increase its readability score. But once you have a halfway decent sense of what constitutes clear drafting, readability scores lose any significance. In particular, it would be pointless to tweak one’s drafting to goose the readability score of a contract.

But I suppose that if one were touting the benefits of an MSCD-style redrafting, one could point to the increase in the readability score as a simple indication of the cumulative impact on readability. Darn—I should have tried that on the client who raised the issue in the first place. (As it happens, the client promptly forgot about the entire subject.)

About the author

Ken Adams is the leading authority on how to say clearly whatever you want to say in a contract. He’s author of A Manual of Style for Contract Drafting, and he offers online and in-person training around the world. He’s also chief content officer of LegalSifter, Inc., a company that combines artificial intelligence and expertise to assist with review of contracts.

6 thoughts on “Readability Tests and the Contract Drafter”

  1. I know I should go to the literature and check, but I wonder how accurate these relatively crude tests are at measuring difficulty.

    A more fundamental issue, with long documents like unfortunately many contracts, is that they are local rather than global measures. That is, if they work, they will say how difficult it to understand some random passage from the entire text. They do not measure how easy it is for a person to find the information she needs in a hurry where she might not know where to look. We really don’t want to read the entire text in hopes we will find our answer somewhere. A better measure of this goal is the usability of the document. This term is used most commonly in web design and in instructional manuals for technical products. To some degree contracts and other transactional documents really are a sort of instructional manual, so this is an interesting concept to consider.

    Reply
  2. Josh: Readability tests are indeed crude. That’s perhaps the point: they’re meant to give a quick-and-dirty sense of how readable a piece of writing is. But given that they’re of limited use to the contract drafter, I won’t be delving any further into the literature on readability tests.

    I agree that whether a document is easy to use involves more than readability. I address in MSCD some of the factors, including layout, use of defined terms, and use of numbers and formulas.

    Reply
  3. Ken: I know quite a bit about readability tests, and I like the Flesch test a lot. I use it frequently in my own writing, and I think it can be useful in some types of legal drafting.

    But let’s be honest about two realities: (1) The Flesch test was designed by a plain-language zealot, Rudolf Flesch (one of my heroes). He wanted the government to communicate more clearly with the governed, and he wanted businesses to communicate more clearly with customers. His work did not focus on complex contract drafting. (2) Flesch designed his readability scale to assess writing for ease of reading by the general public, not for trained lawyers.

    So some might say you shouldn’t care about the readability score unless you are drafting legal text for the general public. Even then, the readability score is useful only as a rough guide. And by the way, a complex business contract that can score in the 30s is pretty good if you ask me.

    By the way, in the May 2006 issue of Clarity, there was an article giving a lot of in-depth insights into the limits of readability scores.

    Reply
  4. Ken,
    I have used the Flesch test and, for most types of writing, like it. That test has certainly improved my writing.

    However, lawyers (as well as engineers and others) produce two different types of documents.

    Highly structured writing is used in contracts, statutes, regulations, and (in my field of engineering) pharmaceutical equipment validation specifications. This writing is often based on long sentences that are broken into bulleted or numbered clauses or phrases.

    Another type of document is represented by your post (Flesch 53.7) and Wayne’s comment (Flesch 59.5). These are similar to court decisions, briefs, journal articles, and my letters to clients explaining the high cost of pharmaceutical equipment validation. These documents are made of normal paragraphs and sentences of shorter length.

    The structured organization of some types of writing are not easily analyzed by the Flesh test, yet this structure can make reading easier.

    I recently sent a lengthy and complex validation protocol to about 10 engineers and technicians (some without a college education) for review. From the comments that were returned I am confident that all were able to understand the document.

    I suspect that the Flesch test is not reliable for highly structured text found in contracts and other documents.

    Another problem with the Flesch score is that it has no way of knowing which words are more familiar to the intended audience. Lawyers, engineers, and auto mechanics all have terms of art that may be lengthy and uncommon but would cause problems for others.

    I think that readability tests work very well for most writing that is intended to be read by the general public.

    Reply
  5. Mary: I take the point, made by you and Wayne, that the Flesch Reading Test is geared to the needs of the general public, not lawyers. But I maintain that the test has modest value for drafters as an absolute guide to readability (if a draft scores in the teens, it probably needs some help) and as a relative guide to readability (if your work on a draft improves its score, that probably represents progress). But when I say modest value, I mean really, really modest.

    Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.