The Impact of ChatGPT on Technical Writing

Update 03/31/2023: On his popular tech writing blog I’d Rather Be Writing, Tom Johnson has a more thorough treatment of this subject than my short comments below. I remain less optimistic than he does about AI language generators, but his concrete examples of how writers could work with ChatGPT rather than eschew it are helpful.

Here’s the Write the Docs Newsletter (February 2023) on whether or not ChatGPT will replace the role of the technical writer:

“ChatGPT is a new conversational AI tool that has been making waves in the tech community for its impressively human responses and wide-ranging capabilities. There’s been a lot of discussion – and concern – in the Write the Docs Slack recently about what ChatGPT means for documentarians. Are our jobs about to disappear?

In several conversations over the last month, the community was pretty unanimous: we don’t believe ChatGPT can do the full job of a technical writer, as it has too many flaws. Top of the list is simply being wrong: ChatGPT will merrily make things up, making statements that are not supported even by its own data. Also, it only has access to certain source material – for example, it can’t tell you what’s in your upcoming software release. One community member said ChatGPT had invented command-line flags for their product that don’t exist. It’s also not able to interview SMEs or figure out from a long technical explanation which parts are actually relevant for an end user.

That said, people saw plenty of potential in AI. There is work that is boring and repetitive, which we’d all appreciate being automated away. ChatGPT is good at creating many suggestions really quickly if you give it the information you have to work with. Some suggestions may be bad or wrong, but some could be a decent first draft. ChatGPT could also potentially help with editing – suggesting ways to make sentences shorter or more grammatical.

It’s a tool, not a replacement. Like the ‘computers’ at NASA early on – once machines got more capable, humans stopped doing repetitive tasks, went up a level of abstraction, and became computer programmers. So it’s something we’ll have to try out and figure out how useful it can be for us.”

I really like this summary, but I have some questions.

First, is it possible that ChatGPT will ever reach a point where its documentation-related flaws are greatly reduced? Presumably with enough data and machine learning capability, that is an open question. Yet it’s hard for me to see how such flaws could be remedied when ChatGPT “only has access to certain source [i.e., pre-existing] material”—meaning ChatGPT can’t write new content for new features since it relies on pre-existing content to generate its responses.

Second, if we grant that ChatGPT could help take on repetitive writing or editing tasks and allow technical writers to go up a “level of abstraction” … well, what does that mean? What are some examples of that abstraction? (Update: Tom Johnson’s blog post provides some examples.)

Third, if ChatGPT can scour existing texts and come up with an answer to a question based on them, then it will be able to present information such that users won’t have to come directly to the source help content. This is sort of what happens when you query Google and see an excerpted answer at the top of your search results. I imagine ChatGPT would normalize that concept even more, and perhaps not even bother to cite sources. What are the implications of that? (Update: I think there are many but I also don’t think anyone cares as long as ChatGPT solves their immediate problem.)

As with the rest of the Write the Docs community, I’m skeptical that ChatGPT will replace the job of a technical writer. But there are questions worth exploring. I’ll post about this again if I get a chance to try the tool and do some more research.