What we think.

Machine learning is undoubtedly on the cusp of forever changing our lives. On the one hand, it’s giving our cars the power to understand our commands and drive down our streets. But on the other, it’s opening a Pandora’s box of deep fakes such as artificially constructed images and videos of imaginary situations that are startlingly convincing as the real thing.

I had coffee with an engineering friend the other day who was convinced that AI would soon take over everybody’s job. I agreed that we’re going to see a lot more AI in the near future but felt that it won’t soon take over my business anytime soon. In particular, I felt pretty confident that an AI could not generate meaningful content. However, once back in the office, this got me to thinking … could companies “outsource” content-generation work to an AI? I decided to do some research.

AI algorithms have been used (with mixed results) to generate text for several years now … but they are becoming much smarter.

I discovered that AI algorithms have been used for marketing purposes to generate text for several years now. As expected, this attempt to game Google and improve a website’s SEO resulted in tons of useless fluff that humans pretty quickly recognized as a poorly reshuffled mashup. Thankfully, Google has since retrained their algorithm to recognize fake content and penalize it in their search prioritization.

But AIs are becoming much, much smarter. A California research lab, OpenAI, has just released a language model called GPT-2 that generates shockingly creative and coherent text from scratch based on a small seed sentence. I thought it would be fun to experiment with the GPT-2 to see just how good it might be while I crossed my fingers that my job was still safe.

I fed it two seed sentences that represent significant concepts in the autotech ecosystem. It then created ten passages, from which I picked the best result for discussion.

Sound cool? Then let’s jump right in!

Can an AI write autotech content?

Our first seed is the first line of our upcoming e-Book and speaks to a much talked about concept, new mobility. If you want to see how we handled the sentence, take a look at our eBook. (See what I did there?)

SEED: What exactly is new mobility, and why is it taking over conversations among automakers across the globe?

GPT-2 (AI), Best of 10: To be honest, we don't know exactly how new mobility will shake out. What we know is that it is happening at a fast pace. In the last 12 months alone, more than $7 billion in new vehicle commitments have been announced from GM, Ford, and Fiat Chrysler. That number includes everything from new vehicles to battery-electric vehicles, self-driving and autonomous vehicles, self-driving trucks, and more. It also includes investments in technology, such as high-definition mapping that will deliver accurate location information for each vehicle.

 In terms of size, this is very significant. There is an average of about 20 vehicle-related investments in a major automaker per year. In the last 12 months, that number has been up to 30 vehicle-related investments.

Holy smokes – that’s pretty impressive! Reading through the other samples generated is similarly quite amazing. They are diverse and profess a general ability to make it seem like real discussions in our space. However, it’s not all coming up roses. In reviewing the output, I learned a couple of important limitations.

Specifics are wrong

You can’t trust the language model to generate text around specific items mentioned like numbers, people, or years. A few examples (there were plenty more):

• It talked about things happening in 2017 as if they were this year.
• It talked about the Fiat 500 as being an FCA product (correct), but it said the Model X was FCA too. (Wouldn’t Mike Manley be happy about that?)
• It named the president of Geely as Alok Jha.  Alok is actually a correspondent for the Economist.
• It also referred to John Krafcik as the CMO of Tesla. I have to admit that even though I know John is the CEO of Waymo, I did Google it to see if I might have missed a “new LinkedIn job” post – which I didn’t.

The thing is, most of the “facts” sound scarily plausible if you don’t look them up or otherwise know better. I have no idea if the $7 billion or the 20 or 30 vehicle-related investments mentioned in the story are real – I suspect not – but it sounds convincing enough to be believed.

Stories are widely different

Some of the AI-generated content used specific facts and figures, some were interviews with various industry people, some were portrayed as reports or papers, and so on. Each of the versions discussed relevant issues, companies, and people, but many times didn’t go anywhere, contradicted themselves, or said nothing of substance. And more importantly, they often read like off-kilter fiction – they used topical words and concepts but mixed them together in strange ways.

No real-world knowledge

The storytelling isn’t always based in the real-world, something that the researchers discussed as a limitation of their model as well. What do I mean by that? A perfect example from one of the ten samples I didn’t use:

“Today's cars can travel hundreds of miles on one charge. They can also be driven long distances in a day, and even across oceans.” 

It’s clear from all of these limitations that generating several samples and cherry-picking the best story will be necessary to get a higher-quality result. Let’s look at another example, a musing that I’ve postulated as a potential blog topic.

SEED: Mixing functional safety with self-driving software is both necessary for public assurance yet introduces a dramatic brake on the needs of an AI evolution.

GPT-2 (AI), Best of 10: This should not come as a surprise. As the U.S. Government Accountability Office (GAO) concluded, autonomous cars are a technology of the future and there are no guarantees that this technology will be safe or reliable. As a result, it is imperative that government and industry work together to ensure a robust regulatory framework for this technology.

A major concern is that without oversight by regulators, automated driving systems could quickly expand into areas that cannot be safely managed by a human driver. Autonomous driving is not just a matter of technology; it is also a matter of policy and ethics. We should not let technology and innovation take over the role of government.

The copy is really quite good and makes a claim that most people would agree with – technology and innovation can’t replace regulation. Unfortunately, it doesn’t really understand the subtext behind the seed question which is about the oil and water nature of AI and ISO26262, which was the original topic.

The copy is actually quite compelling. Too bad it’s fake news.

More critically however, it’s fake news. The GAO doesn’t state any belief that autonomous cars aren’t safe or reliable as best as I’ve been able to determine. The GAO did conduct studies that recommend the Department of Transportation (DOT) develop a plan to manage autonomous vehicles, as well as extend safety regulations and testing to cover autonomous cars. Nowhere do they make an assertion that denigrates self-driving technology. Meaning that although this post makes for good clickbait, it’s not content that I would want to stand behind.

Bold claims need to be researched

Several of the other samples generated made similarly bold claims by IEEE, Google, or VTTI that would need to be researched to determine whether they hold water, but it seems safest to pick examples that don’t rely on stances or quotes from individuals or organizations. Using this guideline, here’s the next best sample the for AI and 26262 seed:

GPT-2 (AI), Second best of 10:  While I think we need to get there, it is important to recognize that there will always be a need to be vigilant against dangerous software, so it is best to start by protecting against the less dangerous software.
 
One possible, but not necessarily ideal, approach to preventing such crashes would be to build a safety net to catch the people who create and deploy these software flaws. It's a powerful idea, but it has not really been tried yet.
 
The problem is that while software engineers are more or less good at spotting errors before they happen, the people that actually make decisions about whether to build a self-driving car in the first place aren't as experienced in the task.

Similar to the first selection, it sounds like it’s addressing the question although it’s not. Almost like a politician, it seems to use words and phrases that sound good but obscure the point or avoid it entirely. So, while it’s most definitely entertaining and might be useful for whipping up a couple of different topic angles as a brainstorming exercise, AI is not yet sophisticated enough to replace a human content creator with market specific knowledge unless the effort was supplemented by a substantive editor and a fact checker.

What’s the final take?

AI has come amazingly far. The concern of GPT-2’s creators about its ability to churn out misleading fake news is legitimate. After my experimentation with the GPT-2, this does seem like a serious concern. From a marketing point-of-view, getting a topic-driven AI content generator isn’t quite there yet, at least in a specialized technical field such as ours. I think that AIs will eventually be able to create boatloads of interesting, relevant, compelling content – will content matter by then? We’ll have to see.