AI Work Assistants Require a Heavy Lift To Get Their Full Value

"Artificial-intelligence work assistants were designed to provide businesses a relatively easy avenue into the cutting edge technology. It isn't quite turning out that way, with chief information officers saying it requires a heavy internal lift to get full value from the pricey tools.

"It has been more work than anticipated," said Sharon Mandell, chief information officer of network-tech company Juniper Networks, who is testing tools from several vendors but doesn't feel ready to put any into production.

Tools like Copilot for Microsoft 365 or Gemini for Google Workspace aim to put the full power of generative-AI capabilities into the hands of corporate workforces -- promising safe, prepackaged ways for enterprises to use the technology.

Working in tandem with the Microsoft or Google enterprise suites and large bodies of enterprise data -- including emails, documents and spreadsheets -- the promise is the tools can deliver reliable answers to questions such as "what are our latest sales figures?"

But that isn't always the case -- in part because the enterprise data they are accessing isn't always up-to-date or accurate and in part because the tools are still maturing.

Mandell said if she asks a question related to 2024 data, the AI tool might deliver an answer based on 2023 data. At Cargill, an AI tool failed to correctly answer a straightforward question about who is on the company's executive team, the agricultural giant said. At Eli Lilly, a tool gave incorrect answers to questions about expense policies, said Diogo Rau, the pharmaceutical firm's chief information and digital officer.

The stumbles come amid surging corporate interest in generative AI as well as efforts by CIOs to test the potential productivity gains promised by tools that can carry a $30-a-month price tag per user.

"I remain an AI optimist and am confident that we'll get there. It's just taking a little longer than perhaps we thought," Mandell said.

CIOs interested in moving forward with the technology are now working hard to clean up and manage their data so they can take full advantage.

Bala Krishnapillai, vice president and head of the information technology group at Hitachi Americas, said the organization has encountered instances of inconsistent, duplicated and incorrect data, leading to contradictory information that confuses AI outputs.

He said the company is regularly updating and refining its data to ensure accurate results from AI tools accessing it. That process includes the organization's data engineers validating and cleaning up incoming data, and curating it into a "golden record" with no contradictory or duplicate information.

"That's the very first thing we focused on," Kyndryl CIO Michael Bradshaw said about cleaning up data as the organization prepared to use Copilot. Bradshaw said Copilot has great capabilities, but a company has to be prepared to use it -- by keeping data organized, up-to-date and secure.

Vendors have noted the issue. "As companies started using Copilot, people started finding data that companies didn't know they had access to, or that they realized wasn't as fresh or as valuable as it could be. And then they realized, 'Oh, we've got to do more,'" said Jared Spataro, corporate vice president of AI at Work at Microsoft.

Complicating matters is the fact Copilot doesn't always know where to go to find an answer to a particular question, Spataro said. When asked a question about revenue, Copilot won't necessarily know to go straight to the enterprise financial system of record rather than picking up any revenue-related numbers that appear in emails or documents, he said.

To combat this, Microsoft recently introduced a tool known as Copilot Studio, which allows companies to direct Copilot to go to authoritative data sources within their systems for particular questions.

Some of the challenges with Copilot are related to the complicated art of prompting, Spataro said. Users might not understand how much context they actually need to give Copilot to get the right answer, he said, but he added that Copilot itself could also get better at asking for more context when it needs it. "A lot of people, I think, are having their first initial encounters with the technology and being a little bit disappointed," Spataro said." [1]

1.  AI Work Assistants Require a Heavy Lift To Get Their Full Value. Bousquette, Isabelle.  Wall Street Journal, Eastern edition; New York, N.Y.. 26 June 2024: B.4.