ChatGPT is Not Your Lawyer (Even If it Sounds Like One), Recent GAO Case Confirms

The world of Artificial Intelligence (AI) is a growing topic that many are eager to share their opinions on. There are people excited to witness this advancement of technology and are eager to implement AI programs into their lives and/or businesses to optimize efficiency. Others are uneasy about the advancements of AI, fearing replaceability or changes in the workforce. Or, there are those who have read one too many science fiction novels and believe that this is the beginning of the end.  

A recent decision prompted GAO to weigh in on the use of AI in the realm of federal contracting. Specifically, should companies use AI to draft legal pleadings such as bid protests?

In Raven Investigations & Security Consulting, LLC, B-423447 (Comp. Gen. May 7, 2025), Raven Investigations & Security Consulting, LLC (Protester) filed a protest for the Department of Homeland Security, U.S. Immigration and Customs Enforcement’s (Agency) issuance of order No. 70CDCR25FR0000032 (Order), under request for quotation (RFQ) No. 70CDCR25Q00000007, for administrative processing support services.

Following the Protester’s filing, the Agency and intervening party requested dismissal.  After the Protester responded to these requests for dismissal, the Agency terminated the Order. As a result, GAO dismissed the protest as academic.

While this appeared to be an open-and-shut case, irregularities found in the Protester’s response to the requests for dismissal prompted GAO to weigh in on the use of AI in the realm of federal contracting.

In GAO’s review of the Protester’s responses, GAO found several irregularities in the case citations. There were cases the Protester cited that GAO struggled to locate in legal databases. The Protester cited other GAO decisions that, upon further review, did not contain the same legal principles for which the Protestor was citing them. Additionally, the Protestor incorporated direct quotations from GAO decisions that did not exist.

GAO sought clarification from the Protester on these irregularities. In response, the Protester stated that the Protester’s representative was not a licensed attorney, and that he had utilized “public databases, [artificial intelligence (AI)]-assisted tools, online case repositories, and secondary legal research summaries to guide [his] understanding of GAO precedent.” The Protester admitted that the irregularities “arose from reliance on these automated or secondary tools, which misrepresented or misattributed certain case law and statutory language.”

As AI has just recently made its way into the court systems, GAO looked to the U.S. Court of Federal Claims’ recent response to the growing use of AI-tools in legal briefs. The court noted that there was a current trend of both attorneys and pro se litigants drafting legal briefs that cited non-existent cases. In response, courts have started sanctioning attorneys and pro se litigants who cite non-existent, AI-generated case law through the court’s rules of procedure. Yes, you read that right, sanctions for using AI.

GAO notes that the rules of procedures by other courts do not apply to GAO. However, GAO points to its own precedent to enforce similar sanctions, stating the following,

Our Office necessarily reserves an inherent right to dismiss any protest and to impose sanctions against a protester, where a protester’s actions undermine the integrity and effectiveness of our process.

GAO notes that the protester used AI-tools to draft responses “without engaging in any review of the material for accuracy.” Likely, the Protester assumed the case citations would be overlooked, so long as the cases stated the correct legal basis. The AI-program probably did provide a response that was well written and addressed the Protester’s key arguments in a grammar-free, easy-to-read-format. However, the legal arguments are just as (if not) more important than the factual arguments. GAO rightfully explains that the use of non-existent cases “wastes the time of all parties and GAO, and is at odds with the statutory mandate that [GAO’s] bid protest forum provide for ‘the inexpensive and expeditious resolution of protests.’”  31 U.S.C. § 3554(a)(1).

Since the protest was dismissed as academic, GAO chose not to sanction the Protester. Rather, GAO used this as a warning for future parties appearing before GAO:

“The submission of filings with citations to non-existent authority may result in the imposition of appropriate sanctions.”

Whether you love, hate, or even fear it, AI cannot effectively represent you in front of GAO. ChatGPT may not send you an invoice at the end of the month for its time, but the repercussions for using it in a GAO filing could be far more costly.

Questions about this post? Email us. Need legal assistance? Call us at 785-200-8919.

Looking for the latest government contracting legal news? Sign up for our free monthly newsletter, and follow us on LinkedInTwitter and Facebook.