How do you think organizations can best use the contractual interpretations provided by LLMs? To expand on that, good lawyers don't just provide contractual interpretations, they provide advice on actions to take, putting the legal interpretation into the context of their client's business objectives and risk profile. Do you see LLMs / tools based on LLMs evolving to "contextualize" and "operationalize" legal advice?
Do you have any views on whether context window limits the ability of LLMs to provide sound contractual interpretations of longer contracts that have interdependent sections that are far apart in the document?
Has your level of optimism for the capabilities of LLMs in the legal space changed at all over the past year?
You mentioned that lawyers hoard templates. Most organizations you would have as clients (law firms or businesses) have a ton of contracts that could be used to fine tune LLMs. There are also a ton of freely available contracts on the SEC's website. There are also companies like PLC, Matthew Boender, etc., that create form contracts and license access to them as a business. Presumably some sort of commercial arrangement could be worked out with them. I assume you are aware of all of these potential training sources, and am curious why they were unsatisfactory.
Not op but someone that currently runs an ai contract review tool.
To answer some of your questions:
- contract review works very well for high volume low risk contract types . Think slas, SaaS… these are contracts comercial legal teams need to review for compliance reasons but hate it.
- it’s less good for custom contracts
- what law firms would benefit from is just natural language search on their own contracts.
- it also works well for due diligence. Normally lawyers can’t review all contracts a company has. With a contract review tool they can extract all the key data/risks
- LLM doesn’t need to provide advice. LLM can just identify if x or y is in the contract. This improving the process of
review.
- context windows keep increasing but you don’t need to send the whole contract to the LLM . You can just identify the right paragraphs and send that.
- things changes a lot in the past year. It would cost us $2 to review a contract now it’s $0.2 . Responses are more accurate and faster
- I don’t do contract generation but have explored this. I think the biggest benefit isn’t generating the whole contract but to help the lawyer rewrite a clause for a specific need. The standard CLM already have contract templates that can be easily filled in. However after the template is filled the lawyer needs to add one or two clauses . Having a model trained on the companies documents would be enough.
Do you think LLMs have meaningfully greater capabilities than existing tools (like Kira)?
I take your point on low stakes contracts vs. sophisticated work. There has been automation at the "low end" of the legal totem pole for a while. I recall even ten years ago banks were able to replace attorneys with automations for standard form contracts. Perhaps this is the next step on that front.
I agree that rewriting existing contracts is more useful than generating new ones--that is what most attorneys do. That said, I haven't been very impressed by the drafting capabilities of the LLM legal tools I have seen. They tend to replicate instructions almost word for word (plain English) rather than draw upon precedent to produce quality legal language. That might be enough if the provisions in question are term/termination, governing law, etc. But it's inadequate for more sophisticiated revisions.
Didn't try Kira but tried zuva.ai, which is a spin off from them. We found that the standard LLM performed at the same level for classification for what we needed. We didn't try everything though. They let you train their model on specific contracts and we didn't do that.
For rewriting contracts keep in mind that you don't have to actually use the LLM to generate the text completely. It is helpful if you can feed all the contracts of that law firm into a vector db and help them find the right clause from previous contracts. Then you can add the LLM to rewrite the template based on what was found in the vector db.
Many lawyers still just use folders to organize their files.
Do you have any views on whether context window limits the ability of LLMs to provide sound contractual interpretations of longer contracts that have interdependent sections that are far apart in the document?
Has your level of optimism for the capabilities of LLMs in the legal space changed at all over the past year?
You mentioned that lawyers hoard templates. Most organizations you would have as clients (law firms or businesses) have a ton of contracts that could be used to fine tune LLMs. There are also a ton of freely available contracts on the SEC's website. There are also companies like PLC, Matthew Boender, etc., that create form contracts and license access to them as a business. Presumably some sort of commercial arrangement could be worked out with them. I assume you are aware of all of these potential training sources, and am curious why they were unsatisfactory.
Thanks for any response you can offer.