How Barclays is addressing its biggest problem with LLMs
Investment banks love touting the uses of LLMs, but they face a problem when trying to implement them. AI models are complex and obtuse, so it's hard to explain to investors why a model has interpreted data the way it has. Barclays, however, has a solution.
Who is your Ideal Employer in finance? 💥 Answer our survey for a chance to win $1k
Quantitative analysts at Barclays have just released a paper on what they claim is the "first application in the finance domain of understanding and utilizing the inner workings of LLMs through mechanistic intepretability." Essentially, this means that they're searching for interpretability in how the hardware of a neural network reacts as it's performing a task.
One way it does this is via logit lens, a technique of analyzing how a model would predict the next word of a response at every layer of the network. The paper details how the Barclays quants created a heatmap and asked a question in the form of an open-ended query ("With good earnings the stock price of company will likely...."), before analyzing how likely the model was to predict the next word throughout the sentence. When the LLM hallucinated or got a question wrong, this highlighted where in the neural network those problems first occur, Barclays said this can then "guide targeted model improvements or risk monitoring."
Barclays said one of the "most promising tools for representing disentangled and interpretable concepts," is something else: Sparse Autoencoders (SAEs). With this method, you're essentially using one neural network to understand another.
SAEs are a different kind of neural network, however. Barclays said LLMs have "polysemantic neurons that activate for multiple, unrelated features," but SAEs will instead "encourag[e] each neuron to activate for a distinct feature." By using both in tandem, you can analyze the frequency at which SAE neurons activate to figure out which tasks neurons in the LLM are being activated for. The caveat is that SAE features "lack direct human-readable interpretations," but Barclays says it uses an "an AI model to analyze the patterns in the texts where a feature is most strongly activated," and assign labels.
Mapping out the network then allows for feature steering, which Barclays said is "remarkably effective at modifying model outputs in specific, interpretable ways." With feature steering, you artificially make certain values worth more or less to augment an LLMs perspective and 'steer' its answer in a certain direction. In an example, it made an LLM focus its analysis of a financial company on its credit score without changing the prompt.
Using these features to explain more complex queries doesn't appear to be a robust option yet, but regular testing via these methods can help banks prove that they are mitigating risk of hallucination and bias within the models they use.
Have a confidential story, tip, or comment you’d like to share? Contact: Telegram: @AlexMcMurray, Signal: @AlexMcMurrayEFC.88 Click here to fill in our anonymous form, or email editortips@efinancialcareers.com.
Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libellous (in which case it won’t.)
Photo by Alexander Lyashkov on Unsplash
