Leah has spent the last two decades working on information representation, processing, and modeling. She started her career as a computational neuroscientist studying sensory integration, and then transitioned into data science and engineering. Leah worked on developing AutoML for Salesforce Einstein and contributed to open sourcing some of the foundational pieces of the Einstein modeling products. She has brought her focus on making it easy to learn from expensive to generate and collect datasets to her work in everything from job search, to sales, to biotech, to engineering productivity. Leah currently works as Machine Learning Engineer at FarosAI (an engineering intelligence platform) developing the native AI capabilities.
LLMs, what are they good for?
Prompt: In the style of Terry Pratchett write an abstract for a tech conference about how to decide on good use cases for and evaluate the performance of large language models.
LLaMA2-70B: Greetings, fellow sentients! Are you tired of feeling like a mere mortal, limited by the constraints of your puny human brain? Fear not, for I have come to impart upon you the divine wisdom of the ages: how to decide on good use cases for and evaluate the performance of large language models.
Firstly, let us consider the humble origins of these magnificent beasts. Born from the fiery depths of the digital underworld, these models have been fed on a feast of data, their hunger for knowledge insatiable. [Truncated to leave space for human generated content]
In this talk, we will take a gander at those fancy, shiny new things that have become oh so trendy - large language models (LLMs). Dusting off my neuroscience PhD hat, we will get a little bit philosophical about intelligence, computational complexity, and how language fits into human cognition. Spoiler alert, the conclusion will not be that LLMs are sentient. However, they are both a powerful multitool and an incredibly compelling interface for humans.
We will then take a look at some use cases for LLMs. We will take a peek at some lesions from the more mature, practically ancient history, explosion of generative art and how various use cases have panned out. We will then play a little game of LLM telephone and try to quantify what happens to the information we want to convey. Qualitatively better things are great, but they are difficult to over-optimize for*. We will then walk through some experiments with using LLM models in the AI native platform of FarosAI. We will get as pedantic as possible about metrics, accuracy, information content, interfaces and cost. After all this, the hope is that you will have some tools to quantitatively decide when a foundational model is the right tool for the job you want to do.
*As a computer, humans are pretty good at it**.
**However, if doing repetitive tasks by hand were our cup of tea we probably would not have become engineers in the first place.