Announcing OpenLLM: An Open-Source Platform for Running Large Language Models in Production

June 21, 2023 • Written By Sherlock Xu

⚠️ Outdated content

Note: The content in this blog post may not be applicable any more. To learn more about the latest features of OpenLLM and its usage, see the OpenLLM readme.

Motivations

Key features

Getting started

pip install openllm

openllm models -o porcelain

flan-t5
dolly-v2
chatglm
starcoder
falcon
stablelm

openllm start dolly-v2

To specify a specific variant of the model to be served, provide the option as follows:

openllm start dolly-v2 --model-id databricks/dolly-v2-7b

import openllm
client = openllm.client.HTTPClient('http://localhost:3000')
client.query('What are large language models?')

5. Alternatively, use the  command to query the model from a separate terminal:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Expected output:

Processing query: What are large language models?

Responses: Large language models (LLMs) are artificial intelligence (AI) systems that can parse natural language and provide responses similar to human responses. These systems can be trained with vast amounts of data in order to produce human-like responses to natural language.

6. A

openllm build dolly-v2

7.

What's next?

About BentoML