Navigating the World of Open-Source Large Language Models

Updated on July 29, 2024 • Written By Sherlock Xu

Over the past year and a half, the AI world has been abuzz with the rapid release of large language models (LLMs), each boasting advancements that push the boundaries of what's possible with generative AI. The pace at which new models are emerging is breathtaking. Just last weekend, Meta AI introduced Llama 3.1, with its 405B variant featuring better flexibility, control, and cutting-edge capabilities that can rival the best closed-source models. The very next day, Mistral launched Mistral Large 2, which matches the performance of leading models like GPT-4o, Claude 3 Opus, and Llama 3 405B in coding and reasoning evaluations.

These models, powered by an ever-increasing number of parameters and trained on colossal datasets, have improved our efficiency to generate text and write complex code. However, the sheer number of options available can feel both exciting and daunting. Making informed decisions about which to use — considering output quality, speed, and cost — becomes a problem.

The answer lies not just in the specifications sheets or benchmark scores but in a holistic understanding of what each model brings to the table. In this blog post, we curate a select list of open-source LLMs making waves over the past year. At the same time, we look to provide answers to some of the frequently asked questions.

Llama 3.1

Meta AI continues to push the boundaries of open-source AI with the release of Llama 3.1, available in 8, 70, and 405 billion parameters. It can be used across a broad spectrum of tasks, including chatbots and various natural language generation applications. Llama 3.1 is the latest addition to the Llama family, which boasts 300 million total downloads of all Llama versions to date.

Why should you use Llama 3.1:

• Performance: Based on Meta AI’s benchmarks, Llama 3.1 8B and 70B demonstrate superior comprehension, reasoning, and general intelligence capabilities compared to other open-source models like Gemma 2 9B IT and Mistral 7B & 8*22B Instruct. Its largest version, 405B, is competitive with leading foundation models across a range of tasks, including GPT-4, GPT-4o, and Claude 3.5 Sonnet.
• Fine-tuning: With three different sizes, Llama 3.1 is an ideal foundation for a wide range of specialized applications. Users can fine-tune these models to meet the unique needs of specific tasks or industries. This also extends to previous versions of the Llama model family like Llama 2 and Llama 3 (over 45,000 search results for “Llama” in Hugging Face Model Hub). These fine-tuned models not only save developers significant time and resources but also highlight Llama 3.1's capacity for customization and improvement.
• Context window: Llama 3.1 significantly improves upon its predecessors with a large context window of 128k tokens. This enhancement makes it useful for enterprise use cases such as handling long chatbot conversations and processing large documents.
• Safety: Meta has implemented extensive safety measures for Llama 3.1, including Red Teaming exercises to identify potential risks. According to Meta’s research paper, Llama 3 performs generally better at refusing inappropriate requests with lower false refusal rates and violation rates. However, they acknowledge that the numbers in the safety tests are not reproducible externally since safety benchmarks are internal to Meta, so they choose to anonymize the competitors in the test.

Challenge with Llama 3.1:

• Resource requirements: Given the large size, the 405 billion model requires substantial computational resources to run. Even with 4-bit quantization, the model remains around 200GB and may need multiple A100 GPUs to run effectively, which can be prohibitive for smaller organizations or individuals.

As it was just released recently, more investigation is still needed to fully understand the potential limitations of Llama 3.1.

Quickly serve a Llama 3.1 server with OpenLLM or click the following links to self-host it with BentoML:

• llama3.1-8b-instruct
• llama3.1-70b-instruct-awq
• llama3.1-405b-instruct-awq

Mixtral 8x7B

Mixtral 8x7B, released by Mistral AI in December 2023, uses a sparse mixture-of-experts architecture. Simply put, it uses many small networks, each specialized in different things. Only a few of these "experts" work on each task, making the process efficient without using the full model's power every time and thus controlling cost and latency.

Licensed under the Apache 2.0 license for commercial use, Mixtral 8x7B demonstrates exceptional versatility across various text generation tasks, including code generation, and features a fine-tuned variant, Mixtral 8x7B Instruct, optimized for chat applications.

Why should you use Mixtral 8x7B:

• State-of-the-art performance: Mixtral 8x7B outperforms leading models like Llama 2 70B and GPT-3.5 across many benchmarks.

• Long context window: Mixtral 8x7B's 32k-token context window significantly enhances its ability to handle lengthy conversations and complex documents. This enables the model to handle a variety of tasks, from detailed content creation to sophisticated retrieval-augmented generation, making it highly versatile for both research and commercial applications.
• Optimized for efficiency: Despite its large parameter count, it offers cost-effective inference, comparable to much smaller models.
• Versatile language support: Mixtral 8x7B handles multiple languages (French, German, Spanish, Italian, and English), making it ideal for global applications.

Challenges with Mixtral 8x7B:

• Lack of built-in moderation mechanisms: Without native moderation, there may be a risk of generating inappropriate or harmful content, especially when the model is prompted with sensitive or controversial inputs. Businesses aiming to deploy this model in environments where content control and safety are important should be careful about this.
• Hardware requirements: The entire parameter set requires substantial RAM for operation, which could limit its use on lower-end systems.

Quickly serve a Mixtral 8x7B server with OpenLLM or self-host it with BentoML.

Zephyr 7B

Zephyr 7B, built on the base of Mistral 7B, has been fine-tuned to achieve better alignment with human intent, outperforming its counterparts in specific tasks and benchmarks. At the time of its release, Zephyr-7B-β is the highest ranked 7B chat model on the MT-Bench and AlpacaEval benchmarks.

Zephyr 7B's training involves refinement of its abilities through exposure to a vast array of language patterns and contexts. This process allows it to comprehend complex queries and generate coherent, contextually relevant text, making it a versatile tool for content creation, customer support, and more.

Why should you use Zephyr 7B:

• Efficiency and performance: Despite its smaller size relative to giants like GPT-3.5 or Llama-2-70B, Zephyr 7B delivers comparable or superior performance, especially in tasks requiring a deep understanding of human intent.
• Multilingual capabilities: Trained on a diverse dataset, Zephyr 7B supports text generation and understanding across multiple languages, including but not limited to English, Spanish, French, German, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, and Korean.
• Task flexibility: Zephyr 7B excels in performing a broad spectrum of language-related tasks, from text generation and summarization to translation and sentiment analysis. This positions it as a highly adaptable tool across numerous applications.

Challenges with Zephyr 7B:

• Intent alignment: While Zephyr 7B has made some progress in aligning with human intent, continuous evaluation and adjustment may be necessary to ensure its outputs meet specific user needs or ethical guidelines.
• Adaptation for specialized tasks: Depending on the application, additional fine-tuning may be required to optimize Zephyr 7B's performance for specialized tasks, like reasoning, math, and coding.

SOLAR 10.7B

SOLAR 10.7B is a large language model with 10.7 billion parameters, using an upscaling technique known as depth up-scaling (DUS). This simplifies the scaling process without complex training or inference adjustments.

SOLAR 10.7B undergoes two fine-tuning stages: instruction tuning and alignment tuning. Instruction tuning enhances its ability to follow instructions in a QA format. Alignment tuning further refines the model to align more closely with human preferences or strong AI outputs, utilizing both open-source datasets and a synthesized math-focused alignment dataset.

Why should you use SOLAR 10.7B:

• Versatility: Fine-tuned variants like SOLAR 10.7B-Instruct offer enhanced instruction-following capabilities, making the model capable for a broad range of applications.
• Superior NLP performance: SOLAR 10.7B demonstrates exceptional performance in NLP tasks, outperforming other pre-trained models like Llama 2 and Mistral 7B.
• Fine-tuning: SOLAR 10.7B is an ideal model for fine-tuning with solid baseline capabilities.

Challenges with SOLAR 10.7B:

• Resource requirements: The model might require substantial computational resources for training and fine-tuning.
• Bias concerns: The model's outputs may not always align with ethical or fair use principles.

Code Llama

Fine-tuned on Llama 2, Code Llama is an advanced LLM specifically fine-tuned for coding tasks. It's engineered to understand and generate code across several popular programming languages, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash, making it an ideal tool for developers.

The model is available in four sizes (7B, 13B, 34B, and 70B parameters) to accommodate various use cases, from low-latency applications like real-time code completion with the 7B and 13B models to more comprehensive code assistance provided by the 34B and 70B models.

Why should you use Code Llama:

• Large input contexts: Code Llama can handle inputs with up to 100,000 tokens, allowing for better understanding and manipulation of large codebases.
• Diverse applications: It's designed for a range of applications such as code generation, code completion, debugging, and even discussing code, catering to different needs within the software development lifecycle.
• Performance: With models trained on extensive datasets (up to 1 trillion tokens for the 70B model), Code Llama can provide more accurate and contextually relevant code suggestions. The Code Llama - Instruct 70B model even scores 67.8 in HumanEval test, higher than GPT 4 (67.0).

Challenges with Code Llama:

• Hardware requirements: Larger models (34B and 70B) may require significant computational resources for optimal performance, potentially limiting access for individuals or organizations with limited hardware.
• Potential for misalignment: While it has been fine-tuned for improved safety and alignment with human intent, there's always a risk of generating inappropriate or malicious code if not properly supervised.
• Not for general natural language tasks: Optimized for coding tasks, Code Llama is not recommended for broader natural language processing applications. Note that only Code Llama Instruct is specifically fine-tuned to better respond to natural language prompts.

Why should I choose open-source models over commercial ones?

All the language models listed in this blog post are open-source, so I believe this is the very first question to answer. In fact, the choice between open-source and commercial models often depends on specific needs and considerations, but the former may be a better option in the following aspects:

• High controllability: Open-source models offer a high degree of control, as users can access and refine-tune the model as needed. This allows for customization and adaptability to specific tasks or requirements that might not be possible with commercial models.
• Data security: Open-source models can be run locally, or within a private cloud infrastructure, giving users more control over data security. With commercial models, there may be concerns about data privacy since the data often needs to be sent to the provider's servers for processing.
• Cost-effectiveness: Utilizing open-source models can be more cost-effective, particularly when considering the cost of API calls or tokens required for commercial offerings. Open-source models can be deployed without these recurring costs, though there may be investments needed for infrastructure and maintenance.
• Community and collaboration: Open-source models benefit from the collective expertise of the community, leading to rapid improvements, bug fixes, and new features driven by collaborative development.
• No vendor lock-in: Relying on open-source models eliminates dependence on a specific vendor's roadmap, pricing changes, or service availability.

How can I optimize LLM inference and serving?

To optimize the inference and serving of LLMs, you can select a specialized inference backend. They can help enhance user experience through faster response time and increase cost efficiency by improving token generation rates and resource utilization. Popular frameworks include:

• LMDeploy: An inference backend focusing on delivering high decoding speed and efficient handling of concurrent requests. It supports various quantization techniques, making it suitable for deploying large models with reduced memory requirements.
• vLLM: A high-performance inference engine optimized for serving LLMs. It is known for its efficient use of GPU resources and fast decoding capabilities.
• TensorRT-LLM: An inference backend that leverages NVIDIA's TensorRT, a high-performance deep learning inference library. It is optimized for running large models on NVIDIA GPUs, providing fast inference and support for advanced optimizations like quantization.
• Hugging Face Text Generation Inference (TGI): A toolkit for deploying and serving LLMs. It is used in production at Hugging Face to power Hugging Chat, the Inference API and Inference Endpoint.
• MLC-LLM: An ML compiler and high-performance deployment engine for LLMs. It is built on top of Apache TVM and requires compilation and weight conversion before serving models.

These frameworks significantly improve inference speed and system performance by implementing targeted optimizations. For detailed comparisons and benchmark results, refer to Benchmarking LLM Inference Backends: vLLM, LMDeploy, MLC-LLM, TensorRT-LLM, and TGI.

How do specialized LLMs compare to general-purpose models?

Specialized LLMs like Code Llama offer a focused performance boost in their areas of specialization. They are designed to excel at specific tasks, providing outputs that are more accurate, relevant, and useful for those particular applications.

In contrast, general-purpose models like Llama 2 are built to handle a wide range of tasks. While they may not match the task-specific accuracy of specialized models, their broad knowledge base and adaptability make them helpful tools for a variety of applications.

The choice between specialized and general-purpose LLMs depends on the specific requirements of the task. Specialized models are preferable for high-stakes or niche tasks where precision is more important, while general-purpose models offer better flexibility and broad utility.

What are the ethical considerations in deploying LLMs at scale?

The ethical deployment of LLMs requires a careful examination of issues such as bias, transparency, accountability, and the potential for misuse. Ensuring that LLMs do not perpetuate existing biases present in their training data is a significant challenge, requiring ongoing vigilance and refinement of training methodologies. Transparency about how LLMs make decisions and the data they are trained on is crucial for building trust and accountability, particularly in high-stakes applications.

What should I consider when deploying LLMs in production?

Deploying LLMs in production can be a nuanced process. Here are some strategies to consider:

1. Choose the right model size: Balancing the model size with your application's latency and throughput requirements is essential. Smaller models can offer faster responses and reduced computational costs, while larger models may provide more accurate and nuanced outputs.
2. Infrastructure considerations: Ensure that your infrastructure can handle the computational load. Using cloud services with GPU support or optimizing models with quantization and pruning techniques can help manage resource demands. A serverless platform with autoscaling capabilities can be a good choice for teams without infrastructure expertise.
3. Plan for scalability: Your deployment strategy should allow for scaling up or down based on demand. Containerization with technologies like Docker and orchestration with Kubernetes can support scalable deployments.
4. Build robust logging and observability: Implementing comprehensive logging and observability tools will help in monitoring the system's health and quickly diagnosing issues as they arise.
5. Use APIs for modularity: APIs can abstract the complexity of model hosting, scaling, and management. They can also facilitate integration with existing systems and allow for easier updates and maintenance.
6. Consider model serving frameworks: Frameworks like BentoML, TensorFlow Serving, TorchServe, or ONNX Runtime can simplify deployment, provide version control, and handle request batching for efficiency.

Final thoughts

As we navigate the expanding universe of large language models, it's clear that their potential is only just beginning to be tapped. The rapid innovation in this field signifies a future where AI can contribute even more profoundly to our work and creative endeavors.

Moving forward, I believe it's vital to continue promoting AI models in open-source communities, pushing for advances that benefit all and ensuring responsible usage of these powerful tools. As we do so, hopefully, we'll find the right balance that maximizes the benefits of LLMs for society while mitigating their risks.

More on LLMs

To learn more about how to serve and deploy LLMs, check out the following resources:

• [Doc] vLLM inference
• [Blog] Deploying A Large Language Model with BentoML and vLLM
• [Blog] From Ollama to OpenLLM: Running LLMs in the Cloud
• Join the BentoML Slack community
• Sign up for BentoCloud for free and deploy your first LLM in the cloud
• Contact us if you have any questions of deploying LLMs