Skip to content

feat: add LiteLLM as AI gateway client#1649

Open
RheagalFire wants to merge 2 commits into
patched-codes:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway client#1649
RheagalFire wants to merge 2 commits into
patched-codes:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented Jun 9, 2026

Copy link
Copy Markdown

PR Checklist

  • The commit message follows our guidelines: Code of conduct
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Does this PR introduce a breaking change?
  • Include PR in release notes?

PR Type

  • Feature

What is the current behavior?

Patchwork supports OpenAI, Anthropic, and Google as LLM providers. Users wanting to use other providers (Groq, Mistral, AWS Bedrock, Ollama, etc.) must use the OpenAI-compatible client_base_url workaround, which doesn't handle provider-specific API differences.

What is the new behavior?

Adds LiteLLMClient as a new LLM client, giving users access to 100+ LLM providers through the LiteLLM AI gateway.

Changes:

  • patchwork/common/client/llm/litellm_.py - New LiteLLMClient(LlmClient) implementing all abstract methods. Uses litellm.completion() with drop_params=True for cross-provider compatibility. Returns OpenAI-format ChatCompletion natively.
  • patchwork/common/client/llm/aio.py - Added litellm_api_key check in create_aio_client() factory
  • pyproject.toml - Added litellm>=1.35.0 dependency

Example usage:

# Use any LiteLLM-supported model via CLI
patchwork AutoFix --litellm_api_key=sk-ant-xxx --model=anthropic/claude-sonnet-4-6

# Or with Groq
patchwork AutoFix --litellm_api_key=gsk_xxx --model=groq/llama-4-scout-17b-16e-instruct

# Or with Ollama (local)
patchwork AutoFix --litellm_api_key=dummy --model=ollama/llama3
# Programmatic usage
from patchwork.common.client.llm.litellm_ import LiteLLMClient

client = LiteLLMClient(api_key="sk-ant-xxx")
response = client.chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is 2+2?"},
    ],
    model="anthropic/claude-sonnet-4-6",
)
print(response.choices[0].message.content)  # "4"

Live E2E proof:

litellm.completion(model='anthropic/claude-sonnet-4-6', messages=[...], drop_params=True)
# Response: "4"
# Type: ModelResponse

Other information

  • is_model_supported() returns True for all models since LiteLLM handles provider routing internally
  • is_prompt_supported() returns 1 (permissive) since LiteLLM manages token limits per-provider
  • drop_params=True silently drops provider-unsupported kwargs so the same call works across OpenAI, Anthropic, Gemini, etc.
  • Unit tests at tests/common/client/llm/test_litellm.py (3 tests: model support, chat completion with mock, drop_params default)

@CLAassistant

CLAassistant commented Jun 9, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@RheagalFire

Copy link
Copy Markdown
Author

cc @whoisarpit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants