Atharva Kanherkar

LLM work and research

A small public index of things I am working on, things I have merged, and writing around model gateways, agents, and evaluation.

Open work

Open pull requests

NVIDIA NeMo GymAggregate metrics helper for agentsOpen PR NVIDIA NeMo OO AgentsMake sync agent methods respect middlewareOpen PR

Open issues

SwitchyardDo not echo a base URL containing a key in a 502Open issue SwitchyardDo not pass client x-* headers upstreamOpen issue NVIDIA NeMo OO AgentsSync agent methods skip agent_call middlewareOpen issue LiteLLMMessages endpoint drops stop sequencesOpen issue LiteLLMProxy accepts a request-body API key when it should notOpen issue

Recently merged

A few upstream fixes

NVIDIA NeMo / Switchyard

Kept system and developer messages as instructions when routing Responses API calls.

NVIDIA NeMo / Switchyard

Stopped clients from forwarding API keys and OpenAI tenant headers through the proxy.

NVIDIA NeMo / Switchyard

Made content-filtered replies show up correctly for Anthropic clients.

Mozilla AI / AnyLLM

Fixed a Messages bridge that could leave out thinking, tool, attachments, and schema details.

SakanaAI / ShinkaEvolve

Fixed Anthropic response parsing when reasoning and text arrive in different block shapes.

SakanaAI / ShinkaEvolve

Kept edit markers on their own line so inserted patches stay readable.

SakanaAI / ShinkaEvolve

Added stable Rust formatting checks for generated patches.

Projects

Things I have built around this work

KairoA research dataset and test harness for LLM gateway translation. It collects real breakages and makes them easier to replay across gateways. AgentClashAgent evaluation for real tasks, comparisons, and regressions. agentic-memorySeparate semantic, episodic, and procedural memory for agents. datasmithTools for building synthetic data and evaluation datasets. ProofSWEA way to capture and score coding-agent sessions. chalkboardA prompt-to-explainer-video tool that can run with local models.

Kairo

A growing record of LLM gateway failures

Kairo is a public dataset and test harness for the awkward parts of moving tool calls between LLM APIs. I built it to keep small gateway bugs from disappearing after a report or a release.

So far it has 44 incident folders covering 48 reproduced bugs across LiteLLM, NVIDIA Switchyard, Bifrost, GoModel, AxonHub, and AnyLLM. The repo has 119 tests that replay recorded requests and responses without provider keys. The cases cover dropped tool settings, missing stop reasons, broken tool IDs, lost images and documents, thinking history, bad JSON-schema handling, and credential leaks. I also added a field-preservation sweep that compares gateways side by side instead of relying on one-off reports.

Read the Kairo repository ↗

Work

The last six months at Rimo

I have worked on model-connected product features, agent tools, OAuth and connector flows, and the background jobs and privacy checks around knowledge extraction. I have also worked on evaluation and model-selection work. This section stays high level because the underlying work is private.

Writing

Research notes on X

I write longer notes on X Articles. A few recent ones:

LLM routers and related researchA look at gateway translation problems, with Kairo as a way to collect and replay the failures. Your LLM proxy’s health check is handing out your API keysTwo reproducible credential problems in LiteLLM and Switchyard, explained with the requests that made them visible. You are probably wrong about open intelligenceNotes from trying lower-cost models in agent work, and why open data and useful evaluation matter as much as open weights.