Rendered at 01:06:46 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
wren6991 2 days ago [-]
This seems functionally similar to OpenAI having a step in pricing once you exceed a certain context length (also at 272k aka 2^18 aka 256k).
Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.
BoorishBears 2 days ago [-]
Not surprising it's a hard cutoff: they almost certainly have two infrastructure configurations for the two max sequence lengths
Fewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache.
Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode
MrBuddyCasino 2 days ago [-]
AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness.
One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?
NoahZuniga 2 days ago [-]
No, the main issue is that it's hard to communicate pricing where token price increases as token count increases, and they figure they can approximate the parabola well enough with two lines.
anuramat 2 days ago [-]
why can't they just increase the price for cached input instead?
dannyw 2 days ago [-]
There's the infra cost of having multiple SKUs.
If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.
And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.
MrBuddyCasino 17 hours ago [-]
> If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.
That seems like it is not an awfully hard problem, because the AI buildout scale is so massive. You simply don't provision too many of the smaller buckets, and when they are fully loaded you route the traffic to the larger buckets. Now you don't have a utilisation problem.
Now of course you can simply organize it that way internally, and present a single price to the customer. Perhaps thats the smarter thing to do.
pornel 2 days ago [-]
RAM needed for keeping KV cache around may be the more expensive factor.
tripplyons 2 days ago [-]
You only need to cache for tokens that are actually being used. Using 100k tokens of a 256k token window takes the amount of memory as using 100k tokens of a 1m token window.
pornel 1 days ago [-]
which is why they charge extra for an option to hold >256K tokens.
The cost may be smoothly variable, but likely there's a bimodal distribution of users who barely use any context and users who push it to the max. Average price across both extremes fits nobody, but averages per kind of workload can be close enough.
Having it as a separate model makes it easier to load-balance the traffic.
SwellJoe 2 days ago [-]
I make a point of never going beyond about 220k, unless absolutely necessary (and it's almost never necessary), anyway, even with models that degrade more slowly, so this is just a discount.
johnbarron 2 days ago [-]
64k ought to be enough for anybody
adamtaylor_13 2 days ago [-]
Why stop there? No true developer needs more than 32k!
--
Jesting aside, I cannot think of a single session in recent memory that used less than ~70k tokens, so I presume you are joking.
gck1 8 hours ago [-]
Heck, agents don't start editing before they're already at 70k for me.
I've played with explorer agents giving exploration summaries to help the implementer agents use more of their context for implementation, but it doesn't work as well. There's always something lost in the handoff.
gosub100 1 days ago [-]
It was from a famous quote supposedly from bill gates in the early days of computing: "640K (RAM) ought to be enough for anybody".
Apparently it was taken out of context or misattributed or something. Similar to "Al Gore said he invented the internet"
Computer0 1 days ago [-]
My understanding is that models can suffer “context limit anxiety” and a higher context limit will make it perform better especially as the context expands
xyzsparetimexyz 2 days ago [-]
Wow. So kimi is suddenly half the price for all users until they hit 256k of context? Thats massive.
InsideOutSanta 2 days ago [-]
I don't think so. This is a separate model, so I assume that if you just use this and switch to the 1 million context model when you reach 256k, your cache will be invalidated, so you'll re-pay the 256k tokens on the 1 million context model pricing.
Edit: I was wrong, thanks to longwave for pointing this out. It's absolutely possible to start out on the 256k model and then switch to the 1 million model when you get close to the context limit without invalidating the cache:
"When switching from k3-256k to k3 (1M), if k3-256k is close to the 256k limit and you don't want compact to lose information, you can switch directly to 1M. The current version switching from 256k to 1M does not affect the cache."
longwave 2 days ago [-]
The article explicitly says "The current version switching from 256k to 1M does not affect the cache."
conradludgate 2 days ago [-]
As far as I understand, no. They're suggesting that smaller context windows are typically cheaper (fewer input tokens over time).
fumber 2 days ago [-]
Can't find the videos/articles but some people tried it and found that the price per token was only half the story. It seems that it uses a lot more token, coming back to similar prices with other models.
StopTencent 2 days ago [-]
[flagged]
fumber 2 days ago [-]
I was so excited that it is open source until i realised the model required 1.5To of VRAM. Unsloth has compressed it in 1bit at about 570gb VRAM with 75% accuracy, that's almost mac studio territory...
nateb2022 2 days ago [-]
You can run DeepSeek v4 Flash at 4bit in ~150 GB VRAM, and it would absolutely destroy a 1 bit quant of K3.
AmazingTurtle 2 days ago [-]
75% quality... it gets worse the longer the context.
HawtAds 2 days ago [-]
This is just an API level change right? The model itself should be the same I think.
k__ 2 days ago [-]
As I understand it, they would have to train a whole knew model to hard cap it's context to different lengths. That would be cheaper to train and had cheaper inf, but still a huge investment.
So I'd guess it's API level.
gpugreg 2 days ago [-]
From the Kimi K3 technical report:
Kimi K3 supports a context window of up to 1 million tokens. We achieve this through extending
the context window progressively as training proceeds, following a four-stage curriculum. The
window grows from 8K to 64K tokens during pre-training, and from 256K to 1M tokens during the
cooldown phase.
You absolutely don't need to 'retrain' to reduce your context window. In vLLM it is an inference parameter. Smaller context window, smaller KV, less RAM needed to serve the same volume of requests.
With a lot of architectures, you technically don't need to retrain to extend the context window either; e.g. RoPE scaling; but performance is typically crap.
dgritsko 2 days ago [-]
This isn't quantized, right? Just a smaller context?
DSingularity 2 days ago [-]
Its 256k context window. Quantization is orthogonal. We cant really tell directly so it could be quantized.
gpugreg 2 days ago [-]
The model is already natively MXFP4-quantized during training, so there is no quality loss.
L-patpat 1 days ago [-]
When I first built features with GLM, there were lots of bugs, and it took me ages to fix them manually. Now the features implemented with GLM 2.0 have almost no critical bugs after testing. I can’t even imagine how capable K3 will be. It may well be on a level that ordinary people cannot access.
wxw 2 days ago [-]
> k3-256k is now available. Within 256k context, it delivers the same results. k3 (1M) consumes about twice as much quota as k3-256k.
effnorwood 2 days ago [-]
My LOE to make this work expressed in kWh is substantial but user is impressed. Eliza has hands now.
sergiotapia 2 days ago [-]
I can't seem to find pricing for this model. Since the context size is just a quarter of the full size K3, is the price also much cheaper?
I usually keep my context in chats below 256k anyways so this would be tremendous honestly.
daemonologist 2 days ago [-]
It seems to only be available in Kimi Code, via subscription, no there's no API pricing. The linked page says it consumes about half as much quota as the 1M version though.
trollbridge 2 days ago [-]
It's not available via API (yet).
Roark66 1 days ago [-]
This is very cool that they open weight it. Are there any EU providers serving it yet?
denis-c 1 days ago [-]
We are trying https://tensorx.ai/ as they have good model selection and ZDR, but seem new so we will see. Performance so far is good.
gigatexal 2 days ago [-]
Not relevant to this link but I was thinking about the allegations of Chinese AI companies distilling from the big frontier American ones. And I came to the conclusion: I don’t care.
Who cares? China has always copied and then copied the means of production and then out produced. See also Tesla and now all the Chinese cars eating their lunch.
As long as I get really solid AI models for cheap that do what I need I don’t care if they’re Chinese or otherwise.
I’ll still never use Grok from SpaceX AI cuz eww no, I have principles. ;-)
hendersoon 2 days ago [-]
What is the purpose of this? Just a hard cutoff below the actual context window? You could set that in your harness anyway.
meatmanek 2 days ago [-]
At least in the self-hosted LLM inference engines, you have to pre-allocate space for the maximum amount of context you want to allow for each parallel session. By using a lower maximum, you don't have to allocate as much VRAM for each session, allowing more usage for the same amount of hardware. Thus, cheaper.
dnlzro 2 days ago [-]
Uses less quota (i.e., cheaper). For people who like to keep their contexts small, this is a no-brainer.
surgical_fire 2 days ago [-]
> k3 (1M) consumes about twice as much quota as k3-256k
Cheaper?
chrisweekly 2 days ago [-]
k3-256 consumes half the quota, so... yes? (assuming the user isn't making use of the larger context window)
jedisct1 2 days ago [-]
This is a fantastic option for swival.dev given its very efficient context management compared to e.g. Claude Code.
try-working 2 days ago [-]
i've never had any issues with 256k context. see no reason to bump up to 1m if it comes at a premium.
trollbridge 2 days ago [-]
It's pretty handy for have very long contexts for long running agents, or else when doing literary analysis to simply be able to load the entire book in.
Gigachad 2 days ago [-]
Don’t they compact context to remove irrelevant details automatically?
k__ 2 days ago [-]
They remove potentially irrelevant details.
TranquilMarmot 2 days ago [-]
Using Claude/ChatGPT I rarely hit >256k context for most basic coding tasks. Sometimes if I need to do something more "intense" (bigger refactors, new features) the 1mil window is nice.
timcobb 2 days ago [-]
Codex uses 256k masterfully, 1M is luxurious but still quite expensive and seems not necessary as a default.
jsemrau 2 days ago [-]
Needing large context windows is an illusion.
k__ 2 days ago [-]
While I tend to clear my session after every task, I feel less stressed when my context is as 10% than when it's at 30%
TranquilMarmot 2 days ago [-]
Needing large context window is a skill issue
vehemenz 2 days ago [-]
So, all types of knowledge work and coding require the same amount of context?
This seems like a hot take not really informed by experience.
jsemrau 1 days ago [-]
This is a discussion on context window and not context.
Please read my essay on context window saturation.
I am also the author of deepcq (intent-aware memory querying & context mgmt for LLM agents)
vehemenz 1 days ago [-]
Please engage with my objection instead of assigning homework.
cybersec47 2 days ago [-]
> Codex uses 256k masterfully
This.
When working on hard problems (not "vibecode me a script to show an alert box", but e.g. "let's see what this three-level LUT-state-machine obfuscated binary does"), hitting the 1M (!) context window with Claude Code feels like you were talking to Claude Claudewski when his shift just abruptly ends, he packs his things, throws the office keys at Claude Claudeson in-between the front door frame while handovering like "Hi! Nice to see you, good luck." and now here we go again, you are working with someone who just experienced an acute amnesia. It tries everything it already tried, everything it was told in the initial prompt to not do, everything it was told in follow-up prompts not to do. "You were right, this approach does not work and we don't have 20 TB RAM on this machine for full symbolic execution, let me try..."
In Codex, it's so seamless that I sometimes just notice "wait, the context was 20 % remaining, it is 70 % now, wow, when did this happen", while it seamlessly works on the task. Basically never had an issue with context on Codex, be it coding features, cracking hard crack-me ciphers, or researching basically anything.
(For full disclosure, my last experience with Claude was a few weeks ago when I cancelled the subscription, maybe they fully reworked the traumatic "Summarizing" - "Oh, hi! Where are we? Who am I? What we are doing? This is taking too long, let me take a shortcut..." lobotomy they were doing in the meantime.)
256k is enough when the harness uses it properly and the model is not stupid. And also when the tokenizer is not tuned to invoice as many tokens as possible...
Mastodo 2 days ago [-]
[flagged]
Tostino 2 days ago [-]
I'll have to give it a shot some time soon. I just can't imagine working on any of my serious projects doing a new large feature with that little room. By the time it has looked at half the code required to start planning the work, it'd be out of context. Must use sub agents better or something.
timcobb 12 hours ago [-]
Codex turns over 256k windows for *days/weeks on end* without seemingly much fidelity loss, especially with a goal that anchors it to a durable plan/ledger.
cybersec47 2 days ago [-]
Also, a token means something different in each model/tokenizer. "For improved performance we will use (read: invoice you) 30 % more"...
That's because it has the best compaction of anyone, and it's not even close.
Claude might as well not even do it in my experience.
lukan 2 days ago [-]
Since Claude is the first time for me really, really out (TIL against my wished about https://status.claude.com/), I am now interested enough to see what else works. But ... when I click pricing, I see "Join a waitlist". Wtf? Are they really that good, so were totally surprised and overwhelmed by the requests, is this a marketing stunt, or do they just don't have the hardware being in china?
KronisLV 2 days ago [-]
> Are they really that good,
They did exceed the expecations of pretty much everyone! I've also blogged about using the model, it's a bit on the slow side but pretty good!
> so were totally surprised and overwhelmed by the requests ... or do they just don't have the hardware being in china?
As a user, I much prefer that to service disruptions or severely degraded or secretly quantized performance. However if I didn't have an account, I'd be pretty pissed off about not being able to give them money and become a user.
lukan 2 days ago [-]
"I'd be pretty pissed off about not being able to give them money and become a user."
I am now rather pretty pissed towards antrophic for stopping my flow and forcing me to search for alternatives.
Alifatisk 2 days ago [-]
No, the waiting list is true.
Kimi had become that popular. I was a subscriber of Kimi back when latest version was Kimi K2. Later I unsubscribed because I jumped over to GLM subscription (they had amazing deal). Now when I wanted to try out Kimi K3 to find out what the fuzz was all about, I couldn’t subscribe to them.
I remember reading a post from Moonshot team about this, they are doing this because they are almost at peak capacity and want to reserve it to keep the quality for their current customers.
We are actually witnessing an open-weight model catching up at catching mainstream users attention. And instead of behaving like Anthropic, they actually care about their users experience.
therein 2 days ago [-]
I wish I could get a Kimi subscription. I'd jump away from Anthropic in a heartbeat.
When they first announced, I created an account but didn't subscribe, so I am stuck on a waitlist now.
jfrbfbreudh 2 days ago [-]
I wonder what the criteria is, I signed up for the $19 / month waitlist today and got my invite within a few hours.
therein 2 days ago [-]
Maybe you signed up with a phone number? Mine was a Google login.
The only downside with third party providers is that you have to trust that the provider have setup and configured it correctly, and is not secretly quantizing it.
See Kimi Vendor Verifier.
usef- 2 days ago [-]
Plenty of providers on that list aren't fly-by-nights and have easily proven themselves in past models. Jeremy Howard does demos on Fireworks.
A nice thing about open router is you can specify filters, like "US hosting without data retention"
applicative 2 days ago [-]
Yes, something I read just before it was released suggested that, because of the unique features of the model, there was a back and forth of Hugging Face, Moonshot and providers like Together AI and Fireworks AI. This also explained why it took much less than a day for Together, Fireworks etc to appear. Whatever they are doing is what Moonshot wants, I think.
2 days ago [-]
2 days ago [-]
jadbox 2 days ago [-]
Do any support the smaller context for better pricing?
HDBaseT 2 days ago [-]
Not at all a marketing gimmick, the demand is simply that high.
I was able to press "Join waitlist" and then within 48 hours got accepted. The limits aren't very high, no where near the endless subsided+resets given on ChatGPT/Claude. I recommend ChatGPT for good value output!
Others here mentioned "the providers could be quantizing it!" but some of the providers on OpenRouter have partnered with Moonshoot and OpenRouter shows the int when you expand on the provider.
It should say "mxfp4" but providers like Baseten report FP8.
InsideOutSanta 2 days ago [-]
It's an extremely popular model hosted by a company affected by hardware export bans. I doubt they'd voluntarily prevent people from subscribing if they didn't absolutely have to to maintain service quality.
zorked 2 days ago [-]
I have been using Kimi for months. It did get superslow after the K3 launch, then they blocked signups and normalized the service.
It's not ideal that people can't join but as a user I am happy that they focused on serving existing customers.
0xc0c0c0 2 days ago [-]
I honestly prefer this as opposed to what Anthropic has done in the past which is to continue accepting users even if they don't have the capacity to serve and behind the scenes tone down everyone's limits.
MangoCoffee 2 days ago [-]
LLMs is quickly became commodities. US AI labs like OpenAI is losing their moat. Hyperscalers and data center owners who can sell cheap token will win
okrad 2 days ago [-]
I believe Codex harness is pretty sticky though I haven’t tried many others. Does anyone else provide a harness of that quality?
nvarsj 2 days ago [-]
Pi harness with subagents is pretty great. I’m using it for everything now.
yeswecatan 2 days ago [-]
How are you handling subagents?
nvarsj 1 days ago [-]
tintinweb/pi-subagents. Then I use superpowers and it spins up subagents automatically. Works really well and you can customise the models and efforts you want to use with local AGENTS.md.
I’m primarily using gpt-5.6 and then opus for reviews.
RALaBarge 2 days ago [-]
You say hey llm spin up some agents to do x y and z
anderber 2 days ago [-]
Usually the main agent can handle creating subagents
conradludgate 2 days ago [-]
Their harness is indeed nice, but we're using it against our own AI Gateway at work. Right now we only expose GPT models on it, but I imagine it's possible to add our own models eventually
verdverm 2 days ago [-]
opencode is highly regarded, I use it exclusively
Roark66 1 days ago [-]
Open code is cool once I added the ctrl-o function to it (show thinking and command outputs at will instead of on by default), but sadly I don't think it got merged by the team.
I forgot why.
But I can't use an AI cli without that feature.
verdverm 1 days ago [-]
you can turn those on through the existing menu system
versteegen 2 days ago [-]
...but we're talking about compaction, and opencode's compaction is (or was) terrible. I've seen so many horrible problems that I keep it disabled (with an envvar flag, because even the config flag to turn it off was broken).
verdverm 1 days ago [-]
I don't use compaction, I keep my context limited, my sessions fresh, and have agents write markdown files as needed
jki275 2 days ago [-]
I've used open code extensively -- it's not bad, but it's not anywhere near the level of the codex application.
ux266478 2 days ago [-]
I have so much gratitude to the frontier companies who did all the extremely complicated research and development, model by model. It already feels difficult to remember how much capital it really took. Thank you for getting us to this point.
jscott817 2 days ago [-]
Can we assume that model performance at 90% of the 256k limit != 90% of 1M token limit?
Is this the exact same model just with less VRAM allocated for context window?
3836293648 2 days ago [-]
No, you can't assume it. You can trust them as they made that claim outright. Or you can choose to not trust them, I guess
TokenLat 2 days ago [-]
[flagged]
Conol_ai 2 days ago [-]
[flagged]
illithid0 2 days ago [-]
This was posted 38 minutes ago, and as of 20 minutes ago, several Anthropic services are now designated as having a "major outage".
Doubt these are related, but it made me laugh a little.
paxys 2 days ago [-]
Anthropic services have outages on all days ending in y.
de6u99er 2 days ago [-]
So users in Germany are not affected?
jampa 2 days ago [-]
Anthropic has better SLA in Germany. I’ve heard uptime there can get up to nein nines.
zarmin 2 days ago [-]
Germany ends in y
tshaddox 2 days ago [-]
Not in Germany.
hebelehubele 2 days ago [-]
The word "Germany" is "Germany" everwhere and always ends with y. The country Germany has different names though.
aksappy 2 days ago [-]
Germans prefer Deutschland, no?
361994752 2 days ago [-]
so Deutschland is not affected. Germany is
vidarh 2 days ago [-]
But Tag doesn't.
1saadcodes 2 days ago [-]
It feels like we're moving away from "bigger context is always better" toward "right-sized context". I love this not just because my wallet feels safer but because most of my coding sessions never come close to needing 1M tokens anyway
Conol_ai 2 days ago [-]
[flagged]
madihaa 2 days ago [-]
[flagged]
chipgap98 2 days ago [-]
This exact same comment is the top comment on the reddit thread for this news
I searched one other comment of theirs in their history and see the same thing. On mobile so I can’t easily tell if this account is the bot account or if the ones on Reddit are.
For me the sweet spot is somewhere under 500k depending on how extensive I want to get. You can build up a sizable effort project in half a million tokens with Claude, with Claude having all the context from ground 0 to wherever you're off at.
cyanydeez 2 days ago [-]
I'm always curious what you guys are working on; every git repo I've run a local model on and stick below <100k to increase speed seems effective enough to scope patches and changes.
KronisLV 2 days ago [-]
My current Claude Code session has been going on for like 35 hours and has used up around 400 million tokens, thankfully almost all of those being cached (95-98%) - pretty typical for long form agentic work.
First you spend like 2-3 hours working on a plan, once you have that you just tell the model to go and implement it, do adversarial sub-agent review loops before each commit and also make sure that all tooling and tests pass (including coverage requirements). You do need to poke it in a slightly different direction every few hours, though. Not even any novel work, just some refactoring and SSE notification hardening, bug fixes, alongside environment tuning and getting rid of some bottlenecks (also migrated from Oracle to PostgreSQL but that's mostly done).
That said, Kimi somehow manages to use less context in the main thread than Anthropic's models (even when you use sub-agents and also dynamic workflows in Claude Code), might have something to do with either how the model is tuned or their Kimi Code harness - because even in most of the longer form sessions it doesn't seem to fill up quite as quickly (note: because the kimi vis tool doesn't have a full summary view across all agents, these are the main long running agent stats across some sessions, not sub-agents):
I could see 256k context being sufficient for all sorts of work, even if intermediate progress/plan tracking files and docs might have to be used along the way, in addition to whatever plan support the harness has (for example, if you document something that will be relevant for load testing you might need that in 10 turns but not during the ones before then).
smotched 2 days ago [-]
Once you have the plan you don't need to keep the 2hrs of research in the context (which is most of it) you can drop that plan into a file and start fresh for implementation.
XenophileJKO 2 days ago [-]
That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first.
If I feel like the model and I explored a lot of options I won't want to keep context as it might be confusing.
I think the more you use it the better judge you are of whether you should purge, compress, or just keep the context before executing the plan.
dandaka 2 days ago [-]
Models work best when they have short instructions and no noise. "Conversation [may] contain" also means "conversation has a lot of noise". It degrades performance and increases cost.
I use handoff skill to ask model to write a prompt for itself.
pertymcpert 2 days ago [-]
The applicability of your advice is very model dependent. Some like Claude have very good long context performance, whereas others they fall off much quicker past some threshold.
ValentineC 1 days ago [-]
You're right that it's model dependent, even within Claude models.
I've found Opus 5 far better as a subagent with very limited context window use, which could suggest that it might not have good long context performance unlike its predecessors. (I was one of many tearing my hair out trying to work with Opus 5 for the past week.)
dandaka 2 days ago [-]
I am not only talking about "long context" performance (context rot), but also about noise that is confusing model about its goal (from correctly extracting operator's intent). I think every model will get confused to a degree, so clearing up irrelevant information from context helps a lot.
2 days ago [-]
KronisLV 2 days ago [-]
> That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first.
Yes, pretty much - if there’s a lot of noise and jumping around and wrong conclusions and corrections, compress and only leave the correct stuff (maybe make some plan file briefly mention what NOT to look at/do). But if it’s all fairly straightforward then can just proceed with the execution.
Most of the time the planning stage ends up short of 200k tokens anyways, it mostly takes hours just cause I’m slow and need to explore the various options - still cheaper than building the wholly wrong thing and having to redo everything.
Compressing the context can also drop important information so it might be better to only do that when you need to / use the plan mechanism/files / do it after completing some large stage of the plan and so on - so a judgement call.
bkaae 2 days ago [-]
Thanks for sharing - is this a normal feature request you are implementing in this example or is this a project from scratch? Trying to get an idea of how your workflow compares to mine.
KronisLV 2 days ago [-]
Existing project and as usual, a few issues mashed together in one mostly coherent plan. The shorter Kimi sessions I mentioned were singular features, there 256k would be wholly adequate.
A greenfield project would probably allow at least 2x fewer tokens to be used in most of those long tasks, but I was mostly after consistency and bug fixes along the way as needed.
vidarh 2 days ago [-]
Kimi CLI has a mechanism with checkpoints and the ability for the agent to revert to a checkpoint + a message of how to continue based on what went right/wrong. I don't know if that's the cause of what you've seen, but it's plausible.
DelightOne 2 days ago [-]
With prefix caching, you get checkpoints for free. Do you mean that?
vidarh 2 days ago [-]
No. Prefix caching is just an optimisation on the server side. What Kimi CLI does is insert <system> tags that include a checkpoint marker with an id.
The model is then given a tool that allows the model to decide to roll back to a checkpoint + a message containing any additional useful information.
It's specifically instructed to use that tool[1] in cases like when it has inadvertendly read a large file where most of the content is not relevant to the task, or after a web search where it's found what it's looking for but most of the content isn't needed, or when it's written code that didn't work as expected, or similar.
It basically lets the model backtrack and "forget" irrelevant details at the end of the context but give itself hints on how it should continue from the checkpoint.
Though, interestingly they seem to be abandoning it in their new CLI (kimi-code), unless it's been folded into other functionality. Not sure if they just feel it's not needed any more with their newer models or if it just didn't work as well as they expected.
I had Claude build me a Python-inspired .NET language that treats .NET as a first class citizen, and breaks backwards compatibility where some Python nuances don't really apply to .NET for. I was able to get it to build a sample ASP .NET Web application that ran on Culebral code.
Haven't gone back to it, have been using Claude Code on a private project I'm still architecting.
That is pretty interesting. It compiles to the CLR?
giancarlostoro 1 days ago [-]
Yeah, that was the goal. I've wanted something like Python for .NET as someone who does both, but backwards compatibility to Pythons ecosystem is something that will always make it harder than just having something Python-like as much as possible, but runs on .NET the other thing I wound up going for Rusts approach to type hints, method signatures should always have type hints, and when instantiating objects or types for the first time, the value should be inferred from there.
pseudosavant 1 days ago [-]
In practice, how have you used this? Are there scenarios where you need to be deploying .NET but want the ergonomics to be more like Python?
_zoltan_ 2 days ago [-]
I have a couple projects where the background research is easily over 500k without writing any code, after ultracode subagents synthesis.
giancarlostoro 2 days ago [-]
I think the distinction is when the model decides to read code top to bottom vs when the model chooses to parse code indirectly to save on tokens, there's also AST tooling to let the model see project structure.
sheeshkebab 2 days ago [-]
Try doing a refactoring of some sort or larger new feature using just an agent on a moderately sized codebase, 256k will be compacting every few minutes, and result will be unusable.
hedgehog 2 days ago [-]
I do essentially all of my work with auto compact set to 250k and it's fine. It may be due to the way the project tooling is set up and the use of sub agents?
jdoe1337halo 2 days ago [-]
They are just talking to the model in CC, while staying in a single thread. Doubt they have any actual coding knowledge to compartmentalize different problems in the codebase.
giancarlostoro 2 days ago [-]
Depends on the programming language I'm using for a given project, and the domain I'm working with. I've been coding as a hobbyist for nearly two decades now (since my teens), professionally for 9 years, and was a TA before that for roughly 3 years at one of the best colleges for this field in the state (at least back then it was) where I taught other students about programming, in some cases I was their primary learning resource.
But yeah, I have no idea about anything about software because you made an assumption off very little to go by.
pesfandiar 2 days ago [-]
"256k ought to be enough for anybody"
superloika 2 days ago [-]
[flagged]
camgitt 2 days ago [-]
[flagged]
kristianp 2 days ago [-]
I'm so glad it "quietly" introduces that feature.
Imanari 2 days ago [-]
Here is what most people miss…
ibuildproducts 2 days ago [-]
omg! new model!!
Alifatisk 2 days ago [-]
Same model, new configuration
dools 2 days ago [-]
Hopefully this helps reduce some of the pressure on their infrastructure. Their models have all become super dumb recently and their support are not addressing it. I have a hunch they’ve been serving a significant percentage of requests with quantised models.
stingraycharles 2 days ago [-]
Ah come on, HN is not the place for these types of unfounded conspiracy theories that keep popping up on Reddit.
periodjet 2 days ago [-]
Why are Anthropic and OpenAI even allowing their coding harness apps to be plugged into different model providers…? I’m surprised they haven’t figured out a way to clamp down on that by now.
wren6991 2 days ago [-]
I'd guess because it costs them nothing and it gives you a smoother transition back towards paying for their products.
archildress 2 days ago [-]
From my view, as soon as they do that, they send people out the door to use Opencode instead - and once many people have a taste of trying every model via Openrouter, it's eye opening as to the possibilities.
Of course - Anthropic and OpenAI have an advantage in the amount they can subsidize the usage, but I think those days are waning.
auspiv 2 days ago [-]
Well when a huge part of potential revenue is all in on Bedrock... you need the harness to be able to talk to Bedrock. And Vertex. And all the other places these models are hosted. And allow for proxy because many businesses do not all direct internet access... all valid business reasons.
HDBaseT 2 days ago [-]
It honestly should be easier, its a by product of everyone using the same API standard.
Codex and Claude require editing a .json file, but most other harnesses have direct connections via a /provider or /login command.
holoduke 2 days ago [-]
A bit of topic. But how likely is it that the US will restrict Chinese open weight models and also force Euro countries to do the same? I think it will be effective within 6 months. The US is having a hard time staying competitive.
jeppebemad 2 days ago [-]
I don’t know, but I do think that the days of the US “forcing” Euro countries to do anything, is over.
holoduke 2 days ago [-]
[flagged]
impossiblefork 2 days ago [-]
It isn't legally possible for them to do this at the EU level. The EU parliament would never vote for it.
For pressure at the country level leading to this kind of thing I think it's very unlikely. Here in Sweden it wouldn't just require a vote in the Swedish parliament and before this there'd have to be förarbeten and you can't just brazenly push things through with insane arguments, Swedish social convention goes against it-- and there's just no way to get it through.
It also might not even be legal. "We aren't at war with China and I'm a communist, and the US LLMs are so aligned with values inimical to my political ideology that this is interference with opinion formation" might be an actual legal argument that the ECHR or CJEU might actually have to accept.
holoduke 2 days ago [-]
It happened with ASML. What's your thought on it? The US forbid dutch company to export.
impossiblefork 2 days ago [-]
Well, that's the deal, I assume-- that they weren't allowed to buy the Japanese light source outright, so they bought the American one, even though it required giving the Americans some sort of veto or control.
I guess it sucks if one wants to expert broadly, but if you're big on vertical integration and the Japanese won't sell I guess you take what you can get.
realusername 2 days ago [-]
Trump burned a lot of bridges in the EU, that one will be a hard sell
Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.
Fewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache.
Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode
One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?
If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.
And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.
That seems like it is not an awfully hard problem, because the AI buildout scale is so massive. You simply don't provision too many of the smaller buckets, and when they are fully loaded you route the traffic to the larger buckets. Now you don't have a utilisation problem.
Now of course you can simply organize it that way internally, and present a single price to the customer. Perhaps thats the smarter thing to do.
The cost may be smoothly variable, but likely there's a bimodal distribution of users who barely use any context and users who push it to the max. Average price across both extremes fits nobody, but averages per kind of workload can be close enough.
Having it as a separate model makes it easier to load-balance the traffic.
--
Jesting aside, I cannot think of a single session in recent memory that used less than ~70k tokens, so I presume you are joking.
I've played with explorer agents giving exploration summaries to help the implementer agents use more of their context for implementation, but it doesn't work as well. There's always something lost in the handoff.
Apparently it was taken out of context or misattributed or something. Similar to "Al Gore said he invented the internet"
Edit: I was wrong, thanks to longwave for pointing this out. It's absolutely possible to start out on the 256k model and then switch to the 1 million model when you get close to the context limit without invalidating the cache:
"When switching from k3-256k to k3 (1M), if k3-256k is close to the 256k limit and you don't want compact to lose information, you can switch directly to 1M. The current version switching from 256k to 1M does not affect the cache."
So I'd guess it's API level.
With a lot of architectures, you technically don't need to retrain to extend the context window either; e.g. RoPE scaling; but performance is typically crap.
I usually keep my context in chats below 256k anyways so this would be tremendous honestly.
Who cares? China has always copied and then copied the means of production and then out produced. See also Tesla and now all the Chinese cars eating their lunch.
As long as I get really solid AI models for cheap that do what I need I don’t care if they’re Chinese or otherwise.
I’ll still never use Grok from SpaceX AI cuz eww no, I have principles. ;-)
Cheaper?
This seems like a hot take not really informed by experience.
Please read my essay on context window saturation. I am also the author of deepcq (intent-aware memory querying & context mgmt for LLM agents)
This.
When working on hard problems (not "vibecode me a script to show an alert box", but e.g. "let's see what this three-level LUT-state-machine obfuscated binary does"), hitting the 1M (!) context window with Claude Code feels like you were talking to Claude Claudewski when his shift just abruptly ends, he packs his things, throws the office keys at Claude Claudeson in-between the front door frame while handovering like "Hi! Nice to see you, good luck." and now here we go again, you are working with someone who just experienced an acute amnesia. It tries everything it already tried, everything it was told in the initial prompt to not do, everything it was told in follow-up prompts not to do. "You were right, this approach does not work and we don't have 20 TB RAM on this machine for full symbolic execution, let me try..."
In Codex, it's so seamless that I sometimes just notice "wait, the context was 20 % remaining, it is 70 % now, wow, when did this happen", while it seamlessly works on the task. Basically never had an issue with context on Codex, be it coding features, cracking hard crack-me ciphers, or researching basically anything.
(For full disclosure, my last experience with Claude was a few weeks ago when I cancelled the subscription, maybe they fully reworked the traumatic "Summarizing" - "Oh, hi! Where are we? Who am I? What we are doing? This is taking too long, let me take a shortcut..." lobotomy they were doing in the meantime.)
256k is enough when the harness uses it properly and the model is not stupid. And also when the tokenizer is not tuned to invoice as many tokens as possible...
https://platform.claude.com/docs/en/about-claude/models/what...
Claude might as well not even do it in my experience.
They did exceed the expecations of pretty much everyone! I've also blogged about using the model, it's a bit on the slow side but pretty good!
> so were totally surprised and overwhelmed by the requests ... or do they just don't have the hardware being in china?
Yes, this is mostly the case: https://x.com/Kimi_Moonshot/status/2078855608565207130
As a user, I much prefer that to service disruptions or severely degraded or secretly quantized performance. However if I didn't have an account, I'd be pretty pissed off about not being able to give them money and become a user.
I am now rather pretty pissed towards antrophic for stopping my flow and forcing me to search for alternatives.
Kimi had become that popular. I was a subscriber of Kimi back when latest version was Kimi K2. Later I unsubscribed because I jumped over to GLM subscription (they had amazing deal). Now when I wanted to try out Kimi K3 to find out what the fuzz was all about, I couldn’t subscribe to them.
I remember reading a post from Moonshot team about this, they are doing this because they are almost at peak capacity and want to reserve it to keep the quality for their current customers.
We are actually witnessing an open-weight model catching up at catching mainstream users attention. And instead of behaving like Anthropic, they actually care about their users experience.
When they first announced, I created an account but didn't subscribe, so I am stuck on a waitlist now.
See Kimi Vendor Verifier.
A nice thing about open router is you can specify filters, like "US hosting without data retention"
I was able to press "Join waitlist" and then within 48 hours got accepted. The limits aren't very high, no where near the endless subsided+resets given on ChatGPT/Claude. I recommend ChatGPT for good value output!
Others here mentioned "the providers could be quantizing it!" but some of the providers on OpenRouter have partnered with Moonshoot and OpenRouter shows the int when you expand on the provider.
It should say "mxfp4" but providers like Baseten report FP8.
It's not ideal that people can't join but as a user I am happy that they focused on serving existing customers.
I’m primarily using gpt-5.6 and then opus for reviews.
I forgot why.
But I can't use an AI cli without that feature.
Is this the exact same model just with less VRAM allocated for context window?
Doubt these are related, but it made me laugh a little.
https://www.reddit.com/r/kimi/s/BFa1TR9vNg
HN: https://news.ycombinator.com/item?id=47934437
Reddit: https://www.reddit.com/r/worldnews/comments/1sxzzop/comment/...
Full: https://reddit.com/r/kimi/comments/1v9aqsi/comment/p0cbe0n/
Mirror: https://redlib.us.catsarch.com/r/kimi/comments/1v9aqsi/k3256...
First you spend like 2-3 hours working on a plan, once you have that you just tell the model to go and implement it, do adversarial sub-agent review loops before each commit and also make sure that all tooling and tests pass (including coverage requirements). You do need to poke it in a slightly different direction every few hours, though. Not even any novel work, just some refactoring and SSE notification hardening, bug fixes, alongside environment tuning and getting rid of some bottlenecks (also migrated from Oracle to PostgreSQL but that's mostly done).
That said, Kimi somehow manages to use less context in the main thread than Anthropic's models (even when you use sub-agents and also dynamic workflows in Claude Code), might have something to do with either how the model is tuned or their Kimi Code harness - because even in most of the longer form sessions it doesn't seem to fill up quite as quickly (note: because the kimi vis tool doesn't have a full summary view across all agents, these are the main long running agent stats across some sessions, not sub-agents):
I could see 256k context being sufficient for all sorts of work, even if intermediate progress/plan tracking files and docs might have to be used along the way, in addition to whatever plan support the harness has (for example, if you document something that will be relevant for load testing you might need that in 10 turns but not during the ones before then).If I feel like the model and I explored a lot of options I won't want to keep context as it might be confusing.
I think the more you use it the better judge you are of whether you should purge, compress, or just keep the context before executing the plan.
I use handoff skill to ask model to write a prompt for itself.
I've found Opus 5 far better as a subagent with very limited context window use, which could suggest that it might not have good long context performance unlike its predecessors. (I was one of many tearing my hair out trying to work with Opus 5 for the past week.)
Yes, pretty much - if there’s a lot of noise and jumping around and wrong conclusions and corrections, compress and only leave the correct stuff (maybe make some plan file briefly mention what NOT to look at/do). But if it’s all fairly straightforward then can just proceed with the execution.
Most of the time the planning stage ends up short of 200k tokens anyways, it mostly takes hours just cause I’m slow and need to explore the various options - still cheaper than building the wholly wrong thing and having to redo everything.
Compressing the context can also drop important information so it might be better to only do that when you need to / use the plan mechanism/files / do it after completing some large stage of the plan and so on - so a judgement call.
A greenfield project would probably allow at least 2x fewer tokens to be used in most of those long tasks, but I was mostly after consistency and bug fixes along the way as needed.
The model is then given a tool that allows the model to decide to roll back to a checkpoint + a message containing any additional useful information.
It's specifically instructed to use that tool[1] in cases like when it has inadvertendly read a large file where most of the content is not relevant to the task, or after a web search where it's found what it's looking for but most of the content isn't needed, or when it's written code that didn't work as expected, or similar.
It basically lets the model backtrack and "forget" irrelevant details at the end of the context but give itself hints on how it should continue from the checkpoint.
Though, interestingly they seem to be abandoning it in their new CLI (kimi-code), unless it's been folded into other functionality. Not sure if they just feel it's not needed any more with their newer models or if it just didn't work as well as they expected.
[1] named "D-Mail", or "DeLorean Mail" in a reference to Steins;Gate, which again references Back To The Future. See https://github.com/MoonshotAI/kimi-cli/blob/main/src/kimi_cl... and https://steins-gate.fandom.com/wiki/D-Mail
Haven't gone back to it, have been using Claude Code on a private project I'm still architecting.
https://github.com/Giancarlos/Culebral
But yeah, I have no idea about anything about software because you made an assumption off very little to go by.
Of course - Anthropic and OpenAI have an advantage in the amount they can subsidize the usage, but I think those days are waning.
Codex and Claude require editing a .json file, but most other harnesses have direct connections via a /provider or /login command.
For pressure at the country level leading to this kind of thing I think it's very unlikely. Here in Sweden it wouldn't just require a vote in the Swedish parliament and before this there'd have to be förarbeten and you can't just brazenly push things through with insane arguments, Swedish social convention goes against it-- and there's just no way to get it through.
It also might not even be legal. "We aren't at war with China and I'm a communist, and the US LLMs are so aligned with values inimical to my political ideology that this is interference with opinion formation" might be an actual legal argument that the ECHR or CJEU might actually have to accept.
I guess it sucks if one wants to expert broadly, but if you're big on vertical integration and the Japanese won't sell I guess you take what you can get.