Trigger an API outage and check that your code survives it

Read this page in French

A 200 is easy to get anywhere. A reproducible 503 is not. Yet refusals are what your retry, your backoff and your error handling are made of, and a real cloud gives you neither on demand nor twice the same way. feint arms them in one request: you name the operation, the status and how many times, then you watch what the official client does with it.

What you will get

By the end of this page, terraform apply will have survived two 503s and succeeded, you will read the refusal in each provider's own dialect, and you will tell a status failure from a truncated response.

• Arm a deterministic failure on a named operation.
• Observe the client's real reaction instead of assuming it.
• Tell apart the refusal, the slow answer and the half-written body.
• State what this test demonstrates, and what it leaves untouched.

Why a failure is harder to obtain than a success

Your code's happy path is tested by accident: every run walks it. The degradation path is only walked the day your provider has an incident, which means at the worst moment and with no witness. Teams work around that hole by simulating the failure in the client's own unit tests, which checks an assumption about the API rather than the API.

feint's injector moves the boundary: the real official client, its real error decoder and its real retry policy all go through a refusal you decided. What you measure is no longer an assumption, it is an observed behaviour of the client you will ship.

The lab: two 503s, then the answer

The example takes the operation everybody calls first on Scaleway, instance/v1/API.ListServers, and makes it fail exactly twice.

1. Start the emulator and point the official client at it.

   feint start
   eval "$(feint env scaleway)"

   feint env exports the variables the Scaleway SDK expects, endpoint included. What it sets is detailed in Wiring the official CLIs.

2. Arm the rule.

   curl -X PUT localhost:4599/_feint/faults -d '{"faults":[
     {"operation":"instance/v1/API.ListServers","status":503,"times":2}]}'

   The emulator answers the list of armed rules, each with its hits counter at zero and its spent flag at false. Nothing else changes: the other operations keep answering normally.

3. Call, and read the refusal.

   scw instance server list zone=fr-par-1

   Feint answered Service Unavailable because a fault rule is armed for this
   operation; GET /_feint/faults lists them

   The body carries its own origin. That is deliberate: a message that imitated a Scaleway incident would invent a fact about the provider, and six months later you would not know whether that 503 came from a test or from a real outage.

4. Check that the rule fired, and how many times.

   curl -s localhost:4599/_feint/faults | jq '.faults[] | {operation, status, times, hits}'

   {
     "operation": "instance/v1/API.ListServers",
     "status": 503,
     "times": 2,
     "hits": 1
   }

   This counter is the heart of the test. A client that gives up on the first refusal leaves hits at 1; a client that retries leaves 2, then gets the real answer on the third call. You do not have to believe the SDK's documentation about its retry policy, you read it.

5. Disarm.

   curl -X DELETE localhost:4599/_feint/faults

   A forgotten rule is a ghost outage in the next test. The DELETE empties the set and answers the empty list.

The case Terraform users actually want

Arm {"status":503,"times":2} on an operation your configuration reads, then run terraform apply. If the apply completes, you have just proved that the provider retries and that your pipeline survives a brief outage of the control plane. That is the scenario which, on a real cloud, breaks a deployment once a quarter with nobody able to reproduce it.

The injector's four guarantees

These properties are not intentions, they are refusals written into the code, and each one is verifiable from your terminal.

Off by default. A freshly started emulator arms nothing. No test can fail because of a rule inherited from somewhere else.

Deterministic. The rule says "the first N calls of this operation", never "one in ten". A random failure cannot be the subject of a test: it produces an error that a replay does not reproduce.

Per operation. The target is the upstream name published by /_feint/routes. A rule aiming at an operation nobody serves is refused when you write it, not silently ignored:

curl -X PUT localhost:4599/_feint/faults -d '{"faults":[
  {"operation":"instance/v1/API.ThisOperationDoesNotExist","status":503,"times":1}]}'

HTTP 400
no route serves the operation "instance/v1/API.ThisOperationDoesNotExist":
a rule that never fires reads exactly like a client that survived the fault

The sentence states the trap it avoids: a rule that never fires reads exactly like a client that survived. Without that refusal, you would conclude your code is resilient when nothing tested it.

In the provider's own dialect. A refusal does not have the same shape on every cloud, and that is precisely what your code has to decode.

How each cloud says the same outage

Three envelopes, one mechanism. The table below is measured on 0.13.0, arming the same rule on each pack.

Provider	What the client receives	What your code must recognise
Scaleway	scaleway-sdk-go: insufficient permissions on a 403	the SDK's error type, here PermissionsDeniedError
Outscale	{"Errors":[{"Type":"FeintInjectedFault",…}],"ResponseContext":{…}}	the Errors envelope and its Type field
Exoscale	{"message":"feint answered Service Unavailable…"}	a bare message, with no code to branch on

Glissez pour voir

This asymmetry belongs to the real clouds, it is not an emulator shortcut: Outscale's envelope has a field for a marker, Exoscale's has none. Code that branches on the message text works on one and breaks on the other, and this test makes that visible before production.

Each pack only renders the statuses it knows how to write. Asking for one outside that list is refused, with the list:

rule "instance/v1/API.ListServers": the scaleway pack does not render 418;
it renders [401 403 429 500 502 503]

Beyond the status: slowness and the truncated body

An API does not always fail cleanly. It drags, or it cuts a response in half, and those two failures break clients that survive a clean 503 perfectly well.

The delay is declared as a Go duration and applies before the answer:

curl -X PUT localhost:4599/_feint/faults -d '{"faults":[
  {"operation":"instance/v1/API.ListServers","delay":"3s","times":1}]}'

Measured on this machine, the first call takes 3.010 s and the next one 0.007 s: the rule is spent and the emulator is back to its normal speed. That is the shape of an API that hangs, the one that exposes a client with no timeout. An absurd delay is bounded rather than accepted, because a value like that is almost always a typo:

rule "…": delay 30m is longer than 5m0s, which is a typo more often than an intent

The truncated body keeps the handler's own status and cuts the response to N bytes. The official client does not recover, and the message it produces is the one you will see in production:

curl -X PUT localhost:4599/_feint/faults -d '{"faults":[
  {"operation":"instance/v1/API.ListServers","truncate_bytes":12,"times":1}]}'
scw instance server list zone=fr-par-1

scaleway-sdk-go: could not parse application/json response body: unexpected EOF

A 200 carrying incomplete JSON is the scenario of the proxy that cuts, the gateway that expires, the interrupted page. It never happens in a happy test, and it produces a named error here that your code can learn to handle.

How to test a client that waits for a resource to be ready

From 0.13.0, the --consistency eventual flag walks resources through the transient states of a real cloud, which finally gives waiting code something to observe. By default the emulator settles every action at once: a volume is born available, a rebooted server answers running from the first read to the last. That is fast and deterministic, but a waiting loop never meets the state it watches for, and a defect in that loop goes unnoticed.

feint serve --consistency eventual

The contrast takes two commands to measure. Without the flag an Outscale volume is available immediately; with it, the volume is born creating and the next read finds it available.

octl iaas api CreateVolume --Size 10 --SubregionName eu-west-2a --VolumeType gp2 --jq '.Volume.State'

creating

octl iaas api ReadVolumes --jq '.Volumes[-1].State'

available

States advance on reads, never on a clock. That is a design decision rather than a shortcut: a state timed on a wall clock would either never end under a frozen test clock, or turn every suite into a waiting room. A four-second suite stays a four-second suite, and the number of reads a transition takes is reproducible from one run to the next.

The mechanism makes refusals reachable that did not exist here before. A snapshot requested while the volume is still creating gets the answer measured against a real account, with its status, its type and its code.

{"Errors":[{"Code":"6007","Details":"the volume vol-a88ea433 is still creating; a snapshot can be taken once it is available","Type":"InvalidVolumeState"}]}

On the Scaleway side it is the lifecycle actions that become observable, in the order recorded on fr-par: poweron answers starting then running, poweroff answers stopping then stopped, and reboot answers stopping, starting, then running. That last case is what justifies the whole mechanism: a reboot's target state is the state it started from, so a client that merely checks state == "running" believes it waited when it waited for nothing.

Two guards are worth knowing, because they decide what your test proves. An action is not an observation: the read an action performs in order to change a resource consumes no state, or a reboot would swallow the stopping it has just pushed. And a failed action walks no chain: a start that failed answers its failed state, without narrating a path towards a running it never reached.

The default mode does not move

Without the flag the emulator behaves exactly as before, byte for byte, and the conformance suite runs in that mode. One deliberate exception: a chain that is an action's only signal, such as a reboot's, is walked in both modes, or a reboot would be invisible to every client that did not ask for the flag.

What this test proves, and what it does not

This is the most important distinction on the page, and it is written into the emulator's counters.

What you have just demonstrated is a fact about your client: that scw decodes a permissions_denied body as a rights refusal rather than as a missing resource, that terraform apply survives two 503s on a read, that your code exits cleanly when the JSON arrives truncated. Those are the facts you need, and they were unobservable otherwise.

What you have not demonstrated is a fact about the real cloud. A 403 injected here does not establish that Scaleway answers 403 to that call, or with those fields. The body's shape comes from the provider's SDK, but the choice of moment comes from you.

An injected answer moves no counter

The emulator counts injected answers separately, under the injected key of /_feint/conformance, and the operation stays marked as not driven:

curl -s localhost:4599/_feint/conformance | jq '{injected}'

{
  "injected": {
    "instance/v1/API.CreateServer": 1,
    "instance/v1/API.ListServers": 1
  }
}

The consequence is deliberate: fault injection cannot inflate the very number it exists to expose. The project's conformance suite even refuses any run that carries one. What those counters measure, and why they never add up, is covered in Measuring fidelity.

Troubleshooting

Symptom	Cause	Fix
HTTP 400 when arming, "no route serves the operation"	Wrong operation name, or the emulator does not serve it	List the exact names: curl -s localhost:4599/_feint/routes
The rule is listed but hits stays at 0	The client calls another operation than the one you targeted	Read the call log: curl -s localhost:4599/_feint/trace
HTTP 400, "the pack does not render 418"	The status has no shape in that pack's dialect	Pick one from the list the message prints
A later test fails for no reason	A rule stayed armed	curl -X DELETE localhost:4599/_feint/faults at the end of each test
The client hangs forever	A delay is armed and the client has no timeout	That is the defect you were looking for: add a client timeout

Glissez pour voir

Key takeaways

• An outage is armed in one request: operation, status, number of times.
• Determinism is the condition of the test: "the first N calls", never a probability.
• A rule that cannot fire is refused up front, because it would look like a success.
• The refusal arrives in the provider's dialect, and the three dialects differ.
• Status, slowness and truncated body are three distinct failures, and the third breaks different clients than the first.
• An injected answer proves nothing about the real cloud and counts towards no fidelity metric.

Next steps

Diagnose a run When a test really fails: doctor, status, logs and the call log.

Test in a pipeline Replay these scenarios in CI, with no cloud credentials.

Measure fidelity The emulator's counters, and why they never add up.

What feint proves The verdict matrix, row by row, with what establishes each one.

×

📖 Voir la documentation →