4 Message delivery and processing (55)

Exactly-once processing refers to the ideal scenario in which each message is processed only once, whereas exactly-once processing semantics means the the system behaves as though each message is processed only once.

Having a concise mental model will help you get the right feel for the system but if you are still working on the model or not quite understanding the entire system these gotchas will have an affect.

4.1 Exchanging messages (56)

A change in a components state is referred to as a side effect.

If we are to avoid any Byzantine failures we can make some assumptions about the overall state of the system:

We might assume the the system is Synchronous most of the time and Asynchronous at other times. 
Components will fail due to crashes, but they may also recover. 
Network may reorder, drop, or duplicate messages.   

For the rest of the chapter we will think in terms of components having a dialog. Both sides will be sending and reviving messages but we will think of the one submitting the request as the sender and the component submitting the response as the receiver

We will loot at a Petri Nets model for the figure 4.1 and 4.2, This is a model for the dynamic aspects of a concurrent system. You can think about this as a board game in some respects as you can use a token (character token) and watch them go from one action to the next. Tokens are used and passed from one state to the next. This is also a way of describing places where uncertainty can take place, or possibly failures.

You can see a place for a failure between any send message and the receiver receiving that message. There is uncertainty from the time that the sender sends the message all the way until the receiver sends a response.

Looking at figure 4.2:

1. We see that we start off with a token at the sender and then we send the request.
2. Then the token moves to a place of possible failure and uncertainty.
3. Then the receiver gets the request and we again are in a possible failure place. 
4. Then the message is processed and again an other point for failure.
5. The receiver then sends a message back to the sender, while it is in transit we have again a possible 
place for failure.
6. Only once the sender gets a message back are we in a state the we have certainty and now points of failure.

We can think of message exchange as being 2 separate aspects:

message delivery the transfer of messages within the network.

message processing generation of response, and the creation of a side effect.

4.2 The uncertainty principle of message delivery and processing (58)

A sender can only know its own state and that of the channel to the network. Again uncertainty is always going to happen after a message is sent.

4.2.1 Before sending the request (59)

Can be certain that no work has been done and no side effect have occurred.

4.2.2 After sending the request and before receiving a response (59)

Now knowledge about any work that has been done. The sender will not be able to determine about any of these state.

¡ The request was lost in the network.
¡ The receiver crashed before processing the message.
¡ The receiver crashed after processing the message.
¡ The response was lost in the network.
¡ The network or the receiver is slow.

4.2.3 After receiving a response (59)

We are now certain of what has occurred. We know that the right work was done or that we had a failure that we would know about.

4.3 Silence and chatter (60)

There are ways to determine if an message was not received in a Synchronous system but in the event of a Asynchronous system it is less likely. With this said we can find ourselves in a situation that we may not know if the receiver has gotten more than one message. There will be different events that can and will happen within our system.

In this way when we send a message and don’t receive a response back we have 2 options: move on or try again. If we where to move on we intend to deliver and process the message at most once. If we are going to try again we are intending to deliver and process the message at least once. Let’s look at some code.

requests := { }

# Event handler
on message request do

  # Request is in the received request.
  if request in received then
    return
  end

  # Tracks the request as received. 
  received := received + request

  # If a crash happens here a retry will not help as we have already 
  # received the request.
  :skull:

  process(request) // e.g. transfer balance

  return

end

# Lets look at an other example

requests := { }

on message request do

  if request in received then
    return response
  end
  process (request) // e.g. transfer balance

  # If we have a crash here the message is all ready processed a 
  # retry will process the message again
  :skull:

  received = received + request
  return response

end

4.4 Exactly-once processing semantics (62)

Exactly-once processing semantics focus on the processing outcome rather than the process itself. The idea here is that processing the message once is the same as processing the message two or more times.

  # === is equivalence not equals
  P(M) === P(M) * ... * P(M)

4.5 Idempotence (62)

idempotence refers to the property of an operation where applying the operation multiple times yields the same value as operation once:

  f(f(x)) = F(x)

You can also look at it the other way. Where once is equal to doing it more than once as well. It’s an iff (if and only if).

There will be guaranteed idempotent data structures like a write-once that sets the value to null.

Charging a credit card will not be idempotent as doing it more than once will result in more than 1 charge.

A way to avoid things like the credit card is to add in a UUID (universally unique identifier) that can be sent with the orginal message and any retries so that we only will process the message once.

requests := { }


on message request do

  if request in received then
    return response
  end
  # Process and track the message
  begin transaction
    # Both the next lines will need to happen or neither will happen
    process(request)
    received = received + request
  
  commit transaction
  return response
end

4.6 Case study: Charging a credit card (64)

We can now look at the process to buy a plane ticket. We want to ensure that the credit care is charged exactly once.

Stripe will include idempotence keys in the API and will only process once and then send the same message back (without processing again) not matter how many times they receive a message with the same key.

  Stripe’s idempotency works by saving the resulting status code and body of the first
  request made for any given idempotency key, regardless of whether it succeeded or
  failed. Subsequent requests with the same key return the same result, including 500
  errors.

This is a good way to deal with things but keep in mind that you will need to generate a UUID for the transaction. But even this will not guarantee a idempotent transaction. As if the entire app crashes or at a vital time you will then need to get an other transaction id so the UUID will be different.