The Network is NOT Reliable

The 8 Fallacies of Distributed Computing (Peter Deutsch):

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

Moving from Monolith to Microservices means accepting these challenges.

Communication Styles

1. Synchronous (Request/Response)

• Protocol: HTTP (REST), gRPC
• Flow: Client waits for a response.
• Pros: Simple, real-time feedback, familiar.
• Cons: Temporal coupling (both must be up), cascading failures, latency adds up.

2. Asynchronous (Event-Driven)

• Protocol: AMQP (RabbitMQ), Kafka
• Flow: Fire and forget.
• Pros: Decoupling, better scaling, resilience.
• Cons: Complexity, eventual consistency, harder to debug.

The CAP Theorem

In a distributed data store, you can only provide two of the following three guarantees:

1. Consistency (C): Every read receives the most recent write or an error.
2. Availability (A): Every request receives a (non-error) response, without the guarantee that it contains the most recent write.
3. Partition Tolerance (P): The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network.

Reality: In distributed systems, P is mandatory.
You must choose between CP (Consistent but unavailable during failures) or AP (Available but potentially inconsistent).

Consistency Models

Strong Consistency (ACID)

• Data is consistent immediately after a write.
• Typical in Monoliths (Single DB Transaction).
• Hard to scale in distributed systems.

Eventual Consistency (BASE)

• Basically Available, Soft state, Eventual consistency.
• Data will become consistent eventually (seconds, minutes).
• "The patient is registered, but might not appear in search results for 2 seconds."

What happens when things fail?

In a Monolith, if function A calls function B and B fails, the transaction rolls back.

In Microservices, if Service A calls Service B via HTTP:

• Network timeout?
• Service B is overloaded (503)?
• Service B throws an error (500)?

Risk: Cascading Failure. Service A waits -> Thread pool fills up -> Service A dies.

Resilience Patterns

Timeouts
Stop waiting after X seconds. Fail fast.

Retries
Try again (maybe it was a blip). Danger: Can DdoS your own service.

Circuit Breaker
If Service B fails repeatedly, "open" the circuit. Stop calling B immediately. Return an error or a default value (Fallback).

Bulkhead
Isolate resources. A crash in "Image Processing" shouldn't crash "User Login".

Exercise: Create a microservice that calls another microservice via REST.

Create a service that handles "Consultations".

When a consultation is created, it needs to fetch Patient details from the Patient Service.

The Problem with "Request/Response"

Imagine Consultation Service needs to do 3 things when a consultation ends:

1. Create an Invoice (Accounting Service).
2. Send a Summary Email (Notification Service).
3. Update Search Index (Search Service).

If using HTTP (REST):

• Consultation Service waits for Accounting (200ms)
• ... waits for Notification (500ms)
• ... waits for Search (100ms)

Result: The Doctor waits ~1 second. If Notification fails, does the whole Consultation transaction roll back?

Inversion of Control

Instead of the Consultation Service telling others what to do...

It simply shouts to the world:
"I just finished Consultation #123!"

It doesn't care who listens. It doesn't wait for a response.

Commands vs. Events

This distinction is critical.

Command (Imperative)

• "Create Invoice"
• Sent to a specific target.
• Expects a result (Success/Fail).
• Intent: I want something to happen.

Event (Declarative)

• "Consultation Completed"
• Broadcast to anyone listening.
• No immediate result expected.
• Intent: Something has happened.

The Architecture Components

1. Producer (Publisher): The service where the state change happened.
2. Message Broker: The post office (RabbitMQ, Kafka). Buffers messages if consumers are down.
3. Consumer (Subscriber): Reacts to the event.

The "Dual Write" Problem

Advanced Warning

// Danger Zone!
database.save(consultation); // 1. Commit to DB
eventBus.send(event);        // 2. Send to Broker

What if Step 1 succeeds but Step 2 fails (Network crash)?

You have a consultation in DB, but no invoice will ever be generated.

Your system is inconsistent.

Solutions: Transactional Outbox Pattern (We will discuss this later).

Idempotency

Network Rule: Messages might be delivered more than once.

If RabbitMQ sends "Consultation #123 Completed" twice... Do we charge the patient twice?

Idempotent Consumer: "The property that applying an operation multiple times has the same effect as applying it once."

if (invoiceRepository.existsFor(consultationId)) {
    return; // Ignore duplicate
}
createInvoice();

Exercise: Modify the Consultation Service to publish an event when a consultation is completed.
Consume that event in the PatientManager service to log the completion.

Example can be found here:

Practice Manager Repo: https://gitlab.com/etherealy/microservices-practicemanager

Consultation Service Repo: https://gitlab.com/etherealy/consultations

/!\ Examples for events are in branches /!\

Decentralized data management

• Data consistency approach:

  • Favor eventual consistency over distributed transactions (asyncronous)
  • Use compensating operations to handle errors
  • Accept trade-off between consistency and responsiveness
  • Can use the Saga Pattern to manage distributed transactions

• Key benefits:

  • Greater flexibility in data storage choices
  • Reduced coupling between services
  • Better alignment with business realities

• If the domain requires strong consistency, you may prefer to use synchronous communication between services. Or reconsider the domain model. Or even the microservices architecture.

Distributed Transactions: The Saga Pattern

The Challenge:
You cannot use a single database transaction (ACID) across multiple microservices.

The Solution:
A Saga is a sequence of local transactions. Each service updates its own database and publishes an event or message to trigger the next step.

The "Undo" Button (Compensation):
Since you cannot "rollback" a committed database transaction, if a step fails, you must execute Compensating Transactions to undo previous changes.

Example: If "Charge Card" fails, trigger "Cancel Hotel Reservation".

This should be used sparingly due to complexity!

Saga Implementation Styles

1. Choreography (Event-Based)

• Decentralized: Service A emits an event, Service B listens and reacts. No central coordinator.
• Pros: Simple to start, loose coupling.
• Cons: Hard to visualize the complex flow, risk of cyclic dependencies.

2. Orchestration (Command-Based)

• Centralized: An "Orchestrator" service (or state machine) tells participants what to do via commands.
• Pros: Clear flow logic, easier error handling.
• Cons: The Orchestrator can become a "God Service".