[LRUG] Testing SOAs

[Paul Robinson]

On 23 June 2014 19:46, Jonathan <j.fantham at gmail.com> wrote:

> Each team I've worked with in a SOA environment has a different approach
> to the way they isolate their services from each other for testing. E.g.
>
>  - use VCR to connect to the genuine service the first time and save a
> response
>  - create clients for your services, and use mock clients during testing,
> or stub the real clients explicitly in tests.
>  - create fake services that respond to the same interface as the real
> service but just return dummy responses
>

<snip>

The design of that SOA might well be wrong.

I'm not being sniffy when I say that.

When I learned SOA I thought that was the way to do it as well, and then
realised that only makes sense inside a bank. I am not sure it makes sense
inside a modern Internet application.

We have a couple of APIs in place on our current product designed this way,
and we had a nightmare getting them tested (we lost weeks to trying to get
VCR to work in our slightly odd scenario), because each service was too
tightly coupled to other services. This meant as I made a request into
service A, it would need to talk to services B and C, one of which needed
to talk one database, and the other needed to talk to another distinct
database.

This meant that ultimately to test service A properly I had to somehow mock
two other services and two databases, even if just mocking it at a contract
level - at minimum I was storing responses from DBs far, far away.

That got painful, and quickly too.

Further, it caused us at a design level to put more and more into larger
APIs because we wanted to reduce the number of services so that testing was
a little less painful. This made things more fragile and we started having
this "one big API to rule them all" that was backed by a monstrously large
DB. We called it an SOA but really - if we were honest with ourselves - we
talked about "the API", not "one of the APIs". When you do that, stop what
you're talking about and take a deep breath in through the nose. That
smell? Pure code smell.

On the new product we're building at the moment we took the opportunity to
step back and make things a bit more flexible: we went for a micro-service
architecture. What I mean by that is that instead of a small number of APIs
each doing a lot, now when we see an API with more than a few controllers,
we split it up - we try and stop it doing too much.

You should be able to read the entire app directory for one of our APIs in
under a half hour, and know what its doing. We have over a dozen such
services. I have one service whose only job is to take thumbnails of live
video every 60 seconds. That has its own entire application stack for
something that to most people would be a single controller action. We
really try and isolate functionality.

Critically, this meant we could decouple the services from each other, and
that makes testing a little easier.

Each service has all the information it needs to fully answer the requests
it alone is responsible for. In answering a request it will likely talk to
a DB, Redis, Neo4j or some other persistence/storage layer, but it does not
start firing off requests to other APIs for the most part (with one
exception around user authorisation checks we've abstracted into a gem and
put some caching in - it gets tested on its own in isolation).

Communication with other services takes one of two forms:

1. I need this other service to do something. I will send it a message to
do so.
2. I have done something. Others may be interested and wish to do something
themselves.

Scenario 1 is addressed with a message queue (we use SQS) and scenario 2 is
addressed with a PubSub type message architecture (we use SNS). Other
technologies exist that are almost drop-in replacements for them if you are
AWS-averse.

This means we only need to test that a message was created, or what the
code would do on receiving a specific kind of message:

   Given a user exists with username "paul"
   When the user updates their username to "pablo_the_magnificent" # I
always wanted to be a magician
   Then a message is created on the SNS queue for "user_updates"
   And the message has ...

Or

  Given a message exists on "user_updates" specifying a username has changed
  When I consume that message
  Then my copy of that data for the user is updated...

At this point I can hear much wailing and gnashing of teeth. You will say
"wait, user data exists in multiple services, and you have to keep it in
synch? What?!" Some will argue this is de-normalisation gone crazy.

The textbooks we all read about SOA and normalised data are no longer
relevant to my mind. They were designed for predictable environments inside
data centres with low latency, high bandwidth interconnections that can be
improved with a patch of cable, and all in an era when storage was
expensive.

None of that applies to a modern web application.

If you have a fault-tolerant distributed service that is able to consume
and create messages, and you're able to bring it back into synch after a
catastrophic failure, it causes no problem to say "this service needs to
know this information about a user" and for that data to be passed around
and replicated in multiple services.

One other caveat you might have is around the app which builds a rendered
view to be returned to the browser: you can't do that with asynchronous
message passing, right? You need that to be a direct connection to the APIs
to build the view, right? Yup, you're right. Which is why on the new
product we got around that in the easiest way possible: we're not using
Rails or any other service for that layer, because we decided that's a job
for the client, not for a server - you don't want a service there, you want
Ember, Angular, etc. running in your browser building the page based on
what it gets back directly from the services.

By building a client-side application that talks to APIs directly, suddenly
you're decoupling even more, you remove a SPoF, and your code gets cleaner.
Tiny APIs that standalone and can be tested independently and a client that
you can serve off a static web server? Beautiful. Even better, I no longer
have to spend a month training a new dev on "the API", I can get them up
and running shipping code on a service in under a day.

Test suites are small and localised, and the only time we need to be
careful is when changing a message format because it might break other
services - so we don't do that very often, and even then there are
workaround (abstract message creating/reception into its own gem).

You might read the above and think "you're utterly crazy" and decide it's
not for you, so I offer an alternative: at a minimum break your app up into
gems, and be clear about separating out what is client side and what is
server side within those gems and have one side test the other.

Design and build functionality in a way that can stand alone outside of
your app and be included as a gem. Test that on its own in isolation. Your
app then just brings it in via your Gemfile and you don't need to test the
internals of that library which is talking to other services within your
app.

I personally think that's harder to do and potentially more brittle (what
does an integration test do at this point), but we've done it in a few
cases where we knew we'd end up repeating code (user auth stuff, some neo4j
graph stuff, some message passing/parsing, etc.).

Hope that helps, or at least is food for thought,

Paul Robinson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20140624/6fa9086a/attachment-0003.html>