RFC-7807 problem details with Spring Boot and JAX-RS

No Comments

Application specific problems, e.g. a missing field in a client request, have to be handled properly with machine readable and human friendly custom business error codes — in RESTful web services using JAX-RS, Spring Boot, or any other technology. Only all too often we pay all too little attention to failures and focus only on the happy path. And if we do think about failures, we want to get it done as quickly as possible. We fear that doing it right, i.e. using standardized, machine readable as well as human readable problem descriptions, is hard. But actually it’s not, you just need to know how to do it. And this post is going to show how easy it can be.

Problems

Say you’re writing an order processing system. In some situations a customer may not be qualified to use a certain payment method for an order. You want to present this problem to the user, either in a web frontend or as the result of an http API call. And the clients need to be able to react to this situation differently than when, e.g., the user account balance is insufficient for the order placed.

You look into the http specs and find code 405, “Method Not Allowed”. Sounds exactly like what you need. It works in your tests, and for some time in production. But then, a routine update to a load balancer breaks your system. Quickly, finger-pointing between dev and ops ensues and eventually a full blame war breaks out. It looks as if the update done by ops caused the problems, but they claim that there is no bug in the load balancer and they have to update due to security issues with the old one. There was no new release of the application, how could dev be responsible? But actually the blame is on you, dear developer: you misused a technical code with a specific semantic in order to express a completely different business semantic — and that’s never a good idea. In this case, it’s explicitly permitted to cache the 405 code, so a load balancer is allowed not to call your service but return the error response directly.

The http status codes (see rfc-7231 or nicely formatted https://httpstatuses.com) precisely specify different situations, mainly fine-grained technical problems. Application specific problems are restricted to the generic 400 Bad Request (and a few others) or 500 Internal Server Error status codes, which can be used to signal general failures on the client side or server side. But we need to differentiate between many situations. How else can we communicate our various issues to our clients?

glass bulbs showing what, who, where, when, how, and why questions

You are probably aware that the http protocol allows to include a body (called ‘entity’ in the RFCs) in almost any response, not only with a 200 OK status after a GET request. I.e. it’s perfectly fine to include a body for a 400 Bad Request after a POST. Most systems display a custom html error page in this case. If we make this body machine readable, our clients can react appropriately. Defining a new document type for every endpoint or even every application is a lot of work: you not only have to write the code but also documentation, tests, and communicate all of it to the clients, etc., and the client has to use exactly this format for one request, and exactly that format for another, that’s just too much hassle. A standard would be nice — and actually, there is one: RFC-7807.

RFC-7807

This standard defines a media type application/problem+json (or +xml) and the standard fields to be used with their exact semantics. Here’s a short summary:

  • type: a URI to identify what type of problem occurred. Ideally it should be a stable URL to the documentation of the details about this type of error, e.g. https://api.myshop.example/problems/not-entitled-for-payment-method; but it also can be a URN, e.g. urn:problem-type:not-entitled-for-payment-method. In any case, changing the type is defined to be a breaking API change, so it’s safe for a client to use this to switch to different problem situations.
  • title: an informal, human readable short description of the general type of problem, e.g. You're not entitled to use this payment method. Can be changed without breaking the API.
  • status: repeats the response status code, e.g. 403 for Forbidden. There might be a difference between what the server threw and the client received due to a proxy changing the http status code. It’s only advisory to help debugging, so it can be changed without breaking the API.
  • detail: a human readable full description about what went wrong, e.g. Customer 123456 has only GOLD status but needs PLATINUM to be entitled to order the sum of USD 1,234.56 on account. can be changed without breaking the API.
  • instance: a URI identifying the specific occurrence of the problem. If this is a URL, it should provide details about this occurrence, e.g. point to your logs https://logging.myshop.example/prod?query=d294b32b-9dda-4292-b51f-35f65b4bf64d — note that just because it’s a URL doesn’t mean it has to be accessible to everybody! If you don’t even want to provide details about your logging system on the web, you can also produce a UUID URN like urn:uuid:d294b32b-9dda-4292-b51f-35f65b4bf64d. Can be changed without breaking the API.
  • All other fields are extensions, i.e. custom, machine readable fields; e.g. customer-status or order-sum. Extensions can also be complex types, i.e. lists or objects containing multiple fields, as long as they are (de)serializable. The client might want to display this to the customer. You can add new extensions without breaking the API, but removing extensions (or changing the semantics) is a breaking API change.

NOTE: It’s easy to say that the type URI needs to be stable. But it must. not. change. Even when you move your documentation to a different host or different wiki, rename packages or class names, or even rewrite your service in a different tech stack. And as error conditions are often not tested as thoroughly as they should be, it may even take some time for the break to become evident. So please be extra careful.

Spring Boot

The ideas and most of the code samples here are essentially the same as for JAX-RS. You may want to skip ahead to the JAX-RS part.

Server

Say we have a REST controller OrderBoundary (I use the BCE term ‘boundary’ here):

@RestController
@RequestMapping(path = "/orders")
@RequiredArgsConstructor ①
public class OrderBoundary {
    private final OrderService service;
 
    @PostMapping
    public Shipment order(@RequestParam("article") String article) {
        return service.order(article);
    }
}

①: We use the Lombok @RequiredArgsConstructor to create a constructor to be auto-wired.

The OrderService may throw an UserNotEntitledToOrderOnAccountException.

Spring Boot already provides a json error body by default, but it’s very technical. It contains these fields:

  • status + error: e.g. 403 and Forbidden
  • message: e.g. You're not entitled to use this payment method
  • path: e.g. /orders
  • timestamp: e.g. 2020-01-10T12:00:00.000+0000
  • trace: the stacktrace

We need to specify the http status code and message by annotating the UserNotEntitledToOrderOnAccountException:

@ResponseStatus(code = FORBIDDEN,
    reason = "You're not entitled to use this payment method")
public class UserNotEntitledToOrderOnAccountException
  extends RuntimeException {
    ...
}

Note that there is no stable field to distinguish different error situations, our main use-case. So we need to take a different route:

Manual Exception Mapping

The most basic approach is to catch and map the exception manually, i.e. in our OrderBoundary we return a ResponseEntity with one of two different body types: either the shipment or the problem detail:

public class OrderBoundary {
    @PostMapping
    public ResponseEntity<?> order(@RequestParam("article") String article) {
        try {
            Shipment shipment = service.order(article);
            return ResponseEntity.ok(shipment);
 
        } catch (UserNotEntitledToOrderOnAccountException e) {
            ProblemDetail detail = new ProblemDetail();
            detail.setType(URI.create("https://api.myshop.example/problems/" +
                "not-entitled-for-payment-method")); ①
            detail.setTitle("You're not entitled to use this payment method");
            detail.setInstance(URI.create(
                "urn:uuid:" + UUID.randomUUID())); ②
 
            log.debug(detail.toString(), exception); ③
 
            return ResponseEntity.status(FORBIDDEN).
                contentType(ProblemDetail.JSON_MEDIA_TYPE)
                .body(detail);
        }
    }
}

①: I chose to use a fixed URL for the type field, e.g. to a Wiki.
②: I chose to use a random UUID URN for the instance.
③: I log the problem detail and the stack trace, so we can search our logs for the UUID instance to see all details in the context of the logs that led to the problem.

Problem Detail

The ProblemDetail class is trivial (thanks to Lombok):

@Data
public class ProblemDetail {
    public static final MediaType JSON_MEDIA_TYPE =
        MediaType.valueOf("application/problem+json");
 
    private URI type;
    private String title;
    private String detail;
    private Integer status;
    private URI instance;
}

Exception Handler

This manual mapping code can grow quite a bit if you have many exceptions to convert. By using some conventions, we can replace it with a generic mapping for all our exceptions. We can revert the OrderBoundary to the simple form and use an exception handler controller advice instead:

@Slf4j
@ControllerAdvice ①
public class ProblemDetailControllerAdvice {
    @ExceptionHandler(Throwable.class)public ResponseEntity<?> toProblemDetail(Throwable throwable) {
        ProblemDetail detail = new ProblemDetailBuilder(throwable).build();
 
        log.debug(detail.toString(), throwable); ③
 
        return ResponseEntity.status(detail.getStatus())
            .contentType(ProblemDetail.JSON_MEDIA_TYPE)
            .body(detail);
    }
}

①: Make the actual exception handler method discoverable by Spring.
②: We handle all exceptions and errors.
③: We log the details (including the instance) and the stack trace.

The interesting part is in the ProblemDetailBuilder.

Problem Detail Builder

The conventions used here are:

  • type: URL to the javadoc of the exception hosted on https://api.myshop.example/apidocs. This may not be the most stable URL, but it’s okay for this demo.
  • title: Use the simple class name, converting camel case to spaces.
  • detail: The exception message.
  • instance: Use a random UUID URN.
  • status: If the exception is annotated as Status use that; otherwise use a 500 Internal Server Error.
@Retention(RUNTIME)
@Target(TYPE)
public @interface Status {
    int value();
}

Note that you should be very careful with conventions: they should never bear any surprises.The ProblemDetailBuilder is a few lines of code, but it should be fun to read:

@RequiredArgsConstructor
class ProblemDetailBuilder {
    private final Throwable throwable;
 
    ProblemDetail build() {
        ProblemDetail detail = new ProblemDetail();
        detail.setType(buildType());
        detail.setTitle(buildTitle());
        detail.setDetail(buildDetailMessage());
        detail.setStatus(buildStatus());
        detail.setInstance(buildInstance());
        return detail;
    }
 
    private URI buildType() {
        return URI.create("https://api.myshop.example/apidocs/" +
            javadocName(throwable.getClass()) + ".html");
    }
 
    private static String javadocName(Class<?> type) {
        return type.getName()
            .replace('.', '/') // the package names are delimited like a path
            .replace('$', '.'); // nested classes are delimited with a period
    }
 
    private String buildTitle() {
        return camelToWords(throwable.getClass().getSimpleName());
    }
 
    private static String camelToWords(String input) {
        return String.join(" ", input.split("(?=\\p{javaUpperCase})"));
    }
 
    private String buildDetailMessage() {
        return throwable.getMessage();
    }
 
    private int buildStatus() {
        Status status = throwable.getClass().getAnnotation(Status.class);
        if (status != null) {
            return status.value();
        } else {
            return INTERNAL_SERVER_ERROR.getStatusCode();
        }
    }
 
    private URI buildInstance() {
        return URI.create("urn:uuid:" + UUID.randomUUID());
    }
}

You can extract this error handling into a separate module, and if you can agree on the same conventions with other teams, you can share it. You may even simply use a problem detail artifact defined by someone else, like mine 😜, which also allows extension fields and other things.

Client

I don’t want to spill technical details all over my domain code, so I extract an OrderServiceClient class to do the call and map those problem details back to exceptions. I want the domain code to look something like this:

@RequiredArgsConstructor
public class MyApplication {
    private final OrderServiceClient client;
    public OrderStatus handleOrder(String articleId) {
        try {
            Shipment shipment = client.postOrder(articleId);
            // store shipment
            return SHIPPED;
        } catch (UserNotEntitledToOrderOnAccount e) {
            return NOT_ENTITLED_TO_ORDER_ON_ACCOUNT;
        }
    }
}

So the interesting part is in the OrderServiceClient.

Manual Problem Detail Mapping

Leaving the error handling aside, the code doesn’t look too bad:

public class OrderServiceClient {
    public Shipment postOrder(String article) {
        MultiValueMap<String, String> form = new LinkedMultiValueMap<>();
        form.add("article", article);
        RestTemplate template = new RestTemplate();
        try {
            return template.postForObject(BASE_URI + "/orders", form, Shipment.class);
        } catch (HttpStatusCodeException e) {
            String json = e.getResponseBodyAsString();
            ProblemDetail problemDetail = MAPPER.readValue(json, ProblemDetail.class);
            log.info("got {}", problemDetail);
            switch (problemDetail.getType().toString()) {
                case "https://api.myshop.example/apidocs/com/github/t1/problemdetaildemoapp/" +
                        "OrderService.UserNotEntitledToOrderOnAccount.html":
                    throw new UserNotEntitledToOrderOnAccount();
                default:
                    log.warn("unknown problem detail type [" +
                        ProblemDetail.class + "]:\n" + json);
                    throw e;
            }
        }
    }
 
    private static final ObjectMapper MAPPER = new ObjectMapper()
        .disable(FAIL_ON_UNKNOWN_PROPERTIES);
}

Response Error Handler

There’s also a mechanism on the Spring REST client side that allows us to generalize this handling:

public class OrderServiceClient {
    public Shipment postOrder(String article) {
        MultiValueMap<String, String> form = new LinkedMultiValueMap<>();
        form.add("article", article);
        RestTemplate template = new RestTemplate();
        template.setErrorHandler(new ProblemDetailErrorHandler());return template.postForObject(BASE_URI + "/orders", form,
            Shipment.class);
    }
}

①: This line replaces the try-catch block.

The ProblemDetailErrorHandler hides all the conventions we use; this time including some error handling. In that case, we log a warning and fall back to the Spring default handling:

@Slf4j
public class ProblemDetailErrorHandler extends DefaultResponseErrorHandler {
    @Override public void handleError(ClientHttpResponse response) throws IOException {
        if (ProblemDetail.JSON_MEDIA_TYPE.isCompatibleWith(
            response.getHeaders().getContentType())) {
            triggerException(response);
        }
        super.handleError(response);
    }
 
    private void triggerException(ClientHttpResponse response) throws IOException {
        ProblemDetail problemDetail = readProblemDetail(response);
        if (problemDetail != null) {
            log.info("got {}", problemDetail);
            triggerProblemDetailType(problemDetail.getType().toString());
        }
    }
 
    private ProblemDetail readProblemDetail(ClientHttpResponse response) throws IOException {
        ProblemDetail problemDetail = MAPPER.readValue(response.getBody(), ProblemDetail.class);
        if (problemDetail == null) {
            log.warn("can't deserialize problem detail");
            return null;
        }
        if (problemDetail.getType() == null) {
            log.warn("no problem detail type in:\n" + problemDetail);
            return null;
        }
        return problemDetail;
    }
 
    private void triggerProblemDetailType(String type) {
        if (isJavadocUrl(type)) {
            String className = type.substring(36, type.length() - 5)
                .replace('.', '$').replace('/', '.');
            try {
                Class<?> exceptionType = Class.forName(className);
                if (RuntimeException.class.isAssignableFrom(exceptionType)) {
                    Constructor<?> constructor = exceptionType.getDeclaredConstructor();
                    throw (RuntimeException) constructor.newInstance();
                }
                log.warn("problem detail type [" + type + "] is not a RuntimeException");
            } catch (ReflectiveOperationException e) {
                log.warn("can't instantiate " + className, e);
            }
        } else {
            log.warn("unknown problem detail type [" + type + "]");
        }
    }
 
    private boolean isJavadocUrl(String typeString) {
        return typeString.startsWith("https://api.myshop.example/apidocs/")
            && typeString.endsWith(".html");
    }
 
    private static final ObjectMapper MAPPER = new ObjectMapper()
        .disable(FAIL_ON_UNKNOWN_PROPERTIES);
}

Recovering the exception type from the URL is not ideal, as it tightly couples the client side to the server side, i.e. it assumes that we use the same classes in the same packages. It’s good enough for the demo, but to do it properly you need a way to register exceptions or scan for them, like in my library, which also allows extension fields and other things.

JAX-RS

If you’re not into JAX-RS, you may want to skip ahead to the Summary.

Server

Say you have a REST boundary OrderBoundary like this:

@Path("/orders")
public class OrderBoundary {
    @Inject OrderService service;
    @POST public Shipment order(@FormParam("article") String article) {
        return service.order(article);
    }
}

The OrderService may throw an UserNotEntitledToOrderOnAccountException and we want to map that to a problem detail.

Manual Exception Mapping

The most basic approach is to map it manually, i.e. we return a Response with one of two different body types: the shipment or the problem detail:

@Path("/orders")
public class OrderBoundary {
    @Inject OrderService service;
    @POST public Response order(@FormParam("article") String article) {
        try {
            Shipment shipment = service.order(article);
            return Response.ok(shipment).build();
        } catch (UserNotEntitledToOrderOnAccount e) {
            ProblemDetail detail = new ProblemDetail();
            detail.setType(URI.create("https://api.myshop.example/problems" +
                "/not-entitled-for-payment-method")); ①
            detail.setTitle("You're not entitled to use this payment method");
            detail.setInstance(URI.create("urn:uuid:" + UUID.randomUUID())); ②
 
            log.debug(detail.toString(), exception); ③
 
            return Response.status(NOT_FOUND)
                .type(ProblemDetail.JSON_MEDIA_TYPE)
                .entity(detail).build();
        }
    }
}

①: I chose to use a fixed URL for the type field, e.g. to a Wiki.
②: I chose to use a random UUID URN for the instance.
③: I log the problem detail and the stack trace, so we can search our logs for the instance UUID to see all details in the context of the logs that led to the problem.

The ProblemDetail class is trivial (shown above).

Exception Mapper

This manual mapping code can grow quite a bit if you have many exceptions to convert. By using some conventions, we can replace it with a generic mapping for all our exceptions:

@Slf4j
@Providerpublic class ProblemDetailExceptionMapper
    implements ExceptionMapper<Throwable> { ②
    @Override public Response toResponse(Throwable throwable) {
        ProblemDetail detail = new ProblemDetailBuilder(throwable).build();
 
        log.debug(detail.toString(), throwable); ③
 
        return Response
            .status(detail.getStatus())
            .entity(detail)
            .header("Content-Type", ProblemDetail.JSON_MEDIA_TYPE)
            .build();
    }
}

①: Automatically register the exception handler method with JAX-RS.
②: We handle all exceptions and errors.
③: We log the details (including the instance) and the stack trace.

The interesting part is again in the ProblemDetailBuilder shown above.

Client

I don’t want to spill technical details all over my domain code, so I extract an OrderServiceClient class to do the call and map those problem details back to exceptions. I want the domain code to look something like this:

public class MyApplication {
    @Inject OrderServiceClient client;
    public ResultEnum handleOrder(String articleId) {
        try {
            Shipment shipment = client.postOrder(articleId);
            // store shipment
            return SHIPPED;
        } catch (UserNotEntitledToOrderOnAccount e) {
            return NOT_ENTITLED_TO_ORDER_ON_ACCOUNT;
        }
    }
}

So the interesting part is in the OrderServiceClient.

Manual Problem Detail Mapping

The code is quite straight forward:

@Slf4j
public class OrderServiceClient {
    public Shipment postOrder(String article) {
        Response response = target()
            .path("/orders").request(APPLICATION_JSON_TYPE)
            .post(Entity.form(new Form().param("article", article)));
        if (ProblemDetail.JSON_MEDIA_TYPE.isCompatible(response.getMediaType())) {
            throw buildProblemDetailException(response);
        }
        return response.readEntity(Shipment.class);
    }
 
    private RuntimeException buildProblemDetailException(Response response) {
        ProblemDetail problemDetail = response.readEntity(ProblemDetail.class);
        requireNonNull(problemDetail.getType(), "no `type` field found in " + problemDetail);
        switch (problemDetail.getType().toString()) {
            case "https://api.myshop.example/apidocs/com/github/t1/problemdetaildemoapp/" +
                    "OrderService.UserNotEntitledToOrderOnAccount.html":
                return new UserNotEntitledToOrderOnAccount();
            default:
                return new IllegalArgumentException("unknown problem detail type [" +
                    problemDetail.getType() + "]:\n" + problemDetail);
        }
    }
}

Response Error Handler

There’s also a mechanism on the JAX-RS client side that allows us to generalize this handling:

public class OrderServiceClient {
    public Shipment order(String article) {
        try {
            Response response = target()
                .request(APPLICATION_JSON_TYPE)
                .post(Entity.form(new Form().param("article", article)));
            return response.readEntity(Shipment.class);
        } catch (ResponseProcessingException e) {
            throw (RuntimeException) e.getCause();
        }
    }
}

We completely removed the problem detail handling and extracted it into an automatically registered ClientResponseFilter instead (see ProblemDetailClientResponseFilter further down). The downside of using the JAX-RS client directly is that exceptions thrown by a ClientResponseFilter are wrapped into a ResponseProcessingException, so we need to unpack it. We don’t have to do that when we use a MicroProfile Rest Client instead:

public class OrderServiceClient {
    @Path("/orders")
    public interface OrderApi {
        @POST Shipment order(@FormParam("article") String article);
    }
 
    private OrderApi api = RestClientBuilder.newBuilder()
            .baseUri(baseUri())
            .build(OrderApi.class);
 
    public Shipment order(String article) {
        return api.order(article);
    }
}

The ProblemDetailClientResponseFilter hides all the conventions we use:

@Slf4j
@Providerpublic class ProblemDetailClientResponseFilter implements ClientResponseFilter {
    private static final Jsonb JSONB = JsonbBuilder.create();
 
    @Override public void filter(ClientRequestContext requestContext, ClientResponseContext response) {
        if (ProblemDetail.JSON_MEDIA_TYPE.isCompatible(response.getMediaType())
          && response.hasEntity()) {
            ProblemDetail problemDetail = JSONB.fromJson(response.getEntityStream(), ProblemDetail.class);
            triggerProblemDetailException(problemDetail);
        }
    }
 
    private void triggerProblemDetailException(ProblemDetail problemDetail) {
        if (problemDetail.getType() == null) {
            log.warn("no type string in problem detail type [" + problemDetail + "]");
        } else {
            String typeString = problemDetail.getType().toString();
            if (isJavadocUrl(typeString)) {
                String className = typeString.substring(35, typeString.length() - 5)
                    .replace('.', '$').replace('/', '.');try {
                    Class<?> exceptionType = Class.forName(className);
                    if (RuntimeException.class.isAssignableFrom(exceptionType)) {
                        throw (RuntimeException) exceptionType.getDeclaredConstructor().newInstance();
                    }
                    log.warn("problem detail type [" + typeString + "] is not a RuntimeException");
                } catch (ReflectiveOperationException e) {
                    log.warn("can't instantiate " + className, e);
                }
            } else {
                throw new IllegalArgumentException("unknown problem detail type [" +
                    problemDetail.getType() + "]:\n" + problemDetail);
            }
        }
    }
 
    private boolean isJavadocUrl(String typeString) {
        return typeString.startsWith("https://api.myshop.example/apidocs/")
            && typeString.endsWith(".html")
    }
}

①: Automatically register the ClientResponseFilter with JAX-RS.
②: Recovering the exception type from the javadoc URL is not ideal, as it tightly couples the client side to the server side, i.e. it assumes that we use the same classes in the same packages. It’s good enough for the demo, but to do it properly you need a way to register exceptions or scan for them, like in my library, which also allows extension fields and other things.

Summary

Avoid misusing http status codes; that’s a snake pit. Produce standardized and thereby interoperable problem details instead, it’s easier than you may think. To not litter your business logic code, you can use exceptions, on the server side as well as on the client side. Most of the code can even be made generic and reused in several applications, by introducing some conventions.

This implementation provides annotations for @Type, @Title, @Status, @Instance, @Detail, and @Extension for the your custom exceptions. It works with Spring Boot as well as JAX-RS and MicroProfile Rest Client. Zalando took a different approach with their Problem library and the Spring integration. problem4j looks usable, too. There are solutions for a few other languages, e.g. on GitHub rfc7807 and rfc-7807.

More on this topic by my colleague Christian in his blog post (in German).

What do you think? Do you know about other good libraries? Shouldn’t this become a standard tool in your belt?

Rüdiger zu Dohna

To Rüdiger, the term agility is not a method but a canon of values founded on ultimate customer orientation, which includes pronouncing also uncomfortable truths. Rüdiger has been practicing this for much longer than the Agile Manifesto exists, in different roles, industries, company cultures, and technology stacks. Since January 2018 he works for codecentric in Karlsruhe, Germany.

Post by Rüdiger zu Dohna

Architecture

DDD vs. Anemic Domain Models (Martin Fowler)

Agile Testing

Structured JUnit 5 testing

More content about Jakarta EE

Comment

Your email address will not be published. Required fields are marked *