Consolidated Design Rules and Standards for Great Web API

In a previous thread named API vs Web service, we have explored how the APIs have evolved through time, generation after generation until we get to the Web API generation. In this thread, we will list and consolidate the most common rules, conventions, and standards for designing Great Web APIs and we will group them into the following categories:

• Security
• Performance and Reliability
• Clarity and Consistency
• Change Management

Security Rules

The following lists the main recommended security rules for Great API Design:

Rule 1: Prefer OAuth over Basic Authentication

Rule 2: Define minimal OAuth Scopes

Rule 3: Isolate sensitive information in a separate scope

Rule 4: Define different scopes for different resources

Rule 5: Always verify the OAuth access token is valid and has the right permission

1. Make sure the access token is not expired
2. Make sure the access token is permitted to perform the requested HTTP method on the requested resource
3. If access token validation failed, an error should be returned
4. It is useful to return more details about the scopes needed and the scopes provided. For example, return these headers: X-OAuth-Scopes lists the scopes provided for the access token and X-Accepted-OAuth-Scopes lists the required scopes for this action

Rule 6: Issue one time use Authorization Codes

Rule 7: Issue short-lived Access Tokens

Rule 8: Support refresh tokens to allow consumers to issue new access tokens.

Rule 9: Support OAuth state parameter

Rule 10: Use SSL/TLS with mutual authentication

Rule 11: Verify provided redirect URL parameter and make sure it matches one of the registered URLs for the client application

Rule 12: Disallow rendering of OAUTH page in iframes by using X-Frame-Options and/or Content-Security-Policy: frame-ancestors header

Rule 13: Use Verification Tokens with WebHook APIs

Rule 14: Use Signatures and Prevent replay attacks by including request timestamp

Rule 15: Avoid sending sensitive information as part of WebHooks

Rule 16: Provide guidelines and documentation for the API security requirements and best practices

Rule 17: Provide consumers with SDKs that implement the best security practices (authentication-using OAuth, rate-limit control, issue and validate the signature, validate request timestamp, etc.)

Performance and Reliability Rules

The following lists the main recommended performance rules for Great API Design:

Rule 18: Use Database Indexes. Note: Create indexes wisely as adding too many indexes can affect the insert, update and delete performance, for that purpose, it is recommended to follow the MENTOR checklist

Rule 19: Use Database Materialized Views

Rule 20: Use Caching. Note: Remember to implement a proper invalidation and/or expiration mechanisms

Rule 21: Use Asynchronous processing for Expensive operations

Rule 22: If your API consumer uses polling to get the changes, consider event-driven APIs (such as Webhooks, WebSockets, and HTTP Streaming)

Rule 23: If your API is too fine-grained where few details are returned and where consumers need to make multiple calls to get all the required details, then you need to introduce new coarser-grained methods to collect the frequently required details in one method (see Service API Denormalization design pattern for more details)

Rule 24: If your API is too coarse-grained where more details are returned than what they need, then you need to introduce finer-grained methods to return as minimum as required (see Lightweight Endpoint design pattern for more details).

Rule 25: If your API consumers’ requirements for the details vary, consider GraphQL API where the consumers can fetch exactly the fields they need.

Rule 26: Support Bulks; Instead of sending bulky details as individual requests, consider supporting bulk requests where multiple entries/objects can be submitted in one request

Rule 27: Support Filtration and Ordering. If not provided, the consumer may request all the data to filter out locally which can overload the database and network.

Rule 28: If your API returns a large dataset, use pagination:

1. Offset-based Pagination (based on Page Id and Page Size)
2. Cursor-based Pagination (server returns in the response, cursor for the next page which need to be submitted by the consumer if he needs to get the next page)
3. Make sure to sort the data as per the consumer preference

Rule 29: Implement Rate-limiting strategies to control how many times an application or a client is allowed to call an API during a given time interval.

Rule 30: Encourage your consumers to use gzip compression

Rule 31: Provide consumers with SDKs that implements the best performance practices (caching, compression, pagination, rate-limit support, etc.)

Clarity and Consistency Rules

Rule 31: Enforce naming convention and standard structure for API methods, resources, and payloads for better consistency.

Rule 32: Adopt standards to name your resources and their fields. The following sites define the list of schemas to be consulted through the API design (schema.org, GeoJSON, Microformats wiki, Dublin Core, Activity Vocabulary, FOAF)

Rule 33: Use standards for field values. Use ISO3166 for countries, ISO4217 for currencies, RFC3339 for dates and times formats, etc.

Rule 34: Provide standard and meaningful response codes and return longer-form response codes “name_too_long”.

Rule 35: define a standard schema for error reporting and use it across all the APIs responses

Rule 36: Provide a human-readable error description in the response payload.

Rule 37: Provide a standard form of the meta-data and a standard list of allowed meta-data fields.

Rule 38: Log errors and provide consumers a dashboard to browse and analyze the logs

Rule 39: Apply the Contract-First Design approach by defining the API before starting the implementation to ensure the consistency among all methods and operations

Rule 40: Pre-define the data models for your API using strict data types before starting designing any API using XML Schema and/or JSON Schema and map them with Application-specific media types.

Rule 41: Don’t use database schema as the basis for your API design.

Rule 42: Do not use Auto-generation schema tools

Rule 43: Reduce the quantity and restrictiveness of validation logic embedded in the API and keep them generic. Detailed validations in the API can make it coupled to the underlying implementation and accordingly it will be always eligible for change whenever the underlying implementation is changed. See the Validation Abstraction design pattern for more details

Rule 44: Pre-define problem detail schema and use it across all the APIs

Rule 45: Support multiple formats (JSON, XML, etc.) and allow the consumer to negotiate for his preferred format (see Content Negotiation design pattern for more details)

Rule 46: Use Open API Specification to describe your API and use $ref to reference the predefined data model Schema (See Rule#40&44) that increases the reusability and improves the consistency

Rule 47: Always validate requests against the defined schema and respond with a meaningful error code and message in case of failure. As per Rule#43 keep the schema constraints to the minimum

Rule 48: You must use the same format for both success and failure responses. For example, If you are using JSON for success response, then you must use JSON for failure response

Rule 49: Provide documentation to help developers get started

1. Provide tutorials and Getting start guides
2. Provide examples for request and response payload
3. Provide sample code
4. Provide API technical specifications (see Rule#46)
5. Provide best practices guidelines and terms of service
6. Provide Frequently Asked Questions (FAQ)

Rule 50: Provide interactive documentation online, where developers can test out the API

Rule 51: Provide consumers with SDKs that implements the best practices (error handling, retry for rate-limit errors, performing OAuth authentication, pagination, etc.)

Change Management Rules

Rule 52: Make sure your changes are consistent and follow your standard naming convention, error coding, logging, etc.

Rule 53: Set up a versioning system and standard versioning scheme from the beginning (see Canonical Versioning design pattern for more details) and use the Version Identification design pattern through the API design

1. Recommended versioning scheme (MAJOR.MINOR.PATCH). MAJOR for backward-incompatible changes (See Rule#55). MINOR for backward-compatible changes (See Rule#54). PATCH for backward-compatible bug fixes
2. Another recommended versioning scheme is to apply the backward compatible changes seamlessly without changing the version number (e.g., v1, v2, v3, etc.)

Rule 54: The following changes can be considered backward compatible and can be applied at any time with a proper communication:

1. Adding a new Operation, a new method, or a new endpoint
2. Adding an optional field to the request payload
3. Deleting optional fields from the request or response (Unless the response payload is expected to change based on this request field)
4. Loosen the request's field restrictions. For example, request field max length is increased from 10 to 20 or changing integer data type to string
5. Tighten response field restrictions. For example, response field max length is decreased from 20 to 10 or changing string data type to integer

Rule 55: The following changes are not backward compatible and accordingly proper change management techniques should be applied to minimize the impact on the consumers:

1. Deleting or renaming operation, method, or endpoint
2. Add a Mandatory field to the request payload
3. Add Field to the response payload
4. Delete or rename a mandatory field from the request payload
5. Delete or rename mandatory field from the response payload
6. Change the order of the fields in case of an XML schema that doesn’t use <xsd:all>. In the case of JSON, the properties are, by default, unordered.
7. Tighten request field restrictions. For example, request field max length decreased from 20 to 10 or changing a string data type to integer
8. Loosen response field restrictions. For example, response field max length increased from 10 to 20 or changing integer data type to string
9. Changes in error codes and error codes meaning
10. Change the default data format (e.g., changing the default from JSON to XML or vice versa)

Rule 56: Apply the following techniques to minimize the impact on the consumer in case of backward-incompatible changes:

1. Utilize the existing versioning system and versioning scheme to introduce a new version (See Rule#2)
2. Introduce Service API proxy design pattern for existing customers. API proxy retains the original API and contains logic to accommodate the changes.
3. In case of endpoint change, apply the Endpoint Redirection design pattern using the Location header to redirect the consumer to the new endpoint
4. Apply Termination Notification design pattern to announce version has been deprecated and announce the date of the old version's retirement
5. Build a change log to track the versions and changes applied in each version
6. Once the old service is switched off, the server should return the status code (410 “Gone”)

Rule 57: Encourage your clients:

1. To not fail if they received a new field in the response
2. To support unordered fields for the data models in the request and response

Don’t forget to like the story and subscribe to get notifications for my new stories.

References

The above list of rules has been collected and consolidated from the following references:

• www.arcitura.com
• RESTful Web APIs, by Leonard Richardson
• REST API Design Rulebook, by Mark Masse
• Designing Web APIs — Building APIs that developers love, by Brenda Jin
• Restful Web Services Cookbook, by Subbu Allamaraju
• W3C Activity Stream Vocabulary
• Dublin Core
• Schema.org
• Microformats Wiki
• http://xmlns.com/foaf/spec/
• https://tools.ietf.org/html/rfc7807
• https://tools.ietf.org/html/rfc3339
• https://www.iso.org/iso-3166-country-codes.html
• https://en.wikipedia.org/wiki/ISO_4217