In this tutorial you'll learn how to handle error conditions in Python from a whole system point of view. Error handling is a critical aspect of design, and it crosses from the lowest levels (sometimes the hardware) all the way to the end users. If you don't have a consistent strategy in place, your system will be unreliable, the user experience will be poor, and you'll have a lot of challenges debugging and troubleshooting.
The key to success is being aware of all these interlocking aspects, considering them explicitly, and forming a solution that addresses each point.
Status Codes vs. Exceptions
There are two main error handling models: status codes and exceptions. Status codes can be used by any programming language. Exceptions require language/runtime support.
Python supports exceptions. Python and its standard library use exceptions liberally to report on many exceptional situations like IO errors, divide by zero, out of bounds indexing, and also some not so exceptional situations like end of iteration (although it is hidden). Most libraries follow suit and raise exceptions.
That means your code will have to handle the exceptions raised by Python and libraries anyway, so you may as well raise exceptions from your code when necessary and not rely on status codes.
Before diving into the inner sanctum of Python exceptions and error handling best practices, let's see some exception handling in action:
def f(): return 4 / 0 def g(): raise Exception("Don't call us. We'll call you") def h(): try: f() except Exception as e: print(e) try: g() except Exception as e: print(e)
Here is the output when calling
h() division by zero Don't call us. We'll call you
Python exceptions are objects organized in a class hierarchy.
Here is the whole hierarchy:
BaseException +-- SystemExit +-- KeyboardInterrupt +-- GeneratorExit +-- Exception +-- StopIteration +-- StandardError | +-- BufferError | +-- ArithmeticError | | +-- FloatingPointError | | +-- OverflowError | | +-- ZeroDivisionError | +-- AssertionError | +-- AttributeError | +-- EnvironmentError | | +-- IOError | | +-- OSError | | +-- WindowsError (Windows) | | +-- VMSError (VMS) | +-- EOFError | +-- ImportError | +-- LookupError | | +-- IndexError | | +-- KeyError | +-- MemoryError | +-- NameError | | +-- UnboundLocalError | +-- ReferenceError | +-- RuntimeError | | +-- NotImplementedError | +-- SyntaxError | | +-- IndentationError | | +-- TabError | +-- SystemError | +-- TypeError | +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError +-- Warning +-- DeprecationWarning +-- PendingDeprecationWarning +-- RuntimeWarning +-- SyntaxWarning +-- UserWarning +-- FutureWarning +-- ImportWarning +-- UnicodeWarning +-- BytesWarning
There are several special exceptions that are derived directly from
GeneratorExit. Then there is the
Exception class, which is the base class for
Warning. All the standard errors are derived from
When you raise an exception or some function you called raises an exception, that normal code flow terminates and the exception starts propagating up the call stack until it encounters a proper exception handler. If no exception handler is available to handle it, the process (or more accurately the current thread) will be terminated with an unhandled exception message.
Raising exceptions is very easy. You just use the
raise keyword to raise an object that is a sub-class of the
Exception class. It could be an instance of
Exception itself, one of the standard exceptions (e.g.
RuntimeError), or a subclass of
Exception you derived yourself. Here is a little snippet that demonstrates all cases:
# Raise an instance of the Exception class itself raise Exception('Ummm... something is wrong') # Raise an instance of the RuntimeError class raise RuntimeError('Ummm... something is wrong') # Raise a custom subclass of Exception that keeps the timestamp the exception was created from datetime import datetime class SuperError(Exception): def __init__(self, message): Exception.__init__(message) self.when = datetime.now() raise SuperError('Ummm... something is wrong')
You catch exceptions with the
except clause, as you saw in the example. When you catch an exception, you have three options:
- Swallow it quietly (handle it and keep running).
- Do something like logging, but re-raise the same exception to let higher levels handle.
- Raise a different exception instead of the original.
Swallow the Exception
You should swallow the exception if you know how to handle it and can fully recover.
For example, if you receive an input file that may be in different formats (JSON, YAML), you may try parsing it using different parsers. If the JSON parser raised an exception that the file is not a valid JSON file, you swallow it and try with the YAML parser. If the YAML parser failed too then you let the exception propagate out.
import json import yaml def parse_file(filename): try: return json.load(open(filename)) except json.JSONDecodeError return yaml.load(open(filename))
Note that other exceptions (e.g. file not found or no read permissions) will propagate out and will not be caught by the specific except clause. This is a good policy in this case where you want to try the YAML parsing only if the JSON parsing failed due to a JSON encoding issue.
If you want to handle all exceptions then just use
except Exception. For example:
def print_exception_type(func, *args, **kwargs): try: return func(*args, **kwargs) except Exception as e: print type(e)
Note that by adding
as e, you bind the exception object to the name
e available in your except clause.
Re-Raise the Same Exception
To re-raise, just add
raise with no arguments inside your handler. This lets you perform some local handling, but still lets upper levels handle it too. Here, the
invoke_function() function prints the type of exception to the console and then re-raises the exception.
def invoke_function(func, *args, **kwargs): try: return func(*args, **kwargs) except Exception as e: print type(e) raise
Raise a Different Exception
There are several cases where you would want to raise a different exception. Sometimes you want to group multiple different low-level exceptions into a single category that is handled uniformly by higher-level code. In order cases, you need to transform the exception to the user level and provide some application-specific context.
Sometimes you want to ensure some cleanup code executes even if an exception was raised somewhere along the way. For example, you may have a database connection that you want to close once you're done. Here is the wrong way to do it:
def fetch_some_data(): db = open_db_connection() query(db) close_db_Connection(db)
query() function raises an exception then the call to
close_db_connection() will never execute and the DB connection will remain open. The
finally clause always executes after a try all exception handler is executed. Here is how to do it correctly:
def fetch_some_data(): db = None try: db = open_db_connection() query(db) finally: if db is not None: close_db_connection(db)
The call to
open_db_connection() may not return a connection or raise an exception itself. In this case there is no need to close the DB connection.
finally, you have to be careful not to raise any exceptions there because they will mask the original exception.
Context managers provide another mechanism to wrap resources like files or DB connections in cleanup code that executes automatically even when exceptions have been raised. Instead of try-finally blocks, you use the
with statement. Here is an example with a file:
def process_file(filename): with open(filename) as f: process(f.read())
Now, even if
process() raised an exception, the file will be closed properly immediately when the scope of the
with block is exited, regardless of whether the exception was handled or not.
Logging is pretty much a requirement in non-trivial, long-running systems. It is especially useful in web applications where you can treat all exceptions in a generic way: Just log the exception and return an error message to the caller.
When logging, it is useful to log the exception type, the error message, and the stacktrace. All this information is available via the
sys.exc_info object, but if you use the
logger.exception() method in your exception handler, the Python logging system will extract all the relevant information for you.
This is the best practice I recommend:
import logging logger = logging.getLogger() def f(): try: flaky_func() except Exception: logger.exception() raise
If you follow this pattern then (assuming you set up logging correctly) no matter what happens you'll have a pretty good record in your logs of what went wrong, and you'll be able to fix the issue.
If you re-raise, make you sure you don't log the same exception over and over again at different levels. It is a waste, and it might confuse you and make you think multiple instances of the same issue occurred, when in practice a single instance was logged multiple times.
The simplest way to do it is to let all exceptions propagate (unless they can be handled confidently and swallowed earlier) and then do the logging close to the top level of your application/system.
Logging is a capability. The most common implementation is using log files. But, for large-scale distributed systems with hundreds, thousands or more servers, this is not always the best solution.
To keep track of exceptions across your whole infrastructure, a service like sentry is super helpful. It centralizes all exception reports, and in addition to the stacktrace it adds the state of each stack frame (the value of variables at the time the exception was raised). It also provides a really nice interface with dashboards, reports and ways to break down the messages by multiple projects. It is open source, so you can run your own server or subscribe to the hosted version.
Dealing With Transient Failure
Some failures are temporary, in particular when dealing with distributed systems. A system that freaks out at the first sign of trouble is not very useful.
If your code is accessing some remote system that is not responding, the traditional solution is timeouts, but sometimes not every system is designed with timeouts. Timeouts are not always easy to calibrate as conditions change.
Another approach is to fail fast and then retry. The benefit is that if the target is responding fast then you don't have to spend a lot of time in sleep condition and can react immediately. But if it failed, you can retry multiple times until you decide it is really unreachable and raise an exception. In the next section, I'll introduce a decorator that can do it for you.
Two decorators that can help with error handling are the
@log_error, which logs an exception and then re-raises it, and the
@retry decorator, which will retry calling a function several times.
Here is a simple implementation. The decorator excepts a logger object. When it decorates a function and the function is invoked, it will wrap the call in a try-except clause, and if there was an exception it will log it and finally re-raise the exception.
def log_error(logger) def decorated(f): @functools.wraps(f) def wrapped(*args, **kwargs): try: return f(*args, **kwargs) except Exception as e: if logger: logger.exception(e) raise return wrapped return decorated
Here is how to use it:
import logging logger = logging.getLogger() @log_error(logger) def f(): raise Exception('I am exceptional')
Here is a very good implementation of the @retry decorator.
import time import math # Retry decorator with exponential backoff def retry(tries, delay=3, backoff=2): '''Retries a function or method until it returns True. delay sets the initial delay in seconds, and backoff sets the factor by which the delay should lengthen after each failure. backoff must be greater than 1, or else it isn't really a backoff. tries must be at least 0, and delay greater than 0.''' if backoff <= 1: raise ValueError("backoff must be greater than 1") tries = math.floor(tries) if tries < 0: raise ValueError("tries must be 0 or greater") if delay <= 0: raise ValueError("delay must be greater than 0") def deco_retry(f): def f_retry(*args, **kwargs): mtries, mdelay = tries, delay # make mutable rv = f(*args, **kwargs) # first attempt while mtries > 0: if rv is True: # Done on success return True mtries -= 1 # consume an attempt time.sleep(mdelay) # wait... mdelay *= backoff # make future wait longer rv = f(*args, **kwargs) # Try again return False # Ran out of tries :-( return f_retry # true decorator -> decorated function return deco_retry # @retry(arg[, ...]) -> true decorator
Error handling is crucial for both users and developers. Python provides great support in the language and standard library for exception-based error handling. By following best practices diligently, you can conquer this often neglected aspect.