Support for Event Hooks

[martsime]

I'm currently using https://github.com/encode/requests-async for one of my projects and now I wanted to switch to HTTPX, but I can't find out if it supports event hooks like this https://requests.readthedocs.io/en/master/user/advanced/#event-hooks.

Or is there another easy way to add hooks onto an AsyncClient object to be called every request?

[lovelydinosaur]

Right now it doesn’t have any API for that, no. It’s certainly something we want to consider, yes.

It’d be really helpful to start getting some user feedback to help us figure out if event hooks are what we’d like to use here, or if we should be considering a more general middleware API.

What’s your particular use case here?

[martsime]

We use hooks to collect metrics about all the requests, mostly for monitoring purposes. It would be nice if the AsyncClient supported it similar to how Session in requests and requests_async does. So one could do something like this:

def response_hook(response, *args, **kwargs): 
    # Log status of response, etc...
    ...

def get_client():
    client = httpx.AsyncClient()
    client.hooks["response"].append(response_hook) 
    return client

A more general middleware API could be a good idea and definitely solves the problem.

It's not a big deal to make a workaround for our specific case, but since the other libraries supported it I thought there might be another way of doing it in HTTPX which I couldn't find.

[florimondmanca]

IMO a general middleware API like the one drafted in #783 would be the most versatile way to provide these "on request/on response" hooks, as well as address a bunch of other use cases…

class MonitoringMiddleware:
    def __init__(self, parent):
        self.parent = parent

    def __call__(self, request, scope, timeout):
        request_hook(request, ...)
        response = yield from self.parent(request, scope, timeout)
        response_hook(response, ...)
        return response

client = httpx.Client()
client.add_middleware(MonitoringMiddleware)

[florimondmanca]

Though we might want to provide a more streamlined API, that might just happen to be implemented as middleware, e.g...

def monitor_requests(request):
    # Process request...
    response = yield request
    # Process response...
    
client = httpx.Client(hooks=[monitor_requests])

(Somewhat inspired by Kludex/starlette#799.)

[jbrockmendel]

Jumping on here instead of creating a new issue:

I've got some project code that patched both requests and the stdlib to allow for hooks that logged requests and redirects. Went to go upstream it to requests and ended up here instead. Is there enough interest here to make a proof of concept worthwhile?

[florimondmanca]

Hey,

Yes, I think we could go for a PR drafting things out...

Forget the middleware stuff I mentioned above at the time, I’d agree a simpler and more integrated « register request/response callbacks » API is probably a better first step.

We could start off back from #790 (comment), however I think we should consider a slightly different API design approach, in the form of registering lists of callbacks on client init (instead of mutating an attribute on the client). This is somewhat in line with Starlette’s event handlers API, and less fiddly than « expose a dict with arbitrary 'request'/'response' keys that might be mutated anytime etc ». So something like...

import httpx

client = httpx.Client(
    on_request=[...],
    on_response=[...],
)

[lovelydinosaur]

Okay, so I've been thinking about this for a bit, and it's time to get this down.
Here's what I'm thinking for the design here...

# Setting on __init__.
client = httpx.Client(event_hooks={'request': ..., 'response': ...})

# Mutating the property.
client.event_hooks['request'] = ...
client.event_hooks['response'] = ...

# Should be included at the top-level API
httpx.get(..., event_hooks={...})

We want to support either none, one, or many hooks on each event type, so I think event_hooks should be a mapping type that always returns lists.

I think we want to support 4 event hooks, that will give us a nice comprehensive hook into the complete request/response flow...

• request - Signature: func(request). Called after everything is prepared on the request, including any initial authentication.
• auth - Signature: func(response, request). For most folks this is a corner case. Only called if the auth flow invokes additional requests beyond the original outgoing request.
• redirect - Signature: func(response, request). Called if the redirect flow invokes multiple requests. Not called for allow_redirects=False.
• response - Signature: func(response). Called once we're done & happy. Obvs.

One thing I don't feel super absolutely clear on is for the async case what we should be expecting from the functions. I think that the sensible thing to do is always require regular callables for the sync case and always require async callables for the async case.

I don't really want to get into "allow either async/sync callables for AsyncClient and we'll magically introspect which you're using".

Sticking with event_hooks on AsyncClient being mandated as always being async function is at least simple & clear.

Aside: One neat pattern that event hooks would give us is this...

client = httpx.Client(event_hooks={'response': lambda r: r.raise_for_status()})