Proxy-fixing HTTPS requests to nameko proxied through nginx (or another server)

googlegroup
1

We're using the *@http* support of Nameko to write simple APIs and are
serving those APIs over HTTPS, with them sitting behind an nginx instance
that terminates SSL. This presents a problem for us because we do various
validation checks on *request.base_url* which shows up as HTTP (because the
request is proxied through as HTTPS) even though the original request was
HTTPS.

In Flask, we've solved this same issue using *werkzeug.fixers.ProxyFix, *a
WSGI middleware that updates the WSGI environ to be in line with what it
should have been before the request was proxied.

I'm trying to apply this same fix to our nameko instances, but am having a
really hard time because the WSGI app instantiation and use is buried deep
in the Webserver
<https://github.com/onefinestay/nameko/blob/7209c1d45fce20cb2ba7ddcd6f79fa120c85bc29/nameko/web/server.py#L82-L86>
.

Has anyone else dealt with this or have any suggestions on how to proceed?

Thanks!
Jesse

2

This kind of problem crops up occasionally as we discover the different
ways the community want to use nameko. There is a similar issue passing
message delivery options into the RPC entrypoint.

Our general suggestion for extending nameko is to subclass the built-in
classes. At Student.com for example, our HTTP entrypoint looks like this:

import json

from nameko.web.handlers import HttpRequestHandler
from nameko.web.server import WebServer as BaseWebServer
from werkzeug.wrappers import Response

class WebServer(BaseWebServer):

    def context_data_from_headers(self, request):
        # specify the mapping between HTTP headers and context data
        context_data = super().context_data_from_headers(request)
        ...
        return context_data

class HttpEntrypoint(HttpRequestHandler):

    server = WebServer()

    def response_from_exception(self, exc):
        # generate a custom response object from the thrown exception
        ...
        return Response(
            json.dumps(payload),
            status=status_code,
            mimetype='application/json',
        )

http = HttpEntrypoint.decorator

These subclasses are light because their parent classes expect to be
extended and are structured appropriately.

Unfortunately the WsgiApp is not exposed in such a helpful way, but it'd be
great if it was!

We could perhaps add a get_wsgi_app method to the WebServer (and maybe a
get_wsgi_server method, while we're at it). These methods could then be
overridden in a subclass and the fixers and other customisations provided
as the user saw fit.

A pull request along those lines would be very welcome. In the mean time,
the sledgehammer approach would be to subclass and override the whole start
method.

Matt.

···

On Wednesday, August 17, 2016 at 8:06:10 AM UTC+8, je...@pollak.io wrote:

We're using the *@http* support of Nameko to write simple APIs and are
serving those APIs over HTTPS, with them sitting behind an nginx instance
that terminates SSL. This presents a problem for us because we do various
validation checks on *request.base_url* which shows up as HTTP (because
the request is proxied through as HTTPS) even though the original request
was HTTPS.

In Flask, we've solved this same issue using *werkzeug.fixers.ProxyFix, *a
WSGI middleware that updates the WSGI environ to be in line with what it
should have been before the request was proxied.

I'm trying to apply this same fix to our nameko instances, but am having a
really hard time because the WSGI app instantiation and use is buried
deep in the Webserver
<https://github.com/onefinestay/nameko/blob/7209c1d45fce20cb2ba7ddcd6f79fa120c85bc29/nameko/web/server.py#L82-L86&gt;
.

Has anyone else dealt with this or have any suggestions on how to proceed?

Thanks!
Jesse

3

Thanks for this reply - really appreciate it! I messed around with a few
different options and got to one that I'm somewhat happy with, though it
relies a little too much on the Nameko internals for me to be comfortable.
Essentially, I've created a *DependencyProvider* that injects the
middleware deep in the *nameko.web.server.WebServer* internals. The two
crappy things about it are: 1) it relies on dot access deep into the
*WebServer* object; 2) it relies on being the "started" after the
*WebServer* has been started (achieved by making the *__hash__ *value
infinity. Here's the quick code snippet:

from __future__ import absolute_import

import sys
from nameko.extensions import DependencyProvider, SharedExtension
from nameko.web.server import WebServer
from werkzeug.contrib.fixers import ProxyFix as _ProxyFix

class ProxyFixMiddleware(SharedExtension):

    def __hash__(self):
        """
        Change the hash so when it is added to the extension set, it is always last.
        """
        return sys.maxint

    def start(self):
        """
        Patch the WebServer on start to use the WSGI Middleware
        """
        if self.container.config.get('NUM_PROXIES', 0) > 0:
            for ext in self.container.subextensions:
                if isinstance(ext, WebServer) and ext._serv:
                    ext._serv.app = _ProxyFix(ext._serv.app, num_proxies=self.container.config.get('NUM_PROXIES'))

class ProxyFix(DependencyProvider):
    middleware = ProxyFixMiddleware()

I can look into opening a PR to make this sort of WSGI middleware insertion
much more straightfoward - would love any and all feedback before I dive
into that!

Cheers,
Jesse

···

On Tuesday, August 16, 2016 at 10:57:37 PM UTC-7, Matt Yule-Bennett wrote:

This kind of problem crops up occasionally as we discover the different
ways the community want to use nameko. There is a similar issue passing
message delivery options into the RPC entrypoint.

Our general suggestion for extending nameko is to subclass the built-in
classes. At Student.com for example, our HTTP entrypoint looks like this:

import json

from nameko.web.handlers import HttpRequestHandler
from nameko.web.server import WebServer as BaseWebServer
from werkzeug.wrappers import Response

class WebServer(BaseWebServer):

    def context_data_from_headers(self, request):
        # specify the mapping between HTTP headers and context data
        context_data = super().context_data_from_headers(request)
        ...
        return context_data

class HttpEntrypoint(HttpRequestHandler):

    server = WebServer()

    def response_from_exception(self, exc):
        # generate a custom response object from the thrown exception
        ...
        return Response(
            json.dumps(payload),
            status=status_code,
            mimetype='application/json',
        )

http = HttpEntrypoint.decorator

These subclasses are light because their parent classes expect to be
extended and are structured appropriately.

Unfortunately the WsgiApp is not exposed in such a helpful way, but it'd
be great if it was!

We could perhaps add a get_wsgi_app method to the WebServer (and maybe a
get_wsgi_server method, while we're at it). These methods could then be
overridden in a subclass and the fixers and other customisations provided
as the user saw fit.

A pull request along those lines would be very welcome. In the mean time,
the sledgehammer approach would be to subclass and override the whole
start method.

Matt.

On Wednesday, August 17, 2016 at 8:06:10 AM UTC+8, je...@pollak.io wrote:

We're using the *@http* support of Nameko to write simple APIs and are
serving those APIs over HTTPS, with them sitting behind an nginx instance
that terminates SSL. This presents a problem for us because we do various
validation checks on *request.base_url* which shows up as HTTP (because
the request is proxied through as HTTPS) even though the original request
was HTTPS.

In Flask, we've solved this same issue using *werkzeug.fixers.ProxyFix, *a
WSGI middleware that updates the WSGI environ to be in line with what it
should have been before the request was proxied.

I'm trying to apply this same fix to our nameko instances, but am having
a really hard time because the WSGI app instantiation and use is buried
deep in the Webserver
<https://github.com/onefinestay/nameko/blob/7209c1d45fce20cb2ba7ddcd6f79fa120c85bc29/nameko/web/server.py#L82-L86&gt;
.

Has anyone else dealt with this or have any suggestions on how to proceed?

Thanks!
Jesse

4

Hi Jesse,

Thanks for this reply - really appreciate it! I messed around with a few
different options and got to one that I'm somewhat happy with, though it
relies a little too much on the Nameko internals for me to be comfortable.
Essentially, I've created a *DependencyProvider* that injects the
middleware deep in the *nameko.web.server.WebServer* internals. The two
crappy things about it are: 1) it relies on dot access deep into the
*WebServer* object; 2) it relies on being the "started" after the
*WebServer* has been started (achieved by making the *__hash__ *value
infinity. Here's the quick code snippet:

from __future__ import absolute_import

import sys
from nameko.extensions import DependencyProvider, SharedExtension
from nameko.web.server import WebServer
from werkzeug.contrib.fixers import ProxyFix as _ProxyFix

class ProxyFixMiddleware(SharedExtension):

    def __hash__(self):
        """
        Change the hash so when it is added to the extension set, it is always last.
        """
        return sys.maxint

    def start(self):
        """
        Patch the WebServer on start to use the WSGI Middleware
        """
        if self.container.config.get('NUM_PROXIES', 0) > 0:
            for ext in self.container.subextensions:
                if isinstance(ext, WebServer) and ext._serv:
                    ext._serv.app = _ProxyFix(ext._serv.app, num_proxies=self.container.config.get('NUM_PROXIES'))

class ProxyFix(DependencyProvider):
    middleware = ProxyFixMiddleware()

This is inventive! I like the __hash__ hack

I would definitely recommend using subclasses rather than this approach,
even if it means overloading the entire start method on the WebServer for
now. There are a few advantages of doing it that way instead:

* Fewer nameko internals exposed or hacked around
* No need for the fake middleware dependency in your service class
* You'll only instantiate the WsgiApp once

I can look into opening a PR to make this sort of WSGI middleware
insertion much more straightfoward - would love any and all feedback before
I dive into that!

That would be awesome!

···

On Wednesday, August 17, 2016 at 2:02:39 PM UTC+8, je...@pollak.io wrote:

Cheers,
Jesse

On Tuesday, August 16, 2016 at 10:57:37 PM UTC-7, Matt Yule-Bennett wrote:

This kind of problem crops up occasionally as we discover the different
ways the community want to use nameko. There is a similar issue passing
message delivery options into the RPC entrypoint.

Our general suggestion for extending nameko is to subclass the built-in
classes. At Student.com for example, our HTTP entrypoint looks like this:

import json

from nameko.web.handlers import HttpRequestHandler
from nameko.web.server import WebServer as BaseWebServer
from werkzeug.wrappers import Response

class WebServer(BaseWebServer):

    def context_data_from_headers(self, request):
        # specify the mapping between HTTP headers and context data
        context_data = super().context_data_from_headers(request)
        ...
        return context_data

class HttpEntrypoint(HttpRequestHandler):

    server = WebServer()

    def response_from_exception(self, exc):
        # generate a custom response object from the thrown exception
        ...
        return Response(
            json.dumps(payload),
            status=status_code,
            mimetype='application/json',
        )

http = HttpEntrypoint.decorator

These subclasses are light because their parent classes expect to be
extended and are structured appropriately.

Unfortunately the WsgiApp is not exposed in such a helpful way, but it'd
be great if it was!

We could perhaps add a get_wsgi_app method to the WebServer (and maybe a
get_wsgi_server method, while we're at it). These methods could then be
overridden in a subclass and the fixers and other customisations provided
as the user saw fit.

A pull request along those lines would be very welcome. In the mean time,
the sledgehammer approach would be to subclass and override the whole
start method.

Matt.

On Wednesday, August 17, 2016 at 8:06:10 AM UTC+8, je...@pollak.io wrote:

We're using the *@http* support of Nameko to write simple APIs and are
serving those APIs over HTTPS, with them sitting behind an nginx instance
that terminates SSL. This presents a problem for us because we do various
validation checks on *request.base_url* which shows up as HTTP (because
the request is proxied through as HTTPS) even though the original request
was HTTPS.

In Flask, we've solved this same issue using *werkzeug.fixers.ProxyFix,
*a WSGI middleware that updates the WSGI environ to be in line with
what it should have been before the request was proxied.

I'm trying to apply this same fix to our nameko instances, but am having
a really hard time because the WSGI app instantiation and use is buried
deep in the Webserver
<https://github.com/onefinestay/nameko/blob/7209c1d45fce20cb2ba7ddcd6f79fa120c85bc29/nameko/web/server.py#L82-L86&gt;
.

Has anyone else dealt with this or have any suggestions on how to
proceed?

Thanks!
Jesse

5

I ended up shipping the subclassed approach and it's working well. As
promised, I've opened up a PR on nameko to make accessing the WSGI app and
WSGI server easier: Add wsgi flexibility to server by jessepollak · Pull Request #352 · nameko/nameko · GitHub.
Any feedback on the PR would be much appreciated!

···

On Wednesday, August 17, 2016 at 1:40:58 AM UTC-7, Matt Yule-Bennett wrote:

Hi Jesse,

On Wednesday, August 17, 2016 at 2:02:39 PM UTC+8, je...@pollak.io wrote:

Thanks for this reply - really appreciate it! I messed around with a few
different options and got to one that I'm somewhat happy with, though it
relies a little too much on the Nameko internals for me to be comfortable.
Essentially, I've created a *DependencyProvider* that injects the
middleware deep in the *nameko.web.server.WebServer* internals. The two
crappy things about it are: 1) it relies on dot access deep into the
*WebServer* object; 2) it relies on being the "started" after the
*WebServer* has been started (achieved by making the *__hash__ *value
infinity. Here's the quick code snippet:

from __future__ import absolute_import

import sys
from nameko.extensions import DependencyProvider, SharedExtension
from nameko.web.server import WebServer
from werkzeug.contrib.fixers import ProxyFix as _ProxyFix

class ProxyFixMiddleware(SharedExtension):

    def __hash__(self):
        """
        Change the hash so when it is added to the extension set, it is always last.
        """
        return sys.maxint

    def start(self):
        """
        Patch the WebServer on start to use the WSGI Middleware
        """
        if self.container.config.get('NUM_PROXIES', 0) > 0:
            for ext in self.container.subextensions:
                if isinstance(ext, WebServer) and ext._serv:
                    ext._serv.app = _ProxyFix(ext._serv.app, num_proxies=self.container.config.get('NUM_PROXIES'))

class ProxyFix(DependencyProvider):
    middleware = ProxyFixMiddleware()

This is inventive! I like the __hash__ hack

I would definitely recommend using subclasses rather than this approach,
even if it means overloading the entire start method on the WebServer for
now. There are a few advantages of doing it that way instead:

* Fewer nameko internals exposed or hacked around
* No need for the fake middleware dependency in your service class
* You'll only instantiate the WsgiApp once

I can look into opening a PR to make this sort of WSGI middleware
insertion much more straightfoward - would love any and all feedback before
I dive into that!

That would be awesome!

Cheers,
Jesse

On Tuesday, August 16, 2016 at 10:57:37 PM UTC-7, Matt Yule-Bennett wrote:

This kind of problem crops up occasionally as we discover the different
ways the community want to use nameko. There is a similar issue passing
message delivery options into the RPC entrypoint.

Our general suggestion for extending nameko is to subclass the built-in
classes. At Student.com for example, our HTTP entrypoint looks like this:

import json

from nameko.web.handlers import HttpRequestHandler
from nameko.web.server import WebServer as BaseWebServer
from werkzeug.wrappers import Response

class WebServer(BaseWebServer):

    def context_data_from_headers(self, request):
        # specify the mapping between HTTP headers and context data
        context_data = super().context_data_from_headers(request)
        ...
        return context_data

class HttpEntrypoint(HttpRequestHandler):

    server = WebServer()

    def response_from_exception(self, exc):
        # generate a custom response object from the thrown exception
        ...
        return Response(
            json.dumps(payload),
            status=status_code,
            mimetype='application/json',
        )

http = HttpEntrypoint.decorator

These subclasses are light because their parent classes expect to be
extended and are structured appropriately.

Unfortunately the WsgiApp is not exposed in such a helpful way, but it'd
be great if it was!

We could perhaps add a get_wsgi_app method to the WebServer (and maybe
a get_wsgi_server method, while we're at it). These methods could then
be overridden in a subclass and the fixers and other customisations
provided as the user saw fit.

A pull request along those lines would be very welcome. In the mean
time, the sledgehammer approach would be to subclass and override the whole
start method.

Matt.

On Wednesday, August 17, 2016 at 8:06:10 AM UTC+8, je...@pollak.io >>> wrote:

We're using the *@http* support of Nameko to write simple APIs and are
serving those APIs over HTTPS, with them sitting behind an nginx instance
that terminates SSL. This presents a problem for us because we do various
validation checks on *request.base_url* which shows up as HTTP
(because the request is proxied through as HTTPS) even though the original
request was HTTPS.

In Flask, we've solved this same issue using *werkzeug.fixers.ProxyFix,
*a WSGI middleware that updates the WSGI environ to be in line with
what it should have been before the request was proxied.

I'm trying to apply this same fix to our nameko instances, but am
having a really hard time because the WSGI app instantiation and use is buried
deep in the Webserver
<https://github.com/onefinestay/nameko/blob/7209c1d45fce20cb2ba7ddcd6f79fa120c85bc29/nameko/web/server.py#L82-L86&gt;
.

Has anyone else dealt with this or have any suggestions on how to
proceed?

Thanks!
Jesse