url-parsing-with-wsgi.txt 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304
  1. URL Parsing With WSGI And Paste
  2. +++++++++++++++++++++++++++++++
  3. :author: Ian Bicking <ianb@colorstudy.com>
  4. :revision: $Rev$
  5. :date: $LastChangedDate$
  6. .. contents::
  7. Introduction and Audience
  8. =========================
  9. This document is intended for web framework authors and integrators,
  10. and people who want to understand the internal architecture of Paste.
  11. .. include:: include/contact.txt
  12. URL Parsing
  13. ===========
  14. .. note::
  15. Sometimes people use "URL", and sometimes "URI". I think URLs are
  16. a subset of URIs. But in practice you'll almost never see URIs
  17. that aren't URLs, and certainly not in Paste. URIs that aren't
  18. URLs are abstract Identifiers, that cannot necessarily be used to
  19. Locate the resource. This document is *all* about locating.
  20. Most generally, URL parsing is about taking a URL and determining what
  21. "resource" the URL refers to. "Resource" is a rather vague term,
  22. intentionally. It's really just a metaphor -- in reality there aren't
  23. any "resources" in HTTP; there are only requests and responses.
  24. In Paste, everything is about WSGI. But that can seem too fancy.
  25. There are four core things involved: the *request* (personified in the
  26. WSGI environment), the *response* (personified inthe
  27. ``start_response`` callback and the return iterator), the WSGI
  28. application, and the server that calls that application. The
  29. application and request are objects, while the server and response are
  30. really more like actions than concrete objects.
  31. In this context, URL parsing is about mapping a URL to an
  32. *application* and a *request*. The request actually gets modified as
  33. it moves through different parts of the system. Two dictionary keys
  34. in particular relate to URLs -- ``SCRIPT_NAME`` and ``PATH_INFO`` --
  35. but any part of the environment can be modified as it is passed
  36. through the system.
  37. Dispatching
  38. ===========
  39. .. note::
  40. WSGI isn't object oriented? Well, if you look at it, you'll notice
  41. there's no objects except built-in types, so it shouldn't be a
  42. surprise. Additionally, the interface and promises of the objects
  43. we do see are very minimal. An application doesn't have any
  44. interface except one method -- ``__call__`` -- and that method
  45. *does* things, it doesn't give any other information.
  46. Because WSGI is action-oriented, rather than object-oriented, it's
  47. more important what we *do*. "Finding" an application is probably an
  48. intermediate step, but "running" the application is our ultimate goal,
  49. and the only real judge of success. An application that isn't run is
  50. useless to us, because it doesn't have any other useful methods.
  51. So what we're really doing is *dispatching* -- we're handing the
  52. request and responsibility for the response off to another object
  53. (another actor, really). In the process we can actually retain some
  54. control -- we can capture and transform the response, and we can
  55. modify the request -- but that's not what the typical URL resolver will
  56. do.
  57. Motivations
  58. ===========
  59. The most obvious kind of URL parsing is finding a WSGI application.
  60. Typically when a framework first supports WSGI or is integrated into
  61. Paste, it is "monolithic" with respect to URLs. That is, you define
  62. (in Paste, or maybe in Apache) a "root" URL, and everything under that
  63. goes into the framework. What the framework does internally, Paste
  64. does not know -- it probably finds internal objects to dispatch to,
  65. but the framework is opaque to Paste. Not just to Paste, but to
  66. any code that isn't in that framework.
  67. That means that we can't mix code from multiple frameworks, or as
  68. easily share services, or use WSGI middleware that doesn't apply to
  69. the entire framework/application.
  70. An example of someplace we might want to use an "application" that
  71. isn't part of the framework would be uploading large files. It's
  72. possible to keep track of upload progress, and report that back to the
  73. user, but no framework typically is capable of this. This is usually
  74. because the POST request is completely read and parsed before it
  75. invokes any application code.
  76. This is resolvable in WSGI -- a WSGI application can provide its own
  77. code to read and parse the POST request, and simultaneously report
  78. progress (usually in a way that *another* WSGI application/request can
  79. read and report to the user on that progress). This is an example
  80. where you want to allow "foreign" applications to be intermingled with
  81. framework application code.
  82. Finding Applications
  83. ====================
  84. OK, enough theory. How does a URL parser work? Well, it is a WSGI
  85. application, and a WSGI server, in the typical "WSGI middleware"
  86. style. Except that it determines which application it will serve
  87. for each request.
  88. Let's consider Paste's ``URLParser`` (in ``paste.urlparser``). This
  89. class takes a directory name as its only required argument, and
  90. instances are WSGI applications.
  91. When a request comes in, the parser looks at ``PATH_INFO`` to see
  92. what's left to parse. ``SCRIPT_NAME`` represents where we are *now*;
  93. it's the part of the URL that has been parsed.
  94. There's a couple special cases:
  95. The empty string:
  96. URLParser serves directories. When ``PATH_INFO`` is empty, that
  97. means we got a request with no trailing ``/``, like say ``/blog``
  98. If URLParser serves the ``blog`` directory, then this won't do --
  99. the user is requesting the ``blog`` *page*. We have to redirect
  100. them to ``/blog/``.
  101. A single ``/``:
  102. So, we got a trailing ``/``. This means we need to serve the
  103. "index" page. In URLParser, this is some file named ``index``,
  104. though that's really an implementation detail. You could create
  105. an index dynamically (like Apache's file listings), or whatever.
  106. Otherwise we get a string like ``/path...``. Note that ``PATH_INFO``
  107. *must* start with a ``/``, or it must be empty.
  108. URLParser pulls off the first part of the path. E.g., if
  109. ``PATH_INFO`` is ``/blog/edit/285``, then the first part is ``blog``.
  110. It appends this to ``SCRIPT_NAME``, and strips it off ``PATH_INFO``
  111. (which becomes ``/edit/285``).
  112. It then searches for a file that matches "blog". In URLParser, this
  113. means it looks for a filename which matches that name (ignoring the
  114. extension). It then uses the type of that file (determined by
  115. extension) to create a WSGI application.
  116. One case is that the file is a directory. In that case, the
  117. application is *another* URLParser instance, this time with the new
  118. directory.
  119. URLParser actually allows per-extension "plugins" -- these are just
  120. functions that get a filename, and produce a WSGI application. One of
  121. these is ``make_py`` -- this function imports the module, and looks
  122. for special symbols; if it finds a symbol ``application``, it assumes
  123. this is a WSGI application that is ready to accept the request. If it
  124. finds a symbol that matches the name of the module (e.g., ``edit``),
  125. then it assumes that is an application *factory*, meaning that when
  126. you call it with no arguments you get a WSGI application.
  127. Another function takes "unknown" files (files for which no better
  128. constructor exists) and creates an application that simply responds
  129. with the contents of that file (and the appropriate ``Content-Type``).
  130. In any case, ``URLParser`` delegates as soon as it can. It doesn't
  131. parse the entire path -- it just finds the *next* application, which
  132. in turn may delegate to yet another application.
  133. Here's a very simple implementation of URLParser::
  134. class URLParser(object):
  135. def __init__(self, dir):
  136. self.dir = dir
  137. def __call__(self, environ, start_response):
  138. segment = wsgilib.path_info_pop(environ)
  139. if segment is None: # No trailing /
  140. # do a redirect...
  141. for filename in os.listdir(self.dir):
  142. if os.path.splitext(filename)[0] == segment:
  143. return self.serve_application(
  144. environ, start_response, filename)
  145. # do a 404 Not Found
  146. def serve_application(self, environ, start_response, filename):
  147. basename, ext = os.path.splitext(filename)
  148. filename = os.path.join(self.dir, filename)
  149. if os.path.isdir(filename):
  150. return URLParser(filename)(environ, start_response)
  151. elif ext == '.py':
  152. module = import_module(filename)
  153. if hasattr(module, 'application'):
  154. return module.application(environ, start_response)
  155. elif hasattr(module, basename):
  156. return getattr(module, basename)(
  157. environ, start_response)
  158. else:
  159. return wsgilib.send_file(filename)
  160. Modifying The Request
  161. =====================
  162. Well, URLParser is one kind of parser. But others are possible, and
  163. aren't too hard to write.
  164. Lets imagine a URL like ``/2004/05/01/edit``. It's likely that
  165. ``/2004/05/01`` doesn't point to anything on file, but is really more
  166. of a "variable" that gets passed to ``edit``. So we can pull them off
  167. and put them somewhere. This is a good place for a WSGI extension.
  168. Lets put them in ``environ["app.url_date"]``.
  169. We'll pass one other applications in -- once we get the date (if any)
  170. we need to pass the request onto an application that can actually
  171. handle it. This "application" might be a URLParser or similar system
  172. (that figures out what ``/edit`` means).
  173. ::
  174. class GrabDate(object):
  175. def __init__(self, subapp):
  176. self.subapp = subapp
  177. def __call__(self, environ, start_response):
  178. date_parts = []
  179. while len(date_parts) < 3:
  180. first, rest = wsgilib.path_info_split(environ['PATH_INFO'])
  181. try:
  182. date_parts.append(int(first))
  183. wsgilib.path_info_pop(environ)
  184. except (ValueError, TypeError):
  185. break
  186. environ['app.date_parts'] = date_parts
  187. return self.subapp(environ, start_response)
  188. This is really like traditional "middleware", in that it sits between
  189. the server and just one application.
  190. Assuming you put this class in the ``myapp.grabdate`` module, you
  191. could install it by adding this to your configuration::
  192. middleware.append('myapp.grabdate.GrabDate')
  193. Object Publishing
  194. =================
  195. Besides looking in the filesystem, "object publishing" is another
  196. popular way to do URL parsing. This is pretty easy to implement as
  197. well -- it usually just means use ``getattr`` with the popped
  198. segments. But we'll implement a rough approximation of `Quixote's
  199. <http://www.mems-exchange.org/software/quixote/>`_ URL parsing::
  200. class ObjectApp(object):
  201. def __init__(self, obj):
  202. self.obj = obj
  203. def __call__(self, environ, start_response):
  204. next = wsgilib.path_info_pop(environ)
  205. if next is None:
  206. # This is the object, lets serve it...
  207. return self.publish(obj, environ, start_response)
  208. next = next or '_q_index' # the default index method
  209. if next in obj._q_export and getattr(obj, next, None):
  210. return ObjectApp(getattr(obj, next))(
  211. environ, start_reponse)
  212. next_obj = obj._q_traverse(next)
  213. if not next_obj:
  214. # Do a 404
  215. return ObjectApp(next_obj)(environ, start_response)
  216. def publish(self, obj, environ, start_response):
  217. if callable(obj):
  218. output = str(obj())
  219. else:
  220. output = str(obj)
  221. start_response('200 OK', [('Content-type', 'text/html')])
  222. return [output]
  223. The ``publish`` object is a little weak, and functions like
  224. ``_q_traverse`` aren't passed interesting information about the
  225. request, but this is only a rough approximation of the framework.
  226. Things to note:
  227. * The object has standard attributes and methods -- ``_q_exports``
  228. (attributes that are public to the web) and ``_q_traverse``
  229. (a way of overriding the traversal without having an attribute for
  230. each possible path segment).
  231. * The object isn't rendered until the path is completely consumed
  232. (when ``next`` is ``None``). This means ``_q_traverse`` has to
  233. consume extra segments of the path. In this version ``_q_traverse``
  234. is only given the next piece of the path; Quixote gives it the
  235. entire path (as a list of segments).
  236. * ``publish`` is really a small and lame way to turn a Quixote object
  237. into a WSGI application. For any serious framework you'd want to do
  238. a better job than what I do here.
  239. * It would be even better if you used something like `Adaptation
  240. <http://www.python.org/peps/pep-0246.html>`_ to convert objects into
  241. applications. This would include removing the explicit creation of
  242. new ``ObjectApp`` instances, which could also be a kind of fall-back
  243. adaptation.
  244. Anyway, this example is less complete, but maybe it will get you
  245. thinking.