manager-objects.html 18 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287
  1. <?xml version="1.0" encoding="utf-8" ?>
  2. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  3. <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  4. <head>
  5. <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  6. <meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" />
  7. <title>Manager objects</title>
  8. <link rel="stylesheet" href="html4css1.css" type="text/css" />
  9. </head>
  10. <body>
  11. <div class="header">
  12. <a class="reference" href="connection-objects.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="processing-ref.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="proxy-objects.html">Next</a>
  13. <hr class="header"/>
  14. </div>
  15. <div class="document" id="manager-objects">
  16. <h1 class="title">Manager objects</h1>
  17. <p>A manager object controls a server process which manages <em>shared
  18. objects</em>. Other processes can access the shared objects by using
  19. proxies.</p>
  20. <p>Manager processes will be shutdown as soon as they are garbage
  21. collected or their parent process exits. The manager classes are
  22. defined in the <tt class="docutils literal"><span class="pre">processing.managers</span></tt> module.</p>
  23. <div class="section">
  24. <h1><a id="basemanager" name="basemanager">BaseManager</a></h1>
  25. <p><tt class="docutils literal"><span class="pre">BaseManager</span></tt> is the base class for all manager classes which use a
  26. server process. It does not possess any methods which create shared
  27. objects.</p>
  28. <p>The public methods of <tt class="docutils literal"><span class="pre">BaseManager</span></tt> are the following:</p>
  29. <blockquote>
  30. <dl class="docutils">
  31. <dt><tt class="docutils literal"><span class="pre">__init__(self,</span> <span class="pre">address=None,</span> <span class="pre">authkey=None)</span></tt></dt>
  32. <dd><p class="first">Creates a manager object.</p>
  33. <p>Once created one should call <tt class="docutils literal"><span class="pre">start()</span></tt> or <tt class="docutils literal"><span class="pre">serveForever()</span></tt> to
  34. ensure that the manager object refers to a started manager
  35. process.</p>
  36. <p>The arguments to the constructor are as follows:</p>
  37. <dl class="last docutils">
  38. <dt><tt class="docutils literal"><span class="pre">address</span></tt></dt>
  39. <dd><p class="first">The address on which the manager process listens for
  40. new connections. If <tt class="docutils literal"><span class="pre">address</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then an arbitrary
  41. one is chosen.</p>
  42. <p class="last">See <a class="reference" href="connection-ref.html#listener-objects">Listener objects</a>.</p>
  43. </dd>
  44. <dt><tt class="docutils literal"><span class="pre">authkey</span></tt></dt>
  45. <dd><p class="first">The authentication key which will be used to check the
  46. validity of incoming connections to the server process.</p>
  47. <p>If <tt class="docutils literal"><span class="pre">authkey</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then <tt class="docutils literal"><span class="pre">currentProcess().getAuthKey()</span></tt>.
  48. Otherwise <tt class="docutils literal"><span class="pre">authkey</span></tt> is used and it must be a string.</p>
  49. <p class="last">See <a class="reference" href="connection-ref.html#authentication-keys">Authentication keys</a>.</p>
  50. </dd>
  51. </dl>
  52. </dd>
  53. <dt><tt class="docutils literal"><span class="pre">start()</span></tt></dt>
  54. <dd>Spawn or fork a subprocess to start the manager.</dd>
  55. <dt><tt class="docutils literal"><span class="pre">serveForever()</span></tt></dt>
  56. <dd>Start the manager in the current process. See <a class="reference" href="#using-a-remote-manager">Using a remote
  57. manager</a>.</dd>
  58. <dt><tt class="docutils literal"><span class="pre">fromAddress(address,</span> <span class="pre">authkey)</span></tt></dt>
  59. <dd>A class method which returns a manager object referring to a
  60. pre-existing server process which is using the given address and
  61. authentication key. See <a class="reference" href="#using-a-remote-manager">Using a remote manager</a>.</dd>
  62. <dt><tt class="docutils literal"><span class="pre">shutdown()</span></tt></dt>
  63. <dd><p class="first">Stop the process used by the manager. This is only available
  64. if <tt class="docutils literal"><span class="pre">start()</span></tt> has been used to start the server process.</p>
  65. <p class="last">This can be called multiple times.</p>
  66. </dd>
  67. </dl>
  68. </blockquote>
  69. <p><tt class="docutils literal"><span class="pre">BaseManager</span></tt> instances also have one read-only property:</p>
  70. <blockquote>
  71. <dl class="docutils">
  72. <dt><tt class="docutils literal"><span class="pre">address</span></tt></dt>
  73. <dd>The address used by the manager.</dd>
  74. </dl>
  75. </blockquote>
  76. <p>The creation of managers which support arbitrary types is discussed
  77. below in <a class="reference" href="#customized-managers">Customized managers</a>.</p>
  78. </div>
  79. <div class="section">
  80. <h1><a id="syncmanager" name="syncmanager">SyncManager</a></h1>
  81. <p><tt class="docutils literal"><span class="pre">SyncManager</span></tt> is a subclass of <tt class="docutils literal"><span class="pre">BaseManager</span></tt> which can be used for
  82. the synchronization of processes. Objects of this type are returned
  83. by <tt class="docutils literal"><span class="pre">processing.Manager()</span></tt>.</p>
  84. <p>It also supports creation of shared lists and dictionaries. The
  85. instance methods defined by <tt class="docutils literal"><span class="pre">SyncManager</span></tt> are</p>
  86. <blockquote>
  87. <dl class="docutils">
  88. <dt><tt class="docutils literal"><span class="pre">BoundedSemaphore(value=1)</span></tt></dt>
  89. <dd>Creates a shared <tt class="docutils literal"><span class="pre">threading.BoundedSemaphore</span></tt> object and
  90. returns a proxy for it.</dd>
  91. <dt><tt class="docutils literal"><span class="pre">Condition(lock=None)</span></tt></dt>
  92. <dd><p class="first">Creates a shared <tt class="docutils literal"><span class="pre">threading.Condition</span></tt> object and returns a
  93. proxy for it.</p>
  94. <p class="last">If <tt class="docutils literal"><span class="pre">lock</span></tt> is supplied then it should be a proxy for a
  95. <tt class="docutils literal"><span class="pre">threading.Lock</span></tt> or <tt class="docutils literal"><span class="pre">threading.RLock</span></tt> object.</p>
  96. </dd>
  97. <dt><tt class="docutils literal"><span class="pre">Event()</span></tt></dt>
  98. <dd>Creates a shared <tt class="docutils literal"><span class="pre">threading.Event</span></tt> object and returns a proxy
  99. for it.</dd>
  100. <dt><tt class="docutils literal"><span class="pre">Lock()</span></tt></dt>
  101. <dd>Creates a shared <tt class="docutils literal"><span class="pre">threading.Lock</span></tt> object and returns a proxy
  102. for it.</dd>
  103. <dt><tt class="docutils literal"><span class="pre">Namespace()</span></tt></dt>
  104. <dd><p class="first">Creates a shared <tt class="docutils literal"><span class="pre">Namespace</span></tt> object and returns a
  105. proxy for it.</p>
  106. <p class="last">See <a class="reference" href="#namespace-objects">Namespace objects</a>.</p>
  107. </dd>
  108. <dt><tt class="docutils literal"><span class="pre">Queue(maxsize=0)</span></tt></dt>
  109. <dd>Creates a shared <tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> object and returns a proxy for
  110. it.</dd>
  111. <dt><tt class="docutils literal"><span class="pre">RLock()</span></tt></dt>
  112. <dd>Creates a shared <tt class="docutils literal"><span class="pre">threading.RLock</span></tt> object and returns a proxy
  113. for it.</dd>
  114. <dt><tt class="docutils literal"><span class="pre">Semaphore(value=1)</span></tt></dt>
  115. <dd>Creates a shared <tt class="docutils literal"><span class="pre">threading.Semaphore</span></tt> object and returns a
  116. proxy for it.</dd>
  117. <dt><tt class="docutils literal"><span class="pre">Array(typecode,</span> <span class="pre">sequence)</span></tt></dt>
  118. <dd>Create an array and returns a proxy for
  119. it. (<tt class="docutils literal"><span class="pre">format</span></tt> is ignored.)</dd>
  120. <dt><tt class="docutils literal"><span class="pre">Value(typecode,</span> <span class="pre">value)</span></tt></dt>
  121. <dd>Create an object with a writable <tt class="docutils literal"><span class="pre">value</span></tt> attribute and returns
  122. a proxy for it.</dd>
  123. <dt><tt class="docutils literal"><span class="pre">dict()</span></tt>, <tt class="docutils literal"><span class="pre">dict(mapping)</span></tt>, <tt class="docutils literal"><span class="pre">dict(sequence)</span></tt></dt>
  124. <dd>Creates a shared <tt class="docutils literal"><span class="pre">dict</span></tt> object and returns a proxy for it.</dd>
  125. <dt><tt class="docutils literal"><span class="pre">list()</span></tt>, <tt class="docutils literal"><span class="pre">list(sequence)</span></tt></dt>
  126. <dd>Creates a shared <tt class="docutils literal"><span class="pre">list</span></tt> object and returns a proxy for it.</dd>
  127. </dl>
  128. </blockquote>
  129. <div class="section">
  130. <h2><a id="namespace-objects" name="namespace-objects">Namespace objects</a></h2>
  131. <p>A namespace object has no public methods but does have writable
  132. attributes. Its representation shows the values of its attributes.</p>
  133. <p>However, when using a proxy for a namespace object, an attribute
  134. beginning with <tt class="docutils literal"><span class="pre">'_'</span></tt> will be an attribute of the proxy and not an
  135. attribute of the referent:</p>
  136. <pre class="literal-block">
  137. &gt;&gt;&gt; manager = processing.Manager()
  138. &gt;&gt;&gt; Global = manager.Namespace()
  139. &gt;&gt;&gt; Global.x = 10
  140. &gt;&gt;&gt; Global.y = 'hello'
  141. &gt;&gt;&gt; Global._z = 12.3 # this is an attribute of the proxy
  142. &gt;&gt;&gt; print Global
  143. Namespace(x=10, y='hello')
  144. </pre>
  145. </div>
  146. </div>
  147. <div class="section">
  148. <h1><a id="customized-managers" name="customized-managers">Customized managers</a></h1>
  149. <p>To create one's own manager one creates a subclass of <tt class="docutils literal"><span class="pre">BaseManager</span></tt>.</p>
  150. <p>To create a method of the subclass which will create new shared
  151. objects one uses the following function:</p>
  152. <blockquote>
  153. <dl class="docutils">
  154. <dt><tt class="docutils literal"><span class="pre">CreatorMethod(callable=None,</span> <span class="pre">proxytype=None,</span> <span class="pre">exposed=None,</span> <span class="pre">typeid=None)</span></tt></dt>
  155. <dd><p class="first">Returns a function with signature <tt class="docutils literal"><span class="pre">func(self,</span> <span class="pre">*args,</span> <span class="pre">**kwds)</span></tt>
  156. which will create a shared object using the manager <tt class="docutils literal"><span class="pre">self</span></tt>
  157. and return a proxy for it.</p>
  158. <p>The shared objects will be created by evaluating
  159. <tt class="docutils literal"><span class="pre">callable(*args,</span> <span class="pre">**kwds)</span></tt> in the manager process.</p>
  160. <p>The arguments are:</p>
  161. <dl class="last docutils">
  162. <dt><tt class="docutils literal"><span class="pre">callable</span></tt></dt>
  163. <dd>The callable used to create a shared object. If the
  164. manager will connect to a remote manager then this is ignored.</dd>
  165. <dt><tt class="docutils literal"><span class="pre">proxytype</span></tt></dt>
  166. <dd><p class="first">The type of proxy which will be used for object returned
  167. by <tt class="docutils literal"><span class="pre">callable</span></tt>.</p>
  168. <p class="last">If <tt class="docutils literal"><span class="pre">proxytype</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then each time an object is
  169. returned by <tt class="docutils literal"><span class="pre">callable</span></tt> either a new proxy type is created
  170. or a cached one is reused. The methods of the shared
  171. object which will be exposed via the proxy will then be
  172. determined by the <tt class="docutils literal"><span class="pre">exposed</span></tt> argument, see below.</p>
  173. </dd>
  174. <dt><tt class="docutils literal"><span class="pre">exposed</span></tt></dt>
  175. <dd><p class="first">Given a shared object returned by <tt class="docutils literal"><span class="pre">callable</span></tt>, the
  176. <tt class="docutils literal"><span class="pre">exposed</span></tt> argument is the list of those method names which
  177. should be exposed via <a class="reference" href="proxy-objects.html#methods-of-baseproxy"><tt class="docutils literal"><span class="pre">BaseProxy._callMethod()</span></tt></a>. <a class="footnote-reference" href="#id3" id="id1" name="id1">[1]</a> <a class="footnote-reference" href="#id4" id="id2" name="id2">[2]</a></p>
  178. <p>If <tt class="docutils literal"><span class="pre">exposed</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> and <tt class="docutils literal"><span class="pre">callable.__exposed__</span></tt> exists then
  179. <tt class="docutils literal"><span class="pre">callable.__exposed__</span></tt> is used instead.</p>
  180. <p>If <tt class="docutils literal"><span class="pre">exposed</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> and <tt class="docutils literal"><span class="pre">callable.__exposed__</span></tt> does not
  181. exist then all methods of the shared object which do not
  182. start with <tt class="docutils literal"><span class="pre">'_'</span></tt> will be exposed.</p>
  183. <p class="last">An attempt to use <tt class="docutils literal"><span class="pre">BaseProxy._callMethod()</span></tt> with a method name which is
  184. not exposed will raise an exception.</p>
  185. </dd>
  186. <dt><tt class="docutils literal"><span class="pre">typeid</span></tt></dt>
  187. <dd>If <tt class="docutils literal"><span class="pre">typeid</span></tt> is a string then it is used as an identifier
  188. for the callable. Otherwise, <tt class="docutils literal"><span class="pre">typeid</span></tt> must be <tt class="docutils literal"><span class="pre">None</span></tt> and
  189. a string prefixed by <tt class="docutils literal"><span class="pre">callable.__name__</span></tt> is used as the
  190. identifier.</dd>
  191. </dl>
  192. </dd>
  193. </dl>
  194. </blockquote>
  195. <table class="docutils footnote" frame="void" id="id3" rules="none">
  196. <colgroup><col class="label" /><col /></colgroup>
  197. <tbody valign="top">
  198. <tr><td class="label"><a class="fn-backref" href="#id1" name="id3">[1]</a></td><td>A method here means any attribute which has a <tt class="docutils literal"><span class="pre">__call__</span></tt>
  199. attribute.</td></tr>
  200. </tbody>
  201. </table>
  202. <table class="docutils footnote" frame="void" id="id4" rules="none">
  203. <colgroup><col class="label" /><col /></colgroup>
  204. <tbody valign="top">
  205. <tr><td class="label"><a class="fn-backref" href="#id2" name="id4">[2]</a></td><td><p class="first">The method names <tt class="docutils literal"><span class="pre">__repr__</span></tt>, <tt class="docutils literal"><span class="pre">__str__</span></tt>, and <tt class="docutils literal"><span class="pre">__cmp__</span></tt> of a
  206. shared object are always exposed by the manager. However, instead
  207. of invoking the <tt class="docutils literal"><span class="pre">__repr__()</span></tt>, <tt class="docutils literal"><span class="pre">__str__()</span></tt>, <tt class="docutils literal"><span class="pre">__cmp__()</span></tt> instance
  208. methods (none of which are guaranteed to exist) they invoke the
  209. builtin functions <tt class="docutils literal"><span class="pre">repr()</span></tt>, <tt class="docutils literal"><span class="pre">str()</span></tt> and <tt class="docutils literal"><span class="pre">cmp()</span></tt>.</p>
  210. <p class="last">Note that one should generally avoid exposing rich comparison
  211. methods like <tt class="docutils literal"><span class="pre">__eq__()</span></tt>, <tt class="docutils literal"><span class="pre">__ne__()</span></tt>, <tt class="docutils literal"><span class="pre">__le__()</span></tt>. To make the proxy
  212. type support comparison by value one can just expose <tt class="docutils literal"><span class="pre">__cmp__()</span></tt>
  213. instead (even if the referent does not have such a method).</p>
  214. </td></tr>
  215. </tbody>
  216. </table>
  217. <div class="section">
  218. <h2><a id="example" name="example">Example</a></h2>
  219. <pre class="literal-block">
  220. from processing.managers import BaseManager, CreatorMethod
  221. class FooClass(object):
  222. def bar(self):
  223. print 'BAR'
  224. def baz(self):
  225. print 'BAZ'
  226. class NewManager(BaseManager):
  227. Foo = CreatorMethod(FooClass)
  228. if __name__ == '__main__':
  229. manager = NewManager()
  230. manager.start()
  231. foo = manager.Foo()
  232. foo.bar() # prints 'BAR'
  233. foo.baz() # prints 'BAZ'
  234. manager.shutdown()
  235. </pre>
  236. <p>See <a class="reference" href="../examples/ex_newtype.py">ex_newtype.py</a> for more examples.</p>
  237. </div>
  238. </div>
  239. <div class="section">
  240. <h1><a id="using-a-remote-manager" name="using-a-remote-manager">Using a remote manager</a></h1>
  241. <p>It is possible to run a manager server on one machine and have clients
  242. use it from other machines (assuming that the firewalls involved allow
  243. it).</p>
  244. <p>Running the following commands creates a server for a shared queue which
  245. remote clients can use:</p>
  246. <pre class="literal-block">
  247. &gt;&gt;&gt; from processing.managers import BaseManager, CreatorMethod
  248. &gt;&gt;&gt; import Queue
  249. &gt;&gt;&gt; queue = Queue.Queue()
  250. &gt;&gt;&gt; class QueueManager(BaseManager):
  251. ... get_proxy = CreatorMethod(callable=lambda:queue, typeid='get_proxy')
  252. ...
  253. &gt;&gt;&gt; m = QueueManager(address=('foo.bar.org', 50000), authkey='none')
  254. &gt;&gt;&gt; m.serveForever()
  255. </pre>
  256. <p>One client can access the server as follows:</p>
  257. <pre class="literal-block">
  258. &gt;&gt;&gt; from processing.managers import BaseManager, CreatorMethod
  259. &gt;&gt;&gt; class QueueManager(BaseManager):
  260. ... get_proxy = CreatorMethod(typeid='get_proxy')
  261. ...
  262. &gt;&gt;&gt; m = QueueManager.fromAddress(address=('foo.bar.org', 50000), authkey='none')
  263. &gt;&gt;&gt; queue = m.get_proxy()
  264. &gt;&gt;&gt; queue.put('hello')
  265. </pre>
  266. <p>Another client can also use it:</p>
  267. <pre class="literal-block">
  268. &gt;&gt;&gt; from processing.managers import BaseManager, CreatorMethod
  269. &gt;&gt;&gt; class QueueManager(BaseManager):
  270. ... get_proxy = CreatorMethod(typeid='get_proxy')
  271. ...
  272. &gt;&gt;&gt; m = QueueManager.fromAddress(address=('foo.bar.org', 50000), authkey='none')
  273. &gt;&gt;&gt; queue = m.get_proxy()
  274. &gt;&gt;&gt; queue.get()
  275. 'hello'
  276. </pre>
  277. </div>
  278. </div>
  279. <div class="footer">
  280. <hr class="footer" />
  281. <a class="reference" href="connection-objects.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="processing-ref.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="proxy-objects.html">Next</a>
  282. </div>
  283. </body>
  284. </html>