heapyc.gsl 19 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583
  1. .c: Standard superkinds
  2. .import:: boolean+, Any+, dict+, Exception+, iterable+, string+, type+
  3. ..from: kindnames
  4. .c: Standard kinds
  5. .import:: Any, boolean, int, frame, iterator, list, None
  6. ..from: kindnames
  7. .import:: ObjectClassifier, NodeSet, RelationStructure, HeapView, NodeGraph
  8. ..from: kindnames
  9. .import:: NodeGraph+, NodeSet+, ObjectClassifier+,
  10. type_with_hiding_tag+
  11. ..from: kindnames
  12. .superkind:: HeapDefs+
  13. ..eg: ()
  14. .superkind:: command+
  15. ..eg: "print 'hello'"
  16. .kind:: thread_id
  17. ..subkind of: int
  18. .superkind:: thread_id+
  19. ..superkind of: thread_id
  20. .superkind:: TargetSpec+
  21. .kind:: module heapyc
  22. ..d: The heapyc module.
  23. ..attr:: HeapView
  24. ...mapping
  25. ....returns: HeapView
  26. .....d: a new HeapView object.
  27. ....arg: root:Any+
  28. .....d: The initial value of the root member.
  29. ....arg: heapdefs:HeapDefs+
  30. .....d: Definitions of specially treated extension types.
  31. ..attr:: interpreter
  32. ...mapping
  33. ....d: Create a new interpreter structure with a new thread.
  34. ....arg: command:command+
  35. .....d: A command that will be exec'd in the new environment.
  36. ....optionals
  37. .....arg: locals: dict+
  38. ......d: Local variables passed to the command when exec'd.
  39. ....returns: thread_id
  40. .....d: the thread identity number.
  41. ....d
  42. The new interpreter and thread is started in a new environment. This
  43. environment consists of a new '__main__' module, with the optional
  44. locals dict as local variables.
  45. ....d
  46. The site-specific initializations are not automatically made. To do
  47. that, 'import site' could be used as the first statement in the
  48. command string.
  49. ....d
  50. The interpreter() function will return after the new thread structure
  51. has been created. The command will execute sooner or later. The
  52. thread will terminate, and the interpreter structure be deallocated,
  53. when the command has been executed, and dependent threads have
  54. terminated.
  55. ..attr:: NodeGraph
  56. ...mapping
  57. ....d: Construct a new NodeGraph object.
  58. ....optionals
  59. .....arg: edges:iterable+
  60. ......d: The edges that will be used to
  61. initialize the new nodegraph. It should yield a
  62. sequence of pairs being edges of the form (source, target).
  63. ......default: ()
  64. .....arg: is_mapping:boolean+
  65. ......d: If True, will cause the nodegraph
  66. to be treated like a 'mapping'. It will then, for the
  67. purpose of indexing, be expected to contain a single
  68. target for each source node.
  69. ......default: False
  70. ....returns: NodeGraph
  71. ..attr:: RootState
  72. ...d: The single instance of RootStateType.
  73. ..attr:: set_async_exc
  74. ...mapping
  75. ....d: Set an exception to be raised asynchronously in a thread.
  76. ....arg: thread_id:thread_id+
  77. ....arg: exception:Exception+
  78. ..attr:: xmemstats
  79. ...mapping
  80. ....d: Print system-dependent memory statistics.
  81. ....d: What is printed depends on the system configuration.
  82. .and: HeapView
  83. ..self: HV
  84. ..d
  85. ...p
  86. A HeapView object provides methods to get memory related information
  87. about the system heap and about individual objects.
  88. ...p
  89. It implements much of the low-level functionality for the Heapy
  90. system. It is intended to provide what can not be done at all or would
  91. be much slower if programmed directly in Python. It is not intended to
  92. be used directly by a user, but to be wrapped in higher level objects.
  93. ...p
  94. Some terms that are referred to in the method descriptions:
  95. ...label: Visible objects
  96. ...h3: Visible objects.
  97. ...p
  98. The HeapView object attempts to restrict its view of the heap to only
  99. the 'visible objects'. This is to make it possible to analyse the heap
  100. via a Python library that inevitably itself is continually allocating
  101. and deallocating objects. These should be hidden from the heap view
  102. presented. This is primarily done via a special tag attribute, see
  103. ....ref: .mykind._hiding_tag_
  104. ....t: and
  105. ....ref: .mykind.register__hiding_tag__type
  106. ....t:. Frames can be hidden
  107. with another mechanism, see
  108. ....ref: .mykind.limitframe
  109. ....t:. For hiding all objects of a special type,
  110. ....ref: .mykind.register_hidden_exact_type
  111. ....t: may be used. It is also
  112. possible to use a separate interpreter and hide its root objects, see
  113. ....ref: .mykind.is_hiding_calling_interpreter
  114. ....t:.
  115. ...h3: Classifiers.
  116. ...text
  117. The methods named cli_* are factory methods that create objects of
  118. type ObjectClassifier. The principal difference between classifiers is
  119. how a single object is classified. The single-object classification
  120. function is available in classifier objects; it is the classify
  121. method. There are also methods that operate on collections of objects,
  122. namely partition and select. These eliminate the per-object
  123. Python-level function call overhead that would occur if the classify
  124. method were to be called from Python for each object in a collection.
  125. See also the ObjectClassifier type.
  126. ...label: Individual size
  127. ...h3: Individual size.
  128. ...p
  129. The individual size of an object is its individually allocated memory size.
  130. ...p
  131. It includes:
  132. ...ul
  133. ....li: The basic object size, as can be found out in a standard way.
  134. ....li: The extra memory for variable size objects.
  135. ....li: For GC collected objects, the size of the GC information.
  136. ....li: An alignment to the next highest multiple of a pointer size.
  137. ....li: The size of any other memory allocated that belongs to the object.
  138. ...p
  139. Some types of objects have extra memory allocated that can not be
  140. accounted for in the standard way. This memory should nevertheless be
  141. included in the individual size. To determine the size of these
  142. objects, special functions are needed. These are defined for standard
  143. builtin types, such as lists and dicts. Other types should be defined
  144. via the heapdefs argument to the HeapView constructor.
  145. ...p: The individual size does not include:
  146. ...ul
  147. ....li: Subobjects that are accounted for separately.
  148. ....li: Overhead for the memory allocation system. This varies depending
  149. on the kind of memory allocator, the requested size, etc.
  150. ..constructor: module heapyc.HeapView
  151. ..attr:: _hiding_tag_
  152. ...self: HV
  153. ...d: The hiding tag defining what objects are hidden from the view defined
  154. by HV. Objects that contain a _hiding_tag_ object which is identical
  155. to HV._hiding_tag_, will be hidden from view, in the following cases:
  156. ....ul
  157. .....li: The object is of a type that has been registered for hiding via
  158. ......ref: .mykind.register__hiding_tag__type
  159. ......t:, or is of a subtype of such a type.
  160. .....li: The object is of instance type. Such an object will be checked
  161. for a _hiding_tag_ item in its __dict__.
  162. ..attr:: cli_class
  163. ...mapping
  164. ....returns: ObjectClassifier
  165. .....d: a classifier that classifies by "class".
  166. .....d:
  167. If an object is of instance type, its classification will be its
  168. class, as given in its C-level member 'in_class', otherwise it will
  169. be its type, as given in 'ob_type'.
  170. ....dwh: Note
  171. This is like classifying by the Python-level '__class__' attribute
  172. in newer versions of Python, except that even though the '__class__'
  173. attribute may be redefined via the class definition, the classification
  174. as made by cli_class isn't affected.
  175. ..attr:: cli_clodo
  176. ...mapping
  177. ....returns: ObjectClassifier
  178. .....d: a classifier, that classifies by "Class Or Dict Owner".
  179. .....d
  180. The classification of an object is its class (as by cli_class),
  181. unless the object is a dict object that is 'owned' by some owner.
  182. If the object is such an owned dict, the classification will be a
  183. 1-tuple containing the class (as by cli_class) of its owner.
  184. ....arg: owners:NodeGraph+
  185. .....d:
  186. Used to map each dict object to its owner, or to None if it has no
  187. owner. The graph will be automatically updated, from heap information
  188. defined by HV, whenever an attempt is made to classify a dict that
  189. maps to nothing.
  190. ....arg: memo:dict+
  191. .....d: Used to memoize the 1-tuples generated when classifying owned dicts.
  192. ..attr:: cli_indisize
  193. ...mapping
  194. ....returns: ObjectClassifier
  195. .....d: a classifier that classifies by "individual size".
  196. .....d: The classification of each object is an int, containing the
  197. object's individual memory size.
  198. ....arg: memo:dict+
  199. .....d: Used to memoize the classification objects.
  200. ..attr:: cli_none
  201. ...mapping
  202. ....returns: ObjectClassifier
  203. .....d: a classifier that classifies all objects the same.
  204. .....d: The classification of each object is None.
  205. ..attr:: cli_rcs
  206. ...mapping
  207. ....returns: ObjectClassifier
  208. .....d: a classifier that classifies by Referrer Classification Set.
  209. .....d: The classification of an object is the classifications of its
  210. referrers, collected in an immutable NodeSet object.
  211. ....arg: referrers: NodeGraph+
  212. .....d: Used to map each object to its referrers.
  213. ....arg: classifier:ObjectClassifier+
  214. .....d: Used to classify each referrer.
  215. ....arg: memo:dict+
  216. .....d: Used to memoize the classification sets.
  217. ..attr:: cli_type
  218. ...mapping
  219. ....returns: ObjectClassifier
  220. .....d: a classifier that classifies by type.
  221. .....d:
  222. The classification of each object is the type, as given by its
  223. C-level member 'ob_type'. (This is the same as the type returned
  224. by the Python-level builtin 'type'.)
  225. ..attr:: delete_extra_type
  226. ...description: For Internal Use
  227. ..attr:: heap
  228. ...mapping
  229. ....args
  230. ....returns: NodeSet
  231. .....d: a set containing all 'visible objects' in the heap view
  232. defined by HV.
  233. ....dwh: See also
  234. .....ref: .mykind.Visible objects
  235. ......t: Visible objects
  236. ..attr:: indisize_sum
  237. ...mapping
  238. ....arg: S:iterable+
  239. ....returns: int
  240. .....d: the sum of the 'individual size' of the objects in S.
  241. ....dwh: See also
  242. .....ref: .mykind.Individual size
  243. ......t: Individual size
  244. ..attr:: numedges
  245. ...mapping
  246. ....arg: src:Any+
  247. ....arg: tgt:Any+
  248. ....returns: int
  249. .....d: the number of edges from src to tgt.
  250. ..attr:: reachable
  251. ...mapping
  252. ....arg: X:NodeSet+
  253. ....arg: Y:NodeSet+
  254. ....returns: NodeSet
  255. .....d: the set of objects reached via a path in the visible heap as
  256. defined by HV, from some object in X, avoiding any object in Y.
  257. ..attr:: reachable_x
  258. ...mapping
  259. ....arg: X:NodeSet+
  260. ....arg: Y:NodeSet+
  261. ....returns: NodeSet
  262. .....d: the set of objects reached via a path in the visible heap as
  263. defined by HV, from some object in X, avoiding any object in Y except
  264. at the end of the path.
  265. ..attr:: register__hiding_tag__type
  266. ...mapping
  267. ....args: type_with_hiding_tag+
  268. ....d:
  269. Register a type of objects that may be hidden from the heap view
  270. defined by HV. The type must have a slot named _hiding_tag_. An object
  271. that is an instance of the type, or of a subtype, is hidden when its
  272. _hiding_tag_ is HV._hiding_tag_.
  273. ..attr:: register_hidden_exact_type
  274. ...mapping
  275. ....args: type+
  276. ....d:
  277. Register a type of objects that should be hidden from the heap view
  278. defined by HV. Objects of the exact type registered -- not including
  279. subtypes -- will be hidden.
  280. ..attr:: relate
  281. ...mapping
  282. ....arg: src:Any+
  283. ....arg: tgt:Any+
  284. ....returns: RelationStructure
  285. .....d: a description of the relation between src and tgt.
  286. .....d: This is used for descriptions of edges in paths.
  287. .....d: [The result is in a special format that I choose to not define here
  288. since it is for special low-level use and subject to change.]
  289. ..attr:: relimg
  290. ...mapping
  291. ....returns: NodeSet
  292. .....d: the 'relational image of HV wrt S'.
  293. .....d: That is, the set of nodes
  294. that are directly referred to from the nodes in S via the visible heap
  295. reachability relation as defined by HV.
  296. ....arg: S:iterable+
  297. ..attr:: shpathstep
  298. ...mapping
  299. ....d: This method implements one step of a shortest path algorithm.
  300. ....arg: G:NodeGraph+
  301. .....d: Updated by the method, with the edges from nodes in the
  302. source set to the new nodes visited.
  303. ....arg: U:NodeSet+
  304. .....d: The source set for this step.
  305. ....arg: S:NodeSet+
  306. .....d: The set of already visited nodes.
  307. ....optionals
  308. .....arg: AvoidEdges:NodeGraph+
  309. ......d: Edges to avoid.
  310. .....arg: find_one:boolean+
  311. ......d: If True, at most one edge will be found from each node
  312. in the source set. Normally, all edges will be found.
  313. ....returns: NodeSet
  314. .....d: the new nodes visited.
  315. .....d: This may be used for the U argument the next time the method is called.
  316. ....dwh: See also
  317. .....t: shpgraph_algorithm in Path.py.
  318. ..attr:: update_dictowners
  319. ...mapping
  320. ....d: Update owners with ownership edges.
  321. ....d: The dict owners graph will be updated with an edge from each dict
  322. object in the heap, to either its owner or to None.
  323. ....arg: owners:NodeGraph+
  324. ..attr:: update_referrers
  325. ...mapping
  326. ....d: Update referrer graph X for Y.
  327. ....d: The visible heap defined by HV will be traversed from the root of HV
  328. so that the edges of every path from the root to nodes in Y will be
  329. represented, inverted, in X.
  330. ....arg: X:NodeGraph+
  331. ....arg: Y:NodeSet+
  332. ..attr:: update_referrers_completely
  333. ...mapping
  334. ....arg: X:NodeGraph+
  335. ....d: Update referrer graph X 'completely'.
  336. ....d: [Experimental algorithm that updates X with the referrers to all
  337. objects in the heap (of visible nodes as defined in HV). It is not
  338. normally used.]
  339. ..attr:: is_hiding_calling_interpreter
  340. ...kind of: boolean
  341. ...d:
  342. If True, the data of the interpreter using the HV will be hidden from
  343. the heap view as seen from RootState.
  344. ...d
  345. This is used when multiple Python interpreters are used. One
  346. interpreter will be monitoring the operation of the other
  347. interpreter(s). It would set is_hiding_calling_interpreter to True in
  348. the HV it is using. Its own data will then be hidden from view, making
  349. memory leak detection more practical.
  350. ..attr:: is_using_traversing_owner_update
  351. ...kind of: boolean
  352. ...d:
  353. True if update_dictowners is using a recursive traversal algorithm to
  354. find the dicts in the heap. When False, the normal case, it will use the
  355. dicts found in the gc collection structure, by gc.get_objects(). This was
  356. found to be much faster in usual cases, but the old version is available
  357. by setting this flag. -- It may be removed in a later release!
  358. ..attr:: limitframe
  359. ...either: None, frame
  360. ...d: The traversal limiting frame.
  361. ...d: If limitframe is set to a frame object, the frames that are more
  362. recently entered than limitframe will be hidden when traversing the
  363. heap from the root RootState. It will start traversing from limitframe
  364. rather than from the most recent frame as it would otherwise do.
  365. .and: NodeGraph
  366. ..d
  367. A NodeGraph object contains pairs of nodes (edges) and can be indexed
  368. on the first node of such a pair (the source of an edge) to find all
  369. second nodes of such pairs (the targets of those edges).
  370. ...p
  371. NodeGraph objects are used internally in the Heapy system, for example
  372. to record dict ownership and shortest-path graphs.
  373. ...p
  374. They may be used generally for mapping and dict-like purposes, but
  375. differ from dicts in the following:
  376. ...ul
  377. ....li: The mapping is based on object identity - no equality or hashing is
  378. assumed, so any object can be used as a key. Only the address is used.
  379. To distinguish this usage from that of ordinary dicts and sets, such
  380. objects are called 'nodes'.
  381. ....li: There may be any number of targets associated with each source.
  382. ....li: Performance characteristics differ from dicts, in somewhat subtle ways.
  383. ..self: NG
  384. ..constructor: module heapyc.NodeGraph
  385. ..attr:: _hiding_tag_
  386. ...d: The hiding tag: if it is the the same object as
  387. ....ref: .myfile.HeapView._hiding_tag_
  388. ....t: of a HeapView object, the nodegraph will be hidden from the corresponding heap view .
  389. ..attr:: is_mapping
  390. ...kind of: boolean
  391. ...d: Read only. True if NG is a 'mapping'. Then, only one edge is
  392. allowed for each source; indexing returns the actual target object
  393. instead of a tuple of targets.
  394. ..attr:: is_sorted
  395. ...kind of: boolean
  396. ...d: Read only. True if NG is sorted.
  397. ...d: It will become unsorted after any update. It will need to be
  398. sorted to make it possible to find edges (implementation uses binary
  399. search). Any indexing operation will automatically sort it if it was
  400. not already sorted. The flag is currently used from Python to see if
  401. the nodegraph has been used at least once after update, so that it
  402. will not be cleared too early.
  403. ..fop: len
  404. ...returns: int
  405. ....d: the number of edges in NG.
  406. ..getitem
  407. ...returns: Any
  408. ....d: the target(s) of all edges with a particular source. The value
  409. depends on if NG was initialized to be a 'mapping' or not:
  410. .....dl
  411. ......dt: NG.is_mapping == False
  412. ......dd: Return a tuple of all targets of edges from the source.
  413. The tuple will be empty if there are no such edges.
  414. ......dt: NG.is_mapping == True
  415. ......dd: Return the target of the edge from the source.
  416. If there is no such edge, KeyError will be raised.
  417. If there is more than one edge, ValueError will be raised.
  418. ...arg: source: Any+
  419. ..setitem
  420. ...d: Assign new target(s) to the edge(s) from a given source. How the
  421. targets are specified, depends on whether NG is a 'mapping' or not:
  422. ....dl:
  423. .....dt: NG.is_mapping == False
  424. .....dd: The target_spec argument is a tuple of targets.
  425. There must already be the same number of edges in NG from the given
  426. source as the number of objects in target_spec.
  427. .....dt: NG.is_mapping == True
  428. .....dd: The target_spec argument is the target itself.
  429. There must already be exactly one edge in NG from the given source.
  430. ...d:
  431. If there were not already the same number of edges as the targets specified,
  432. ValueError will be raised. The reason for the restriction is to make sure
  433. assignment will be efficient when interleaved with indexing.
  434. ...dwh: See also
  435. ....ref: .mykind.add_edge
  436. ....t:,
  437. ....ref: .mykind.add_edges_n1
  438. ...arg: source: Any+
  439. ...arg: target_spec: TargetSpec+
  440. ..method:: add_edge
  441. ...d: Add to NG, an edge from source to target.
  442. ...arg: source: Any+
  443. ...arg: target: Any+
  444. ..method:: add_edges_n1
  445. ...d: Add to NG, for each src in srcs, an edge from src to tgt.
  446. ...arg: srcs: iterable+
  447. ...arg: tgt: Any+
  448. ..method:: as_flat_list
  449. ...returns: list
  450. ....d: the edges of NG in the form [src0, tgt0, src1, tgt1 ...].
  451. ..method:: clear
  452. ...d: Remove all items from NG.
  453. ..method:: copy
  454. ...returns: NodeGraph
  455. ....d: a copy of NG.
  456. ..method:: domain_covers
  457. ...returns: boolean
  458. ....d: True if each node in X is the source of some edge in NG, False otherwise.
  459. ...arg: X: iterable+
  460. ..method:: domain_restricted
  461. ...returns: NodeGraph
  462. ....d: a new NodeGraph, containing those edges in NG that have the source in X.
  463. ...arg: X: iterable+
  464. ..method:: get_domain
  465. ...returns: NodeSet
  466. ....d: the set of nodes that are the source of some edge in NG.
  467. ..method:: get_range
  468. ...returns: NodeSet
  469. ....d: the set of nodes that are the target of some edge in NG.
  470. ..method:: invert
  471. ...d: Invert the edges of NG.
  472. ..method:: inverted
  473. ...returns: NodeGraph
  474. ....d: a copy of NG with the edges inverted.
  475. ..fop: iter
  476. ...returns: iterator
  477. ....d: an iterator yielding a pair (source, target) for each edge in NG.
  478. ..method:: relimg
  479. ...returns: NodeSet
  480. ....d: the relational image of NG wrt X.
  481. ....d: That is, the set of nodes that are the target of some edge that have
  482. its source in X.
  483. ...arg: X:iterable+
  484. ..method:: update
  485. ...d: Update NG with the edges from X,
  486. specified as pairs of the form (source, target).
  487. ...arg: X:iterable+
  488. ..method:: updated
  489. ...returns: NodeGraph
  490. ....d: a copy of NG updated with the edges from X,
  491. specified as pairs of the form (source, target).
  492. ...arg: X:iterable+