| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583 |
- .c: Standard superkinds
- .import:: boolean+, Any+, dict+, Exception+, iterable+, string+, type+
- ..from: kindnames
- .c: Standard kinds
- .import:: Any, boolean, int, frame, iterator, list, None
- ..from: kindnames
- .import:: ObjectClassifier, NodeSet, RelationStructure, HeapView, NodeGraph
- ..from: kindnames
- .import:: NodeGraph+, NodeSet+, ObjectClassifier+,
- type_with_hiding_tag+
- ..from: kindnames
- .superkind:: HeapDefs+
- ..eg: ()
- .superkind:: command+
- ..eg: "print 'hello'"
- .kind:: thread_id
- ..subkind of: int
- .superkind:: thread_id+
- ..superkind of: thread_id
- .superkind:: TargetSpec+
- .kind:: module heapyc
- ..d: The heapyc module.
- ..attr:: HeapView
- ...mapping
- ....returns: HeapView
- .....d: a new HeapView object.
- ....arg: root:Any+
- .....d: The initial value of the root member.
- ....arg: heapdefs:HeapDefs+
- .....d: Definitions of specially treated extension types.
- ..attr:: interpreter
- ...mapping
- ....d: Create a new interpreter structure with a new thread.
- ....arg: command:command+
- .....d: A command that will be exec'd in the new environment.
- ....optionals
- .....arg: locals: dict+
- ......d: Local variables passed to the command when exec'd.
- ....returns: thread_id
- .....d: the thread identity number.
- ....d
- The new interpreter and thread is started in a new environment. This
- environment consists of a new '__main__' module, with the optional
- locals dict as local variables.
- ....d
- The site-specific initializations are not automatically made. To do
- that, 'import site' could be used as the first statement in the
- command string.
- ....d
- The interpreter() function will return after the new thread structure
- has been created. The command will execute sooner or later. The
- thread will terminate, and the interpreter structure be deallocated,
- when the command has been executed, and dependent threads have
- terminated.
- ..attr:: NodeGraph
- ...mapping
- ....d: Construct a new NodeGraph object.
- ....optionals
- .....arg: edges:iterable+
- ......d: The edges that will be used to
- initialize the new nodegraph. It should yield a
- sequence of pairs being edges of the form (source, target).
- ......default: ()
- .....arg: is_mapping:boolean+
- ......d: If True, will cause the nodegraph
- to be treated like a 'mapping'. It will then, for the
- purpose of indexing, be expected to contain a single
- target for each source node.
- ......default: False
- ....returns: NodeGraph
- ..attr:: RootState
- ...d: The single instance of RootStateType.
- ..attr:: set_async_exc
- ...mapping
- ....d: Set an exception to be raised asynchronously in a thread.
- ....arg: thread_id:thread_id+
- ....arg: exception:Exception+
- ..attr:: xmemstats
- ...mapping
- ....d: Print system-dependent memory statistics.
- ....d: What is printed depends on the system configuration.
- .and: HeapView
- ..self: HV
- ..d
- ...p
- A HeapView object provides methods to get memory related information
- about the system heap and about individual objects.
- ...p
- It implements much of the low-level functionality for the Heapy
- system. It is intended to provide what can not be done at all or would
- be much slower if programmed directly in Python. It is not intended to
- be used directly by a user, but to be wrapped in higher level objects.
- ...p
- Some terms that are referred to in the method descriptions:
- ...label: Visible objects
- ...h3: Visible objects.
- ...p
- The HeapView object attempts to restrict its view of the heap to only
- the 'visible objects'. This is to make it possible to analyse the heap
- via a Python library that inevitably itself is continually allocating
- and deallocating objects. These should be hidden from the heap view
- presented. This is primarily done via a special tag attribute, see
- ....ref: .mykind._hiding_tag_
- ....t: and
- ....ref: .mykind.register__hiding_tag__type
- ....t:. Frames can be hidden
- with another mechanism, see
- ....ref: .mykind.limitframe
- ....t:. For hiding all objects of a special type,
- ....ref: .mykind.register_hidden_exact_type
- ....t: may be used. It is also
- possible to use a separate interpreter and hide its root objects, see
- ....ref: .mykind.is_hiding_calling_interpreter
- ....t:.
- ...h3: Classifiers.
- ...text
- The methods named cli_* are factory methods that create objects of
- type ObjectClassifier. The principal difference between classifiers is
- how a single object is classified. The single-object classification
- function is available in classifier objects; it is the classify
- method. There are also methods that operate on collections of objects,
- namely partition and select. These eliminate the per-object
- Python-level function call overhead that would occur if the classify
- method were to be called from Python for each object in a collection.
- See also the ObjectClassifier type.
- ...label: Individual size
- ...h3: Individual size.
- ...p
- The individual size of an object is its individually allocated memory size.
- ...p
- It includes:
- ...ul
- ....li: The basic object size, as can be found out in a standard way.
- ....li: The extra memory for variable size objects.
- ....li: For GC collected objects, the size of the GC information.
- ....li: An alignment to the next highest multiple of a pointer size.
- ....li: The size of any other memory allocated that belongs to the object.
- ...p
- Some types of objects have extra memory allocated that can not be
- accounted for in the standard way. This memory should nevertheless be
- included in the individual size. To determine the size of these
- objects, special functions are needed. These are defined for standard
- builtin types, such as lists and dicts. Other types should be defined
- via the heapdefs argument to the HeapView constructor.
- ...p: The individual size does not include:
- ...ul
- ....li: Subobjects that are accounted for separately.
- ....li: Overhead for the memory allocation system. This varies depending
- on the kind of memory allocator, the requested size, etc.
- ..constructor: module heapyc.HeapView
- ..attr:: _hiding_tag_
- ...self: HV
- ...d: The hiding tag defining what objects are hidden from the view defined
- by HV. Objects that contain a _hiding_tag_ object which is identical
- to HV._hiding_tag_, will be hidden from view, in the following cases:
- ....ul
- .....li: The object is of a type that has been registered for hiding via
- ......ref: .mykind.register__hiding_tag__type
- ......t:, or is of a subtype of such a type.
- .....li: The object is of instance type. Such an object will be checked
- for a _hiding_tag_ item in its __dict__.
- ..attr:: cli_class
- ...mapping
- ....returns: ObjectClassifier
- .....d: a classifier that classifies by "class".
- .....d:
- If an object is of instance type, its classification will be its
- class, as given in its C-level member 'in_class', otherwise it will
- be its type, as given in 'ob_type'.
- ....dwh: Note
- This is like classifying by the Python-level '__class__' attribute
- in newer versions of Python, except that even though the '__class__'
- attribute may be redefined via the class definition, the classification
- as made by cli_class isn't affected.
- ..attr:: cli_clodo
- ...mapping
- ....returns: ObjectClassifier
- .....d: a classifier, that classifies by "Class Or Dict Owner".
- .....d
- The classification of an object is its class (as by cli_class),
- unless the object is a dict object that is 'owned' by some owner.
- If the object is such an owned dict, the classification will be a
- 1-tuple containing the class (as by cli_class) of its owner.
- ....arg: owners:NodeGraph+
- .....d:
- Used to map each dict object to its owner, or to None if it has no
- owner. The graph will be automatically updated, from heap information
- defined by HV, whenever an attempt is made to classify a dict that
- maps to nothing.
- ....arg: memo:dict+
- .....d: Used to memoize the 1-tuples generated when classifying owned dicts.
- ..attr:: cli_indisize
- ...mapping
- ....returns: ObjectClassifier
- .....d: a classifier that classifies by "individual size".
- .....d: The classification of each object is an int, containing the
- object's individual memory size.
- ....arg: memo:dict+
- .....d: Used to memoize the classification objects.
- ..attr:: cli_none
- ...mapping
- ....returns: ObjectClassifier
- .....d: a classifier that classifies all objects the same.
- .....d: The classification of each object is None.
- ..attr:: cli_rcs
- ...mapping
- ....returns: ObjectClassifier
- .....d: a classifier that classifies by Referrer Classification Set.
- .....d: The classification of an object is the classifications of its
- referrers, collected in an immutable NodeSet object.
- ....arg: referrers: NodeGraph+
- .....d: Used to map each object to its referrers.
- ....arg: classifier:ObjectClassifier+
- .....d: Used to classify each referrer.
- ....arg: memo:dict+
- .....d: Used to memoize the classification sets.
- ..attr:: cli_type
- ...mapping
- ....returns: ObjectClassifier
- .....d: a classifier that classifies by type.
- .....d:
- The classification of each object is the type, as given by its
- C-level member 'ob_type'. (This is the same as the type returned
- by the Python-level builtin 'type'.)
- ..attr:: delete_extra_type
- ...description: For Internal Use
- ..attr:: heap
- ...mapping
- ....args
- ....returns: NodeSet
- .....d: a set containing all 'visible objects' in the heap view
- defined by HV.
- ....dwh: See also
- .....ref: .mykind.Visible objects
- ......t: Visible objects
- ..attr:: indisize_sum
- ...mapping
- ....arg: S:iterable+
- ....returns: int
- .....d: the sum of the 'individual size' of the objects in S.
- ....dwh: See also
- .....ref: .mykind.Individual size
- ......t: Individual size
- ..attr:: numedges
- ...mapping
- ....arg: src:Any+
- ....arg: tgt:Any+
- ....returns: int
- .....d: the number of edges from src to tgt.
- ..attr:: reachable
- ...mapping
- ....arg: X:NodeSet+
- ....arg: Y:NodeSet+
- ....returns: NodeSet
- .....d: the set of objects reached via a path in the visible heap as
- defined by HV, from some object in X, avoiding any object in Y.
- ..attr:: reachable_x
- ...mapping
- ....arg: X:NodeSet+
- ....arg: Y:NodeSet+
- ....returns: NodeSet
- .....d: the set of objects reached via a path in the visible heap as
- defined by HV, from some object in X, avoiding any object in Y except
- at the end of the path.
- ..attr:: register__hiding_tag__type
- ...mapping
- ....args: type_with_hiding_tag+
- ....d:
- Register a type of objects that may be hidden from the heap view
- defined by HV. The type must have a slot named _hiding_tag_. An object
- that is an instance of the type, or of a subtype, is hidden when its
- _hiding_tag_ is HV._hiding_tag_.
- ..attr:: register_hidden_exact_type
- ...mapping
- ....args: type+
- ....d:
- Register a type of objects that should be hidden from the heap view
- defined by HV. Objects of the exact type registered -- not including
- subtypes -- will be hidden.
- ..attr:: relate
- ...mapping
- ....arg: src:Any+
- ....arg: tgt:Any+
- ....returns: RelationStructure
- .....d: a description of the relation between src and tgt.
- .....d: This is used for descriptions of edges in paths.
- .....d: [The result is in a special format that I choose to not define here
- since it is for special low-level use and subject to change.]
- ..attr:: relimg
- ...mapping
- ....returns: NodeSet
- .....d: the 'relational image of HV wrt S'.
- .....d: That is, the set of nodes
- that are directly referred to from the nodes in S via the visible heap
- reachability relation as defined by HV.
- ....arg: S:iterable+
- ..attr:: shpathstep
- ...mapping
- ....d: This method implements one step of a shortest path algorithm.
- ....arg: G:NodeGraph+
- .....d: Updated by the method, with the edges from nodes in the
- source set to the new nodes visited.
- ....arg: U:NodeSet+
- .....d: The source set for this step.
- ....arg: S:NodeSet+
- .....d: The set of already visited nodes.
- ....optionals
- .....arg: AvoidEdges:NodeGraph+
- ......d: Edges to avoid.
- .....arg: find_one:boolean+
- ......d: If True, at most one edge will be found from each node
- in the source set. Normally, all edges will be found.
- ....returns: NodeSet
- .....d: the new nodes visited.
- .....d: This may be used for the U argument the next time the method is called.
- ....dwh: See also
- .....t: shpgraph_algorithm in Path.py.
- ..attr:: update_dictowners
- ...mapping
- ....d: Update owners with ownership edges.
- ....d: The dict owners graph will be updated with an edge from each dict
- object in the heap, to either its owner or to None.
- ....arg: owners:NodeGraph+
- ..attr:: update_referrers
- ...mapping
- ....d: Update referrer graph X for Y.
- ....d: The visible heap defined by HV will be traversed from the root of HV
- so that the edges of every path from the root to nodes in Y will be
- represented, inverted, in X.
- ....arg: X:NodeGraph+
- ....arg: Y:NodeSet+
- ..attr:: update_referrers_completely
- ...mapping
- ....arg: X:NodeGraph+
- ....d: Update referrer graph X 'completely'.
- ....d: [Experimental algorithm that updates X with the referrers to all
- objects in the heap (of visible nodes as defined in HV). It is not
- normally used.]
- ..attr:: is_hiding_calling_interpreter
- ...kind of: boolean
- ...d:
- If True, the data of the interpreter using the HV will be hidden from
- the heap view as seen from RootState.
- ...d
- This is used when multiple Python interpreters are used. One
- interpreter will be monitoring the operation of the other
- interpreter(s). It would set is_hiding_calling_interpreter to True in
- the HV it is using. Its own data will then be hidden from view, making
- memory leak detection more practical.
- ..attr:: is_using_traversing_owner_update
- ...kind of: boolean
- ...d:
- True if update_dictowners is using a recursive traversal algorithm to
- find the dicts in the heap. When False, the normal case, it will use the
- dicts found in the gc collection structure, by gc.get_objects(). This was
- found to be much faster in usual cases, but the old version is available
- by setting this flag. -- It may be removed in a later release!
- ..attr:: limitframe
- ...either: None, frame
- ...d: The traversal limiting frame.
- ...d: If limitframe is set to a frame object, the frames that are more
- recently entered than limitframe will be hidden when traversing the
- heap from the root RootState. It will start traversing from limitframe
- rather than from the most recent frame as it would otherwise do.
- .and: NodeGraph
- ..d
- A NodeGraph object contains pairs of nodes (edges) and can be indexed
- on the first node of such a pair (the source of an edge) to find all
- second nodes of such pairs (the targets of those edges).
- ...p
- NodeGraph objects are used internally in the Heapy system, for example
- to record dict ownership and shortest-path graphs.
- ...p
- They may be used generally for mapping and dict-like purposes, but
- differ from dicts in the following:
- ...ul
- ....li: The mapping is based on object identity - no equality or hashing is
- assumed, so any object can be used as a key. Only the address is used.
- To distinguish this usage from that of ordinary dicts and sets, such
- objects are called 'nodes'.
- ....li: There may be any number of targets associated with each source.
- ....li: Performance characteristics differ from dicts, in somewhat subtle ways.
- ..self: NG
- ..constructor: module heapyc.NodeGraph
- ..attr:: _hiding_tag_
- ...d: The hiding tag: if it is the the same object as
- ....ref: .myfile.HeapView._hiding_tag_
- ....t: of a HeapView object, the nodegraph will be hidden from the corresponding heap view .
- ..attr:: is_mapping
- ...kind of: boolean
- ...d: Read only. True if NG is a 'mapping'. Then, only one edge is
- allowed for each source; indexing returns the actual target object
- instead of a tuple of targets.
- ..attr:: is_sorted
- ...kind of: boolean
- ...d: Read only. True if NG is sorted.
- ...d: It will become unsorted after any update. It will need to be
- sorted to make it possible to find edges (implementation uses binary
- search). Any indexing operation will automatically sort it if it was
- not already sorted. The flag is currently used from Python to see if
- the nodegraph has been used at least once after update, so that it
- will not be cleared too early.
- ..fop: len
- ...returns: int
- ....d: the number of edges in NG.
- ..getitem
- ...returns: Any
- ....d: the target(s) of all edges with a particular source. The value
- depends on if NG was initialized to be a 'mapping' or not:
- .....dl
- ......dt: NG.is_mapping == False
- ......dd: Return a tuple of all targets of edges from the source.
- The tuple will be empty if there are no such edges.
- ......dt: NG.is_mapping == True
- ......dd: Return the target of the edge from the source.
- If there is no such edge, KeyError will be raised.
- If there is more than one edge, ValueError will be raised.
- ...arg: source: Any+
- ..setitem
- ...d: Assign new target(s) to the edge(s) from a given source. How the
- targets are specified, depends on whether NG is a 'mapping' or not:
- ....dl:
- .....dt: NG.is_mapping == False
- .....dd: The target_spec argument is a tuple of targets.
- There must already be the same number of edges in NG from the given
- source as the number of objects in target_spec.
- .....dt: NG.is_mapping == True
- .....dd: The target_spec argument is the target itself.
- There must already be exactly one edge in NG from the given source.
- ...d:
- If there were not already the same number of edges as the targets specified,
- ValueError will be raised. The reason for the restriction is to make sure
- assignment will be efficient when interleaved with indexing.
- ...dwh: See also
- ....ref: .mykind.add_edge
- ....t:,
- ....ref: .mykind.add_edges_n1
- ...arg: source: Any+
- ...arg: target_spec: TargetSpec+
- ..method:: add_edge
- ...d: Add to NG, an edge from source to target.
- ...arg: source: Any+
- ...arg: target: Any+
- ..method:: add_edges_n1
- ...d: Add to NG, for each src in srcs, an edge from src to tgt.
- ...arg: srcs: iterable+
- ...arg: tgt: Any+
- ..method:: as_flat_list
- ...returns: list
- ....d: the edges of NG in the form [src0, tgt0, src1, tgt1 ...].
- ..method:: clear
- ...d: Remove all items from NG.
- ..method:: copy
- ...returns: NodeGraph
- ....d: a copy of NG.
- ..method:: domain_covers
- ...returns: boolean
- ....d: True if each node in X is the source of some edge in NG, False otherwise.
- ...arg: X: iterable+
- ..method:: domain_restricted
- ...returns: NodeGraph
- ....d: a new NodeGraph, containing those edges in NG that have the source in X.
- ...arg: X: iterable+
- ..method:: get_domain
- ...returns: NodeSet
- ....d: the set of nodes that are the source of some edge in NG.
- ..method:: get_range
- ...returns: NodeSet
- ....d: the set of nodes that are the target of some edge in NG.
- ..method:: invert
- ...d: Invert the edges of NG.
- ..method:: inverted
- ...returns: NodeGraph
- ....d: a copy of NG with the edges inverted.
- ..fop: iter
- ...returns: iterator
- ....d: an iterator yielding a pair (source, target) for each edge in NG.
- ..method:: relimg
- ...returns: NodeSet
- ....d: the relational image of NG wrt X.
- ....d: That is, the set of nodes that are the target of some edge that have
- its source in X.
- ...arg: X:iterable+
- ..method:: update
- ...d: Update NG with the edges from X,
- specified as pairs of the form (source, target).
- ...arg: X:iterable+
- ..method:: updated
- ...returns: NodeGraph
- ....d: a copy of NG updated with the edges from X,
- specified as pairs of the form (source, target).
- ...arg: X:iterable+
|