| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583 |
- .macro:: header
- ..header
- ...link
- ....rel=stylesheet
- ....href=css/guppy.css
- .document: index
- ..output: html
- ..document_title: Guppy-PE: A Python Programming Environment
- ..use: header
- ..div
- ...dl
- ....dt
- .....h1:
- ......big: Guppy-PE
- ....dd
- .....h1: A Python Programming Environment
- ...hr
- ...table
- ....tr
- .....td
- ......a: Guppy
- .......href=#Guppy
- .....td: A fish swimming in Python
- ....tr
- .....td
- ......a: Heapy
- .......href=#Heapy
- .....td: Heap Analysis Toolset
- ....tr
- .....td
- ......a: GSL
- .......href=#GSL
- .....td: Guppy Specification Language
- ....tr
- .....td
- ......a: Documentation
- .......href=#Documentation
- ....tr
- .....td
- ......a: Download
- .......href=#Download
- ....tr
- .....td
- ......a: Credits
- .......href=#Credits
- ....tr
- .....td
- ......a: Contact
- .......href=#Contact
- ...hr
- ..p: This is the home page for
- ...big:
- ....strong: Guppy-PE
- ...t:, a programming environment providing object and heap memory
- sizing, profiling and analysis. It includes a prototypical specification
- language that can be used to formally specify aspects of
- ...a: Python
- ....href= http://www.python.org
- ...t: programs and generate tests and documentation from a common
- source.
- ..a
- ...name=Guppy
- ...h2: Guppy
- ..p: Guppy is an umbrella package combining Heapy and GSL with support
- utilities such as the Glue module that keeps things together.
- ..p: The name guppy was chosen because I found it in a backward-dictionary
- as a word ending with py and I thought it was cute enough and that it
- would not so likely conflict with some other package name. It was to
- be a general name since all kinds of packages should fit under this
- top level name.
- ..p: The name guppy-pe is because there was another project named guppy in
- Sourceforge when I was about to register guppy. The other guppy was
- not in Python, so I added
- ...strong: -pe
- ...t: which means Programming Environment. The Python package is just guppy.
- ..a
- ...name=Heapy
- ...h2: Heapy
- ..p: The aim of Heapy is to support debugging and optimization
- regarding memory related issues in Python programs.
- ..p: Such issues can make a program use too much memory, making it
- slow by itself as well as slowing down an entire server, or it may
- fail to run at all in a limited memory device such as a mobile phone.
- ..p: The primary motivation for Heapy is that there has been a lack of
- support for the programmer to get information about the memory usage
- in Python programs. Heapy is an attempt to improve this situation. A
- project with a similar intent is
- ...a: PySizer.
- ....href=http://pysizer.8325.org
- ..p: The problem situation has a number of aspects, which I think can be
- characterised, for example, as follows.
- ..ul
- ...li: What data about memory and objects to get from the system.
- ...li: How to get that data technically.
- ...li: How portable to be across platforms and versions.
- ...li: How to process data to compress it and find structure in it.
- ...li: How to present the resulting information to the user.
- ...li: How to make a coherent, useful & usable system of it all
- that doesn't interfer too much with the target system it is analysing.
- ..p: As Heapy has evolved, with considerations like this in mind,
- it currently provides the following features.
- ..div
- ...h3: Data gathering
- ...div
- ....ul
- .....li: Finds reachable and/or unreachable objects in the object
- heap, and collects them into special C-implemented 'nodesets'. Can get
- data about the objects such as their sizes and how they refer to each
- other.
- .....li: Uses a C library that can get data about non-standard
- types from extension modules, given a function table.
- .....li: Optionally uses multiple Python interpreters in the same
- process, so one can monitor the other transparently.
- ...h3: Data processing
- ...div
- ....ul:
- .....li: Algebraic set operations, for example the set difference can
- be used to extract the objects allocated after a reference point in
- time.
- .....li: Various classifications of object sets, and different
- classifiers can be combined.
- .....li: Shortest paths to a set of objects from other objects,
- which can be used to find out why the objects are retained in memory.
- .....li: Calculation of the 'dominated' set from a set of root
- objects which yields the set of objects that would be deallocated if the
- root objects were deallocated.
- ...h3: Presentation
- ...div
- ....ul
- .....li: Tables where each row represents a classification of data.
- .....li: Lists of shortest paths where the edges show the
- relationships found between the underlying C objects.
- .....li: Reference pattern, presenting a spanning tree of the
- graph with sets of objects treated as a unit.
- .....li: Limits the number of rows when presentation objects are
- shown, without depending on an external pager.
- .....li: An interactive graphical browser program can show a time
- sequence of classified heap data sets as a graph together with a table
- detailing the data at a specific time or the difference between two
- points in time.
- ...h3: Portability aspects
- ...div
- ....ul
- .....li: Can be used with an unmodified C Python, back to version 2.3
- AFAIK. Does not depend on any external unix-specific or other
- utilities.
- .....li: Requires Tk if the graphical browser is to be used.
- .....li: Can not be used with Jython or other non-C Python versions.
- ...h3: System aspects
- ...div
- ....ul
- .....li: A general 'glue' model provides a session context that
- imports modules and creates objects automatically when accessed. The
- glue model is not Heapy specific but is used throughout Guppy and
- could be used by other packages as well.
- .....li: The glue model makes it practical to have everything in Guppy
- being dynamically allocated in a session context, so there is no need
- for any global module-level variables. The modules themself are stored
- as usual in sys.modules but they are not modified.
- .....li: To be true there is one exception I come to think of but
- it is really exceptional.
- ..p: Heapy has been used during development of itself and of the other
- parts of Guppy. It has been used to tell how much memory the parts of
- compound objects use, to see what could be worthwhile to optimize. It
- was used to find a memory leak in the Heapy profile browser, and to
- find out the cause, which as far as I can tell was due to a bug in a
- library routine which I have reported.
- ..a
- ...name=GSL
- ...h2: GSL
- ..p: The Guppy Specification Language is an evolving specification
- language. I started experimenting with this language because I felt
- the need to have a way to specify documentation and tests from the
- same source. GSL can describe aspects of a system, especially its API,
- in a way that can be automatically converted to tests as well as to
- documents. The documents generated have a formal structure for
- describing the formal aspects of the specification, complemented with
- descriptive text from the same source documents. A language that is
- similar in intent is the
- ...a: Assertion Definition Language
- ....href=http://adl.opengroup.org/
- ...t:.
- ..p: Specifications written in GSL can be used for:
- ..ul
- ...li: Generating documentation
- ....p: Documents are generated from a combination of formally
- described aspects and descriptive text. The formal aspects include
- specifications of attributes and methods and their parameters and
- return values. The descriptive text is written together with the
- formal specifications. It can include HTML tags and attributes,
- specified in the GSL dotted tree format. Output modules can convert
- from the GSL form to formats such as HTML or Latex.
- ...li: Generating tests
- ....p: The tests generated can check the formal aspects of the
- specification against an implementation. It can test whether objects
- have the promised attributes, and then if the kind of attribute is
- specified it is tested again and so on. Methods are checked to see if
- they can be called with the forms of parameters specified. The return
- value can then be checked up to some limit of recursion.
- ...li: Resolving compatibility issues
- ....p: A specification can be compared to a previous specification, to
- tell according to formalized rules whether or not the new one is
- backwards compatible with the old one, and if not, in what way they
- differ. For example, according to such rules, an attribute or
- parameter type can be added to a specification, but it can not be
- removed. The idea is that all tests possibly generated from the old
- specification should still succeed with objects conforming to the new
- specification. -- A program could likely compare specifications for
- compatibility automatically, but this is not yet implemented.
- ..p: GSL has been used to generate the documentation for this Guppy
- distribution. Some part of the specification has been checked against
- the implementation using the generated tests, which did reveal some
- discrepancies that were subsequently corrected.
- ..p: The documents generated by GSL use a formal syntax to describe
- parameter modes.
- ...a: This
- ....href=docexample.html
- ...t: document contains examples of such parameter descriptions and
- explains what they mean.
- ..a
- ...name=Documentation
- ...h2: Documentation
- ..p: Some documentation is included with the source code distribution
- and can also be browsed here via the following links.
- ...table:
- ....class=list
- ....tr:
- .....class=even
- .....td
- ......a: Getting started with Heapy
- .......href=heapy_tutorial.html
- .....td: An example of how to get started with Heapy
- ....tr:
- .....class=odd
- .....td
- ......a: Document example
- .......href=docexample.html
- .....td: Explains the meaning of some aspects of the documents.
- ....tr:
- .....class=even
- .....td
- ......a: Guppy
- .......href=guppy.html
- .....td: Specification of
- ......code: guppy
- ......t:, the top level module.
- ....tr:
- .....class=odd
- .....td
- ......a: Profile Browser
- .......href=ProfileBrowser.html
- .....td: How to use the graphical heap profile browser.
- ....tr:
- .....class=even
- .....td
- ......a: Screenshot
- .......href=pbscreen.jpg
- .....td: Example showing the graphical heap profile browser in action.
- ....tr:
- .....class=odd
- .....td
- ......a: GSL
- .......href=gsl.html
- .....td: The Guppy Specification Language.
- ....tr:
- .....class=even
- .....td
- ......a: heapyc
- .......href=heapyc.html
- .....td: Specification of the heapyc extension module. Note that this
- is an internal interface and may be subject to change.
- ....tr:
- .....class=odd
- .....td
- ......a: sets
- .......href=sets.html
- .....td: Specification of the interface to the
- setsc extension module which contains bitsets and nodesets.
- ....tr:
- .....td
- ......id=last
- ......colspan=5
- ..p: The following documentation is not included with the source code.
- ...table
- ....class=list
- ....tr:
- .....class=odd
- .....td
- ......a: heapy-thesis.pdf
- .......href=http://guppy-pe.sourceforge.net/heapy-thesis.pdf
- .....td: The master's thesis, "Heapy: A Memory Profiler and Debugger
- for Python", which presents background, design, implementation,
- rationale and some use cases for Heapy (version 0.1).
- ....tr
- .....class=even
- .....td
- ......a: Metadata and Abstract
- .......href=http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-7247
- .......c:href=http://www.diva-portal.org/liu/abstract.xsql?dbid=7247
- .....td: Published at
- ......a: Linköping University Electronic Press.
- .......href= http://www.ep.liu.se/
- ....tr:
- .....class=odd
- .....td
- ......a: heapy-presentation.pdf
- .......href=http://guppy-pe.sourceforge.net/heapy-presentation.pdf
- .....td: Slides from the presentation.
- ....tr
- .....td
- ......id=last
- ......colspan=5
- ..h3: External resources
- ..p: Thanks to all that have written about Heapy on various blogs. I
- think the following links may be especially useful to get
- started. More tips are very welcome!
- ..ul
- ...li
- ....a:How to use guppy/heapy for tracking down memory usage
- .....href=http://www.pkgcore.org/trac/pkgcore/doc/dev-notes/heapy.rst
- ....t: (pkgcore - Trac)
- ...li
- ....a:Debugging Django memory leak with TrackRefs and Guppy
- .....href=http://blog.redinnovation.com/2008/03/07/debugging-django-memory-leak-with-trackrefs-and-guppy
- ....t: (Redi for Life)
- ..a
- ...name=Download
- ...h2: Download
- ..p
- The latest version is in the svn trunk directory.
- ..p
- In Version 0.1.9 I have introduced interactive help and compatibility
- for MS compilers (see ANNOUNCE and ChangeLog).
- ..p: As of 2009-06-23, I have tested the latest revision successfully on
- an AMD64 in 64 bits mode with Ubuntu 7.10 with Python 2.3, 2.4, 2.5
- and 2.6, and in 32 bits mode with Ubuntu 9.04 with Python 2.4, 2.5 and
- 2.6 .
- ..p: To check out the latest (HEAD) revision, you can do:
- ..pre
- svn co https://guppy-pe.svn.sourceforge.net/svnroot/guppy-pe/trunk/guppy guppy
- ..p: To check out the latest release as of this writing
- (2009-06-23, guppy-0.1.9), you can do:
- ..pre
- svn co -r79 https://guppy-pe.svn.sourceforge.net/svnroot/guppy-pe/trunk/guppy guppy
- ..h3: Source code releases in tarball format
- ..table
- ...class=list
- ...style=margin-bottom: 10px;
- ...tr
- ....class=even
- ....td
- .....a: guppy-0.1.9
- ......href=http://pypi.python.org/pypi/guppy/0.1.9
- ....td: Updated 2009-06-23. Added interactive help and improved
- MS compatibility.
- ...tr
- ....class=odd
- ....td
- .....a: guppy-0.1.8
- ......href=http://pypi.python.org/pypi/guppy/0.1.8
- ....td: Updated 2008-04-08. Works with 64 bits and Python 2.6 .
- ...tr
- ....class=even
- ....td
- .....a: guppy-0.1.6.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.6.tar.gz
- ....td: Updated 2006-10-16. Doesn't work with 64 bits. -- Quick bug fix version,
- correcting the name of the Root object in the hpy instance. This is a
- kind of bug the automatic test generation should have catched, since
- it was specified with another name in the documentation, but I yet
- need some time to get that to work... so I am just uploading this
- quick fix now.
- ...tr
- ....class=odd
- ....td
- .....a: guppy-0.1.5.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.5.tar.gz
- ....td: Updated 2006-10-12. Fixed bugs wrt remote
- monitoring and HTML rendering. New features include the shorthand sp
- for shpaths and representing the source of the shortest paths in terms
- of a Root object in the hpy instance. See changelog.
- ...tr
- ....class=even
- ....td
- .....a: guppy-0.1.4.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.4.tar.gz
- ....td: Updated 2006-10-11. Most changes are to make it work with
- Python 2.5; other changes include improved error reporting in Glue.py
- and some test improvements.
- ...tr
- ....class=odd
- ....td
- .....a: guppy-0.1.3.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.3.tar.gz
- ....td: Updated 2006-03-02.
- Updates to Monitor so multiple lines work. It also got a command to
- interrupt the remote process. Cleanups and bugfixes especially todo
- with Python2.4 (used to crash with array objects). A bunch of other
- fixes, see changelog.
- ...tr
- ....class=even
- ....td
- .....a: guppy-0.1.2.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.2.tar.gz
- ....td: Pointer comparison bugs and test portability problems were
- fixed. See the included changelog.
- ...tr
- ....class=odd
- ....td
- .....a: guppy-0.1.1.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.1.tar.gz
- ....td: The C source code for the extension modules
- was changed to be ANSI compatible and I also changed some help text that
- had become outdated.
- ...tr
- ....class=even
- ....td
- .....a: guppy-0.1.tar.gz
- ......href=http://guppy-pe.sourceforge.net/guppy-0.1.tar.gz
- ....td: Original version. Extension modules could not be compiled using
- strict ANSI C compilers.
- ...tr
- ....td
- .....id=last
- .....colspan=5
- ..c:
- ...a: http://sourceforge.net/project/showfiles.php?group_id=105577
- ....href=http://sourceforge.net/project/showfiles.php?group_id=105577
- ..a
- ...name=Credits
- ...h2: Credits
- ..ul
- ...li: Claudio Grondi has tested in Microsoft XP and reported the
- problems that occurred. The ANSI-compatibility problems I think have
- been fixed but there also seemed to be problems with the .NET C compiler
- which I haven't had a chance to sort out yet.
- ...li: Patrik Andersson has helped me installing Ubuntu Linux on my new
- computer and to make it dual-boot with Microsoft XP. (But we don't have
- any C compiler for Windows XP (yet).)
- ...li: Chad Austin has supplied a patch for compiling on Windows w/
- Visual Studio 2003 and Python 2.5. I think this may fix similar
- problems with other MS compilers, Cygwin & Mingw as well.
- ...li: Yaroslav Halchenko is packaging and uploading Guppy into the
- Debian distribution.
- ..a
- ...name=Contact
- ...h2: Contact
- ..dl
- ...dt: The author, Sverker Nilsson, may be contacted at:
- ...dd
- ....address
- .....a: svenil@users.sourceforge.net
- ......href=mailto:svenil@users.sourceforge.net
- ...dt: I have registered a mailing list for discussions, questions,
- announcements etc. The list information, subscription form and archives
- are available at:
- ...dd
- ....a: http://lists.sourceforge.net/mailman/listinfo/guppy-pe-list
- .....href=http://lists.sourceforge.net/mailman/listinfo/guppy-pe-list
- ...dt: Please report bugs preferably via the SourceForge bug tracking
- system:
- ...dd
- ....a: http://sourceforge.net/tracker/?group_id=105577&atid=641821
- .....href=http://sourceforge.net/tracker/?group_id=105577&atid=641821
- ...dt: The Sourceforge project summary page is:
- ...dd
- ....a: http://sourceforge.net/projects/guppy-pe
- .....href= http://sourceforge.net/projects/guppy-pe
|