ordereddict

Version: ordereddict 0.4.5
Author: Anthon van der Neut
Contact: anthon@mnt.org
Date: 2012-08-13

The ordereddict module in short

This is an implementation of an ordered dictionary with Key Insertion Order (KIO: updates of values do not affect the position of the key), Key Value Insertion Order (KVIO, an existing key's position is removed and put at the back).

Sorted dictionaries are also provided. Currently only with Key Sorted Order (KSO, no sorting function can be specified, but you can specify a transform to apply on the key before comparison (e.g. string.lower)).

Usage:

from _ordereddict import ordereddict
kio = ordereddict()
kvio = ordereddict(kvio=True)
# without relax unordered initalisation is not allowed
d = ordereddict({'a':1, 'b': 2}, relax=True)
sd = sorteddict({'a':1, 'b': 2}) # sorteddict is always relaxed

please note the underscore which is new since version 0.3

This module has been tested under:

OS compiler Python
Ubuntu 12.04 gcc 4.6.3 2.7.3
Ubuntu 8.04 gcc 4.2.4 2.6.4
Ubuntu 8.04 gcc 4.2.4 2.5.2
Windows XP Visual C++ 2008 Express 2.7.3
Windows XP Visual C++ 2008 Express 2.6.5
Windows MingGW 4.7.0 2.7.3
Solaris 10 GCC 4.4.x 2.7.3

MingGW and Solaris were tested and reported to work by Wladimir with version 0.4.5

Older versions of this module has been tested under and I expect those to still work:

OS compiler Python
Ubuntu 8.10 gcc 4.3.2 2.5.4
Ubuntu 8.10 gcc 4.3.2 2.4.6
Ubuntu 7.04 gcc 4.1.2 2.5.1
Ubuntu 7.04 gcc 4.1.2 2.4.4
Ubuntu 6.06 gcc 2.5.1
Windows XP Visual Studio 2003 2.5.1

Version 0.4.1 was tested and found working on SuSE Linux Enterprise Server (GCC 4.1.0 and Intel C/C++ 10.1) by Stuart Stock.

Home/Downloading

http://www.xs4all.nl/~anthon/Python/ordereddict is ordereddict's home on the web.

Downloading:
ordereddict-0.4.5.tar.bz2
ordereddict-0.4.5.zip

Installation

To install the package unzip/untar and run:

python setup.py install

For Windows users without a compiler, the .pyd file for Python 2.7.3 has been included in the .zip download. Just copy that into your site-packages directory.

Bugreporting

If you find any problems, please let me know, but also realise that I have a spamfilter that catches over 100 emails a day and yours might get in there unnoticed. So if there is no response within a few days please try again.

Functionality

ordereddict has all of the functionality of dict() except that there is no keyword based initialisation and that you cannot pass a normal dict to the initialisation of the basic ordereddict (however see the relax-ed keyword below). sorteddict cannot be initialised from keywords either, but can be initialised from normal dict (ie. they are always relaxed).

As you probably would expect .keys(), .values(), .items(), .iterkeys(), itervalues(), iteritems() and "for i in some_ordereddict" have elements ordered based on the key insertion order (or key value insertion order if kvio is specified, or sort order for sorteddict).

ordered/sorteddicts can be pickled.

Some methods have been slightly changed:

In addition to that ordereddict and sorteddict have some extra methods:

and ordereddict only also has:

The new OrderedDict in the standard collections module

With Python 3.1 and backported to 2.7 there is an OrderedDict class available in the collections modules. Raymond Hettinger indicated in 2009 at EuroPython that he preferred to start from a minimal OrderedDict instead of using the Larosa/Foord implementation. Unfortunately the available tests (for the functionality that the simple collections.OrderedDict supports) were not used either resulting in preventable bugs like repr initially not working on recursive OrderedDicts.

ordereddict (and the Larosa/Foord implementation) is essentially a superset of collections.OrderedDict, but there are a few differences:

All of the differences can be straightened out in small (70 lines of Python) OrderedDict wrapper around ordereddict. With this wrapper the OrderedDict tests in the standard test_collections.py all pass.

Testing

testordereddict.py in the test subdirectory has been used to test the module. You can use:

python testordereddict

to run the tests (py.test support has been dropped as newer versions of py.test were not compatible).

There is a somewhat patched copy of the python lib/Test dictionary testing routines included as well, it fails on the _update test however because the default is not to use a relaxed ordereddict. You can run it with:

cd test/unit
python test_dict.py

To Do

To Consider

Background information

ordereddict is directly derived from Python's own dictobject.c file. The extensions and the representation of ordereddicts() are based on Larosa/Foord's excellent pure Python OrderedDict() module (http://www.voidspace.org.uk/python/odict.html).

The implemenation adds a vector of pointers to elements to the basic dictionary structure and keeps this vector compact (and in order) so indexing is fast. The elements do not know about their position (so nothing needs to be updated there if that position changes, but then finding an item's index is expensive. Insertion/deletion is also relatively expensive in that on average half of the vector of pointers needs to be memmove-d one position. There is also a long value for bit info like kvio, relaxed.

The sorteddict structure has an additional 3 pointers of which only one (sd_key) is currently used (the others are sd_cmp and sd_value).

Speed

Based on some tests with best of 10 iterations of 10000 iterations of various functions under Ubuntu 7.10 (see test/timeordereddict.py and test/ta.py):

Results in seconds:

-------------------------------   dict         ordereddict  Larosa/Ford  collections
                                                            OrderedDict  OrderedDict
empty                             0.023        0.025        0.023        0.024
create_empty                      0.028        0.031        0.147        0.329
create_five_entry                 0.037        0.042        0.384        0.558
create_26_entry                   0.187        0.203        1.494        1.602
create_676_entry                  5.330        5.574       36.797       34.810
get_keys_from_26_entry            0.209        0.231        1.501        1.762
pop_5_items_26_entry              0.219        0.247        1.952        1.864
pop_26_items_676_entry            7.550        8.127       46.578       41.851
popitem_last_26_entry             0.203        0.225        1.624        1.734
popitem_last_676_entry            5.285        5.534       36.912       34.799
popitem_100_676_entry          --------        5.552       36.577     --------
walk_26_iteritems              --------        0.494        2.792        2.238
-------------------------------   dict         ordereddict  Larosa/Ford  collections
                                                            OrderedDict  OrderedDict

empty                             0.930     1.000           0.950        0.966
create_empty                      0.909     1.000           4.728       10.594
create_five_entry                 0.892     1.000           9.201       13.374
create_26_entry                   0.923     1.000           7.368        7.901
create_676_entry                  0.956     1.000           6.601        6.245
get_keys_from_26_entry            0.908     1.000           6.508        7.641
pop_5_items_26_entry              0.888     1.000           7.916        7.559
pop_26_items_676_entry            0.929     1.000           5.732        5.150
popitem_last_26_entry             0.901     1.000           7.222        7.712
popitem_last_676_entry            0.955     1.000           6.670        6.288
popitem_100_676_entry          --------     1.000           6.588     --------
walk_26_iteritems              --------     1.000           5.653        4.532

Why

Because I am orderly ;-O, and because I use dictionaries to store key/value information read from some text file quite often. Unfortunately comparing those files with diff when written from normal dictionaries often obfucates changes because of the reordering of lines when key/value pairs are added and then written.

I have special routine for YAML files that takes lines like:

- key1: val1
- key2: val3
- key3:
    - val3a
    - val3b

(i.e. a list of key-value pairs) directly to a single ordered dictionary and back. (I find it kind of strange to finally have a structured, human readeable, format that does not try to preserve the order of key-value pairs so that comparing files is difficult with 'standard' text tools).

History

0.4.5  2012-06-17
Fix for a bug while inserting last item again beyond last position (reported
by Volkan Çetin / volki tolki ( cetinv at gmail.com )
Fix for repeated deletion and insertion fail. Found by and solution provided
by Darren Dowker (including tests). Also found by Fabio Zadronzy (including
a less elegant fix).
applied reindent to .py and astyle to .c files

0.4.3  2009-05-11
Fix for a bug in slicing SortedDicts.
Found by, and fix provided by, Migel Anguel (linos.es)

0.4.2  2009-03-27
Bug found and by Alexandre Andrade and Fabio Zadrozny in
doing deepcopy

0.4.1  2007-11-06
Bug found and fixed by Fabio Zadrozny on resizing dictionaries

0.4   2007-10-30
added pickling, added relaxed initialisation/update (from unordered dicts)
added KVIO (Key Value Insertion Order ie. key moves to back on update)
implemented sorteddict, with KSO, Key Sorted Order. You can specify
a function for key transformation before comparison (such as string.lower)
sorteddict does not have all of the ordereddict methods as not all make
sense (eg. slice assignment, rename, setkeys)

0.3   2007-10-24
added setkeys/setvalues/setitems; slice retrieval, deletion, assigment
.rename(oldkey, newkey) rename a key keeping same value and position
.index() of non-existing key now returns ValueError instead of SystemError
Changed the module name to _ordereddict (from ordereddict), as Jason
Kirstland probably rightfully suggested that any private implementation
likely has the (file)name ordereddict.py. A modulename with leading
underscore seams more common for extension modules anyway.

0.2a  2007-10-16
Solved the potential GC problem on Windows

0.2   2007-10-16
First release, with some tests, and possible still a GC problem
with Windows.

0.1   2007-10-..
This version was never released. While testing it I was far in writing
an email to comp.lang.python about why timing with timeit did seem to
be memory hungry ....
and then I realiased ordereddict had a memory leak %-)

http://ruamel.eu/dwnld/1x1/ordereddict/dot.gif