A conversion document has been created, CONVERTING_TO_5.0.0 in the main
directory of this package. Also available online, at
https://github.com/kata198/indexedredis/blob/5.0branch/CONVERTING_TO_5.0.0
Use this document to prepare your code for the changes in IndexedRedis 5.0.0.
Also please review this Changelog to see new features, bug fixes, etc which
will not be listed in the conversion document.
- Improve IRPickleField to work with ALL types (including now, strings and
bytes)
THIS WAS PREVIOUSLY BACKWARDS-INCOMPATIBLE, BUT HAS BEEN MADE COMPATIBLE
WITHOUT CHANGES TO YOUR CODE. **DO NOT RUN THE COMPAT FUNCTION SHIPPED WITH
4.1.3 - 4.1.4**
- Fixup the way connecting to Redis works. Now, instead of having to define
REDIS_CONNECTION_PARAMS on each model, you can call
"setDefaultRedisConnectionParams" to set a default connection configuration.
This configuration will be used on all models unless REDIS_CONNECTION_PARAMS has been
defined and is not empty.
- Rename "connect" method to "connectAlt" on IndexedRedisModel (to better
convey what it does). Also fix it up to better support multiple alternate
connection models
- Remove formerly deprecated IndexedRedisModel.BASE64_FIELDS - You've been
warned for a while now to use fields.IRField.IRBase64Field
**BACKWARDS INCOMPATIBLE**
To convert, remove the BASE64_FIELDS array from your model. For any items
that are in it, change it in the FIELDS array to be:
IRBase64Field
as found in
from IndexedRedis.fields import IRBase64Field
If you already have a type, and were using it to base64 encode that type
(like maybe you are compressing then base64 encoding), use a IRFieldChain
with base64 as the last element to retain same functionality.
Like:
IRFieldChain('fieldName', [IRUnicodeField(encoding='utf-16'),
IRBase64Field()])
Models with BASE64_FIELDS defined will generate an error upon validation.
- Remove IndexedRedis.BINARY_FIELDS - Use the new IRBytesField to act the same
way as a BINARY_FIELD did on python3 or python2, use IRField(valueType=bytes)
[python 3 only], or use IRRawField for no encoding/decoding at all. You can
combine these in a chain just like IRBase64Field.
**BACKWARDS INCOMPATIBLE**
Models with BINARY_FIELDS defined will generate an error upon validation.
- Make copy.copy and copy.deepcopy on an IndexedRedisModel call
IndexedRedisModel.copy(False), in order to copy the data, but NOT to have the
copy be linked to the database. A save on the copy will be an insert, the
parent will remain an update.
**BACKWARDS INCOMPATIBLE**
- Removed the old, deprecated, compat global "defaultEncoding", retaining just
defaultIREncoding. Also, removed deprecated getEncoding and setEncoding, use getDefaultIREncoding and setDefaultIREncoding
**BACKWARDS INCOMPATIBLE**
You shouldn't notice this. You should be using setDefaultIREncoding or
getDefaultIREncoding for default encodings
- Introduce a new IRField method called "fromInput", which is like
"convert", but is used when the input is guarenteed to NOT come from storage.
This will simplify a lot of things, remove some conditional and weird garbage,
etc. Constructing objects is now a lot cheapear.
- Store "was validated" result for model by the class itself. This allows models with the same keyname (like for conversion, tests, etc) to be validated for each unique class.
- Add IRClassicField as a replacement for the plain-string name in FIELDS array. validateModel will automatically convert these to IRClassicField and warn you that these may go away. I have not yet decided if 5.0.0 will generate error or just warning for now.
- Default IRField's which have not been set to irNull (instead of empty string).
Classic string fields (just a string name, not an IRField object) will retain
the empty string.
This allows there to be a difference from not-set and empty-string value items
(which can be meaningful in several contexts).
You can override this by setting defaultValue='something else' (like empty
string for previous default on some types, irNull was previous default on
others)
**BACKWARDS INCOMPATIBLE**
Consider if you are testing like
if myObj.someField == ''
that this will stop working on newly created objects if you aren't setting
the value, and expecting that to test such a condition. You can use:
if not myObj.someField
or
if (myObj.someField in ('', b'', irNull)
to support both.
There is no "compat convert" method for this, because it's unknown if an
item was set to default string, or empty value. It's on you to make sure
your code either handles both, or you convert your datasets accordingly.
You can override the defaultValue by passing defaultValue=X to any IRField
or subclass, so to retain old behaviour set defaultValue='' on your fields.
- Allow specifying a defaultValue on IRFields. This specifies what the default
value will be (was not set) for each field. Every IRField type (Except
IRClassicField) supports specifying a default value.
- repr on FIELDS now shows the field type and any properties specific to that object. asDict now takes a new param, (strKeys, default False), which will str the keys instead of using the IRField (so if you are pretty-printing the asDict repr, you'll want to use this to prevent the field types from creeping into the printout)
- Use "fromInput" every time a field is set on an IndexedRedisModel
( via myModel.xfield = someVal ), same as if it were done via the constructor.
This ensures that everything always has an expected type, and can fix some
weird issues when folks set unconverted values.
** Backwards Incompatible, if you were doing it wrong before... ;) **
- Rename "convert" to "fromStorage" on IRField. Instead of implementing
convert/toStorage/fromInput, the subclasses should implement the
underscore-prefixed version,
* _fromStorage
* _fromInput
* _toStorage
The non-underscore prefixed versions have been reworked to handle irNull and
such.
- Change the way irNull is handled, pull pretty well all of the work and
flip-flopping out of IndexedRedisModel and make it part of IRField
- Ensure that "toIndex" is ALWAYS called when setting indexes. This fixes some
issues with hashIndex=True (like that irNull would not previously hash).
- Fix some cases where irNull would equal empty string on indexes
- Work around a strange "bug" in python that could cause some builtin types to
be overriden within the "fields.__init__" module
- Make IRBytesField able to index (by hashing the bytes)
- Force IRUnicodeField to always have a hashed index, when used as an index
**BACKWARDS INCOMPATIBLE**
If you were using an IRUnicodeField as an index before, you will need to
reindex your data. This is because without hasing, there are issues with
many codecs at obtaining the index key. But it's okay, it's simple to
reindex and move forward.
For any model that has an index on an IRUnicodeField, you must call:
MyModel.objects.compat_convertHashedIndexes()
This existing method will take care of ensuring that ALL fields are using
the proper index format, be it hashed or unhashed, and even takes care of
if you accidently let your application run and have a mix of both hashed
and unhashed values within the index.
Please ensure that your application is offline before executing this
function, for each model that may need index conversion.
- Update runTests.py to the latest provided with the new version of
GoodTests.py
- Some minor to moderate cleanups/fixups/optimizations
- Add pprint method to both IndexedRedisModel and IRQueryableList. This will
pretty-print a dict representation of the model/collection of models, to a
given stream (default sys.stdout)
- Add __eq__ and __ne__ to IndexedRedisModel, to test if the models are equal
(including ID).
- Add "hasSameValues" method to IndexedRedisModel, to test if two objects have
the same field values, even if the _id primary key is different (i.e.
different objects in the database)
- Fix where on insert the forStorage data would be put in _origData instead of
the converted data, which would cause that field to be listed as needing to be
updated if that same object was saved again
- Fix a strange bug where some builtin types could be overriden in the
IndexedRedis.fields.__init__.py file
- Allow specifying an explicit encoding on IRBytesField, to use an alternative
than the default encoding
- Make IRField's copyable
- Add "copyModel" method to IndexedRedisModel as a class method which will
return a copy of the model class. This way when converting field types etc you
can easily have the same model, copy it, and change one.
- Add "name" property to IRField to make it more obvious how to access ( you
can still use str(fieldObj) )
- Improve compat_convertHashedIndexes method to work with all field types, and
to only touch fields which can both hash and not hash
- Update IRFixedPointField to ensure the value is always rounded to
decimalPlaces property, not just after fetch from storage.
- Ensure IRRawField always actually does no encoding/decoding, does not support
irNull (well, it just treats it same as empty string)
- Ensure that IRBytesField always encodes to bytes as soon as you set the
value on the object
- Ensure that every field type converts the value to its consumable form (so
same before-save and after-fetch) when setting the attribute through the
dot-operator (myobj.value = X ) or constructor.
- Fix bug where toIndex could be calling toStorage twice on a value (which
affected things like base64 fields, which could be base64-of-base64 on index).
Didn't matter before because base64 fields couldn't index before, but maybe
some case hit this bug.
- Allow IRFieldChain to index if the right-most field in the chain supports
indexing
- Added many more tests, like for each field type, lots more usage scenarios,
etc.
- Make IRCompressedField indexable.
The only non-indexable fields now are
IRPickleField (Different py2 vs py3 repr), IRRawField (because, no
encoding/decoding), and IRField(valueType= (float or dict or json) ) (float because
floats are machine-dependent, use IRFixedPointField to define a precision and
support indexes. And dicts are not ordered. And json has too many nesting
issues. )
- Support "lzma" ( aka "xz" ) compression in IRCompressedField. This is
provided by the core in python3. For python2, IndexedRedis also supports
backports.lzma and lzmaffi as alternative implementations. You can also
explicitly set IndexedRedis.fields.compressed._lzmaMod if you have some other
external implementation for python2
- Some cleanups, refactoring, optimizations, simplifications
- Rename the fields.bytes fields.unicode and fields.pickle module names to not
conflict with builtins. You should be inheriting all your fields like:
from IndexedRedis.fields import IRPickleField
anyway, so should not notice this change.
- More updates on the way Redis connection parameters work.
Now -- instead of using the "fixed" defaults of host="127.0.0.1",
port=6379, db=0 for connections, IndexedRedis will use what you set
through setDefaultRedisConnectionParams. The old defaults remain the
initial version of the defaultRedisConnectionParams, so existing code
will not break.
This means now that models that define their own REDIS_CONNECTION_PARAMS
are filled in by the defaultRedisConnectionParams for any missing slots,
rather than the aforementioned hard defaults.
So for example, for the following:
setDefaultRedisConnectionParams( { 'host' : '192.168.1.1' , 'port' :
15000, 'db' : 0 } )
class MySpecialModel(IndexedRedisModel):
...
REDIS_CONNECTION_PARAMS = { 'db' : 1 }
The default connection for everything will use the first dict
(host='192.168.1.1', port=15000, db=0 ).
MySpecialModel will override the db portion, but inherit host and port,
so MySpecialModel will use (host='192.168.1.1', port=15000, db=1)
This only affects connection pools that IndexedRedis manages. If you
define your own (by setting the 'connection_pool' key on your connection
params), as before you will not get any of the inheritance,
connect/disconnect, size management, etc that IndexedRedis would
otherwise provide.
Some other changes because of this:
- getDefaultRedisConnectionParams will return a COPY of the default
params, to prevent updates (such that IndexedRedis can track and apply
them properly). setDefaultRedisConnectionParams already made copies of
the values and set them on the global instance.
- Introduce as "clearRedisPools" method which will disconnect and clear
the connection_pool ConnectionPools off of all non-managed connections.
- Calling setDefaultRedisConnectionParams will now call clearRedisPools,
disconnecting any active managed connections and clearing the cached
connection_pool associated with each model (except managed pools). This
allows inheritance to happen again with the new properties, for example
on the above example code, if 'host' was changed in the global,
MySpecialModel on next connection will inherit that value (and cache the
inherited settings and connection pool).