nti.externalization¶

A flexible, schema-driven system for converting Python objects to and from external formats such as JSON and YAML. This works well in a zope.component environment such as Pyramid.

Documentation is hosted at https://ntiexternalization.readthedocs.io/

Introduction and Basics¶

Basic Usage¶

This package was originally developed for use in a Pyramid web application server in 2011. The desire was to be able to communicate Python application objects back and forth with browsers and native iOS applications using the most efficient serialization mechanisms possible.

This was before Pyramid’s built-in JSON Renderer supported adapting arbitrary objects to JSON (or even used the __json__ method), and before iOS had a native JSON reader.

So there were two mandates: support adapters for externalizing arbitrary objects and support multiple objects representations (specifically plist for iOS, which didn’t support None, and JSON for browsers.)

This document provides an overview of nti.externalization and shows some simple examples of its usage.

Reading through the Glossary before beginning is highly recommended.

Motivation and Use-cases¶

This package provides a supporting framework for transforming to and from Python objects and an arbitrary binary or textual representation. That representation is self-describing and intended to be stable. Uses for that representation include:

Communicating with browsers or other clients over HTTP (e.g., AJAX) or sockets;
Storing persistently on disk for later reconstituting the Python objects;
Using as a human-readable configuration format (with the proper choice of representation)

We expect that there will be lots of such objects in an application, and we want to make it as easy as possible to communicate them. Ideally, we want to be able to completely automate that, handing the entire task off to this package.

It is also important that when we read external input, we validate that it meets any internal constraints before further processing. For example, numbers should be within their allowed range, or references to other objects should actually result in an object of the expected type.

Finally, we don’t want to have to intermingle any code for reading and writing objects with our actual business (application) logic. The two concerns should be kept as separated as possible from our model objects. Ideally, we should be able to use third-party objects that we have no control over seamlessly in external and internal data.

Getting Started¶

In its simplest form, there are two functions you’ll use to externalize and internalize objects:

>>> from nti.externalization import to_external_object
>>> from nti.externalization import update_from_external_object

We can define an object that we want to externalize:

Caution

The examples in this section are not representative of best practices or preferred patterns. Please keep reading.

class InternalObject(object):

    def __init__(self, id=''):
        self._field1 = 'a'
        self._field2 = 42
        self._id = id

    def toExternalObject(self, request=None, **kwargs):
        return {'A Letter': self._field1, 'The Number': self._field2}

    def __repr__(self):
        return '<%s %r letter=%r number=%d>' % (
            self.__class__.__name__, self._id, self._field1, self._field2)

And we can externalize it with to_external_object:

>>> from pprint import pprint
>>> pprint(to_external_object(InternalObject()))
{'A Letter': 'a', 'The Number': 42}

If we want to update it, we need to write the corresponding method:

class UpdateInternalObject(InternalObject):

    def updateFromExternalObject(self, external_object, context=None):
         self._field1 = external_object['A Letter']
         self._field2 = external_object['The Number']

Updating it uses update_from_external_object:

>>> internal = UpdateInternalObject('internal')
>>> internal
<UpdateInternalObject 'internal' letter='a' number=42>
>>> update_from_external_object(internal, {'A Letter': 'b', 'The Number': 3})
<UpdateInternalObject 'internal' letter='b' number=3>

That’s Not Good Enough¶

Notice that we had to define procedurally the input and output steps in our classes. For some (small) applications, that may be good enough, but it doesn’t come anywhere close to meeting our motivations:

By mingling the externalization code into our business objects, it makes them larger and muddies their true purpose.
There’s nothing doing any validation. Any such checking is left up to the object itself.
It’s manual code to write and test for each of the many objects we can communicate. There’s nothing automatic about it.

Let’s see how this package helps us address each of those concerns in turn.

Adapters and Configuration¶

This package makes heavy use of the Zope Component Architecture to abstract away details and separate concerns. Most commonly this is configured using ZCML, and this package ships with a configure.zcml that you should load:

>>> from zope.configuration import xmlconfig
>>> import nti.externalization
>>> xmlconfig.file('configure.zcml', nti.externalization)
<zope.configuration.config.ConfigurationMachine ...>

The toExternalObject method is defined by the nti.externalization.interfaces.IInternalObjectExternalizer interface, and the updateFromExternalObject method is defined by nti.externalization.interfaces.IInternalObjectUpdater interface. Because it is common that one object both reads and writes the external representation, the two interfaces are joined together in nti.externalization.interfaces.IInternalObjectIO. Let’s create a new internal object:

class InternalObject(object):
    def __init__(self, id=''):
        self._field1 = 'a'
        self._field2 = 42
        self._id = id

    def __repr__(self):
        return '<%s %r letter=%r number=%d>' % (
            self.__class__.__name__, self._id, self._field1, self._field2)

Now we will write an IInternalObjectIO adapter for it:

from zope.interface import implementer
from zope.component import adapter

from nti.externalization.interfaces import IInternalObjectIO
from nti.externalization.datastructures import StandardInternalObjectExternalizer

@implementer(IInternalObjectIO)
@adapter(InternalObject)
class InternalObjectIO(StandardInternalObjectExternalizer):
    def __init__(self, context):
        super().__init__(context)
        # Setting this is optional, if we don't like the default
        self.__external_class_name__ = 'ExternalObject'

    def toExternalObject(self, **kwargs):
       result = super(InternalObjectIO, self).toExternalObject(**kwargs)
       result.update({
           'Letter': self.context._field1,
           'Number': self.context._field2
       })
       return result

    def updateFromExternalObject(self, external_object, context=None):
         self.context._field1 = external_object['Letter']
         self.context._field2 = external_object['Number']

Tip

It is a best practice for custom externalizers to either extend an existing datastructure, typically StandardInternalObjectExternalizer for simple cases (as in the example above), or to begin with result = to_standard_external_dictionary(self.context) and update that mapping in place. (The StandardInternalObjectExternalizer calls to_standard_external_dictionary under the covers.)

Caution

The signature for toExternalObject is poorly defined right now. The suitable keyword arguments should be enumerated and documented, but they are not. See https://github.com/NextThought/nti.externalization/issues/54

We can register the adapter (normally this would be done in ZCML) and use it:

<configure xmlns="http://namespaces.zope.org/zope">
    <include package="nti.externalization" />
    <adapter factory=".InternalObjectIO" />
</configure>

Because we don’t have a Python package to put this ZCML in, we’ll register it manually.

>>> from zope import component
>>> component.provideAdapter(InternalObjectIO, provides=IInternalObjectIO)
>>> internal = InternalObject('original')
>>> internal
<InternalObject 'original' letter='a' number=42>
>>> pprint(to_external_object(internal))
{'Class': 'ExternalObject', 'Letter': 'a', 'Number': 42}
>>> update_from_external_object(internal, {'Letter': 'b', 'Number': 3})
<InternalObject 'original' letter='b' number=3>

Notice that the external form included the Class key; this is one of the StandardExternalFields automatically recognized by the built-in externalizers, whose value is taken from the corresponding key named in StandardInternalFields. There are others:

>>> internal.creator = u'sjohnson'
>>> internal.createdTime = 123456
>>> pprint(to_external_object(internal))
{'Class': 'ExternalObject',
 'CreatedTime': 123456,
 'Creator': 'sjohnson',
 'Letter': 'b',
 'Number': 3}

Note

Notice how the names of the internal fields, creator and createdTime because Creator and CreatedTime in the external object. The convention used by this library is that fields that cannot be modified directly by the client are always capitalized. Your custom externalizers and interface definitions should follow this convention.

By using adapters like this, we can separate out externalization from our core logic. Of course, that’s still a lot of manual code to write.

Using Schemas for Validation and Automatic Externalization¶

Most application objects will implement one or more interfaces. When those interfaces contain attributes from zope.schema.field or nti.schema.field, they are also called schemas. This package can automate the entire externalization process, including validation, based on the schemas an object implements.

Let’s start by writing a simple schema.

from zope.interface import Interface
from zope.interface import taggedValue

from nti.schema.field import ValidTextLine

class IAddress(Interface):

    full_name = ValidTextLine(title=u"First name", required=True)

    street_address_1 = ValidTextLine(title=u"Street line 1",
                                     max_length=75, required=True)

    street_address_2 = ValidTextLine(title=u"Street line 2",
                                     required=False, max_length=75)

    city = ValidTextLine(title=u"City name", required=True)

    state = ValidTextLine(title=u"State name",
                          required=False, max_length=10)

    postal_code = ValidTextLine(title=u"Postal code",
                                required=False, max_length=30)

    country = ValidTextLine(title=u"Nation name", required=True)

And now an implementation of that interface.

from nti.schema.fieldproperty import createDirectFieldProperties
from nti.schema.schema import SchemaConfigured

@implementer(IAddress)
class Address(SchemaConfigured):
     createDirectFieldProperties(IAddress)

Externalizing (and updating!) based on the schema is done with InterfaceObjectIO. We’ll create a subclass to configure it.

from nti.externalization.datastructures import InterfaceObjectIO

@adapter(IAddress)
class AddressIO(InterfaceObjectIO):
    _ext_iface_upper_bound = IAddress

Now we can register and use it as before:

>>> component.provideAdapter(AddressIO)
>>> address = Address(full_name=u'Steve Jobs',
...    street_address_1=u'One Infinite Loop',
...    city=u'Cupertino',
...    state=u'CA',
...    postal_code=u'95014',
...    country=u'USA')
>>> external = to_external_object(address)
>>> pprint(external)
{'Class': 'Address',
 'city': 'Cupertino',
 'country': 'USA',
 'full_name': 'Steve Jobs',
 'postal_code': '95014',
 'state': 'CA',
 'street_address_1': 'One Infinite Loop',
 'street_address_2': None}

Oops, One Infinte Loop was Apple’s old address. They’ve since moved into their new headquarters:

>>> external['street_address_1'] = u'One Apple Park Way'
>>> _ = update_from_external_object(address, external)
>>> address.street_address_1
'One Apple Park Way'

Notice that our schema declared a number of constraints. For instance, the full_name is required, and the state cannot be longer than ten characters. Let’s see what happens when we try to violate these conditions:

>>> external['state'] = u'Commonwealth of Massachusetts'
>>> update_from_external_object(address, external)
Traceback (most recent call last):
...
TooLong: ('State is too long.', 'state', u'Commonwealth of Massachusetts')
>>> external['state'] = u'CA'
>>> external['full_name'] = None
>>> update_from_external_object(address, external)
Traceback (most recent call last):
...
zope.schema._bootstrapinterfaces.RequiredMissing: full_name

Much better! We get validation of our constraints and we didn’t have to write much code. But, we still had to write some code, one class for each object we’re externalizing. Can we do better?

autoPackageIO: Handing responsibility to the framework¶

The answer is yes, we can do much better, with the ext:registerAutoPackageIO ZCML directive.

Note

ext:registerAutoPackageIO is biased for a conventional setup of a single package: one or more root interfaces in interfaces.py, one or more modules defining factories (classes) implementing those interfaces. To an extent this can be changed using the iobase argument.

The above example schema is taken from the tests distributed with this package in nti.externalization.tests.benchmarks. That package provides the schema (as shown above), an implementation of it, and the ZCML file that pulls it all together with one directive.

Here’s the schema, along with several other schema to define a rich user profile, in interfaces.py:

# -*- coding: utf-8 -*-
"""
A rich user profile.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from zope.interface import Interface
from zope.interface import taggedValue

from zope.schema import Object
from zope.schema import URI

from nti.schema.field import Dict
from nti.schema.field import TextLine
from nti.schema.field import ValidTextLine
from nti.schema.field import DecodingValidTextLine

from nti.externalization.tests.benchmarks.bootstrapinterfaces import IRootInterface
from nti.externalization.tests.benchmarks.bootstrapinterfaces import checkEmailAddress
from nti.externalization.tests.benchmarks.bootstrapinterfaces import checkRealname

# pylint:disable=inherit-non-class
class IFriendlyNamed(Interface):

    alias = TextLine(title=u'Display alias',
                     description=u"Enter preferred display name alias, e.g., johnnyboy."
                     u"Your site may impose limitations on this value.",
                     required=False)

    realname = TextLine(title=u'Full Name aka realname',
                        description=u"Enter full name, e.g. John Smith.",
                        required=False,
                        constraint=checkRealname)

class IAvatarURL(Interface):
    """
    Something that features a display URL.
    """

    avatarURL = URI(title=u"URL of your avatar picture",
                    description=u"If not provided, one will be generated for you.",
                    required=False)


class IBackgroundURL(Interface):

    backgroundURL = URI(title=u"URL of your background picture",
                        description=u"If not provided, one will be generated for you.",
                        required=False)


class IProfileAvatarURL(IAvatarURL, IBackgroundURL):
    pass


class IAddress(IRootInterface):

    full_name = ValidTextLine(title=u"First name", required=True)

    street_address_1 = ValidTextLine(title=u"Street line 1",
                                     max_length=75, required=True)

    street_address_2 = ValidTextLine(title=u"Street line 2",
                                     required=False, max_length=75)

    city = ValidTextLine(title=u"City name", required=True)

    state = ValidTextLine(title=u"State name",
                          required=False, max_length=10)

    postal_code = ValidTextLine(title=u"Postal code",
                                required=False, max_length=30)

    country = ValidTextLine(title=u"Nation name", required=True)
    taggedValue('__external_class_name__',
                'Address')


class IUserContactProfile(Interface):

    addresses = Dict(title=u"A mapping of address objects.",
                     key_type=DecodingValidTextLine(title=u"Adresss key"),
                     value_type=Object(IAddress),
                     min_length=0,
                     required=False)

    phones = Dict(title=u"A mapping of phone numbers objects.",
                  key_type=ValidTextLine(title=u"Phone key"),
                  value_type=ValidTextLine(title=u"A phone"),
                  min_length=0,
                  required=False)

    contact_emails = Dict(title=u"A mapping of contact emails.",
                          key_type=DecodingValidTextLine(title=u"Email key"),
                          value_type=ValidTextLine(title=u'Email',
                                                   constraint=checkEmailAddress),
                          min_length=0,
                          required=False)

class IUserProfile(IProfileAvatarURL, # pylint:disable=too-many-ancestors
                   IUserContactProfile,
                   IFriendlyNamed,
                   IRootInterface):
    """A user profile"""
    taggedValue('__external_class_name__',
                'UserProfile')

They are implemented in objects.py very simply (as above):

from zope import interface
from zope.schema.fieldproperty import createFieldProperties
from nti.schema.eqhash import EqHash
from nti.externalization.representation import WithRepr
from nti.externalization.tests.benchmarks import interfaces

@interface.implementer(interfaces.IAddress)
@EqHash('full_name', 'street_address_1', 'postal_code')
@WithRepr
class Address(SchemaConfigured):
    createDirectFieldProperties(interfaces.IAddress)

@interface.implementer(interfaces.IUserProfile)
@EqHash('addresses', 'alias', 'phones', 'realname')
@WithRepr
class UserProfile(SchemaConfigured):
     createFieldProperties(interfaces.IUserProfile)

Finally, the ZCML file contains one directive that ties everything together:

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:ext="http://nextthought.com/ntp/ext">

    <include package="zope.component" />

    <include package="nti.externalization" file="meta.zcml" />
    <include package="nti.externalization" />

    <ext:registerAutoPackageIO
        root_interfaces=".interfaces.IRootInterface"
        modules=".objects"
        />
</configure>

If we configure this file, we can create and update addresses. We’ll do so through their container object, the UserProfile, thus demonstrating that nested schemas and objects are possible.

>>> import nti.externalization.tests.benchmarks
>>> _ = xmlconfig.file('configure.zcml', nti.externalization.tests.benchmarks)
>>> from nti.externalization.tests.benchmarks.objects import Address
>>> from nti.externalization.tests.benchmarks.objects import UserProfile
>>> home_address = Address(
...     full_name=u'Steve Jobs',
...     street_address_1=u'1313 Mockingbird Lane',
...     city=u'Salem',
...     state=u'MA',
...     postal_code=u'6666',
...     country=u'USA',
... )
>>> work_address = Address(
...     full_name=u'Apple',
...     street_address_1=u'1 Infinite Loop',
...     city=u'Cupertino',
...     state=u'CA',
...     postal_code=u'55555',
...     country=u'USA',
...  )
>>> user_profile = UserProfile(
...     addresses={u'home': home_address, u'work': work_address},
...     phones={u'home': u'405-555-1212', u'work': u'405-555-2323'},
...     contact_emails={u'home': u'steve.jobs@gmail.com', u'work': u'steve@apple.com'},
...     avatarURL='http://apple.com/steve.png',
...     backgroundURL='https://apple.com/bg.jpeg',
...     alias=u'Steve',
...     realname=u'Steve Jobs',
... )
>>> external = to_external_object(user_profile)
>>> pprint(external)
{'Class': 'UserProfile',
 'MimeType': 'application/vnd.nextthought.benchmarks.userprofile',
 'addresses': {'home': {'Class': 'Address',
                        'MimeType': 'application/vnd.nextthought.benchmarks.address',
                        'city': 'Salem',
                        'country': 'USA',
                        'full_name': 'Steve Jobs',
                        'postal_code': '6666',
                        'state': 'MA',
                        'street_address_1': '1313 Mockingbird Lane',
                        'street_address_2': None},
               'work': {'Class': 'Address',
                        'MimeType': 'application/vnd.nextthought.benchmarks.address',
                        'city': 'Cupertino',
                        'country': 'USA',
                        'full_name': 'Apple',
                        'postal_code': '55555',
                        'state': 'CA',
                        'street_address_1': '1 Infinite Loop',
                        'street_address_2': None}},
 'alias': 'Steve',
 'avatarURL': 'http://apple.com/steve.png',
 'backgroundURL': 'https://apple.com/bg.jpeg',
 'contact_emails': {'home': 'steve.jobs@gmail.com', 'work': 'steve@apple.com'},
 'phones': {'home': '405-555-1212', 'work': '405-555-2323'},
 'realname': 'Steve Jobs'}

Notice that there are some additional bits of data in the external form that are not specified in the interface. Here, that’s Class and MimeType. These are two of the Standard Fields.

Let’s make a change to the work address:

>>> external['addresses'][u'work']['street_address_1'] = u'One Apple Park Way'
>>> _ = update_from_external_object(user_profile, external)
>>> user_profile.addresses['work'].street_address_1
'One Apple Park Way'

Importantly, note that, by default, the nested objects are created fresh and not mutated.

>>> user_profile.addresses['work'] is work_address
False

This is described in more detail in Factories.

Representations¶

Being able to get a Python dictionary from an object, and update an object given a Python dictionary, is nice, but it doesn’t go all the way toward solving the goals of this package, interoperating with remote clients using a text (or byte) based stream.

For that, we have the nti.externalization.representation module, and its key interface IExternalObjectIO.

A representation is a format that can serialize Python dictionaries to text, and given that text, produce a Python dictionary. This package provides two representations by default, JSON and YAML. These are named utilities providing IExternalObjectIO. The function nti.externalization.to_external_representation is a shortcut for dumping to a string:

>>> from nti.externalization import to_external_representation
>>> from nti.externalization.interfaces import EXT_REPR_JSON, EXT_REPR_YAML
>>> to_external_representation(address, EXT_REPR_JSON)
'{"Class": "Address", "city": "Cupertino",...
>>> to_external_representation(address, EXT_REPR_YAML)
"{Class: Address, city: Cupertino, country: USA,...

Loading from a string doesn’t have a shortcut, we need to use the utility:

>>> from nti.externalization.interfaces import IExternalObjectIO
>>> external = to_external_object(address)
>>> yaml_io = component.getUtility(IExternalObjectIO, EXT_REPR_YAML)
>>> ext_yaml_str = yaml_io.dump(external)
>>> external_from_yaml = yaml_io.load(ext_yaml_str)
>>> external_from_yaml == external
True

Schemas¶

This package is heavily driven by schemas. The schema determines what attributes of an internal object are externalized based on its fields, and how incoming data in an external object are internalized, including all sorts of validation options.

Here is an example of some schemas.

from zope.interface import Interface
from zope.interface import taggedValue

from zope.schema import List
from zope.schema import Object
from zope.schema import URI

from nti.schema.field import Dict
from nti.schema.field import TextLine
from nti.schema.field import ValidTextLine
from nti.schema.field import DecodingValidTextLine

class IRootInterface(Interface):
    pass

class IFriendlyNamed(Interface):

    alias = TextLine(title=u'Display alias',
                     description=u"Enter preferred display name alias, e.g., johnnyboy."
                     u"Your site may impose limitations on this value.",
                     required=False)

    realname = TextLine(title=u'Full Name aka realname',
                        description=u"Enter full name, e.g. John Smith.",
                        required=False,
                        constraint=checkRealname)

class IAvatarURL(Interface):
    """
    Something that features a display URL.
    """

    avatarURL = URI(title=u"URL of your avatar picture",
                    description=u"If not provided, one will be generated for you.",
                    required=False)


class IBackgroundURL(Interface):

    backgroundURL = URI(title=u"URL of your background picture",
                        description=u"If not provided, one will be generated for you.",
                        required=False)


class IProfileAvatarURL(IAvatarURL, IBackgroundURL):
    pass


class IAddress(IRootInterface):

    full_name = ValidTextLine(title=u"First name", required=True)

    street_address_1 = ValidTextLine(title=u"Street line 1",
                                     max_length=75, required=True)

    street_address_2 = ValidTextLine(title=u"Street line 2",
                                     required=False, max_length=75)

    city = ValidTextLine(title=u"City name", required=True)

    state = ValidTextLine(title=u"State name",
                          required=False, max_length=10)

    postal_code = ValidTextLine(title=u"Postal code",
                                required=False, max_length=30)

    country = ValidTextLine(title=u"Nation name", required=True)
    taggedValue('__external_class_name__',
                'Address')


class IUserContactProfile(Interface):

    addresses = Dict(title=u"A mapping of address objects.",
                     key_type=DecodingValidTextLine(title=u"Adresss key"),
                     value_type=Object(IAddress),
                     min_length=0,
                     required=False)

    phones = Dict(title=u"A mapping of phone numbers objects.",
                  key_type=ValidTextLine(title=u"Phone key"),
                  value_type=ValidTextLine(title=u"A phone"),
                  min_length=0,
                  required=False)

    contact_emails = Dict(title=u"A mapping of contact emails.",
                          key_type=DecodingValidTextLine(title=u"Email key"),
                          value_type=ValidTextLine(title=u'Email',
                                                   constraint=checkEmailAddress),
                          min_length=0,
                          required=False)

class IUserProfile(IProfileAvatarURL,
                   IUserContactProfile,
                   IFriendlyNamed,
                   IRootInterface):
    """A user profile"""
    taggedValue('__external_class_name__',
                'UserProfile')

Externalization¶

Standard Fields¶

Certain fields are generally useful for almost all objects. The names of these fields along with descriptions of their expected contents are contained in the namespace StandardExternalFields.

The function to_standard_external_dictionary is used to find and populate many of these fields. It is called automatically by the datastructures such as InterfaceObjectIO

Decorating¶

Many times, we have additional information we want to include with an external object that is somehow derived or not explicitly represented in the interfaces implemented by an object. For example, in a web application, we may want to provide an href value giving the URL at which a particular object may be found. This URL is derived from the currently executing request object in addition to the object being externalized.

For this purpose, we use decorators. Decorators are subscription adapters (or subscribers), meaning that there can be many of them registered for any given object, that implement IExternalObjectDecorator. They can be registered just for the object, or for the object and the request. Each time an object is externalized by to_external_object, the registered decorators will be invoked before the external object is returned.

Let’s continue with the example address we used before.

>>> import nti.externalization.tests.benchmarks
>>> from zope.configuration import xmlconfig
>>> _ = xmlconfig.file('configure.zcml', nti.externalization.tests.benchmarks)
>>> from nti.externalization.tests.benchmarks.objects import Address
>>> from nti.externalization.tests.benchmarks.objects import UserProfile
>>> home_address = Address(
...     full_name=u'Steve Jobs',
...     street_address_1=u'1313 Mockingbird Lane',
...     city=u'Salem',
...     state=u'MA',
...     postal_code=u'6666',
...     country=u'USA',
... )

This time we’ll create and register a decorator. Let’s pretend that we work in a sensitive system and we need to redact the addresses of users to meet security concerns. Notice that decorators are usually stateless, so it is faster to make them inherit from Singleton.

from zope.interface import implementer
from zope.component import adapter

from nti.externalization.interfaces import IExternalObjectDecorator
from nti.externalization.tests.benchmarks.interfaces import IAddress
from nti.externalization.singleton import Singleton

@implementer(IExternalObjectDecorator)
@adapter(IAddress)
class PrivateAddressDecorator(Singleton):

    def decorateExternalObject(self, address, external):
        for key in 'street_address_1', 'street_address_2', 'state', 'city', 'postal_code':
            del external[key]

We’ll register our adapter and externalize:

>>> from nti.externalization import to_external_object
>>> from zope import component
>>> component.provideSubscriptionAdapter(PrivateAddressDecorator)
>>> from pprint import pprint
>>> pprint(to_external_object(home_address))
{'Class': 'Address',
 'MimeType': 'application/vnd.nextthought.benchmarks.address',
 'country': 'USA',
 'full_name': 'Steve Jobs'}

If we provide a request, adapters for the (object, request) are also found:

class Request(object):
   url = 'http://example.com/path/'

@implementer(IExternalObjectDecorator)
@adapter(IAddress, Request)
class LinkAddressDecorator(object):

    def __init__(self, context, request):
        self.request = request

    def decorateExternalObject(self, address, external):
        external['href'] = self.request.url + 'address'

We can now provide a request when we externalize (if no request argument is given, the hook function get_current_request is used to look for a request):

>>> component.provideSubscriptionAdapter(LinkAddressDecorator)
>>> pprint(to_external_object(home_address, request=Request()))
{'Class': 'Address',
 'MimeType': 'application/vnd.nextthought.benchmarks.address',
 'country': 'USA',
 'full_name': 'Steve Jobs',
 'href': 'http://example.com/path/address'}

IExternalStandardDictionaryDecorator¶

There is also IExternalStandardDictionaryDecorator. It is called by to_standard_external_dictionary. Typically that’s well before most of the object-specific fields have been filled in (e.g., from the object schema), and it is always before IExternalObjectDecorator is used. There may be occasional uses for this, but it’s best to stick to IExternalObjectDecorator.

Dublin Core Metadata¶

Decorators for zope.dublincore metadata are installed for all objects by default. See nti.externalization.dublincore for more information.

Internalization¶

We can create or update an existing object using external data with the functions new_from_external_object() or update_from_external_object(), respectively.

In a web framework like Pyramid where an application object is located by route matching or traversal, update_from_external_object makes the most sense.

Factories¶

While updating objects, internalization will, by default, create a new instance for every mapping that contains a MimeType or Class key using Zope Component factories. Factories are named utilities, that implement IFactory (essentially, a callable of no arguments) This package uses extensions of that interface, namely IMimeObjectFactory, IClassObjectFactory and IAnonymousObjectFactory (whose default implementations are found in nti.externalization.factory).

The factory matching MimeType is preferred, but we can fall back to one matching Class if needed.

Factories are usually registered automatically by ext:registerAutoPackageIO at the same time it creates the InterfaceObjectIO adapters.

You can manually register factories from particular modules using ext:registerMimeFactories.

Lets look at the factories registered for our address:

>>> from zope.configuration import xmlconfig
>>> from zope import component
>>> from nti.externalization.interfaces import IMimeObjectFactory
>>> import nti.externalization.tests.benchmarks
>>> _ = xmlconfig.file('configure.zcml', nti.externalization.tests.benchmarks)
>>> factory = component.getUtility(IMimeObjectFactory, 'application/vnd.nextthought.benchmarks.address')
>>> factory
<MimeObjectFactory titled 'Address' using <class 'nti.externalization.tests.benchmarks.objects.Address'>...>
>>> factory()
<nti.externalization.tests.benchmarks.objects.Address ...>

Mime Types are found in the class attribute mimeType; ext:registerAutoPackageIO will add computed mimeType values to factory objects if needed.

Special Attributes¶

This document covers some of the special attributes used by this module.

When we discuss interfaces or the contents of attributes, we are referring to tagged values. Unless otherwise noted, tagged values must be set directly on the object in question; they are not inherited from parent objects.

__external_class_name__

Used an a class or interface to determine the value of the Class standard external value. Usually this is a string, but when using InterfaceObjectIO (including ext:registerAutoPackageIO) it can be a callable returning a string or None

This value is inherited; the first non-None value in the resolution order will be used (the distinction matters when using callables).

__external_can_create__

This boolean value is set to true on factory functions (e.g., classes). ext:registerAutoPackageIO sets it to true automatically.

mimeType

Part of the zope.mimetype.interfaces.IContentTypeAware interface. This is read from Factories when creating factory registrations. It also forms one of the standard external fields.

_ext_excluded_out

A tagged value on individual attributes of an interface to prevent them from being externalized. See InterfaceObjectIO.

__external_factory_wants_arg__

(Provisional). Attribute of a factory. When creating sub-objects and invoking a factory, should we pass the external object to the factory? If not true or not set, the factory receives no arguments.

New in version 1.0a3.

__external_default_implementation__

(Provisional). Tagged value of an interface implemented by one of the factories discovered by ext:registerAutoPackageIO holding the primary factory discovered for that interface. Used when internalizing anonymous external data.

This value is inherited.

New in version 1.0a8.

__external_accept_id__

For attributes. Documentation needed.

_ext_is_marker_interface

When searching for a schema to use for externalization, interfaces with this tagged value directly set will not be considered.

Glossary¶

schema

A zope.interface.Interface interface, whose attributes are zope.schema.Field instances. zope.schema defines a number of useful field types such as zope.schema.Date. nti.schema.field provides even more, such as nti.schema.field.Variant.

See Schemas for more information.

internal object

A Python object in the application domain (sometimes known as a “model” object). This may be a complex object consisting of multiple nested objects. It may use inheritance. It will implement one or more schema.

external object

A dictionary with string keys, and values that are strings, numbers (including booleans), None, lists of external objects, or external objects.

In other words, a simplified interchange format capable of being represented by many programming languages and text formats.

standard external object

An external object produced by this package and having some defined keys and values. The most important of these is StandardExternalFields.MIMETYPE, which helps identify the class of the internal object.

anonymous external object

An external object often not produced by this package and lacking the defining metadata this package produces.

external representation

The byte string resulting from converting an external object into a particular textual interchange format such as JSON or YAML.

External representations are read and written using IExternalObjectIO utilities.

externalization

The process of creating a external object from a internal object. The API for this is nti.externalization.to_external_object(), and it is customized with IInternalObjectExternalizer adapters.

Sometimes this also means creating the external representation. If done at the same time, the API for this is nti.externalization.to_external_representation().

internalization

The process of taking an external object and using it to mutate an internal object. The API for this is nti.externalization.update_from_external_object().

Sub-objects are freshly created using factories.

factory

A callable object taking no arguments and returning a particular type of internal object. Typically these are (wrappers around) Python classes; the classes typically need to have an attribute __external_can_create__ set to a true value.

Changes¶

2.3.1 (unreleased)¶

Nothing changed yet.

2.3.0 (2021-08-02)¶

Add a new base class, StandardInternalObjectExternalizer. See PR 120 and issue 117.
Rename IExternalMappingDecorator to IExternalStandardDictionaryDecorator to emphasize that it is only used if you (directly or through a super class) call to_standard_external_dictionary. A compatibility alias remains. See PR 120 and issue 118.
Docs-deprecated aliases in nti.externalization.interfaces now also emit deprecation warnings at runtime.
Other documentation improvements. Sphinx can now run all the doctests (on Python 3); many doctests are still run an Python 2.

2.2.0 (2021-04-14)¶

Add support for Python 3.9.
Depend on BTrees 4.8 and above. This simplifies externalization checks. See issue 111.

2.1.0 (2020-08-03)¶

Add support for “externalization policies.” These are instances of ExternalizationPolicy that can be used to tweak certain low-level details of externalization without replacing externalization objects wholesale. They are intended to have a very low performance impact.

The only supported detail that can be changed right now is whether the standard created and last modified fields are externalized as Unix timestamps (the default) or as ISO 8601 strings.

See https://github.com/NextThought/nti.externalization/issues/109

2.0.0 (2020-07-02)¶

Change ILocatedExternalMapping: Previously it extended the legacy zope.interface.common.mapping.IFullMapping. Now it extends the modern zope.interface.common.collections.IMapping. Note that this does not require mutability unlike the older interface. (The LocatedExternalDict class provided by this package is fully mutable and implements IMutableMapping. It also continues to implement IFullMapping, but use of that interface is discouraged.)
Change ILocatedExternalSequence: Previously it extended the legacy zope.interface.common.sequence.ISequence. Now it extends the modern zope.interface.common.collections.ISequence. Note that this does not require mutability unlike the older interface. (The LocatedExternalList class provided by this package is fully mutable and implements IMutableSequence.)
Fix the interface resolution order for LocatedExternalList. Previously, with zope.interface 5, it began implementing both IMutableSequence (the new interface from zope.interface.common.collections) as well as the older interface ISequence (from zope.interface.common.sequence); the two have inconsistent resolution orders. Now, it only implements IMutableSequence and a subset of the legacy interfaces that do not conflict. See issue 105.

1.1.3 (2020-06-25)¶

Correctly fire IObjectWillUpdateFromExternalEvent events before updating an object.

1.1.2 (2020-04-07)¶

Adapt to a change in zope.container 4.4.0 that exposed unsafe assumptions that providedBy(obj) would return the exact same object with the exact same state on a subsequent call. This was always a bug in the case of concurrency (e.g., if a different thread called directlyProvides on the same object, or adjusted the __bases__ of some interface in the IRO); the zope.container changes made it possible without concurrency. See https://github.com/zopefoundation/zope.container/issues/38 and https://github.com/NextThought/nti.externalization/issues/104.

1.1.1 (2020-03-27)¶

Fix a faulty assertion error. See issue 102.

1.1.0 (2020-03-27)¶

Make instances of fractions.Fraction externalize as a string such as "1/3". When received by a schema field that can parse this format, such as zope.schema.Rational (or higher on the numeric tower), this means fractions can be round-tripped.
Support externalizing decimal.Decimal objects in the YAML representation.

1.0.0 (2020-03-19)¶

Add compatibility with, and require, zope.interface 5.0.
Document which tagged values are inherited and which are not.
Stop inheriting _ext_is_marker_interface.

1.0.0a14 (2019-11-13)¶

Build with Cython 0.29.14 using ‘3str’ as the language level.
Add support for Python 3.8.
Update PyYAML to 5.1 and change the default output style slightly.
Fix tests with Persistent 4.4.3 and above.
Support zope.interface 4.7, which lets tagged values on interfaces be inherited, when using <registerAutoPackageIO> on a module that had multiple objects implementing a derived interface. See issue 97.

1.0.0a13 (2018-09-20)¶

Support IFromBytes fields introduced by zope.schema 4.8.0. See issue 92.
Make validate_field_value (and by extension InterfaceObjectIO.update_from_external_object) call fromObject defined by any fields for non-byte and non-text data. Previously, only if the field raised a WrongContainedTypeError would fromObject be called.

1.0.0a12 (2018-09-11)¶

Add support for zope.schema 4.7.0 and nti.schema 1.5.0. Drop support for older versions, which includes dropping support for dm.zope.schema.Object fields.

1.0.0a11 (2018-08-29)¶

The @WithRepr decorator takes into account the updated default repr of Persistent objects with persistent 4.4 and doesn’t hide it.
Subclasses of ExternalizableInstanceDict that have non-str (unicode on Python 2, bytes on Python 3) keys in their __dict__ do not throw TypeError when externalizing. Instead, the non-str values are converted to strs (using ASCII encoding) and the _p_changed attribute, if any, is set.

1.0.0a10 (2018-08-21)¶

The registry argument to most functions is deprecated and ignored. Instead of making calls to registry.queryAdapter, we now invoke the interface directly. For example, IInternalObjectExternalizer(containedObject). This lets individual objects have a say if they already provide the interface without going through the legacy code paths (it also calls __conform__ on the object if needed).

1.0.0a9 (2018-08-20)¶

Allow subclasses of InterfaceObjectIO to have non-frozenset values for _ext_primitive_out_ivars_. This issues a warning and in the future will be a TypeError.

1.0.0a8 (2018-08-16)¶

Better support for internalizing anonymous value objects discovered in a Dict value. Now, they won’t raise a ComponentLookupError when require_updater is True, and they will be given a MimeType based on the schema (if they don’t have one).

1.0.0a7 (2018-07-31)¶

Avoid a TypeError from validate_named_field_value when external objects have unicode keys.
LocatedExternalDict objects accept more constructor arguments and allow arbitrary attributes.

1.0.0a6 (2018-07-31)¶

InterfaceObjectIO only returns an anonymous factory for IDict fields when it wants objects for the value.
StandardExternalFields and StandardInternalFields are deprecated aliases in nti.externalization.externalization.
update_from_external_object properly handles the case where INamedExternalizedObjectFactoryFinder and IInternalObjectUpdater are registered with different levels of specificity, and the finder also implements IInternalObjectUpdater. Before, the finder would, perhaps incorrectly, be used as the updater.

1.0.0a5 (2018-07-30)¶

Objects inheriting from InterfaceObjectIO and registered with the component registry (in ZCML) for IInternalObjectIO can still be found and used as INamedExternalizedObjectFactoryFinder, an interface implemented by InterfaceObjectIO through IInternalObjectIOFinder. A warning will be issued to update the registration (which generally means removing the provides line in ZCML).
ExternalizableInstanceDict no longer inherits from AbstractDynamicIO, it just implements the same interface (with the exception of many of the _ext methods). This class is deprecated.
Formally document the notify_modified member of nti.externalization.internalization. notifyModified is a deprecated alias.

1.0.0a4 (2018-07-30)¶

Make InterfaceObjectIO._ext_self readable from Python, even though that is not documented (and may change again in the future). Document the intended API, _ext_replacement(). See issue 73.
Make AbstractDynamicObjectIO._ext_getattr handle a default value, and add _ext_replacement_getattr for when it will only be called once. See issue 73.

1.0.0a3 (2018-07-28)¶

The @NoPickle decorator also works with Persistent subclasses (and may or may not work with multiple-inheritance subclasses of Persistent, depending on the MRO, but that’s always been the case for regular objects). A Persistent subclass being decorated with @NoPickle doesn’t make much sense, so a RuntimeWarning is issued. A warning is also issued if the class directly implements one of the pickle protocol methods.
Updating objects that use createFieldProperties or otherwise have FieldProperty objects in their type is at least 10% faster thanks to avoiding double-validation due to a small monkey-patch on FieldProperty. See issue 67.
Proxies around objects that implement toExternalObject are allowed again; the proxied object’s toExternalObject will be called.
The signature for updateFromExternalObject() has been tightened. It should be (self, external_object, context, **kwargs), where **kwargs is optional, as is context. **kwargs currently contains nothing useful. Uses of dataserver=None in the signature will generate a warning. This may be tightened further in the future. See issue 30.
__ext_ignore_updateFromExternalObject__ is officially deprecated and generates a warning.
update_from_external_object caches certain information about the types of the updater objects, making it 8-25% faster.
update_from_external_object mutates sequences contained in a dict in-place instead of overwriting with a new list.
update_from_external_object mutates sequences at the top level instead of returning a new list.
Add support for finding factories for incoming data which do not specify a MIME type or class field based on the key they are assigned to. This aids in consuming data produced by foreign systems or using Dict schema fields that require modelled values. See issue 51 and PR 68.
Schemas that use InterfaceObjectIO (including through the ZCML directive registerAutoPackageIO) can use Dict fields more easily on internalization (externalization has always worked): They automatically internalize their values by treating the Dict as anonymous external data.
Strings can automatically be adapted into ITimeDelta objects.

1.0.0a2 (2018-07-05)¶

The low levels of externalization no longer catch and hide POSKeyError. This indicates a problem with the database. See https://github.com/NextThought/nti.externalization/issues/60
Remove support for object_hook in update_from_external_object. See https://github.com/NextThought/nti.externalization/issues/29.
A number of deprecated aliases for moved functions have been removed.
On CPython, some of the modules are compiled as extension modules using Cython for a 10-30% increase in speed. Set the PURE_PYTHON environment variable to disable this at runtime.
The unused, undocumented method stripSyntheticKeysFromExternalDictionary was removed from instances of ExternalizableDictionaryMixin. Use the import instead.
Unused keyword arguments for to_standard_external_dictionary and to_minimal_standard_external_dictionary now produce a warning. In the future, extra keyword arguments will be an error.
notifyModified no longer accepts the eventFactory argument.
The notify_modified alias for notifyModified has been removed.
Decorating external mappings and external objects handled decorate_callback differently. This argument is only used when decorate is false. This argument is also confusing and should be considered deprecated.
choose_field no longer has the undocumented conversion behaviour for the CREATOR external field name.

1.0.0a1 (2017-09-29)¶

First PyPI release.
Add support for Python 3.
Drop support for externalizing to plists. See https://github.com/NextThought/nti.externalization/issues/21
Reach 100% test coverage and ensure we remain there through CI.
Introduce nti.externalization.extension_points to hold hook functions. Move the Pyramid integration there (and deprecate that). Also move the NTIID support there (but the old name works too). See https://github.com/NextThought/nti.externalization/issues/27
Deprecate nti.externalization.internalization.register_legacy_search_module. See https://github.com/NextThought/nti.externalization/issues/35
Stop ext:registerAutoPackageIO from registering the legacy class-name based factories by default. If you need class-name based factories, there are two options. The first is to explicitly register IClassObjectFactory objects in ZCML (we could add a scanning directive to make that more convenient for large numbers of classes), and the second is to set register_legacy_search_module to a true value in the ZCML directive for ext:registerAutoPackageIO. Note that we expect the behaviour of this attribute to change in the near future. See https://github.com/NextThought/nti.externalization/issues/33
Make ext:registerAutoPackageIO perform legacy class registrations when the configuration context executes, not when the directive runs. This means that conflicts in legacy class names will be detected at configuration time. It also means that legacy class names can be registered locally with z3c.baseregistry (previously they were always registered in the global site manager). See https://github.com/NextThought/nti.externalization/issues/28
Drop dependency on zope.preference and zope.annotation. They were not used by this package, although our configure.zcml did include them. If you use zope.preference or zope.annotation, please include them in your own ZCML file.
Drop hard dependency on Acquisition. It is still used if available and is used in test mode.
Add public implementations of IMimeObjectFactory and IClassObjectFactory in nti.externalization.factory.
Drop dependency on nti.zodb and its PersistentPropertyHolder. The datastructures in nti.externalization.persistence no longer extend that class; if you have further subclasses that add nti.zodb.peristentproperty.PropertyHoldingPersistent properties, you’ll need to be sure to mixin this class now. See https://github.com/NextThought/nti.externalization/issues/43
Add the <ext:classObjectFactory> directive for registering Class based factories. (Note: MIME factories are preferred.)
Callers of to_standard_external_dictionary (which includes AutoPackageScopedInterfaceIO) will now automatically get a MimeType value if one can be found. Previously only callers of to_minimal_standard_external_dictionary would.

API Details¶

Basics¶

`nti.externalization.interfaces`: Interfaces and constants¶

Externalization Interfaces

class ExternalizationPolicy(use_iso8601_for_unix_timestamp=False)[source]¶

Bases: object

Adjustment knobs for making tweaks across an entire externalization.

These knobs will tweak low-level details of the externalization format, details that are often in a hot code path where overhead should be kept to a minimum.

Instances of this class are used by registering them as named components in the global site manager. Certain low-level functions accept an optional policy argument that must be an instance of this class; higher level functions accept a policy_name argument that is used to find the registered component. If either argument is not given, then DEFAULT_EXTERNALIZATION_POLICY is used instead.

Instances are immutable.

This class must not be subclassed; as such, there is no interface for it, merely the class itself.

use_iso8601_for_unix_timestamp¶: Should unix timestamp fields be output as their numeric value, or be converted into an ISO 8601 timestamp string? By default, the numeric value is output. This is known to specifically apply to “Created Time” and “Last Modified.”

class LocatedExternalDict(*args, **kwargs)[source]¶

Bases: dict

A dictionary that implements ILocatedExternalMapping. Returned by to_standard_external_dictionary().

This class is not IContentTypeAware, and it indicates so explicitly by declaring a mime_type value of None.

interface IInternalObjectExternalizer[source]¶

Implemented by, or adapted from, an object that can be externalized.

__external_can_create__¶: This must be set to true, generally at the class level, for objects that can be created by specifying their Class name.

__external_class_name__¶: If present, the value is a string that is used for the ‘Class’ key in the external dictionary. If not present, the local name of the object’s class is used instead.

toExternalObject(**kwargs)¶: Optional, see this to_external_object().

interface INonExternalizableReplacement[source]¶: This interface may be applied to objects that serve as a replacement for a non-externalized object.

interface INonExternalizableReplacementFactory[source]¶

An factory object called to make a replacement when some object cannot be externalized.

__call__(obj)¶

Returns:	An externalized object to replace the given object. Possibly the given object itself if some higher level will handle it. The returned object may have the `INonExternalizableReplacement` interface.

interface IExternalObjectDecorator[source]¶

Used as a subscription adapter (of the object or the object and request) to provide additional information to the externalization of an object after it has been externalized by the primary implementation of IInternalObjectExternalizer. Allows for a separation of concerns. These are called in no specific order, and so must operate by mutating the external object.

These are called after IExternalStandardDictionaryDecorator.

decorateExternalObject(origial, external)¶

Decorate the externalized object (which is almost certainly a mapping, though this is not guaranteed).

Parameters:	original – The object that is being externalized. Passed to facilitate using non-classes as decorators. external – The externalization of that object, produced by an implementation of `IInternalObjectExternalizer` or default rules.
Returns:	Undefined.

interface IExternalStandardDictionaryDecorator[source]¶

Used as a subscription adapter (of the object or the object and request) to provide additional information to the externalization of an object after it has been externalized by the primary implementation of IInternalObjectExternalizer (which in turn must have invoked to_standard_external_dictionary()).

Allows for a separation of concerns. These are called in no specific order, and so must operate by mutating the external object.

These are called before IExternalObjectDecorator.

Changed in version 2.3.0: Previously this was called IExternalMappingDecorator; that name remains as a backward compatibility alias.

decorateExternalMapping(original, external)¶

Decorate the externalized object mapping.

Parameters:	original – The object that is being externalized. Passed to facilitate using non-classes as decorators. external – The externalization of that object, an `ILocatedExternalMapping`, produced by an implementation of `IInternalObjectExternalizer` or default rules.
Returns:	Undefined.

interface IExternalizedObject[source]¶: An object that has already been externalized and needs no further transformation.

interface ILocatedExternalMapping[source]¶

Extends: nti.externalization.interfaces.IExternalizedObject, zope.location.interfaces.ILocation, zope.interface.common.collections.IMapping

The externalization of an object as a dictionary, maintaining its location information.

interface ILocatedExternalSequence[source]¶

Extends: nti.externalization.interfaces.IExternalizedObject, zope.location.interfaces.ILocation, zope.interface.common.collections.ISequence

The externalization of an object as a sequence, maintaining its location information.

class LocatedExternalList[source]¶

Bases: list

A list that implements ILocatedExternalSequence. Returned by to_external_object().

This class is not IContentTypeAware, and it indicates so explicitly by declaring a mimeType value of None.

interface IExternalObjectRepresenter[source]¶

Something that can represent an external object as a sequence of bytes.

These will be registered as named utilities and may each have slightly different representation characteristics.

dump(obj, fp=None)¶: Write the given object. If fp is None, then the string representation will be returned, otherwise, fp specifies a writeable object to which the representation will be written.

interface IExternalRepresentationReader[source]¶

Something that can read an external string, as produced by IExternalObjectRepresenter and return an equivalent external value.`

load(stream)¶

Load from the stream an external value. String values should be read as unicode.

All objects must support the stream being a sequence of bytes, some may support an open file object.

interface IExternalObjectIO[source]¶

Extends: nti.externalization.interfaces.IExternalObjectRepresenter, nti.externalization.interfaces.IExternalRepresentationReader

Something that can read and write external values.

EXT_REPR_JSON = u'json'¶: Constant requesting JSON format data

EXT_REPR_YAML = u'yaml'¶: Constant requesting YAML format data

interface IMimeObjectFactory[source]¶

Extends: zope.component.interfaces.IFactory

A factory named for the external mime-type of objects it works with.

interface IClassObjectFactory[source]¶

Extends: zope.component.interfaces.IFactory

A factory named for the external class name of objects it works with.

interface IAnonymousObjectFactory[source]¶

Extends: zope.component.interfaces.IFactory

A factory for external data that doesn’t identify its object type.

This data is not produced by this library but comes from external sources.

When these are registered as factories (utilities) care must be taken to avoid name clashes (since there are no “natural” unique names).

See the ZCML directive IAnonymousObjectFactoryDirective for a simple way to do this.

New in version 1.0a3.

interface IExternalizedObjectFactoryFinder[source]¶

An adapter from an externalized object to something that can find factories.

find_factory(externalized_object)¶

Given an externalized object, return a zope.component.interfaces.IFactory to create the proper internal types.

Returns:	An `zope.component.interfaces.IFactory`, or `None`.

interface IExternalReferenceResolver[source]¶

Used as a multi-adapter from an internal object and an external reference to something that can resolve the reference.

resolve(reference)¶: Resolve the external reference and return it.

interface INamedExternalizedObjectFactoryFinder[source]¶

An object that can find factories for particular named external objects.

This is registered as an adapter for particular internal objects, so that internal object, and any schema it implements, can be used to choose the factory for nested objects being updated into it.

find_factory_for_named_value(name, value)¶

Find a factory for the external object value when it is the value with the name name.

This function has three key pieces of information to work with.

First, it is an adapter from an internal object, so it knows the ultimate destination object (the context) where the results of the factory will be set.

Second, it knows the incoming name of the external value.

Third, it knows the actual incoming external value.

For example, if the external data looked like {'key': 'value'} a call to update_from_external_object(internal_object, data) would conceptually result in a call that looked like this:

adapter = INamedExternalizedObjectFactoryFinder(internal_object)
factory = adapter.find_factory_for_named_value('key', 'value')

When the value for the external data is a mutable sequence, this function will be called once for each item in the sequence. So external data of {'key': [1, 2, 3]} would result in calls ('key', 1), ('key', 2) and ('key', 3).

This function can return actual factories that produce fresh objects, or it can return the current object assigned to the ultimate attribute in the context to update that exact object in place. This can be beneficial for persistent objects.

interface IInternalObjectUpdater[source]¶

An adapter that can be used to update an internal object from its externalized representation.

__external_resolvers__¶: For objects who need to perform arbitrary resolution from external forms to internal forms, this attribute is a map from key path to a function of three arguments, the dataserver, the parsed object, and the value to resolve. It should return the new value. Note that the function here is at most a class or static method, not an instance method.

__external_oids__¶: For objects whose external form includes object references (OIDs), this attribute is a list of key paths that should be resolved. The values for the key paths may be singleton items or mutable sequences. Resolution may involve placing a None value for a key.

updateFromExternalObject(externalObject, context, **kwargs)¶

Update the object this is adapting from the external object.

Alternately, the signature can be updateFromExternalObject(externalObject) or simply updateFromExternalObject(externalObject, **kwargs). In this last case, context will be passed as a value in **kwargs.

Returns:	If not `None`, a value that can be interpreted as a boolean, indicating whether or not the internal object actually underwent updates. If `None`, the caller should assume that the object was updated (to allow older code that doesn’t return at all.)

interface IInternalObjectIO[source]¶

Extends: nti.externalization.interfaces.IInternalObjectExternalizer, nti.externalization.interfaces.IInternalObjectUpdater

A single object responsible for both reading and writing internal objects in external forms. This is convenient for keeping code organized.

interface IInternalObjectIOFinder[source]¶

Extends: nti.externalization.interfaces.INamedExternalizedObjectFactoryFinder, nti.externalization.interfaces.IInternalObjectIO

Like IInternalObjectIO, but this object also gets the chance to find factories for objects.

The class InterfaceObjectIO implements this interface.

interface IObjectWillUpdateFromExternalEvent[source]¶

Extends: zope.interface.interfaces.IObjectEvent

An object will be updated from an external value.

root¶: The object initially passed to update_from_external_object. For nested objects, this will be some ancestor of the object this event is for. For updaters that manually update sub-objects, this isn’t guaranteed to be the actual true root object being updated.

external_value¶: The external value. This is not necessarily a pristine object as decoded from, e.g., JSON. It will be mutated as sub-objects get updated and parsed. For example, strings may get replaced with datetimes, and so on. The consequences of modifying this object in an event subscriber are undefined.

interface IObjectModifiedFromExternalEvent[source]¶

Extends: zope.lifecycleevent.interfaces.IObjectModifiedEvent

An object has been updated from an external value.

external_value¶: The external value

kwargs¶: The keyword arguments

class ObjectModifiedFromExternalEvent(obj, *descriptions, **kwargs)[source]¶

Bases: zope.lifecycleevent.ObjectModifiedEvent

Default implementation of IObjectModifiedFromExternalEvent.

class StandardExternalFields[source]¶

Bases: object

Namespace object defining constants whose values are the keys used in external mappings.

These are text (unicode).

Not all external objects will have all possible keys.

Two special values are collections of metadata, not strings: ALL and EXTERNAL_KEYS.

ID¶: An id

OID¶: An identifier specific to this exact object instance

HREF¶: A hyperlink to reach this object

INTID¶: An integer uniquely identifying the object in some scope

NTIID¶: A structured identifier similar to a hyperlink

CREATOR¶: The name of the creator of the object

CONTAINER_ID¶: The name of the container holding the object

CREATED_TIME¶: The floating point value giving the Unix epoch time of the object’s creation

LAST_MODIFIED¶: The floating point value giving the Unix epoch time of the last modification of the object

CLASS¶: ‘Class’: The class of the object. If the object provides __external_class_name__ it will be used to populate this.

LINKS¶: A dictionary mapping “rel” to more hrefs.

MIMETYPE¶: The MIME type of this object

ITEMS¶: A list or dictionary of external objects contained within this object

TOTAL¶: A counter

ITEM_COUNT¶: The total number of items contained in this object

ALL¶

A collection of all names of all the attributes of this class.

That is, the contents of this collection are the attribute names that give standard external fields. You can iterate this and use getattr() to get the corresponding values.

EXTERNAL_KEYS¶

A collection of all values of all attributes of this class.

That is, the contents of this collection are the keys that a standard external object would be expected to have.

class StandardInternalFields[source]¶

Bases: object

Namespace object defining constants whose values are the property/attribute names looked for on internal objects.

These must be native strings.

CONTAINER_ID¶: ‘containerId’: The ID of the container of this object. Fills StandardExternalFields.CONTAINER_ID.

CREATED_TIME¶: ‘createdTime’: The Unix timestamp of the creation of this object. If no value can be found, we will attempt to adapt to zope.dublincore.interfaces.IDCTimes and use its ‘created’ attribute. Fills StandardExternalFields.CREATED_TIME

CREATOR¶: ‘creator’: An object that created this object. This will be converted to a text string to fill in StandardExternalFields.CREATOR.

ID¶: ‘id’: An object ID

LAST_MODIFIED¶: ‘lastModified’: The Unix timestamp of the last modification of this object. If no value can be found, we will attempt to adapt to zope.dublincore.interfaces.IDCTimes` and use its ‘modified’ attribute. Fills StandardExternalFields.LAST_MODIFIED

NTIID¶: ‘ntiid’: An object’s structured ID.

DEFAULT_EXTERNALIZATION_POLICY = ExternalizationPolicy(use_iso8601_for_unix_timestamp=False)¶: The default externalization policy.

Deprecated Names¶

IList = <InterfaceClass zope.interface.common.builtins.IList>[source]¶: Marker interface for lists.

Deprecated since version 2.1.0: Use zope.interface.common.builtins.IList directly. This is just an alias.

IIterable = <ABCInterfaceClass zope.interface.common.collections.IIterable>[source]¶: Base interface for iterable types.

Deprecated since version 2.1.0: Use zope.interface.common.collections.IIterable directly. This is just an alias.

IExternalObject = <InterfaceClass nti.externalization.interfaces.IInternalObjectExternalizer>¶: Backwards compatibility alias.

Deprecated since version 2.0.0: Use IInternalObjectExternalizer directly.

INonExternalizableReplacer = <InterfaceClass nti.externalization.interfaces.INonExternalizableReplacementFactory>¶: Backwards compatibility alias.

Deprecated since version 2.0.0: Use INonExternalizableReplacement directly.

IExternalMappingDecorator = <InterfaceClass nti.externalization.interfaces.IExternalStandardDictionaryDecorator>¶: Backwards compatibility alias.

Deprecated since version 2.3.0: Use IExternalStandardDictionaryDecorator directly.

Externalization¶

Functions related to actually externalizing objects.

Only import from this module. Sub-modules of this package are implementation details.

exception NonExternalizableObjectError[source]¶: Bases: exceptions.TypeError

get_last_modified_time(context, default=None, policy=ExternalizationPolicy(use_iso8601_for_unix_timestamp=False), _write_into=None)[source]¶

Find and return a number representing the time since the epoch in fractional seconds at which the context was last modified. This is the same value that is used by to_standard_external_dictionary(), and takes into account whether something is nti.dataserver.interfaces.ILastModified or zope.dublincore.interfaces.IDCTimes.

Returns:	A number if it can be found, or the value of `default`

get_created_time(context, default=None, policy=ExternalizationPolicy(use_iso8601_for_unix_timestamp=False), _write_into=None)[source]¶

Find and return a number representing the time since the epoch in fractional seconds at which the context was created. This is the same value that is used by to_standard_external_dictionary(), and takes into account whether something is nti.dataserver.interfaces.ILastModified or zope.dublincore.interfaces.IDCTimes.

Returns:	A number if it can be found, or the value of `default`

to_standard_external_dictionary(self, mergeFrom=None, decorate=True, request=NotGiven)[source]¶

Returns a dictionary representing the standard externalization of the object self. This function takes care of many of the standard external fields:

External identifiers like StandardExternalFields.OID and StandardExternalFields.NTIID using set_external_identifiers.

The StandardExternalFields.CREATOR.

The StandardExternalFields.LAST_MODIFIED.

The StandardExternalFields.CREATED_TIME.

The StandardExternalFields.CONTAINER_ID.

The StandardExternalFields.CLASS and StandardExternalFields.MIMETYPE (from the mimeType attribute of the object).

If the object has any IExternalStandardDictionaryDecorator subscribers registered for it, they will be called to decorate the result of this method before it returns (unless decorate is set to False; only do this if you know what you are doing! ) This is the only part of nti.externalization that invokes this decorator.

Custom externalization should begin by calling this function, or, preferably, by using an existing externalizer (which invokes this function, such as StandardInternalObjectExternalizer or InterfaceObjectIO ) or subclassing such an existing type and mutating the dictionary returned from super’s toExternalObject in your own implementation.

Parameters:	mergeFrom (dict) – For convenience, if mergeFrom is not `None`, then values it contains will be added to the dictionary created by this method. The keys and values in mergeFrom should already be external. policy (ExternalizationPolicy) – The `ExternalizationPolicy` to use. Must not be None.
Returns:	A `LocatedExternalDict`. For further externalization, this object should be mutated in place.

Changed in version 1.0a1: Arbitrary keyword arguments not used by this function are deprecated and produce a warning.

Changed in version 2.1: Add the policy keyword.

to_minimal_standard_external_dictionary(self, mergeFrom=None)[source]¶: Does no decoration. Useful for non-‘object’ types. self should have a mime_type field.

to_external_object(obj, name=<default value>, registry=<default value>, catch_components=(), catch_component_action=None, request=<default value>, decorate=True, useCache=True, decorate_callback=<default value>, default_non_externalizable_replacer=<function DefaultNonExternalizableReplacer>, policy_name=<default value>, policy=<default value>)[source]¶

Translates the object into a form suitable for external distribution, through some data formatting process. See SEQUENCE_TYPES and MAPPING_TYPES for details on what we can handle by default.

Parameters:

name (string) – The name of the adapter to IInternalObjectExternalizer to look for. Defaults to the empty string (the default adapter). If you provide a name, and an adapter is not found, we will still look for the default name (unless the name you supply is None).
catch_components (tuple) – A tuple of exception classes to catch when externalizing sub-objects (e.g., items in a list or dictionary). If one of these exceptions is caught, then catch_component_action will be called to raise or replace the value. The default is to catch nothing.
catch_component_action (callable) – If given with catch_components, a function of two arguments, the object being externalized and the exception raised. May return a different object (already externalized) or re-raise the exception. There is no default, but catch_replace_action() is a good choice.
default_non_externalizable_replacer (callable) – If we are asked to externalize an object and cannot, and there is no INonExternalizableReplacer registered for it, then call this object and use the results.
request – If given, the request that the object is being externalized on behalf of. If given, then the object decorators will also look for subscribers to the object plus the request (like traversal adapters); this is a good way to separate out request or user specific code.
decorate_callback – Callable to be invoked in case there is no decaration
policy – The ExternalizationPolicy to use. Takes priority over policy_name. If this is not given (and neither is policy_name), the thread local state is consulted for the current policy established by the most recent caller to this method; if there is no such caller, then the DEFAULT_EXTERNALIZATION_POLICY is used.
policy_name (str) – If no policy is given, then this is used to lookup a utility. If this is used, the utility must exist.

catch_replace_action(obj, exc)[source]¶: Replaces the external component object obj with an object noting a broken object.

Internalization¶

Functions for taking externalized objects and creating application model objects.

update_from_external_object(containedObject, externalObject, context=None, require_updater=False, notify=True)[source]¶

Central method for updating objects from external values.

Parameters:

containedObject – The object to update.
externalObject – The object (typically a mapping or sequence) to update the object from. Usually this is obtained by parsing an external format like JSON.
context – An object passed to the update methods.
require_updater – If True (not the default) an exception will be raised if no implementation of IInternalObjectUpdater can be found for the containedObject.
notify (bool) – If True (the default), then if the updater for the containedObject either has no preference (returns None) or indicates that the object has changed, then an IObjectModifiedFromExternalEvent will be fired. This may be a recursive process so a top-level call to this object may spawn multiple events. The events that are fired will have a descriptions list containing one or more IAttributes each with attributes for each attribute we modify (assuming that the keys in the externalObject map one-to-one to an attribute; if this is the case and we can also find an interface declaring the attribute, then the IAttributes will have the right value for interface as well).
pre_hook (callable) – If given, called with the before update_from_external_object is called for every nested object. Signature f(k,x) where k is either the key name, or None in the case of a sequence and x is the external object. Deprecated.

Returns:

containedObject after updates from externalObject

Notifies IObjectModifiedFromExternalEvent for each object that is modified, and IObjectWillUpdateFromExternalEvent before doing so.

See also

INamedExternalizedObjectFactoryFinder

Changed in version 1.0.0a2: Remove the object_hook parameter.

Changed in version 1.1.3: Correctly file IObjectWillUpdateFromExternalEvent before updating each object.

new_from_external_object(external_object, *args, **kwargs)[source]¶

Like update_from_external_object, but creates a new object to update using find_factory_for.

All remaining arguments are passed to update_from_external_object.

If no factory can be found, raises a zope.interface.interfaces.ComponentLookupError.

Returns the new object.

New in version 1.0a3.

register_legacy_search_module(module_name)[source]¶

The legacy creation search routines will use the modules registered by this method.

Note that there are no order guarantees about how the modules will be searched. Duplicate class names are thus undefined.

Parameters:	module_name – Either the name of a module to look for at runtime in `sys.modules`, or a module-like object having a `__dict__`.

Deprecated since version 1.0: Use explicit mime or class factories instead. See https://github.com/NextThought/nti.externalization/issues/35

find_factory_for(externalized_object) → factory[source]¶

Given a IExternalizedObject, locate and return a factory to produce a Python object to hold its contents.

If there is a IExternalizedObjectFactoryFinder adapter registered for the externalized object, we return the results of its find_factory method. Note that since externalized objects are typically simple lists or dicts, such adapters have the capability to hijack all factory finding, probably unintentionally.

Otherwise, we examine the contents of the object itself to find a registered factory based on MIME type (preferably) or class name.

Changed in version 1.0a10: The registry argument is deprecated and ignored.

notify_modified(containedObject, externalObject, updater=None, external_keys=None, **kwargs)¶

Create and send an ObjectModifiedFromExternalEvent for containedObject using zope.event.notify.

The containedObject is the subject of the event. The externalObject is the dictionary of data that was used to update the containedObject.

external_keys is list of keys from externalObject that actually changed containedObject. If this is not given, we assume that all keys in externalObject were changed. Note that these should to correspond to the interface fields of interfaces that the containedObject implements in order to properly be able to create and populate the zope.lifecycleevent IAttributes.

updater, if given, is the IInternalObjectUpdater instance that was used to handle the updates. If this object implements an _ext_adjust_modified_event method, it will be called to adjust (and return) the event object that will be notified.

kwargs are the keyword arguments passed to the event constructor.

Returns:	The event object that was notified.

validate_field_value(self, field_name, field, value)[source]¶

Given a zope.schema.interfaces.IField object from a schema implemented by self, validates that the proposed value can be set. If the value needs to be adapted to the schema type for validation to work, this method will attempt that.

Parameters:	field_name (str) – The name of the field we are setting. This implementation currently only uses this for informative purposes. field (`zope.schema.interfaces.IField`) – The schema field to use to validate (and set) the value.
Raises:	zope.interface.Invalid – If the field cannot be validated, along with a good reason (typically better than simply provided by the field itself)
Returns:	A callable of no arguments to call to actually set the value (necessary in case the value had to be adapted).

validate_named_field_value(self, iface, field_name, value)[source]¶

Given a zope.interface.Interface and the name of one of its attributes, validate that the given value is appropriate to set. See validate_field_value() for details.

Parameters:	field_name (str) – The name of a field contained in `iface`. May name a regular `zope.interface.Attribute`, or a `zope.schema.interfaces.IField`; if the latter, extra validation will be possible.
Returns:	A callable of no arguments to call to actually set the value.

Datastructures¶

Datastructures to help externalization.

class ExternalizableDictionaryMixin[source]¶

Bases: object

Implements a toExternalDictionary method as a base for subclasses.

_ext_replacement()[source]¶

Return the object that we are externalizing.

This class returns self, but subclasses will typically override this.

toExternalDictionary(mergeFrom=None, *unused_args, **kwargs)[source]¶

Produce the standard external dictionary for this object.

Uses _ext_replacement.

class StandardInternalObjectExternalizer(context)[source]¶

Bases: nti.externalization.datastructures.ExternalizableDictionaryMixin

An adapter that can be used to implement IInternalObjectExternalizer.

The result of externalizing is the standard external dictionary for this adapter’s context argument.

This can be registered as-is, or subclassed to add additional items in the external dictionary. In that case, always begin by calling this implemention first and updating the result.

New in version 2.3.0.

The constructor sets __external_can_create__ to False (because creating from just an externalizer makes no sense) and __external_class_name__ to None (if you override this value, it will replace the Class value in the returned dictionary; it must be a native str).

_ext_replacement()[source]¶: Returns this adapter’s context argument.

class AbstractDynamicObjectIO[source]¶

Bases: nti.externalization.datastructures.ExternalizableDictionaryMixin

Base class for objects that externalize based on dynamic information.

Abstractions are in place to allow subclasses to map external and internal names independently (this type never uses getattr/setattr/hasattr, except for some standard fields).

See InterfaceObjectIO for a complete implementation.

_ext_accept_external_id(ext_self, parsed)[source]¶

If the object we’re updating does not have an id set, but there is an ID in the external object, should we be able to use it?

Returns:	boolean

_ext_accept_update_key(k, ext_self, ext_keys)[source]¶

Returns whether or not this key should be accepted for setting on the object, or silently ignored.

Parameters:	ext_keys – As an optimization, the value of `_ext_all_possible_keys()` is passed. Keys are only accepted if they are in this list.

_ext_all_possible_keys()[source]¶: This method must return a frozenset of native strings.

_ext_getattr(object, name[, default]) → value[source]¶: Return the attribute of the ext_self object with the internal name name. If the attribute does not exist, should raise (typically AttributeError), unless default is given, in which case it returns that.

Changed in version 1.0a4: Add the default argument.

_ext_keys()[source]¶

Return only the names of attributes that should be externalized. These values will be used as keys in the external dictionary.

See _ext_all_possible_keys(). This implementation then filters out private attributes (those beginning with an underscore), and those listed in _excluded_in_ivars_.

This method must return a set of native strings.

_ext_primitive_keys()[source]¶

Return a container of string keys whose values are known to be primitive. This is an optimization for writing.

This method must return a frozenset.

_ext_replacement()[source]¶

Return the object that we are externalizing.

This class returns self, but subclasses will typically override this.

_ext_replacement_getattr(name, default=<default value>)[source]¶: Like _ext_getattr, but automatically fills in _ext_replacement for the ext_self argument.

New in version 1.0a4.

find_factory_for_named_value(key, value)[source]¶

Uses find_factory_for to locate a factory.

This does not take into account the current object (context) or the key. It only handles finding factories based on the class or MIME type found within value.

toExternalDictionary(mergeFrom=None, *unused_args, **kwargs)[source]¶

Produce the standard external dictionary for this object.

Uses _ext_replacement.

class ExternalizableInstanceDict[source]¶

Bases: object

Externalizes to a dictionary containing the members of __dict__ that do not start with an underscore.

Meant to be used as a super class; also can be used as an external object superclass.

Consider carefully before using this class. Generally, an interface and InterfaceObjectIO are better.

Changed in version 1.0a5: No longer extends AbstractDynamicObjectIO, just delegates to it. Most of the _ext_` prefixed methods can no longer be overridden.

_ext_replacement()[source]¶: See ExternalizableDictionaryMixin._ext_replacement.

toExternalDictionary(mergeFrom=None, *unused_args, **kwargs)[source]¶: See ExternalizableDictionaryMixin.toExternalDictionary

toExternalObject(mergeFrom=None, *args, **kwargs)[source]¶: See toExternalObject. Calls toExternalDictionary.

updateFromExternalObject(parsed, *unused_args, **unused_kwargs)[source]¶: See updateFromExternalObject

class InterfaceObjectIO(context, iface_upper_bound=None, validate_after_update=True)[source]¶

Bases: nti.externalization.datastructures.AbstractDynamicObjectIO

Externalizes the context to a dictionary based on getting the attributes of an object defined by an interface. If any attribute has a true value for the tagged value _ext_excluded_out, it will not be considered for reading or writing.

This is an implementation of IInternalObjectIOFinder, meaning it can both internalize (update existing objects) and externalize (producing dictionaries), and that it gets to choose the factories used for sub-objects when internalizing.

This class is meant to be used as an adapter, so it accepts the object to externalize in the constructor, as well as the interface to use to guide the process. The object is externalized using the most-derived version of the interface given to the constructor that it implements.

If the interface (or an ancestor) has a tagged value __external_class_name__, it can either be the value to use for the Class key, or a callable __external_class_name__(interface, object ) -> name.

(TODO: In the future extend this to multiple, non-overlapping interfaces, and better interface detection (see ModuleScopedInterfaceObjectIO for a limited version of this.)

This class overrides _ext_replacement to return the context.

Parameters:

iface_upper_bound – The upper bound on the schema to use to externalize ext_self; we will use the most derived sub-interface of this interface that the object implements. Subclasses can either override this constructor to pass this parameter (while taking one argument themselves, to be usable as an adapter), or they can define the class attribute _ext_iface_upper_bound
validate_after_update (bool) – If True (the default) then the entire schema will be validated after an object has been updated with update_from_external_object(), not just the keys that were assigned.

_ext_accept_external_id(ext_self, parsed)[source]¶: If the interface we’re working from has a tagged value of __external_accept_id__ on the id field, then this will return that value; otherwise, returns false.

_ext_all_possible_keys()[source]¶: This method must return a frozenset of native strings.

_ext_getattr(object, name[, default]) → value[source]¶: Return the attribute of the ext_self object with the internal name name. If the attribute does not exist, should raise (typically AttributeError), unless default is given, in which case it returns that.

Changed in version 1.0a4: Add the default argument.

_ext_replacement()[source]¶

Return the object that we are externalizing.

This class returns self, but subclasses will typically override this.

find_factory_for_named_value(key, value)[source]¶

If AbstractDynamicObjectIO.find_factory_for_named_value cannot find a factory based on examining value, then we use the context objects’s schema to find a factory.

If the schema contains an attribute named key, it will be queried for the tagged value __external_factory__. If present, this tagged value should be the name of a factory object implementing IAnonymousObjectFactory registered in registry (typically registered in the global site).

The ZCML directive IAnonymousObjectFactoryDirective sets up both the registration and the tagged value.

This is useful for internalizing data from external sources that does not provide a class or MIME field within the data.

The most obvious limitation of this is that if the value is part of a sequence, it must be a homogeneous sequence. The factory is called with no arguments, so the only way to deal with heterogeneous sequences is to subclass this object and override this method to examine the value itself.

A second limitation is that the external data key must match the internal schema field name. Again, the only way to remove this limitation is to subclass this object.

If no registered factory is found, and the schema field is a zope.schema.Dict with a value type of zope.schema.Object, then we return a factory which will update the object in place.

Changed in version 1.0a6: Only return an anonymous factory for IDict fields when it wants objects for the value.

schema¶: The schema we will use to guide the process

class ModuleScopedInterfaceObjectIO(context, iface_upper_bound=None, validate_after_update=True)[source]¶

Bases: nti.externalization.datastructures.InterfaceObjectIO

Only considers the interfaces provided within a given module (usually declared as a class attribute) when searching for the schema to use to externalize an object; the most derived version of interfaces within that module will be used. Subclasses must declare the class attribute _ext_search_module to be a module (something with the __name__) attribute to locate interfaces in.

Suitable for use when all the externalizable fields of interest are declared by an interface within a module, and an object does not implement two unrelated interfaces from the same module.

Note

If the object does implement unrelated interfaces, but one (set) of them is a marker interface (featuring no schema fields or attributes), then it can be tagged with _ext_is_marker_interface and it will be excluded when determining the most derived interfaces. This can correct some cases that would otherwise raise a TypeError. This tag is not inherited.

Parameters:

iface_upper_bound – The upper bound on the schema to use to externalize ext_self; we will use the most derived sub-interface of this interface that the object implements. Subclasses can either override this constructor to pass this parameter (while taking one argument themselves, to be usable as an adapter), or they can define the class attribute _ext_iface_upper_bound
validate_after_update (bool) – If True (the default) then the entire schema will be validated after an object has been updated with update_from_external_object(), not just the keys that were assigned.

ZCML¶

Directives to be used in ZCML.

These directives are all in the http://nextthought.com/ntp/ext namespace, and are loaded using the meta.zcml file.

Example (non-sensical) of all the directives:

<configure xmlns:ext="http://nextthought.com/ntp/ext">
    <include package="nti.externalization" file="meta.zcml" />

    <ext:registerMimeFactories module="the.module" />

    <ext:registerAutoPackageIO
        root_interfaces="other_module.interfaces.IExtRoot"
        modules="other_module.factories"
        iobase="other_module.externalization.IOBase"
        register_legacy_search_module="yes" />

    <ext:classObjectFactory
        factory="third_module.Factory"
        name="AName" />

    <ext:anonymousObjectFactory
        factory="module3.Factory"
        for="interfaces.ISchema"
        field="field" />
    <ext:anonymousObjectFactoryInPlace
        for="interfaces.ISchema"
        field="field2" />
</configure>

interface IRegisterInternalizationMimeFactoriesDirective[source]¶

Defines the ext:registerMimeFactories directive.

Poke through the classes defined in module. If a class defines the mimeType attribute and can be created externally, (because it defines __external_can_create__ to be true), registers a factory utility under the mimeType name. (For backwards compatibility, mime_type is accepted if there is no mimeType.)

Factories are discovered using find_factories_in_module.

See nti.externalization.internalization.find_factory_for() for how factories are used.

module¶

Module to scan for Mime factories to add

Implementation:	`zope.configuration.fields.GlobalObject`
Read Only:	False
Required:	True
Default Value:	None

interface IAutoPackageExternalizationDirective[source]¶

Defines the ext:registerAutoPackageIO directive.

This directive combines the effects of IRegisterInternalizationMimeFactoriesDirective with that of autopackage, removing all need to repeat root interfaces and module names.

After this directive is complete, a new class that descends from AutoPackageSearchingScopedInterfaceObjectIO will be registered as the IInternalObjectIO adapter for all of the root_interface objects, and the modules (or factory_modules) will be searched for object factories via IRegisterInternalizationMimeFactoriesDirective.

Changed in version 1.0: Add the register_legacy_search_module keyword argument, defaulting to False. Previously legacy search modules would always be registered, but now you must explicitly ask for it.

root_interfaces¶

The root interfaces defined by the package.

Implementation:	`zope.configuration.fields.Tokens`
Read Only:	False
Required:	True
Default Value:	None
Allowed Type:	`list`

Value Type

Implementation:	`zope.configuration.fields.GlobalInterface`
Read Only:	False
Required:	True
Default Value:	None

Value Type

Implementation:	`zope.schema.InterfaceField`
Read Only:	False
Required:	True
Default Value:	None

factory_modules¶

If given, module names that should be searched for internalization factories.

If not given, all modules will be examined. If given, only these modules will be searched.

Implementation:	`zope.configuration.fields.Tokens`
Read Only:	False
Required:	False
Default Value:	None
Allowed Type:	`list`

Value Type

Implementation:	`zope.configuration.fields.GlobalObject`
Read Only:	False
Required:	True
Default Value:	None

register_legacy_search_module¶

Register found factories by their class name.

If true (not the default), then, in addition to registering factories by their mime type, also register them all by their class name. This is not recommended; currently no conflicts are caught and the order is ill-defined. See https://github.com/NextThought/nti.externalization/issues/33

Implementation:	`zope.configuration.fields.Bool`
Read Only:	False
Required:	False
Default Value:	False
Allowed Type:	`bool`

iobase¶

If given, a base class that will be used. You can customize aspects of externalization that way.

This class should descend from object, and it should implement the extension methods documented to customize AutoPackageSearchingScopedInterfaceObjectIO.

Implementation:	`zope.configuration.fields.GlobalObject`
Read Only:	False
Required:	False
Default Value:	None

modules¶

Module names that contain the implementations of the root_interfaces.

Implementation:	`zope.configuration.fields.Tokens`
Read Only:	False
Required:	True
Default Value:	None
Allowed Type:	`list`

Value Type

Implementation:	`zope.configuration.fields.GlobalObject`
Read Only:	False
Required:	True
Default Value:	None

interface IClassObjectFactoryDirective[source]¶

Defines the ext:classObjectFactory directive.

This directive registers a single nti.externalization.interfaces.IClassObjectFactory.

The factory will be registered for a class object.

factory¶

The class object that will be created.

This must define the __external_can_create__ attribute to be true.

Implementation:	`zope.configuration.fields.GlobalObject`
Read Only:	False
Required:	True
Default Value:	None

description¶

Description

Provides a description for the object.

Implementation:	`zope.configuration.fields.MessageID`
Read Only:	False
Required:	False
Default Value:	None
Allowed Type:	`unicode`

name¶

The name for the factory.

If not given, the __external_class_name__ of the class will be used. If that’s not available, the __name__ will be used.

Implementation:	`zope.configuration.fields.PythonIdentifier`
Read Only:	False
Required:	False
Default Value:	None
Allowed Type:	`str`

trusted¶

Ignore any value for __external_can_create__ on the factory.

Implementation:	`zope.configuration.fields.Bool`
Read Only:	False
Required:	False
Default Value:	False
Allowed Type:	`bool`

title¶

Title

Provides a title for the object.

Implementation:	`zope.configuration.fields.MessageID`
Read Only:	False
Required:	False
Default Value:	None
Allowed Type:	`unicode`

interface IBaseAnonymousObjectFactoryDirective[source]¶

The common parts of anonymous object factory directives.

New in version 1.0a3.

field¶

The name of the schema field

The factory results will be assigned to this field.

Implementation:	`zope.configuration.fields.PythonIdentifier`
Read Only:	False
Required:	True
Default Value:	None
Allowed Type:	`str`

for_¶

The interface that is the parent object this will be used for

Implementation:	`zope.configuration.fields.GlobalInterface`
Read Only:	False
Required:	True
Default Value:	None

Value Type

Implementation:	`zope.schema.InterfaceField`
Read Only:	False
Required:	True
Default Value:	None

interface IAnonymousObjectFactoryDirective[source]¶

Extends: nti.externalization.zcml.IBaseAnonymousObjectFactoryDirective

Defines the ext:anonymousObjectFactory directive.

This directive registers a single nti.externaliaztion.interfaces.IAnonymousObjectFactory for a single field used within a single object.

New in version 1.0a3.

trusted¶

Ignore any value for __external_can_create__ on the factory.

Implementation:	`zope.configuration.fields.Bool`
Read Only:	False
Required:	False
Default Value:	False
Allowed Type:	`bool`

factory¶

The class object that will be created.

Implementation:	`zope.configuration.fields.GlobalObject`
Read Only:	False
Required:	True
Default Value:	None

description¶

Description

Provides a description for the object.

Implementation:	`zope.configuration.fields.MessageID`
Read Only:	False
Required:	False
Default Value:	None
Allowed Type:	`unicode`

pass_external_object_to_factory¶

Pass the external object to the factory.

If true (not the default), then, the factory will recieve one argument, the anonymous external data. Otherwise, it gets no arguments.

Implementation:	`zope.configuration.fields.Bool`
Read Only:	False
Required:	False
Default Value:	False
Allowed Type:	`bool`

title¶

Title

Provides a title for the object.

Implementation:	`zope.configuration.fields.MessageID`
Read Only:	False
Required:	False
Default Value:	None
Allowed Type:	`unicode`

interface IAnonymousObjectFactoryInPlaceDirective[source]¶

Extends: nti.externalization.zcml.IBaseAnonymousObjectFactoryDirective

Defines the anonymousObjectFactoryInPlace directive.

This directive causes the external object itself to be used and updated in place. This is helpful when the object itself is not modelled, but its values are.

Such data might look like this:

{'home': {'MimeType': 'Address', ...},
 'work': {'MimeType': "Address', ...}}

And it might have a schema field that looks like this:

class IAddress(Interface):
    pass

class IHasAddresses(Interface):

    addresses = Dict(
             title=u"A mapping of address objects.",
             key_type=TextLine(title=u"Adresss key"),
             value_type=Object(IAddress))

The ZCML would then look like this:

<ext:anonymoustObjectFactoryInPlace
     for="IHasAddresses"
     field="addresses" />

New in version 1.0a3.

find_factories_in_module(module, case_sensitive=False)[source]¶

Look through the vars of module to find any eligible factory functions.

An eligible factory is a callable object with a True value for the attribute __external_can_create__.

If module is really a module, then only objects that are defined in that module will be found. Otherwise (module is some namespace object) any callable object is acceptable.

Parameters:	case_sensitive (bool) – If False (the default), then the results will have each factory twice, once with its found name, and once with its name lower cased.
Returns:	An iterable of found (name, factory).

Package IO¶

Support for handling the IO for all the objects in a package, typically via a ZCML directive.

class AutoPackageSearchingScopedInterfaceObjectIO(context, iface_upper_bound=None, validate_after_update=True)[source]¶

Bases: nti.externalization.datastructures.ModuleScopedInterfaceObjectIO

A special, magic, type of interface-driven input and output, one designed for the common use case of a package that provides the common pattern:

interfaces.py
externalization.py (where a subclass of this object lives)
configure.zcml (where the subclass is registered as an adapter for each object; you may also then provide mime-factories as well)
other modules, where types are defined for the external interfaces.

Once you derive from this class and implement the abstract methods, you need to call __class_init__() (exactly once) on your subclass.

You do not have to derive from this class; the common case is handled via the ZCML directive <ext:registerAutoPackageIO> (nti.externalization.zcml.IAutoPackageExternalizationDirective). You can still customize the behaviour by providing the iobase argument.

Parameters:

iface_upper_bound – The upper bound on the schema to use to externalize ext_self; we will use the most derived sub-interface of this interface that the object implements. Subclasses can either override this constructor to pass this parameter (while taking one argument themselves, to be usable as an adapter), or they can define the class attribute _ext_iface_upper_bound
validate_after_update (bool) – If True (the default) then the entire schema will be validated after an object has been updated with update_from_external_object(), not just the keys that were assigned.

classmethod __class_init__()[source]¶

Class initializer. Should be called exactly once on each distinct subclass.

First, makes all interfaces returned by _ap_enumerate_externalizable_root_interfaces() externalizable by setting the __external_class_name__ tagged value (to _ap_compute_external_class_name_from_interface_and_instance()). (See InterfaceObjectIO.)

Then, find all of the object factories and initialize them using _ap_find_factories(). A namespace object representing these factories is returned.

Changed in version 1.0: Registering the factories using register_legacy_search_module() is no longer done by default. If you are using this class outside of ZCML, you will need to subclass and override this method to make that call yourself. If you are using ZCML, you will need to set the appropriate attribute to True.

classmethod _ap_compute_external_class_name_from_concrete_class(a_type)[source]¶

Return the string value of the external class name.

By default this will be either the value of __external_class_name__ or, if not found, the value of __name__.

Subclasses may override.

classmethod _ap_compute_external_class_name_from_interface_and_instance(unused_iface, impl)[source]¶: Assigned as the tagged value __external_class_name__ to each interface. This will be called on an instance implementing iface.

See also

InterfaceObjectIO

classmethod _ap_compute_external_mimetype(package_name, unused_a_type, ext_class_name)[source]¶

Return the string value of the external mime type for the given type in the given package having the given external name (probably derived from _ap_compute_external_class_name_from_concrete_class()).

For example, given the arguments (‘nti.assessment’, FooBar, ‘FooBar’), the result will be ‘application/vnd.nextthought.assessment.foobar’.

Subclasses may override.

classmethod _ap_enumerate_externalizable_root_interfaces(interfaces)[source]¶

Return an iterable of the root interfaces in this package that should be externalized.

Subclasses must implement.

classmethod _ap_enumerate_module_names()[source]¶

Return an iterable of module names in this package that should be searched to find factories.

Subclasses must implement.

classmethod _ap_find_factories(package_name)[source]¶

Return a namespace object whose attribute names are external class names and whose attribute values are classes that can be created externally.

For each module returned by _ap_enumerate_module_names(), we will resolve it against the value of package_name (normally that given by _ap_find_package_name()). The module is then searched for classes that live in that module. If a class implements an interface that has a tagged value of __external_class_name__, it is added to the return value. The external class name (the name of the attribute) is computed by _ap_compute_external_class_name_from_concrete_class().

Each class that is found has an appropriate mimeType added to it (derived by _ap_compute_external_mimetype()), if it does not already have one; these classes also have the attribute __external_can_create__ set to true on them if they do not have a value for it at all. This makes the classes ready to be used with registerMimeFactories(), which is done automatically by the ZCML directive IAutoPackageExternalizationDirective.

Each class that is found is also marked as implementing zope.mimetype.interfaces.IContentTypeAware.

classmethod _ap_find_package_interface_module()[source]¶

Return the module that should be searched for interfaces.

By default, this will be the interfaces sub-module of the package returned from _ap_find_package_name().

Subclasses may override.

classmethod _ap_find_package_name()[source]¶

Return the package name to search for modules.

By default we look at the module name of the cls object given.

Subclasses may override.

classmethod _ap_find_potential_factories_in_module(module)[source]¶

Given a module that we’re supposed to examine, iterate over the types that could be factories.

This includes only types defined in that module. Any given type will only be returned once.

`nti.externalization`¶

The nti.externalization module has the top-level APIs.

to_external_object(obj, name=<default value>, registry=<default value>, catch_components=(), catch_component_action=None, request=<default value>, decorate=True, useCache=True, decorate_callback=<default value>, default_non_externalizable_replacer=<function DefaultNonExternalizableReplacer>, policy_name=<default value>, policy=<default value>)[source]¶

Translates the object into a form suitable for external distribution, through some data formatting process. See SEQUENCE_TYPES and MAPPING_TYPES for details on what we can handle by default.

Parameters:

name (string) – The name of the adapter to IInternalObjectExternalizer to look for. Defaults to the empty string (the default adapter). If you provide a name, and an adapter is not found, we will still look for the default name (unless the name you supply is None).
catch_components (tuple) – A tuple of exception classes to catch when externalizing sub-objects (e.g., items in a list or dictionary). If one of these exceptions is caught, then catch_component_action will be called to raise or replace the value. The default is to catch nothing.
catch_component_action (callable) – If given with catch_components, a function of two arguments, the object being externalized and the exception raised. May return a different object (already externalized) or re-raise the exception. There is no default, but catch_replace_action() is a good choice.
default_non_externalizable_replacer (callable) – If we are asked to externalize an object and cannot, and there is no INonExternalizableReplacer registered for it, then call this object and use the results.
request – If given, the request that the object is being externalized on behalf of. If given, then the object decorators will also look for subscribers to the object plus the request (like traversal adapters); this is a good way to separate out request or user specific code.
decorate_callback – Callable to be invoked in case there is no decaration
policy – The ExternalizationPolicy to use. Takes priority over policy_name. If this is not given (and neither is policy_name), the thread local state is consulted for the current policy established by the most recent caller to this method; if there is no such caller, then the DEFAULT_EXTERNALIZATION_POLICY is used.
policy_name (str) – If no policy is given, then this is used to lookup a utility. If this is used, the utility must exist.

to_external_representation(obj, ext_format='json', name=NotGiven) → str[source]¶

Transforms (and returns) the obj into its external (string) representation.

Uses nti.externalization.to_external_object(), passing in the name.

Parameters:	ext_format (str) – One of `EXT_REPR_JSON` or `EXT_REPR_YAML`, or the name of some other utility that implements `IExternalObjectRepresenter`

new_from_external_object(external_object, *args, **kwargs)[source]¶

Like update_from_external_object, but creates a new object to update using find_factory_for.

All remaining arguments are passed to update_from_external_object.

If no factory can be found, raises a zope.interface.interfaces.ComponentLookupError.

Returns the new object.

New in version 1.0a3.

update_from_external_object(containedObject, externalObject, context=None, require_updater=False, notify=True)[source]¶

Central method for updating objects from external values.

Parameters:

containedObject – The object to update.
externalObject – The object (typically a mapping or sequence) to update the object from. Usually this is obtained by parsing an external format like JSON.
context – An object passed to the update methods.
require_updater – If True (not the default) an exception will be raised if no implementation of IInternalObjectUpdater can be found for the containedObject.
notify (bool) – If True (the default), then if the updater for the containedObject either has no preference (returns None) or indicates that the object has changed, then an IObjectModifiedFromExternalEvent will be fired. This may be a recursive process so a top-level call to this object may spawn multiple events. The events that are fired will have a descriptions list containing one or more IAttributes each with attributes for each attribute we modify (assuming that the keys in the externalObject map one-to-one to an attribute; if this is the case and we can also find an interface declaring the attribute, then the IAttributes will have the right value for interface as well).
pre_hook (callable) – If given, called with the before update_from_external_object is called for every nested object. Signature f(k,x) where k is either the key name, or None in the case of a sequence and x is the external object. Deprecated.

Returns:

containedObject after updates from externalObject

Notifies IObjectModifiedFromExternalEvent for each object that is modified, and IObjectWillUpdateFromExternalEvent before doing so.

See also

INamedExternalizedObjectFactoryFinder

Changed in version 1.0.0a2: Remove the object_hook parameter.

Changed in version 1.1.3: Correctly file IObjectWillUpdateFromExternalEvent before updating each object.

Strings¶

`nti.externalization.integer_strings`: Short readable strings from large integers¶

Functions to represent potentially large integers as the shortest possible human-readable and writable strings. The motivation is to be able to take int ids as produced by an zc.intid.IIntId utility and produce something that can be written down and typed in by a human. To this end, the strings produced have to be:

One-to-one and onto the integer domain;
As short as possible;
While not being easily confused;
Or accidentaly permuted

To meet those goals, we define an alphabet consisting of the ASCII digits and upper and lowercase letters, leaving out troublesome pairs (zero and upper and lower oh and upper queue, one and upper and lower ell) (actually, those troublesome pairs will all map to the same character).

We also put a version marker at the end of the string so we can evolve this algorithm gracefully but still honor codes in the wild.

to_external_string(integer)[source]¶

Turn an integer into a native string representation.

>>> from nti.externalization.integer_strings import to_external_string
>>> to_external_string(123)
'xk'
>>> to_external_string(123456789)
'kVxr5'

from_external_string(key)[source]¶

Turn the string in key into an integer.

>>> from nti.externalization.integer_strings import from_external_string
>>> from_external_string('xkr')
6773

Parameters:	key (str) – A native string, as produced by `to_external_string`. (On Python 2, unicode keys are also valid.)
Raises:	ValueError – If the key is invalid or contains illegal characters. UnicodeDecodeError – If the key is a Unicode object, and contains non-ASCII characters (which wouldn’t be valid anyway)

`nti.externalization.representation`: Reading and writing objects to strings¶

External representation support.

The provided implementations of IExternalObjectIO live here. We provide and register two, one for JSON and one for YAML.

to_external_representation(obj, ext_format='json', name=NotGiven) → str[source]¶

Transforms (and returns) the obj into its external (string) representation.

Uses nti.externalization.to_external_object(), passing in the name.

Parameters:	ext_format (str) – One of `EXT_REPR_JSON` or `EXT_REPR_YAML`, or the name of some other utility that implements `IExternalObjectRepresenter`

to_json_representation(obj)[source]¶: A convenience function that calls to_external_representation() with EXT_REPR_JSON.

WithRepr(default=<function _default_repr>)[source]¶

A class decorator factory to give a __repr__ to the object. Useful for persistent objects.

Parameters:	default – A callable to be used for the default value.

Advanced Customization¶

`nti.externalization.extension_points`: Pluggable integrations¶

Extension points for integrating with other applications, frameworks, and libraries.

The normal extension mechanisms for this package are zope.component and zope.interface. The particular extension points found here are different for two reasons:

There is expected to be only one way that a given application will want to configure the extension point.
They are believed to be so performance critical to normal operations that the use of a component utility lookup would be noticeably detrimental.

For those two reasons, these extension points are both developed with zope.hookable, which provides a very low-overhead way to invoke a function while allowing for it to be extended. Applications that need to changed the behaviour of the built-in functions supplied here will need to call their zope.hookable.hookable.sethook() method at startup.

New in version 1.0.

get_current_request() → request¶

In a request/response system like a WSGI server, return an object representing the current request.

In some cases, this may be used to find adapters for objects. It is also passed to the toExternalObject function of each object as a keyword parameter.

In version 1.0, this will default to using Pyramid’s pyramid.threadlocal.get_current_request() if pyramid is installed. However, in a future version, an application wishing to use Pyramid’s request will explicitly need to set the hook.

Deprecated since version 1.0: The automatic fallback to Pyramid. It will be removed in 1.1 or before.

set_external_identifiers(self, result) → None¶

Place the stable external identifiers for self in the dictionary result.

By default, this function uses the result of to_external_oid as the value to put in result, under the keys OID as well as NTIID.

This is called from to_standard_external_dictionary, a default part of producing the external dictionary for all objects.

`nti.externalization.proxy`: Support for transparent proxies¶

Support for working with transparent proxies.

There are times during the externalization process (such as when computing object identifiers) that we need to be working with the “real” object, stripped of any security or other proxes placed around it. This module provides removeAllProxies for that purpose.

It is extensible with registerProxyUnwrapper.

removeAllProxies(proxy)[source]¶

If the object in proxy is proxied by one of the types of proxies known about by this module, remove all of the known proxies, unwrapping down to the original base object.

This module may know about zope.proxy, zope.container.contained, and Acquisition, if they are installed.

>>> from nti.externalization.proxy import removeAllProxies
>>> from zope.container.contained import ContainedProxy
>>> obj = object()
>>> proxy = ContainedProxy(obj)
>>> proxy == obj
True
>>> proxy is obj
False
>>> removeAllProxies(obj) is obj
True
>>> removeAllProxies(proxy) is obj
True

Changed in version 1.0: The default proxy unwrappers are all optional and will only be registered if they can be imported.

registerProxyUnwrapper(func)[source]¶: Register a function that can unwrap a single proxy from a proxied object. If there is nothing to unwrap, the function should return the given object.

New in version 1.0: This is a provisional way to extend the unwrapping functionality (where speed is critical). It may not be supported in the future.

ZODB Integration¶

`nti.externalization.oids`: Stable object identifiers¶

Functions for finding and parsing OIDs.

to_external_oid(self, default=None, add_to_connection=False, add_to_intids=False) → bytes[source]¶

For a persistent object, returns its persistent OID in a parseable external format (see from_external_oid()). This format includes the database name (so it works in a ZODB multi-database) and the integer ID from the closest zope.intid.interfaces.IIntIds utility.

If the object implements a method toExternalOID(), that method will be called and its result (or the default) will be returned. This should generally be considered legacy behaviour.

If the object has not been saved, and add_to_connection is False (the default) returns the default.

Parameters:

add_to_connection (bool) – If the object is persistent but not yet added to a connection, setting this to true will attempt to add it to the nearest connection in its containment tree, thus letting it have an OID.
add_to_intids (bool) – If we can obtain an OID for this object, but it does not have an intid, and an intid utility is available, then if this is True (not the default) we will register it with the utility.

Returns:

A bytes string.

from_external_oid(ext_oid)[source]¶

Given a byte string, as produced by to_external_oid(), parses it into its component parts.

Parameters:	ext_oid (bytes) – As produced by `to_external_oid()`. (Text/unicode is also accepted.)
Returns:	A three-tuple, `ParsedOID`. Only the OID is guaranteed to be present; the other fields may be empty (`db_name`) or `None` (`intid`).

class ParsedOID(oid, db_name, intid)¶

Bases: tuple

The fields of a parsed OID: oid, db_name and intid

db_name¶: Alias for field number 1

intid¶: Alias for field number 2

oid¶: Alias for field number 0

`nti.externalization.persistence`: Pickling and ZODB support¶

Classes and functions for dealing with persistence (and pickling) in an external context.

Note

Importing this module monkey-patches the class persistent.wref.WeakRef to directly implement toExternalObject() and toExternalOID(). It’s not clear why we don’t simply register class adapters for those things, but we don’t.

getPersistentState(obj)[source]¶

For a persistent.Persistent object, returns one of the constants from the persistent module for its state: persistent.CHANGED and persistent.UPTODATE being the most useful.

If the object is not Persistent and doesn’t implement a getPersistentState method, this method will be pessimistic and assume the object has been persistent.CHANGED.

setPersistentStateChanged(obj)[source]¶: Explicitly marks a persistent object as changed.

class PersistentExternalizableDictionary(**kwargs)[source]¶

Bases: persistent.mapping.PersistentMapping, nti.externalization.datastructures.ExternalizableDictionaryMixin

Dictionary mixin that provides toExternalDictionary() to return a new dictionary with each value in the dict having been externalized with toExternalObject().

Changed in version 1.0: No longer extends nti.zodb.persistentproperty.PersistentPropertyHolder. If you have subclasses that use writable properties and which should bypass the normal attribute setter implementation, please mixin this superclass (first) yourself.

toExternalDictionary(*args, **kwargs)[source]¶

Produce the standard external dictionary for this object.

Uses _ext_replacement.

class PersistentExternalizableList(initlist=None)[source]¶

Bases: persistent.list.PersistentList

List mixin that provides toExternalList() to return a new list with each element in the sequence having been externalized with toExternalObject().

Changed in version 1.0: No longer extends nti.zodb.persistentproperty.PersistentPropertyHolder. If you have subclasses that use writable properties and which should bypass the normal attribute setter implementation, please mixin this superclass (first) yourself.

values()[source]¶: For compatibility with zope.generations.utility, this object defines a values method which does nothing but return itself. That makes these objects transparent and suitable for migrations.

class PersistentExternalizableWeakList(initlist=None)[source]¶

Bases: nti.externalization.persistence.PersistentExternalizableList

Stores persistent.Persistent objects as weak references, invisibly to the user. Any weak references added to the list will be treated the same.

Weak references are resolved on access; if the referrant has been deleted, then that access will return None.

Changed in version 1.0: No longer extends nti.zodb.persistentproperty.PersistentPropertyHolder. If you have subclasses that use writable properties and which should bypass the normal attribute setter implementation, please mixin this superclass (first) yourself.

append(item)[source]¶: S.append(object) – append object to the end of the sequence

count(value) → integer -- return number of occurrences of value[source]¶

extend(other)[source]¶: S.extend(iterable) – extend sequence by appending elements from the iterable

index(value) → integer -- return first index of value.[source]¶: Raises ValueError if the value is not present.

insert(i, item)[source]¶: S.insert(index, object) – insert object before index

pop([index]) → item -- remove and return item at index (default last).[source]¶: Raise IndexError if list is empty or index is out of range.

remove(item)[source]¶: S.remove(value) – remove first occurrence of value. Raise ValueError if the value is not present.

NoPickle(cls)[source]¶

A class decorator that prevents an object from being pickled. Useful for ensuring certain objects do not get pickled and thus avoiding ZODB backward compatibility concerns.

Warning

Subclasses of such decorated classes are also not capable of being pickled, without appropriate overrides of __reduce_ex__ and __getstate__. A “working” subclass, but only for ZODB, looks like this:

@NoPickle
class Root(object):
    pass

class CanPickle(Persistent, Root):
    pass

Warning

This decorator emits a warning when it is used directly on a Persistent subclass, as that rather defeats the point (but it is still allowed for backwards compatibility).

Helpers¶

`nti.externalization.factory`: Object factories¶

Implementations of object factories.

New in version 1.0.

class ObjectFactory(callable=None, title='', description='', interfaces=None)[source]¶

Bases: zope.component.factory.Factory

A convenient zope.component.interfaces.IFactory meant to be registered as a named utility.

The callable object SHOULD be a type/class object, because that’s the only thing we base equality off of (class identity).

This class can be subclassed and trivially instantiated by setting class properties default_factory and (optionally) default_title and default_description. Constructor arguments will override these, but if not given, the class values will be used.

For example:

>>> class MyObjectFactory(ObjectFactory):
...    default_factory = object
...    default_title = 'A Title'
...    default_description = 'A Description'
>>> factory = MyObjectFactory()
>>> factory 
<MyObjectFactory titled 'A Title' using <... 'object'> producing []>
>>> print(factory.title)
A Title
>>> print(factory.description)
A Description

default_description = u''¶: The default description, if none is given to the constructor.

default_factory = None¶: The default callable argument, if none is given to the constructor.

default_title = u''¶: The default title, if none is given to the constructor.

class MimeObjectFactory(callable=None, title='', description='', interfaces=None)[source]¶

Bases: nti.externalization.factory.ObjectFactory

Default implementation of IMimeObjectFactory.

class ClassObjectFactory(callable=None, title='', description='', interfaces=None)[source]¶

Bases: nti.externalization.factory.ObjectFactory

Default implementation of IClassObjectFactory.

class AnonymousObjectFactory(callable=None, title='', description='', interfaces=None)[source]¶

Bases: nti.externalization.factory.ObjectFactory

Default implementation of IAnonymousObjectFactory.

New in version 1.0a3.

Datetime¶

Support for reading, writing and converting date and time related objects.

See the datetime module, as well as the zope.interface.common.idatetime module for types of objects.

These are generally meant to be used as zope.interface adapters once this package has been configured, but they can be called manually as well.

class date_to_string(date)[source]¶

Bases: object

Produce an IOS8601 string from a date.

Registered as an adapter from zope.interface.common.idatetime.IDate to IInternalObjectExternalizer.

>>> import datetime
>>> from nti.externalization.externalization import to_external_object
>>> from nti.externalization.datetime import date_to_string
>>> from zope import component
>>> component.provideAdapter(date_to_string)
>>> to_external_object(datetime.date(1982, 1, 31))
'1982-01-31'

date_from_string(string)[source]¶

This adapter allows any field which comes in as a string in IOS8601 format to be transformed into a date. The schema field must be an zope.schema.Object field with a type of zope.interface.common.idatetime.IDate.

If you need a schema field that accepts human input, rather than programattic input, you probably want to use a custom field that uses zope.datetime.parse() in its fromUnicode method.

>>> from nti.externalization.datetime import date_from_string
>>> date_from_string('1982-01-31')
datetime.date(1982, 1, 31)

class datetime_to_string(date)[source]¶

Bases: object

Produce an IOS8601 string from a datetime.

Registered as an adapter from zope.interface.common.idatetime.IDateTime to IInternalObjectExternalizer.

>>> from zope.component import provideAdapter
>>> import datetime
>>> from nti.externalization import to_external_object
>>> from nti.externalization.datetime import datetime_to_string
>>> provideAdapter(datetime_to_string)
>>> to_external_object(datetime.datetime(1982, 1, 31))
'1982-01-31T00:00:00Z'

datetime_from_string(string, assume_local=False, local_tzname=None)[source]¶

This adapter allows any field which comes in as a string in IOS8601 format to be transformed into a datetime.datetime. The schema field should be an nti.schema.field.Object field with a type of zope.interface.common.idatetime.IDateTime or an instance of nti.schema.field.ValidDateTime. Wrap this with an nti.schema.fieldproperty.AdaptingFieldProperty.

Datetime values produced by this object will always be in GMT/UTC time, and they will always be datetime naive objects.

If you need a schema field that accepts human input, rather than programattic input, you probably want to use a custom field that uses zope.datetime.parse() in its fromUnicode method.

When used as an adapter, no parameters are accepted.

>>> from zope.interface.common.idatetime import IDateTime
>>> from zope.component import provideAdapter
>>> from nti.externalization.datetime import datetime_from_string
>>> provideAdapter(datetime_from_string)
>>> IDateTime('1982-01-31T00:00:00Z')
datetime.datetime(1982, 1, 31, 0, 0)

Parameters:

assume_local (bool) – If False, the default, then when we parse a string that does not include timezone information, we will assume that it is already meant to be in UTC. Otherwise, if set to true, when we parse such a string we will assume that it is meant to be in the “local” timezone and adjust accordingly. If the local timezone experiences DST, then the time will be interpreted with the UTC offset as-of the DST rule in effect on the date parsed, not the current date, if possible. If not possible, the current rule will be used.
local_tzname (str) – If given, either a string acceptable to pytz.timezone() to produce a tzinfo object, or a two-tuple as given from time.timezone. If not given, local timezone will be determined automatically.

datetime_from_timestamp(value)[source]¶

Produce a datetime.datetime from a UTC timestamp.

This is a registered adapter for both integers and floats.

>>> from zope.interface.common.idatetime import IDateTime
>>> from zope.component import provideAdapter
>>> from nti.externalization.datetime import datetime_from_timestamp
>>> provideAdapter(datetime_from_timestamp, (int,))
>>> provideAdapter(datetime_from_timestamp, (float,))
>>> IDateTime(123456)
datetime.datetime(1970, 1, 2, 10, 17, 36)
>>> IDateTime(654321.0)
datetime.datetime(1970, 1, 8, 13, 45, 21)

class duration_to_string(date)[source]¶

Bases: object

Produce an IOS8601 format duration from a datetime.timedelta object.

Timedelta objects do not represent years or months (the biggest duration they accept is weeks) and internally they normalize everything to days and smaller. Thus, the format produced by this transformation will never have a field larger than days.

Registered as an adapter from zope.interface.common.idatetime.ITimeDelta to IInternalObjectExternalizer.

>>> import datetime
>>> from zope.component import provideAdapter
>>> from nti.externalization import to_external_object
>>> from nti.externalization.datetime import duration_to_string
>>> provideAdapter(duration_to_string)
>>> to_external_object(datetime.timedelta(weeks=16))
'P112D'

duration_from_string(value)[source]¶

Produce a datetime.timedelta from a ISO8601 format duration string.

>>> from zope.interface.common.idatetime import ITimeDelta
>>> from zope.component import provideAdapter
>>> from nti.externalization.datetime import duration_from_string
>>> provideAdapter(duration_from_string)
>>> ITimeDelta('P0D')
datetime.timedelta(0)

Dublincore¶

Externalization support for things that implement the interfaces of zope.dublincore.interfaces.

Note

We are “namespacing” the dublincore properties, since they have defined meanings we don’t control. We are currently doing this by simply prefixing them with ‘DC’ for ease of access in JavaScript.

These objects are typically used as decorators, registered from ZCML.

class DCExtendedExternalMappingDecorator[source]¶

Bases: nti.externalization.singleton.Singleton

Adds the extended properties of dublincore to external objects as defined by zope.dublincore.interfaces.IDCExtended.

Note

We are currently only mapping ‘Creator’ since that’s the only field that ever gets populated.

Implements IExternalStandardDictionaryDecorator for zope.dublincore.interfaces.IDCExtended objects.

class DCDescriptivePropertiesExternalMappingDecorator[source]¶

Bases: nti.externalization.singleton.Singleton

Supports the ‘DCTitle’ and ‘DCDescription’ fields, as defined in zope.dublincore.interfaces.IDCDescriptiveProperties.

Implements IExternalStandardDictionaryDecorator.

Singleton¶

Support for fast, memory efficient singleton objects.

Why is this here? The externalization system can use lots of objects as it goes through its process. Many of those objects are adapters (for example, the decorator objectes), meaning a factory callable is called to (create and) return an object (given a particular context, and possibly request).

But the API of many adapter objects accept all the information they need to have in the functions defined in the interface. That is, the constructor does nothing useful with the context (and request). The objects are stateless, and so constructing a new one for each adapter invocation can be a waste of time and memory.

By either using the SingletonMetaclass as your metaclass, or subclassing Singleton, that cost is paid only once, replacing a call to a constructor and an object allocation with a faster call to return a constant object.

class SingletonMetaclass[source]¶

Bases: type

Metaclass for singleton classes most commonly used as external object decorators (adapters). These objects accept one or two context arguments to their __init__ function, but they do not actually use them (because the same object is passed to their decoration method). Thus they can usefully be constructed just once and shared. This metaclass ensures the singleton property, ensures that the __init__ method is a no-op, and ensures that the instance has no dictionary or instance variable.

A singleton class has only one instance which is returned every time the class is instantiated.

Caution

We cannot be used with six.with_metaclass() because it introduces temporary classes. You’ll need to use the metaclass constructor directly:

AClass = SingletonMetaclass('AClass', (object,), {})

Alternatively, you can inherit from Singleton.

Implementation Notes

The class is instantiated immediately at the point where it is defined by calling cls.__new__(cls). This instance is cached and cls.__new__ is rebound to return it directly.

The original constructor is also cached to allow subclasses to access it and have their own instance.

>>> from nti.externalization.singleton import Singleton
>>> class TopSingleton(Singleton):
...    def __init__(self):
...        print("I am never called")
>>> inst = TopSingleton()
>>> isinstance(inst, TopSingleton)
True
>>> TopSingleton() is inst
True
>>> class DerivedSingelton(TopSingleton):
...     pass
>>> derived = DerivedSingelton()
>>> isinstance(derived, DerivedSingelton)
True
>>> DerivedSingelton() is derived
True
>>> derived is inst
False

class Singleton¶

Bases: object

A base class for singletons. Can be more convenient than a metaclass for Python2/Python3 compatibility.

nti.externalization¶

Introduction and Basics¶

Basic Usage¶

Motivation and Use-cases¶

Getting Started¶

That’s Not Good Enough¶

Adapters and Configuration¶

Using Schemas for Validation and Automatic Externalization¶

autoPackageIO: Handing responsibility to the framework¶

Representations¶

Schemas¶

Externalization¶

Standard Fields¶

Decorating¶

IExternalStandardDictionaryDecorator¶

Dublin Core Metadata¶

Internalization¶

Factories¶

Special Attributes¶

Glossary¶

Changes¶

2.3.1 (unreleased)¶

2.3.0 (2021-08-02)¶

2.2.0 (2021-04-14)¶

2.1.0 (2020-08-03)¶

2.0.0 (2020-07-02)¶

1.1.3 (2020-06-25)¶

1.1.2 (2020-04-07)¶

1.1.1 (2020-03-27)¶

1.1.0 (2020-03-27)¶

1.0.0 (2020-03-19)¶

1.0.0a14 (2019-11-13)¶

1.0.0a13 (2018-09-20)¶

1.0.0a12 (2018-09-11)¶

1.0.0a11 (2018-08-29)¶

1.0.0a10 (2018-08-21)¶

1.0.0a9 (2018-08-20)¶

1.0.0a8 (2018-08-16)¶

1.0.0a7 (2018-07-31)¶

1.0.0a6 (2018-07-31)¶

1.0.0a5 (2018-07-30)¶

1.0.0a4 (2018-07-30)¶

1.0.0a3 (2018-07-28)¶

1.0.0a2 (2018-07-05)¶

1.0.0a1 (2017-09-29)¶

API Details¶

API Details¶

Basics¶

nti.externalization.interfaces: Interfaces and constants¶

Deprecated Names¶

Externalization¶

Internalization¶

Datastructures¶

ZCML¶

Package IO¶

nti.externalization¶

Strings¶

nti.externalization.integer_strings: Short readable strings from large integers¶

nti.externalization.representation: Reading and writing objects to strings¶

Advanced Customization¶

nti.externalization.extension_points: Pluggable integrations¶

nti.externalization.proxy: Support for transparent proxies¶

ZODB Integration¶

nti.externalization.oids: Stable object identifiers¶

nti.externalization.persistence: Pickling and ZODB support¶

Helpers¶

nti.externalization.factory: Object factories¶

Datetime¶

Dublincore¶

Singleton¶

Indices and tables¶

`nti.externalization.interfaces`: Interfaces and constants¶

`nti.externalization`¶

`nti.externalization.integer_strings`: Short readable strings from large integers¶

`nti.externalization.representation`: Reading and writing objects to strings¶

`nti.externalization.extension_points`: Pluggable integrations¶

`nti.externalization.proxy`: Support for transparent proxies¶

`nti.externalization.oids`: Stable object identifiers¶

`nti.externalization.persistence`: Pickling and ZODB support¶

`nti.externalization.factory`: Object factories¶