Create classes at runtnime

[Marc Aymerich]

Hi!
I need to create a pretty complex class at runtime. something like
this one:

(note: "...." means that the number of attributes can be variable)

class VirtualUserLimitForm(ModelForm):
swap_limit = forms.CharField(max_length=100,
initial=monitor1.default_limit)
memory_limit = forms.CharField(max_length=100,
initial=monitor2.default_limit)
...

class Meta:
model = model

def __init__(self, *args, **kwargs):
super(VirtualUserLimitForm, self).__init__(*args, **kwargs)
if 'instance' in kwargs:
self.fields['swap_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor1)[0].current
self.fields['memory_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor2)[0].current
...

I can generate all the needed code as string and then use exec(), but
it seems ugly to me. I'm wondering if there is another way more
elegant to do that? metaclasses maybe? What is your recommendation?

Thanks!!

 

[Peter Otten]

I need to create a pretty complex class at runtime. something like
this one:

I have a hunch that you've never heard the famous Kernighan quote:

"Everyone knows that debugging is twice as hard as writing a program in the
first place. So if you're as clever as you can be when you write it, how
will you ever debug it?"

Or that if you've heard it you don't heed it.

(note: "...." means that the number of attributes can be variable)

class VirtualUserLimitForm(ModelForm):
swap_limit = forms.CharField(max_length=100,
initial=monitor1.default_limit)
memory_limit = forms.CharField(max_length=100,
initial=monitor2.default_limit)
...

class Meta:
model = model

def __init__(self, *args, **kwargs):
super(VirtualUserLimitForm, self).__init__(*args, **kwargs)
if 'instance' in kwargs:
self.fields['swap_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor1)[0].current
self.fields['memory_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor2)[0].current
...

I can generate all the needed code as string and then use exec(), but
it seems ugly to me. I'm wondering if there is another way more
elegant to do that? metaclasses maybe?

The metaclass does indeed take a dictionary argument where you can provide
class attributes, e. g.:
(1, 42)

What is your recommendation?

Find something that is simple and robust. Something else.

Peter

 

[Carl Banks]

I need to create a pretty complex class at runtime. something like
this one:

I have a hunch that you've never heard the famous Kernighan quote:

"Everyone knows that debugging is twice as hard as writing a program in the
first place. So if you're as clever as you can be when you write it, how
will you ever debug it?"

Or that if you've heard it you don't heed it.
[snip]

What is your recommendation?

Find something that is simple and robust. Something else.

I've done things like this before, and I would say the Kernigan quote
is not appropriate here, and the implication that isn't not simple and
robust is incorrect. It's repetitive code like the OP posted that's
complex and flimsy. Repetitive code is the exact opposite of the DRY
principle, and there's a reason DRY is one of the most often cited
principles here.

Using advanced techniques like metaclasses to exploit the similarity
to reduce or eliminate the repetitiveness can, if done well, simplify
code. By a lot.

Advanced is not the same thing as complex.


In this particular case I'd say the recommendation to do something
else is a good one, but it's not true in general.


Carl Banks

 

[Carl Banks]

Hi!
I need to create a pretty complex class at runtime. something like
this one:

(note: "...." means that the number of attributes can be variable)

class VirtualUserLimitForm(ModelForm):
    swap_limit = forms.CharField(max_length=100,
initial=monitor1.default_limit)
    memory_limit = forms.CharField(max_length=100,
initial=monitor2.default_limit)
    ...

    class Meta:
        model = model

    def __init__(self, *args, **kwargs):
        super(VirtualUserLimitForm, self).__init__(*args, **kwargs)
        if 'instance' in kwargs:
            self.fields['swap_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor1)[0].current
            self.fields['memory_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor2)[0].current
            ...

I can generate all the needed code as string and then use exec(), but
it seems ugly to me. I'm wondering if there is another way more
elegant to do that?  metaclasses maybe? What is your recommendation?

I'd recommend against using metaclasses (in the normal way) with
Django ORM, since it (and pretty much all ORMs) already makes heavy
use of metaclasses, and I doubt those metaclasses were designed to
share metaclassing duties.

At a minimum you'd have to derive your own metaclasses from Djangos,
and that's not likely to work out well at all.

The right way to do this might be to reorganize your database. It
seems like you have all kinds of tables with different sets of
columns, and the tables themselves are dynamic? Like if you get some
data somewhere and it doesn't fit into an existing schema you make a
new table for it? If that's the case then you'd be better off,
perhaps, to reorganize the tables into some kind of association list.
Put the "relational" aspect to use. (Left as an exercise for now but
if you want pointers feel free to follow up.)

But maybe you have no control of the tables; you simply have a lot of
them and want to cut down on repetition. Then I think a factory
function that builds a dict and passes it to type() would be
reasonable. See the documentation of type() for explanation of
arguments.

def create_user_form(name,fields,_model):
class Meta:
model = _model
dct = { 'Meta': Meta }
for field in fields:
dct[field] = forms.CharField(max_length=100,

initial=monitor1.default_limit)

return type(name,(ModelForm,),dct)


Carl Banks

 

[Ian]

(note: "...." means that the number of attributes can be variable)

class VirtualUserLimitForm(ModelForm):
    swap_limit = forms.CharField(max_length=100,
initial=monitor1.default_limit)
    memory_limit = forms.CharField(max_length=100,
initial=monitor2.default_limit)
    ...

    class Meta:
        model = model

    def __init__(self, *args, **kwargs):
        super(VirtualUserLimitForm, self).__init__(*args, **kwargs)
        if 'instance' in kwargs:
            self.fields['swap_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor1)[0].current
            self.fields['memory_limit'].initial =
kwargs['instance'].monitoring.filter(monitor=monitor2)[0].current
            ...

I can generate all the needed code as string and then use exec(), but
it seems ugly to me. I'm wondering if there is another way more
elegant to do that?  metaclasses maybe?

The metaclass does indeed take a dictionary argument where you can provide
class attributes, e. g.:

(1, 42)

This approach will basically work, but since you're subclassing the
Django ModelForm, note that it already has its own metaclass:
<class 'django.forms.models.ModelFormMetaclass'>

So you will need to either subclass ModelFormMetaclass for your
metaclass and override the __init__ method, or modify what Peter
posted like so:

Cheers,
Ian

 

[Steven D'Aprano]

Hi!
I need to create a pretty complex class at runtime. something like this
one:

(note: "...." means that the number of attributes can be variable)

class VirtualUserLimitForm(ModelForm):
swap_limit = forms.CharField(max_length=100,
initial=monitor1.default_limit)
memory_limit = forms.CharField(max_length=100,
initial=monitor2.default_limit)
...
[...]
I can generate all the needed code as string and then use exec(), but it
seems ugly to me. I'm wondering if there is another way more elegant to
do that? metaclasses maybe? What is your recommendation?

Just add in the class attributes you want after creating the class.

class VirtualUserLimitForm(ModelForm):
pass


f = forms.CharField # alias for brevity
for name, value in [
('swap_limit', f(max_length=100, initial=monitor1.default_limit)),
('memory_limit', f(max_length=100, initial=monitor2.default_limit)),
('spam', 'spam spam spam spam'),
('eggs', 'fried sunny side up'),
]:
setattr(VirtualUserLimitForm, name, value)


There's probably a name for this sort of technique, and an entire chapter
in "Design Patterns For People With Too Much Time On Their Hands" (15th
edition), but I don't remember it

 

[Ian]

Just add in the class attributes you want after creating the class.

class VirtualUserLimitForm(ModelForm):
    pass

f = forms.CharField  # alias for brevity
for name, value in [
    ('swap_limit', f(max_length=100, initial=monitor1.default_limit)),
    ('memory_limit', f(max_length=100, initial=monitor2.default_limit)),
    ('spam', 'spam spam spam spam'),
    ('eggs', 'fried sunny side up'),
    ]:
    setattr(VirtualUserLimitForm, name, value)

This would probably cause some problems with the Django metaclass. It
inspects the class dictionary for fields at the time the class in
constructed, and so it would miss any fields added after construction
time.

 

[Steven D'Aprano]

Just add in the class attributes you want after creating the class.

[...]
This would probably cause some problems with the Django metaclass. It
inspects the class dictionary for fields at the time the class in
constructed, and so it would miss any fields added after construction
time.

What Django metaclass? The OP didn't say anything about using Django.

 

[Chris Rebert]

Just add in the class attributes you want after creating the class.

[...]
This would probably cause some problems with the Django metaclass.  It
inspects the class dictionary for fields at the time the class in
constructed, and so it would miss any fields added after construction
time.

What Django metaclass? The OP didn't say anything about using Django.

His code mentions "forms.CharField" and declares a nested class named
"Meta"; this is indicative of Django.

Cheers,
Chris

 

[Marc Aymerich]

Thank you all for the answers!

I'll try to give you the context in which I need to generate classes
like this.

I'm developing a reusable control panel for an ISP. I have several
modules (VirtualUser, SystemUser, VPS, VirtualHost, ...) and they all
share the fact that you can have limits on resource consumption. for
example.

VPS: has limits on swap, memory, cpu and disk.
SystemUser: has limits on disk and traffic.

As all modules share this property of being limited I have decided to
split the limit functionality in a separate module (called resources).
Resources are associated dinamically with (vps, systemuser ...)
through a Monitor class. So it's up to you decide what limits you are
interested to put in the VPS or in the SystemUser ...

So when you attach the "disk limit" to SystemUser model is necessary
to create a web form for SystemUser a bit different than if you decide
to have disk and traffic limits. That is what my posted code is
supposed to do.

Do you think that I am on the wrong way designing the control panel
like this?

Btw, I've seen several interesting ideas to create the class
dinamically (with class factorys, MetaClass, dictionaries ..), but I
have not yet clear whether this will be usefull for create the
__init__ function at runtime too. Any clue on this?

Many, many thanks!!
Marc

 

[Peter Otten]

I need to create a pretty complex class at runtime. something like
this one:

I have a hunch that you've never heard the famous Kernighan quote:

"Everyone knows that debugging is twice as hard as writing a program in
the first place. So if you're as clever as you can be when you write it,
how will you ever debug it?"

Or that if you've heard it you don't heed it.
[snip]

What is your recommendation?

Find something that is simple and robust. Something else.

I've done things like this before, and I would say the Kernigan quote
is not appropriate here, and the implication that isn't not simple and
robust is incorrect. It's repetitive code like the OP posted that's
complex and flimsy. Repetitive code is the exact opposite of the DRY
principle, and there's a reason DRY is one of the most often cited
principles here.

Using advanced techniques like metaclasses to exploit the similarity
to reduce or eliminate the repetitiveness can, if done well, simplify
code. By a lot.

Advanced is not the same thing as complex.


In this particular case I'd say the recommendation to do something
else is a good one, but it's not true in general.

The trouble with metaclasses is that they are not completely transparent.
Instead of two levels (class and instance) you have to consider three.
Steven D'Aprano has involuntarily given an example of what may go wrong.

Marc has asked another question with a problem that could be attacked with a
metaclass before. Had he been comfortable with metaclasses by now we
wouldn't have seen the above post. So he is operating at the limits of his
current understanding of Python in an area where it is not unlikely that
he'll run into unexpected side effects of the django database wrapper.

I think a reminder that he has to debug the whole thing is in order

Peter

 

[Marc Aymerich]

Thank you all for the answers!

I'll try to give you the context in which I need to generate classes
like this.

I'm developing a reusable control panel for an ISP. I have several
modules (VirtualUser, SystemUser, VPS, VirtualHost, ...) and they all
share the fact that you can have limits on resource consumption. for
example.

VPS: has limits on swap, memory, cpu and disk.
SystemUser: has limits on disk and traffic.

As all modules share this property of being limited I have decided to
split the limit functionality in a separate module (called resources).
Resources are associated dinamically with (vps, systemuser ...)
through a Monitor class. So it's up to you decide what limits you are
interested to put in the VPS or in the SystemUser ...

So when you attach the "disk limit" to SystemUser model is necessary
to create a web form for SystemUser a bit different than if you decide
to have disk and traffic limits. That is what my posted code is
supposed to do.

Do you think that I am on the wrong way designing the control panel
like this?

Btw, I've seen several interesting ideas to create the class
dinamically (with class factorys, MetaClass, dictionaries ..), but I
have not yet clear whether this will be usefull for create the
__init__ function at runtime too. Any clue on this?

this seems to work

def makeLimitForm(name, monitors, _model):

class Meta:
model = _model
dct = { 'Meta': Meta }

for monitor in monitors:
field_name = monitor.resource + "_limit"
dct[field_name] = forms.CharField(max_length=100,
initial=monitor.default_limit)

def __init__(self, *args, **kwargs):
ModelForm.__init__(self, *args, **kwargs)
if 'instance' in kwargs:
for monitor in monitors:
field_name = monitor.resource + "_limit"
print monitor
self.fields[field_name].initial =
kwargs['instance'].monitoring.filter(monitor=monitor)[0].current

dct['__init__'] = __init__

return type(name,(ModelForm,),dct)

 

[Peter Otten]

Thank you all for the answers!

I'll try to give you the context in which I need to generate classes
like this.

I'm developing a reusable control panel for an ISP. I have several
modules (VirtualUser, SystemUser, VPS, VirtualHost, ...) and they all
share the fact that you can have limits on resource consumption. for
example.

VPS: has limits on swap, memory, cpu and disk.
SystemUser: has limits on disk and traffic.

As all modules share this property of being limited I have decided to
split the limit functionality in a separate module (called resources).
Resources are associated dinamically with (vps, systemuser ...)
through a Monitor class. So it's up to you decide what limits you are
interested to put in the VPS or in the SystemUser ...

So when you attach the "disk limit" to SystemUser model is necessary
to create a web form for SystemUser a bit different than if you decide
to have disk and traffic limits. That is what my posted code is
supposed to do.

Do you think that I am on the wrong way designing the control panel
like this?

Btw, I've seen several interesting ideas to create the class
dinamically (with class factorys, MetaClass, dictionaries ..), but I
have not yet clear whether this will be usefull for create the
__init__ function at runtime too. Any clue on this?

this seems to work

def makeLimitForm(name, monitors, _model):

class Meta:
model = _model
dct = { 'Meta': Meta }

for monitor in monitors:
field_name = monitor.resource + "_limit"
dct[field_name] = forms.CharField(max_length=100,
initial=monitor.default_limit)

def __init__(self, *args, **kwargs):
ModelForm.__init__(self, *args, **kwargs)
if 'instance' in kwargs:
for monitor in monitors:
field_name = monitor.resource + "_limit"
print monitor
self.fields[field_name].initial =
kwargs['instance'].monitoring.filter(monitor=monitor)[0].current

dct['__init__'] = __init__

return type(name,(ModelForm,),dct)

I've looked around to see whether Django offers an API for your usecase, but
only found

http://code.djangoproject.com/wiki/DynamicModels

which seems to describe what you are already doing.