use of __new__ to permit "dynamic" completion within (any?) IDE ?

[gst]

Hi,

I met a situation where I was passing an object created in/with an
upper level module class to a lower level module class' instance in
one of its __init__ argument and saving a ref of the upper object in
that lower level class' new instance.

But in my IDE I want the completion to also work from within the lower
level module when it's refering to the object passed from the upper
level:


Well, I'm sure I'm not very clear in my terms (and probably a bit long
in the sentence) so here it is in code:


files:
module1.py
subpackage/module2.py


file module1.py:

from subpackage.module2 import class2

class class1(object):

def __new__(cls, _self=None, *args, **kwargs):
if _self: ## we've been passed an instance already
initialyzed
## so directly return it instead of creating a
new object.
return _self
return object.__new__(cls)

def __init__(self, _self=None, *args, **kwargs):
if _self: ## we've been passed an instance already
initialyzed
## so directly returns
## assert(self is _self) ?
return
self.object2 = class2(object1=self, "blip", "blop")
# others init


file module2.py:

class class2(object):

def __init__(self, object1, *args, **kwargs):

from ..module1 import class1

self.object1 = class1(_self=object1) ## instead of:
self.object1 = object1

## others functions and/or init..
## where now I've completion working on self.object1 :
## if I add(or remove) fields/methods in module1 (and save) then
## I have them available(or disappeared) in the completion when
executed from this submodule.
## This ofcourse permits to save me of remembering all of the
class1 attributes/methods when I'm working with self.object1 from
within one of class2 methods.


What do you think of this ?

I guess there can be others ways of doing this.. ?

Thanks,

Regards,

Greg.

note: I'm using Eclipse 3.5.2 with pydev so (I don't really know how
others IDE can handle this case) .

 

[Steven D'Aprano]

Hi,

I met a situation where I was passing an object created in/with an upper
level module class to a lower level module class' instance in one of its
__init__ argument and saving a ref of the upper object in that lower
level class' new instance.

But in my IDE I want the completion to also work from within the lower
level module when it's refering to the object passed from the upper
level:

"The completion"? What do you mean?

Well, I'm sure I'm not very clear in my terms (and probably a bit long
in the sentence) so here it is in code:

I'm afraid I have to agree with your, your description is not clear to me.

Unfortunately, neither is your code, because I don't understand *why* you
do the things you do.

files:
module1.py
subpackage/module2.py


file module1.py:

from subpackage.module2 import class2

Is it relevant that class2 comes from another module? Would your concept
work if class1 and class2 were defined in the same file?

By the way, it's conventional (but not compulsory) to name classes with
an initial capital letter, and instances in all lowercase.

class class1(object):

def __new__(cls, _self=None, *args, **kwargs):
if _self: ## we've been passed an instance already
initialyzed
## so directly return it instead of creating a
new object.
return _self
return object.__new__(cls)

Depending on your application, this may be an unsafe assumption. If the
caller says class1("spam"), your class will return "spam". This may or
may not be what you want.

def __init__(self, _self=None, *args, **kwargs):
if _self: ## we've been passed an instance already
initialyzed
## so directly returns
## assert(self is _self) ?
return
self.object2 = class2(object1=self, "blip", "blop") # others
init

Okay, this seems like a fairly straightforward case of adding an
attribute to your instance that includes a reference to itself. No big
deal... there's nothing wrong with this, although you are creating a
reference cycle, which is somewhat of a (mild) code-smell.

file module2.py:

class class2(object):

def __init__(self, object1, *args, **kwargs):

from ..module1 import class1

Now you have a circular import, and that's a pretty major code smell.
That's dangerous. Try to avoid it. Perhaps the easiest way to avoid it is
to place class1 and class2 in the same module, and stop writing Java

self.object1 = class1(_self=object1) ## instead of:
self.object1 = object1

## others functions and/or init..
## where now I've completion working on self.object1 : ## if I
add(or remove) fields/methods in module1 (and save) then ## I have
them available(or disappeared) in the completion when
executed from this submodule.
## This ofcourse permits to save me of remembering all of the
class1 attributes/methods when I'm working with self.object1 from within
one of class2 methods.

I'm afraid I don't understand what you mean here in these comments. What
do you mean, "save me of (from?) remembering all the class1 attributes/
methods..." -- how does it save you from knowing the methods? Whether you
write this:

instance = class1()
instance.method()

or this:

instance = class1()
another_instance = class2(instance)
another_instance.object1.method()

you still need to know the name "method".

It seems to me that this scheme is unnecessarily complex, convoluted and
confusing, for very little gain. Perhaps you could try explaining what
you hope to accomplish, giving examples?

 

[gst]

"The completion"? What do you mean?

Hi,

I mean the autocompletion feature of my IDE :

when I type "object." then on the "." it shows me possible members/
methods of the 'object'. depending on what my IDE can guess about the
object possible type(s) ; in the worst case it can't guess nothing
about it and that's my point below..

I'm afraid I have to agree with your, your description is not clear to me..

I hope I'm now

Unfortunately, neither is your code, because I don't understand *why* you
do the things you do.

damn. ok let's go..

Is it relevant that class2 comes from another module?

yes. hmmm or maybe not completely.. I think it only highly depends on
what the IDE can guess on an object type used inside some methods
depending on the files hierarchy..

Would your concept work if class1 and class2 were defined in the same file?

I guess yes also.

Depending on your application, this may be an unsafe assumption. If the
caller says class1("spam"), your class will return "spam". This may or
may not be what you want.

I'm always passing the good object type when I do this.

Okay, this seems like a fairly straightforward case of adding an
attribute to your instance that includes a reference to itself. No big
deal... there's nothing wrong with this, although you are creating a
reference cycle, which is somewhat of a (mild) code-smell.

that is a worry effectively. (hmm when you say it's a case of adding
an attribute that includes a ref to itself it's not really that (or i
badly understand) : it's adding an attribute to a lower level class
instance that contains a ref to an upper level class instance (which
also contains a ref to the lower level class' instance and that's the
cycle (and that's a bit my worry))

Now you have a circular import, and that's a pretty major code smell.
That's dangerous. Try to avoid it. Perhaps the easiest way to avoid it is
to place class1 and class2 in the same module, and stop writing Java

damn if this looks like Java it's really not my will ; I've not made
Java anymore for probably 10 years !

I'll try by putting class1 and class2 in same module (or at same
directory level first) and see if that can do it..

I'm afraid I don't understand what you mean here in these comments. What
do you mean, "save me of (from?) remembering all the class1 attributes/
methods..." -- how does it save you from knowing the methods? Whether you
write this:

instance = class1()
instance.method()

or this:

instance = class1()
another_instance = class2(instance)
another_instance.object1.method()

you still need to know the name "method".

It seems to me that this scheme is unnecessarily complex, convoluted and
confusing, for very little gain. Perhaps you could try explaining what
you hope to accomplish, giving examples?

Effectively I recognise that this seems like a bit silly/odd .. but as
it's working I wanted some advises so

As I told I try to have some kind of "ultimate" autocompletion, always
working for this kind of case (I know exactly that a certain member of
a certain class' instance will always have the same type but my IDE
doesn't seem to guess/know it.. ; I guess the best answer would be :
bad IDE, change IDE but I'm guessing that others IDE can't do a lot
better in this case.

Anyway many thanks for your reply ; I'll do some further tests with
your comments already..

regards,

greg.

nb: so this "hack" is only relevant during dev ; once the project
would be finished the "hack" could be removed (i.e : in class2 init I
would directly do : self.object1 = object1)

 

[Steve Holden]

As I told I try to have some kind of "ultimate" autocompletion, always
working for this kind of case (I know exactly that a certain member of
a certain class' instance will always have the same type but my IDE
doesn't seem to guess/know it.. ; I guess the best answer would be :
bad IDE, change IDE but I'm guessing that others IDE can't do a lot
better in this case.

If you'd told us which IDE you were using we might have offered better
advice, but you seem to want to keep that a secret ("my IDE" tells us
nothing).

regards
Steve

 

[gst]

If you'd told us which IDE you were using we might have offered better
advice, but you seem to want to keep that a secret ("my IDE" tells us
nothing).

sorry it was totally bottom of my first message :

note: I'm using Eclipse 3.5.2 with pydev so (I don't really know how

others IDE can handle this case) .

now I realyze that my question has not a lot to do with python
language itself and I should resubmit that probably on pydev forums/..

but as I told I wanted to have some advises on the "good" (or not) use
of this "way of doing" with python itself. You already explained me
it's not the best so far.. and I agree although during (my) dev it
helps (but I do this only with very few classes that have lot of
attributes/methods ofcourse).

regards,

greg.

 

[Jean-Michel Pichavant]

greg.

nb: so this "hack" is only relevant during dev ; once the project
would be finished the "hack" could be removed (i.e : in class2 init I
would directly do : self.object1 = object1)

Expect some bugs then on the 'release' version.
I'm not sure I understood everything you mentioned in your OP, however
writing python code on class creation to make Eclipse completion 'work'
is ... a very bad idea IMO.
Just don't do it.

quoting eclipse page:

"Pydev [...] uses advanced type inference techniques to provide features
such code completion and code analysis"

I don't know exactly what's hidden behind this marketing stuff. Did you
try to document your method with a markup language supported by Eclipse
(if there is any)?

class class2(object):
def __init__(self, object1, *args, **kwargs):
"""blablabla

@param object1: a L{class1} reference
"""
self.object1 = object1

The docstring would be the only way for an IDE to infer a argument type.
Note that I used the epydoc markup language because it's the only one I
know but it's unlikely supported by Eclipse. You better try
reStructuredText.

JM

 

[gst]

Hi,

Expect some bugs then on the 'release' version.

even with the "assert(self is _self)" in the "if _self:" condition in
the new and/or init methods of the class1 ?

I'm not sure I understood everything you mentioned in your OP, however
writing python code on class creation to make Eclipse completion 'work'
is ... a very bad idea IMO.
Just don't do it.

yes I'm trying to find better ways for having completion works out of
the box in this case.. (that would be a high help to my opinion).

quoting eclipse page:

"Pydev [...] uses advanced type inference techniques to provide features
such code completion and code analysis"

I don't know exactly what's hidden behind this marketing stuff. Did you
try to document your method with a markup language supported by Eclipse
(if there is any)?

pydev completion apparently is restricted (as far as i see) to some
very specific cases (basically it works when you import a module at
top of another one and that you instantiate objects from the imported
module within the init methods of the classes of the module which is
importing the other one (but so "circular" references created on some
objects from the first module (the one imported) in the second one
won't have the completion working for them (that's what I see)).

class class2(object):
    def __init__(self, object1, *args, **kwargs):
    """blablabla

    @param object1: a L{class1} reference
    """
    self.object1 = object1

The docstring would be the only way for an IDE to infer a argument type.
Note that I used the epydoc markup language because it's the only one I
know but it's unlikely supported by Eclipse. You better try
reStructuredText.

JM

well, I'm not very used to docstrings in fact but it's effectively a
way to achieve this ; well certainly better than doing this kind of
"ugly" hack so.
I'll have a try at that.

Thanks for your reply,

regards,

greg.

 

[gst]

Hi,



even with the "assert(self is _self)" in the "if _self:" condition in
the new and/or  init methods of the class1 ?

note: I don't know if that matters but I'm using __slots__ in this
class1 ..

greg.

 

[gst]

even with the "assert(self is _self)" in the "if _self:" condition in
the new and/or  init methods of the class1 ?

damn : in the init method only of course..

 

[Jean-Michel Pichavant]

quoting eclipse page:

"Pydev [...] uses advanced type inference techniques to provide features
such code completion and code analysis"

I don't know exactly what's hidden behind this marketing stuff. Did you
try to document your method with a markup language supported by Eclipse
(if there is any)?

pydev completion apparently is restricted (as far as i see) to some
very specific cases (basically it works when you import a module at
top of another one and that you instantiate objects from the imported
module within the init methods of the classes of the module which is
importing the other one (but so "circular" references created on some
objects from the first module (the one imported) in the second one
won't have the completion working for them (that's what I see)).

To understand what is possible and what is not, you simply need to think
like a completion feature.

case 1:
self.object = MyClass()

self.object is a MyClass instance, easy stuff I need to parse MyClass
and get the list of attributes/method

case 2:

self.object = object1

There may be simply no way to know the type/class of object1, object1
could be even of an inconsistent type, sometimes None, int, MyClass, who
knows ?
Your trick worked because you found a way to write self.object =
MyClass(object1) and getting barely the same effect than self.object =
object1.

I'm still thinking that's it's a bad idea. In any case your IDE can
still be fooled by the dynamic of python (like pylint is)

How to write python code then ? Well I guess most people either knows
all the attributes by heart, have the documentation, or a split screen
to the class definition, possibly folded using advanced text editor
feature. You need to forget about code completion with any dynamic
language. I also sometimes use an ipython shell to instanciate objects
and inspect their attributes.

JM

 

[nn]

Hi,

I met a situation where I was passing an object created in/with an
upper level module class to a lower level module class' instance in
one of its __init__ argument and saving a ref of the upper object in
that lower level class' new instance.

But in my IDE I want the completion to also work from within the lower
level module when it's refering to the object passed from the upper
level:

Well, I'm sure I'm not very clear in my terms (and probably a bit long
in the sentence) so here it is in code:

files:
module1.py
subpackage/module2.py

file module1.py:

from subpackage.module2 import class2

class class1(object):

    def __new__(cls, _self=None, *args, **kwargs):
        if _self:  ## we've been passed an instance already
initialyzed
                     ## so directly return it instead of creating a
new object.
            return _self
        return object.__new__(cls)

    def __init__(self, _self=None, *args, **kwargs):
        if _self:  ## we've been passed an instance already
initialyzed
                     ## so directly returns
            ## assert(self is _self) ?
            return
        self.object2 = class2(object1=self, "blip", "blop")
        # others init

file module2.py:

class class2(object):

    def __init__(self, object1, *args, **kwargs):

        from ..module1 import class1

        self.object1 = class1(_self=object1)    ## instead of:
self.object1 = object1

    ## others functions and/or init..
    ## where  now I've completion working on self.object1 :
    ## if I add(or remove) fields/methods in module1 (and save) then
    ## I have them available(or disappeared) in the completion when
executed from this submodule.
    ## This ofcourse permits to save me of remembering all of the
class1 attributes/methods when I'm working with self.object1 from
within one of class2 methods.

What do you think of this ?

I guess there can be others ways of doing this..  ?

Thanks,

Regards,

Greg.

note: I'm using Eclipse 3.5.2 with pydev so (I don't really know how
others IDE can handle this case) .

You might want to test with Wing IDE http://wingware.com/
using the 10 day trial version to see if that works just for kicks.
I've heard it has the best autocomplete of all IDEs out there.