files,folders,paths

E

ecu_jon

i have a samba server at home (acting as my test environment) with one
of the 'shares' mounted as v: on my windows box. inside of that are 4
folders week[1-4]. i have created c:\users\name\backup , and put a few
files/folders in there to test with. please ignore the wxpython parts
of the script, the last 1/3, that part is functional. initial test was
looking good. (in the code, ver011 source1, destination1) coping files/
folders and not clobbering files. going to version 12a, it seems like
it is trying to copy the files under the folder, before making the
folder. or doing the copy2 part before the copytree part. i dont know
why it works on ver011, and not in ver12a, the only difference being
the destination.
<a href="http://thanksforallthefish.endofinternet.net/
facbac-011.py">facbac 011</a>
<a href="http://thanksforallthefish.endofinternet.net/
facbac-012a.py">facbac 012a</a>
Traceback (most recent call last):
File "I:\college\spring11\capstone-project\facbac-012a.py", line
161, in button1Click
backupall()
File "I:\college\spring11\capstone-project\facbac-012a.py", line 81,
in backupall
shutil.copy2(filesource, filedest1)
File "C:\Python27\lib\shutil.py", line 127, in copy2
copyfile(src, dst)
File "C:\Python27\lib\shutil.py", line 82, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: 'V:\\week2\\configs\
\apache2.conf'

here, apache2.conf = c:\users\name\backup\config\apache2.conf
trying to copy to v:\week2\config\apache2.conf
but there is no v:\week2\config
as it has not made the config folder yet
i think the problem might be the \\ in the destination name.
 
A

aspineux

Without trying to understand your code, I see one difference

if not os.path.isfile(destination+leftover+fname):

become

if not os.path.isfile(destination+leftover):


Regards
 
E

ecu_jon

i just tried changing that in ver12a, still get
IOError: [Errno 2] No such file or directory: 'V:\\week2\\configs\
\apache2.conf'
good catch tho as that is needed. thanks.
weekchoice is used to pick week 1-4. really jsut returns 1-4. i plan
to do full weekly backups with another script.
getusername i was using when i was trying to build the path as part of
a unc path. isdir/isfile did not play well with that.
homedir returns the users home directory. works in xp,7, and linux.
source and destination should be self explanatory.
the backupall() gets called from clicking the button. for now its the
meat and potatoes im working on. it is trying to recursively copy
files/folders from source to dest. the for ... os.walk line digs
through the backup folder. i use os.chgdir , os.getcwd and leftover to
build the path for file/folder names on dest. basically start at c:
\users\name\backup\somefolder and end with just somefolderadded to
dest.
if os.path.isdir(fname): checks if it is a folder. copytree is a
recursive folder copy.
elif os.path.isfile(fname): if it is a file. copy2 copies files and
metadata.
class MyForm(wx.Frame): and down is wxpython, that works. ( tho i will
take suggestions but i know this is not the right forum to as
questions for that part)
 
S

Steven D'Aprano

i have a samba server at home (acting as my test environment) with one
of the 'shares' mounted as v: on my windows box. inside of that are 4
folders week[1-4]. i have created c:\users\name\backup , and put a few
files/folders in there to test with. please ignore the wxpython parts of
the script, the last 1/3, that part is functional.

Please don't dump code on us and tell us to ignore parts of it. We're
offering our time and experience without charging you, the least you can
do is take the time to eliminate the irrelevant parts of the code
yourself, and show us the smallest piece of code which demonstrates the
problem.

Doing so will not only demonstrate respect for others, but it might even
help you solve the problem yourself.

initial test was
looking good. (in the code, ver011 source1, destination1) coping files/
folders and not clobbering files. going to version 12a, it seems like it
is trying to copy the files under the folder, before making the folder.
or doing the copy2 part before the copytree part. i dont know why it
works on ver011, and not in ver12a, the only difference being the
destination.

That's not exactly true, is it? I count at least three other differences:

* facbac-012a.py imports getpass and defines a redundant
getusername function (redundant because all it does is
wrap getpass.getuser);

* it also defines a function destination2 which is different
from destination1, and uses that;

* there's a difference in the call to os.path.isfile;

I don't know if these are relevant.

You say the difference between the script that works and the script that
doesn't is the destination. And then you show an error that appears to be
because the destination doesn't exist:

Traceback (most recent call last):
File "I:\college\spring11\capstone-project\facbac-012a.py", line
161, in button1Click
backupall()
File "I:\college\spring11\capstone-project\facbac-012a.py", line 81,
in backupall
shutil.copy2(filesource, filedest1)
File "C:\Python27\lib\shutil.py", line 127, in copy2
copyfile(src, dst)
File "C:\Python27\lib\shutil.py", line 82, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: 'V:\\week2\\configs\
\apache2.conf'


Seems like an open and shut case to me.


here, apache2.conf = c:\users\name\backup\config\apache2.conf trying to
copy to v:\week2\config\apache2.conf but there is no v:\week2\config
as it has not made the config folder yet i think the problem might be
the \\ in the destination name.


There is no double-backslash in the destination name. Backslashes are
special in Python, and when displaying strings, they are escaped:

x\yz


Your suggestion, that the problem is that the config directory doesn't
exist, sounds reasonable to me. Only you can test that, as we don't have
access to your file system.
 
D

Dave Angel

i just tried changing that in ver12a, still get
IOError: [Errno 2] No such file or directory: 'V:\\week2\\configs\
\apache2.conf'
good catch tho as that is needed. thanks.
weekchoice is used to pick week 1-4. really jsut returns 1-4. i plan
to do full weekly backups with another script.
getusername i was using when i was trying to build the path as part of
a unc path. isdir/isfile did not play well with that.
homedir returns the users home directory. works in xp,7, and linux.
source and destination should be self explanatory.
the backupall() gets called from clicking the button. for now its the
meat and potatoes im working on. it is trying to recursively copy
files/folders from source to dest. the for ... os.walk line digs
through the backup folder. i use os.chgdir , os.getcwd and leftover to
build the path for file/folder names on dest. basically start at c:
\users\name\backup\somefolder and end with just somefolderadded to
dest.
if os.path.isdir(fname): checks if it is a folder. copytree is a
recursive folder copy.
elif os.path.isfile(fname): if it is a file. copy2 copies files and
metadata.
class MyForm(wx.Frame): and down is wxpython, that works. ( tho i will
take suggestions but i know this is not the right forum to as
questions for that part)

I haven't tried to run your script, since much of it is Windows
dependent, and I'm on Linux. So observations are just by inspection.

I suspect your problem is that you're trying to change drives as well as
directories. In Windows, there's a current directory on each drive,
that Windows remembers. So changing directories on a different drive
doesn't mean what you might think it does. I'd recommend doing no
chdir()'s at all, and using only absolute paths.

But I see other complications or maybe risks in the code, concentrating
on function backupall(). You're using os.walk(), which walks through a
whole tree of files. But then when you encounter a directory, you're
doing a copytree(). So you'll be copying the same files many-many times.

Similarly, you have logic for copying a single file, but you invoke it
on a name from dirlist, which is a list of directories, so the isfile()
probably never fires.

os.walk() assumes the current directory is unchanged during its run.
See: http://docs.python.org/library/os.html#module-os
"walk() never changes the current directory, and assumes that its
caller doesn’t either."

You're not copying the files at the top level at all; you don't process
the list called 'files', while you could probably safely copy *only* the
files.

Your name manipulations in this same function look risky, using replace
when you presumably know that a string is either at the begin or end.

In function homeDir(), you never return the value calculated in the try
block. If you don't get an exception, you return None.

DaveA
 
E

ecu_jon

i just tried changing that in ver12a, still get
IOError: [Errno 2] No such file or directory: 'V:\\week2\\configs\
\apache2.conf'
good catch tho as that is needed. thanks.
weekchoice is used to pick week 1-4. really jsut returns 1-4. i plan
to do full weekly backups with another script.
getusername i was using when i was trying to build the path as part of
a unc path. isdir/isfile did not play well with that.
homedir returns the users home directory. works in xp,7, and linux.
source and destination should be self explanatory.
the backupall() gets called from clicking the button. for now its the
meat and potatoes im working on. it is trying to recursively copy
files/folders from source to dest. the for ... os.walk line digs
through the backup folder. i use os.chgdir , os.getcwd and leftover to
build the path for file/folder names on dest. basically start at c:
\users\name\backup\somefolder and end with just somefolderadded to
dest.
if os.path.isdir(fname): checks if it is a folder. copytree is a
recursive folder  copy.
elif os.path.isfile(fname): if it is a file. copy2 copies files and
metadata.
class MyForm(wx.Frame): and down is wxpython, that works. ( tho i will
take suggestions but i know this is not the right forum to as
questions for that part)

I haven't tried to run your script, since much of it is Windows
dependent, and I'm on Linux.  So observations are just by inspection.

I suspect your problem is that you're trying to change drives as well as
directories.  In Windows, there's a current directory on each drive,
that Windows remembers.  So changing directories on a different drive
doesn't mean what you might think it does.  I'd recommend doing no
chdir()'s at all, and using only absolute paths.

But I see other complications or maybe risks in the code, concentrating
on function  backupall().  You're using os.walk(), which walks through a
whole tree of files.  But then when you encounter a directory, you're
doing a copytree().  So you'll be copying the same files many-many times.

Similarly, you have logic for copying a single file, but you invoke it
on a name from dirlist, which is a list of directories, so the isfile()
probably never fires.

os.walk() assumes the current directory is unchanged during its run.
See:  http://docs.python.org/library/os.html#module-os
   "walk() never changes the current directory, and assumes that its
caller doesn’t either."

You're not copying the files at the top level at all;  you don't process
the list called 'files', while you could probably safely copy *only* the
files.

Your name manipulations in this same function look risky, using replace
when you presumably know that a string is either at the begin or end.

In function homeDir(), you never return the value calculated in the try
block.  If you don't get an exception, you return None.

DaveA

first let me say, you guys are helping me and i thank you for that. i
do not want to sound confrontational. i have don my best to make this
os agnostic. yes i know, right now in 21a, destination2 is windows
specific. that's mostly because os.path.isdir() and unc paths do not
play well with each other.

the os.chdir() came about out of need to build relative path names.
some examples :
c:\users\name\backup\ #win7
c:\docs and settings\name\app data\backup\ #winxp
/home/name/backup #linux and mac

i wanted to start from 'backup' in all these cases. my destination is
going to look like
\\servername\username\week[1-4]

yes os.walk goes thru the whole tree (under backup in my case). the
os.chdir() under the for line in def backupall(): basically juct
chages folders to whever the os.walk() is. it never tried to change to
a different drive letter.the folder copy thing, these 2 lines
if os.path.isdir(fname): #if it is a folder
if not os.path.isdir(destination+leftover): #and id
dont exist on dest drive
work on version 011, determine if its a folder, then if it exists.
only copy if both: it is a folder, and not exist on destination.
this "return homedir" needed to be moved 1 tab left. copy error from
random code to functionalizing parts. good catch thanks.
dirlsit, returns the names of folders and files. i was using a
different method
http://thanksforallthefish.endofinternet.net/date-linuxversion.py

try version 11 for yourself. in your home directory (ie: /home/name)
make a backup folder, put a few files/folders in it. then make a
backup2 folder, and make week1,week2week3week4 folders. version 11
will copy the top level files under /home/name/backup , and recreate
the directory structure. so "You're not copying the files at the top
level at all; you don't process
the list called 'files', while you could probably safely copy *only*
the
files. " atleats for version 11 is not true. atleast not within the
bounds i have tested it.

"Your name manipulations in this same function look risky, using
replace
when you presumably know that a string is either at the begin or end.
"
i know. i have several things that need to build up. like starting
from a relative position and going to a different position. username
would be part of the unc path if i could get that to work, and
week[1-4] is part of the name.

the thing that gets me is version 11 works. well. but its is reading/
writing to almost the same place (from backup, to backup2). defiantly
open to suggestions.
 
E

ecu_jon

"Please don't dump code on us and tell us to ignore parts of it. We're
offering our time and experience without charging you, the least you
can
do is take the time to eliminate the irrelevant parts of the code
yourself, and show us the smallest piece of code which demonstrates
the
problem. " i appreciate your opinion. im fairly new here so i might
not get etiquette right.i only asked you all to ignore it for 2
reasons, it's wxpython to not really targets to the crowd, and it does
not interact with the python parts. other than simply as a means to
call the functions. i could chop off the bottom and replace it with a
functname(), it would be the same for my question.

i know backslashes are special. there a special pain in my #$%.
without a windows environment, you would have to change destination2
significantly to try it out. but like i said above you can try version
11 in a linux environment. i know config does not exist, for some
reason the isfolder check is passing it off saying it exists, when it
does not. then moving on to the isfiles parts. tries to copy a folder
deep files with no folder to write to, and fails.
i jsut don't understand why the folder check is not working right in
version 12a.

and destination2, IS basically the difference between 11 and 12a. it
should be the only difference. the isfile difference someone else
pointed out i have since changed that, stil lsame problem.
and again thanks all for time/trouble.
 
S

Steven D'Aprano

i know config does not exist, for some reason the isfolder check is
passing it off saying it exists, when it does not.


[steve@sylar ~]$ grep isfolder facbac-011.py
[steve@sylar ~]$ grep isfolder facbac-012a.py
[steve@sylar ~]$


There is no such "isfolder" check in your source code.
 
E

ecu_jon

If they are a pain in your #$%, then don't use them.  You can use forward
slashes everywhere in Windows, except when typing commands at the command
prompt.  ALL of the Windows file system APIs accept forward slashes.

can you give an example of how to build a folder name, maybe as a
string, with parts of it as variables, that have to be strung
together. like x = "//servername/+variable+"/"+variable+"/"
when i did something liek that the end result string was like
//servername/folder/folder\file and i got an error
 
D

Dave Angel

first let me say, you guys are helping me and i thank you for that. i
do not want to sound confrontational. i have don my best to make this
os agnostic. yes i know, right now in 21a, destination2 is windows
specific. that's mostly because os.path.isdir() and unc paths do not
play well with each other.

the os.chdir() came about out of need to build relative path names.
some examples :
c:\users\name\backup\ #win7
c:\docs and settings\name\app data\backup\ #winxp
/home/name/backup #linux and mac

In what way do those imply a need for chdir() ? What's wrong with
os.path.join() ?
i wanted to start from 'backup' in all these cases. my destination is
going to look like
\\servername\username\week[1-4]

yes os.walk goes thru the whole tree (under backup in my case). the
os.chdir() under the for line in def backupall(): basically juct
chages folders to whever the os.walk() is. it never tried to change to
a different drive letter.

And I quote from facbac-012a.py:
os.chdir("v:")

the folder copy thing, these 2 lines
if os.path.isdir(fname): #if it is a folder
if not os.path.isdir(destination+leftover): #and id
dont exist on dest drive
work on version 011, determine if its a folder, then if it exists.
only copy if both: it is a folder, and not exist on destination.
this "return homedir" needed to be moved 1 tab left. copy error from
random code to functionalizing parts. good catch thanks.
dirlsit, returns the names of folders and files. i was using a
different method
http://thanksforallthefish.endofinternet.net/date-linuxversion.py

try version 11 for yourself. in your home directory (ie: /home/name)
make a backup folder, put a few files/folders in it. then make a
backup2 folder, and make week1,week2week3week4 folders. version 11
will copy the top level files under /home/name/backup , and recreate
the directory structure. so "You're not copying the files at the top
level at all; you don't process
the list called 'files', while you could probably safely copy *only*
the
files. " atleats for version 11 is not true. atleast not within the
bounds i have tested it.

"Your name manipulations in this same function look risky, using
replace
when you presumably know that a string is either at the begin or end.
"
i know. i have several things that need to build up. like starting
from a relative position and going to a different position. username
would be part of the unc path if i could get that to work, and
week[1-4] is part of the name.

the thing that gets me is version 11 works. well. but its is reading/
writing to almost the same place (from backup, to backup2). defiantly
open to suggestions.

I didn't spot the dirlist before, since it's totally unnecessary. It
just builds the same list you already have as 'files'. I thought you
were looping over the names in dirs. copytree() is also redundant, as
it'll cause the same files to be repeatedly copied. If you plan to use
copytree, why bother using os.walk at all?

I suggest you actually try writing a simple loop using os.walk to just
display the names of all the files in and under a specified directory.
Then maybe you'll understand what path, dirs, and files actually are.

Once you have that, use string manipulation to change the source path
name into a destination path, and print, rather than actually doing the
shutilcopy2.



DaveA
 
E

ecu_jon

ecu_jon said:
can you give an example of how to build a folder name, maybe as a
string, with parts of it as variables, that have to be strung
together. like x = "//servername/+variable+"/"+variable+"/"

    import os.path

    infile_path = os.path.join(["//servername", start_dir, other_dir])

Read the documentation for ‘os.path.join’. It's fundamental if you're
working with filesystem paths.
when i did something liek that the end result string was like
//servername/folder/folder\file and i got an error

That path should work on MS Windows. If you want us to comment on the
error, you'll need to show the code here *and* the error traceback.

--
 \        “If this is your first visit to the USSR, you are welcome to |
  `\                                          it.” —hotel room, Moscow |
_o__)                                                                  |
Ben Finney

here take a look.
http://thanksforallthefish.endofinternet.net/testing.py
the output looks like
.....
folder to copy = \\\\Mothera\\jon\week2
file to copy: C:\Users\jon\backup\configs\apache2.conf \\\\Mothera\\jon
\week2\configs\apache2.conf
folder to copy = \\\\Mothera\\jon\week2\configs
.....#some lines ommited for brevity.
 
E

ecu_jon

I think you're confusing yourself by shoving lots of experimental lines
into a large program and not isolating the problem. I see that you are
trying lots of different ways of doing the same thing, and commenting
some lines out arbitrarily.

Instead, focus on the issue that's causing you confusion: constructing a
filesystem path and testing whether it exists.

Make a *minimal* program that shows the problem you're having. Post it
*here* (that's why it's important that it be minimal), along with the
exact error traceback you get from Python when you run it.

Once you have something minimal that we, too, can run, we can comment
better on its behaviour.

--
 \        “Members of the general public commonly find copyright rules |
  `\        implausible, and simply disbelieve them.” —Jessica Litman, |
_o__)                                              _Digital Copyright_ |
Ben Finney

this thing evolved i piece a a time. i got help from the wxpython
people, and now that code has been stable for weeks.

the function weekChoice came after talking to 2 CS grad studenst for 2
hours. deciding how to decide what week to pick. it seems to be solid.
was picking week 1 up till 12:02 when i tested it on the 8th, then
went to week 2.

getusername probably should be unfunctionalized, and maybe dropped
completely.

homeDir is good on the 3 differnt platforms i have tested it on,
linux, xp and 7. Ubuntu returned /home/name, 7 returned c:\Users\name
\, xp returned c:\docs and settings\name\app data\ or something close
to that.

in version 011, source1 and destination1 worked fine. selectivly
coping files or folders as needed.

backupall has been volatile, but relatively stable last 2 weeks. in
version 011, it does the if not folder and file detection. in version
12a, it does not. all of the functions cascade so im not really sure
how i could cut something out, with out serious rewrite. i already did
shrink it by gutting the wxpython parts. after uncommenting the 2
lines
shutil.copytree(os.getcwd(), destination+leftover)
shutil.copy2(filesource, filedest1)
i get this as an traceback
folder to copy = \\\\Mothera\\jon\week2

Traceback (most recent call last):
File "I:\college\spring11\capstone-project\testing.py", line 86, in
<module>
backupall()
File "I:\college\spring11\capstone-project\testing.py", line 73, in
backupall
shutil.copytree(os.getcwd(), destination+leftover)
File "C:\Python27\lib\shutil.py", line 174, in copytree
os.makedirs(dst)
File "C:\Python27\lib\os.py", line 150, in makedirs
makedirs(head, mode)
File "C:\Python27\lib\os.py", line 150, in makedirs
makedirs(head, mode)
File "C:\Python27\lib\os.py", line 150, in makedirs
makedirs(head, mode)
File "C:\Python27\lib\os.py", line 157, in makedirs
mkdir(name, mode)
WindowsError: [Error 161] The specified path is invalid: '\\\\\\\\'
 
E

ecu_jon

I don't doubt it. But clearing up confusion over what a big wad of code
does is best done by trying to isolate the problem into a small piece of
code.



This all sounds as though you're doing ball-of-mud programming: keep
throwing more code at the program, and whatever sticks must stay there
forever without understanding what it's doing.

That's a bad idea. I hope you can already see why.

Really, please make a new, minimal, complete example that still exhibits
the specific confusing behaviour, and *post it in a message* (not at a
URL), so we can see the problem in isolation. If it's more than a dozen
lines, it's doing too much for the problem you're talking about here.

--
 \        “We cannot solve our problems with the same thinking we used |
  `\                           when we created them.” —Albert Einstein |
_o__)                                                                  |
Ben Finney

well it seems not to be the difference between versions 11,12,12a.
it works great on my linux box. i get all these errors on the windows
boxes.
i think it has to do with the folder name, and strings etc. got to se
if i can use os.join.path for the whole naming.
 
E

ecu_jon

i think i got it. my dirlist wasn't the right way to go, i needed to
iterate thu what oswalk was giving me in dirs, and files. not
bruteforce strip to a string. i have been starting at an early version
and a later version for the last 4 hours or so, and i think i have it
this only handles the folders. i wanted to try out the logic before i
doubled up and parsed files too. substitute this in for the backup
function. and for now the source and destination functions are in this
to. ill clean it up once i test it more.
backupdir = os.path.join(homedir, "backup")
remotedir = os.path.join(homedir, "backup2")
weekstr = "week"+str(week)
remotedirweek = os.path.join(remotedir, weekstr)
print
'################################################################'
for (path, dirs, files) in os.walk(backupdir):
print "current path is : ",path
print "current dir is : ",dirs
print "current files are: ",files
for fname in dirs:
os.chdir(path)
leftover = os.getcwd().replace(backupdir, '')
print "leftover is:",leftover
print "remotedirweek is:",remotedirweek
currentdir1 = remotedirweek+leftover
if not os.path.isdir(currentdir1):
print "currentdir1:",currentdir1
print "i should copy teh
folderz",os.path.join(currentdir1,fname)

shutil.copytree(os.getcwd(),os.path.join(currentdir1,fname))
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top