O.T. FTP upload

[Roedy Green]

There are hundreds of FTP utilities. What do you use for FTP
unattended uploads?

My ISP is still waiting for a round TUIT to set up a VPN to replace
FTP.
--
Roedy Green Canadian Mind Products
http://mindprod.com
It's difficult to be rigorous about whether a machine really knows,
thinks, etc., because we’re hard put to define these things.
We understand human mental processes only slightly better than
a fish understands swimming.
~ John McCarthy (born: 1927-09-04 died: 2011-10-23 at age: 84).
Inventor of the term AI (Artificial Intelligence),
the short-circuit OR operator (|| in Java),
and LISP (LIst Processing Language) that makes EMACS
(Extensible MACro System) so addictive.

 

[Daniel Pitts]

There are hundreds of FTP utilities. What do you use for FTP
unattended uploads?

My ISP is still waiting for a round TUIT to set up a VPN to replace
FTP.

I use sftp or scp.

Alternatively I set up a webserver on my local machine and use wget or
curl from the remote machine.

I've also been known to use gzip | nc on the sending host and nc |
gunzip on the receiving host.

Also, never underestimate the bandwidth of a pickup truck full of
physical media.

 

[Tom Anderson]

I use sftp or scp.

I use rsync or Mercurial. Why transfer all those bytes which haven't
changed?

We are lucky, though, to be working with hosts that support modern,
efficient, reliable protocols like those. There are plenty of poor souls
out there who don't have access to anything more sophisticated than FTP.
Of course, our privilege means that we have no idea what the best tools
for that are!

Roedy, you mention that your ISP is going to set up a VPN. That seems like
overkill. Have you discussed rsync, running over SSH, as an option? It's
simple for users, simple for administrators, very widely used, secure,
flexible, and extremely efficient.

On a unix machine, rsync will be available from the package manager. The
administrators will need to set up system accounts for every user, but
they would normally do that anyway for FTP, i think, and they can restrict
them to use for rsync using scripts included in the rsync distribution.

A quick look suggests your ISP may be using Netware (although i find this
slightly hard to believe!), in which case they might be interested in:

http://wiki.novell.com/index.php/RSYNC
http://www.novell.com/coolsolutions/trench/470.html
http://www.novell.com/coolsolutions/appnote/654.html

(and many other writeups)

Also, never underestimate the bandwidth of a pickup truck full of
physical media.

Particularly when fired from a GIANT CANNON. It's not clear that the ISP
would support this, though, in ballistic mode or not.

tom

 

[Arne Vajhøj]

There are hundreds of FTP utilities. What do you use for FTP
unattended uploads?

I coded one in Java (using Commons Net).

That may not be a good solution in general, but given the
group, then it should be considered.

Arne

 

[Roedy Green]

Also, never underestimate the bandwidth of a pickup truck full of
physical media.

That reminds me of a story. Circa 1978, I was working at the research
labs of BC Hydro on solar energy research. I would collect a tape full
of data each day. It turned out mainly to be a PR thing -- to
impress visitors. The brass insisted I send the data to the mainframe
20 miles away, by modem. Modems back then were 300 baud (about 30
cps) and did not have error correction. Further we were out in the
boonies with phone lines that liked to disconnect frequently and were
very noisy. I protested this would be very slow and would take
special programming, especially since the mainframe did not support
any sensible protocol, and liked to go to sleep frequently to focus
its limited attention on higher priority tasks.

I wrote a robust error-correcting, stall-tolerant, re-establishing
protocol. It was similar to my proposed SAX protocol (see
http://mindprod.com/project/sax.html) I then invited the brass to
watch. I fired up my DEC MINC (a PDP11) and then took the tape under
my arm and headed off on my bicycle for downtown.

Of course the tape got there long before the transfer completed.

In my particular case, I do an automated upload every hour. The
problem is the ancient abandoned NETLOAD utility grinds everything
else to a halt when it runs, and it keeps going to sleep, and needs
occasional prodding to wakefulness by minimizing/restoring. I doubt
it will survive yet another Windows "upgrade".
--
Roedy Green Canadian Mind Products
http://mindprod.com
It's difficult to be rigorous about whether a machine really knows,
thinks, etc., because we’re hard put to define these things.
We understand human mental processes only slightly better than
a fish understands swimming.
~ John McCarthy (born: 1927-09-04 died: 2011-10-23 at age: 84).
Inventor of the term AI (Artificial Intelligence),
the short-circuit OR operator (|| in Java),
and LISP (LIst Processing Language) that makes EMACS
(Extensible MACro System) so addictive.

 

[Roedy Green]

I use rsync or Mercurial. Why transfer all those bytes which haven't
changed?

When my ISP finally get around to letting me run code on the server, I
will use a VPN, WebDav, Rsync etc. He has procrastinated for years. I
don't pay him, so I can't very well complain. For now, the only route
in I have in is FTP. There are an astounding number of FTP utilities
out there, but even after years of trying them out, I have not found
one suited for unattended bulk upload of deltas.

This does not make sense. Surely thousands of people do unattended FTP
uploads. Maybe they have all graduated to non-FTP tools. I would have
been better off to write my FTP uploader own from scratch or using
Apache libarries than waste all that time testing duds.
--
Roedy Green Canadian Mind Products
http://mindprod.com
It's difficult to be rigorous about whether a machine really knows,
thinks, etc., because we’re hard put to define these things.
We understand human mental processes only slightly better than
a fish understands swimming.
~ John McCarthy (born: 1927-09-04 died: 2011-10-23 at age: 84).
Inventor of the term AI (Artificial Intelligence),
the short-circuit OR operator (|| in Java),
and LISP (LIst Processing Language) that makes EMACS
(Extensible MACro System) so addictive.

 

[Martin Gregorie]

There are hundreds of FTP utilities. What do you use for FTP unattended
uploads?

Short answer: I don't - I generally use gftp, a GUI tool which has the
ability to whole directory structures. This is a Linux tool (the g stands
for Gnome) but near equivalents are Filezilla (supports Linux, Windows,
OS X but has an over-busy GUI) and WS-FTP (Windows-specific if it still
exists, very similar to gftp).

If I was going to use unattended ftp transfers I'd just use the standard
Linux command line ftp client because its very easily scriptable. If SSH
was available I'd use sftp instead and for the same reasons.

Are you aware of J-ftp - http://sourceforge.net/projects/j-ftp/
I haven't tried it, but it has a similar GUI to gftp and handles a
variety of protocols including scp.

My ISP is still waiting for a round TUIT to set up a VPN to replace FTP.

If your ISP supports SSH, and most do, that's a good argument for
investigating sftp and/or PuTTY (which provides a decent scp command line
tool for Windows).

 

[Jim Janney]

There are hundreds of FTP utilities. What do you use for FTP
unattended uploads?

My ISP is still waiting for a round TUIT to set up a VPN to replace
FTP.

My ftp use is rare enough that I just run it manually inside Emacs. But
if I needed to automate it, I'd look at curl, or wget, or kermit. I've
written kermit scripts in the past and they're ugly as sin, but they
usually get the job done.

-
Jim Janney

 

[Wojtek]

Roedy Green wrote :

There are hundreds of FTP utilities. What do you use for FTP
unattended uploads?

My ISP is still waiting for a round TUIT to set up a VPN to replace
FTP.

I use ScriptFTP, mind it is a Windows only product:

http://www.scriptftp.com

 

[Lew]

Short answer: I don't - I generally use gftp, a GUI tool which has the
ability to whole directory structures. This is a Linux tool (the g stands
for Gnome) but near equivalents are Filezilla (supports Linux, Windows,
OS X but has an over-busy GUI) and WS-FTP (Windows-specific if it still
exists, very similar to gftp).

If I was going to use unattended ftp transfers I'd just use the standard
Linux command line ftp client because its very easily scriptable. If SSH
was available I'd use sftp instead and for the same reasons.

Are you aware of J-ftp - http://sourceforge.net/projects/j-ftp/
I haven't tried it, but it has a similar GUI to gftp and handles a
variety of protocols including scp.

If your ISP supports SSH, and most do, that's a good argument for
investigating sftp and/or PuTTY (which provides a decent scp command line
tool for Windows).

Or Cygwin, which sports "ssh".

 

[Roedy Green]

I use ScriptFTP, mind it is a Windows only product:

http://www.scriptftp.com

That looks promising. The one feature I would like is caching the
dates of files on the server so it would not have to download them on
the SYNC command. That is quite time consuming. I suppose I could
cook up something specific to my needs to do that and generate a
script of specific uploads and only use sync once a day to pick up
stragglers.

The poor thing has to upload a dummy file then download it again to
find out what the server thinks the time is. FTP works off local time
with unknown timezone. I guess FTP was invented when no one imagined
inter-time zone file transfers. Such a protocol designed today would
surely have UTC timestamps on everything and use NTP to keep the
clocks accurate.

I would have thought FTP would have evolved to use UTC on all servers.

I commend the authors. They have both examples and reference
documentation. I was able to figure out how to use it in under a
minute.


--
Roedy Green Canadian Mind Products
http://mindprod.com
Capitalism has spurred the competition that makes CPUs faster and
faster each year, but the focus on money makes software manufacturers
do some peculiar things like deliberately leaving bugs and deficiencies
in the software so they can soak the customers for upgrades later.
Whether software is easy to use, or never loses data, when the company
has a near monopoly, is almost irrelevant to profits, and therefore
ignored. The manufacturer focuses on cheap gimicks like dancing paper
clips to dazzle naive first-time buyers. The needs of existing
experienced users are almost irrelevant. I see software rental as the
best remedy.

 

[Gunter Herrmann]

Hi!

The poor thing has to upload a dummy file then download it again to
find out what the server thinks the time is. FTP works off local time
with unknown timezone. I guess FTP was invented when no one imagined
inter-time zone file transfers. Such a protocol designed today would
surely have UTC timestamps on everything and use NTP to keep the
clocks accurate.

I would have thought FTP would have evolved to use UTC on all servers.

ftp uses the time of the server. Any decent *x server runs on UTC
and does the adjustment with locale settings at log in.
Assume it is 12:00:00 ZULU.
For a user from Germany on a ssh session it will be 13:00:00 MEZ (CET).
A user from NYC will see 8:00 am
(Note the different time formats as part of the locale settings).
In an ftp session the client does not tell the server where it is located,
so the server time is provided.

Gunter in Orlando, Fl

 

[Gunter Herrmann]

There are hundreds of FTP utilities. What do you use for FTP
unattended uploads?

To get on topic again:
Why not use Java? We just implemented a servlet to do sftp
in unattended mode.
With the command line client you need to type the password.
The workaround would be the use of the "expect" utility.
But since for that application we already had a servlet
running we just extended it (password in properties file)
and added an nfs mount to provide the files.

Gunter in Orlando

 

[Roedy Green]

Why not use Java? We just implemented a servlet to do sftp
in unattended mode.

The essential problem is I don't have permission to run any of my
chosen code on the server. I have no control over the HTTP server and
FTP server. Once I get that I have all kinds of options. Real soon
now has stretched to over a decade.
--
Roedy Green Canadian Mind Products
http://mindprod.com
Capitalism has spurred the competition that makes CPUs faster and
faster each year, but the focus on money makes software manufacturers
do some peculiar things like deliberately leaving bugs and deficiencies
in the software so they can soak the customers for upgrades later.
Whether software is easy to use, or never loses data, when the company
has a near monopoly, is almost irrelevant to profits, and therefore
ignored. The manufacturer focuses on cheap gimicks like dancing paper
clips to dazzle naive first-time buyers. The needs of existing
experienced users are almost irrelevant. I see software rental as the
best remedy.

 

[Gunter Herrmann]

Hi!

The essential problem is I don't have permission to run any of my
chosen code on the server. I have no control over the HTTP server and
FTP server.

You don't need anything on the ftp server. The servlet running on
your box implements the sftp client.

Gunter

 

[Arne Vajhøj]

The essential problem is I don't have permission to run any of my
chosen code on the server. I have no control over the HTTP server and
FTP server. Once I get that I have all kinds of options. Real soon
now has stretched to over a decade.

A custom Java FTP client can talk to a standard FTP server.

Arne

 

[Martin Gregorie]

Or Cygwin, which sports "ssh".

I was assuming that Roedy doesn't have either installed: if PuTTY can do
what he needs its the best choice since all it does is implement SSH for
Windows. Yes, I know it does/did implement Telnet as well, but that's
less added cruft than pulling in a shell and the core UNIX utilities.

 

[Lew]

I was assuming that Roedy doesn't have either installed: if PuTTY can do
what he needs its the best choice since all it does is implement SSH for
Windows. Yes, I know it does/did implement Telnet as well, but that's
less added cruft than pulling in a shell and the core UNIX utilities.

Two points:
- How is that "cruft"?
- These posts are public and long-lived. Many people will read them besides the OP. Those folks might find the benefits of Cygwin compelling, and should be aware of the alternative. That "cruft" that you disparage could bethe very added value that makes Cygwin better than Putty for some individuals. It does for me. Why would I opt for Putty, which is a non-standard solution and does not include "find", "ls", "grep" or the rest? Blecch!

YMMV.

 

[Daniel Pitts]

Two points:
- How is that "cruft"?
- These posts are public and long-lived. Many people will read them besides the OP. Those folks might find the benefits of Cygwin compelling, and should be aware of the alternative. That "cruft" that you disparage could be the very added value that makes Cygwin better than Putty for some individuals. It does for me. Why would I opt for Putty, which is a non-standard solution and does not include "find", "ls", "grep" or the rest? Blecch!

YMMV.

I like to combine the two and used "MinTTY" for the terminal used by Cygwin.

Though now I'm on a MacBook Pro, so I needn't worry about such nonsense ;-)

 

[Roedy Green]

These posts are public and long-lived.

Amen. Questions should be answered in general, not just tuned for
the OP, me in this particular case. This is not a help desk. It is a
discussion meant for a wide audience.

When OPs complain they are given information they already know or that
is not specifically germane to their case, I have to refrain myself
from smacking them with some comment that reminds them they are not
the only centre of the universe.
--
Roedy Green Canadian Mind Products
http://mindprod.com
Capitalism has spurred the competition that makes CPUs faster and
faster each year, but the focus on money makes software manufacturers
do some peculiar things like deliberately leaving bugs and deficiencies
in the software so they can soak the customers for upgrades later.
Whether software is easy to use, or never loses data, when the company
has a near monopoly, is almost irrelevant to profits, and therefore
ignored. The manufacturer focuses on cheap gimicks like dancing paper
clips to dazzle naive first-time buyers. The needs of existing
experienced users are almost irrelevant. I see software rental as the
best remedy.