Net::FTP and IBM TSO :-(

[Chris]

Anyone have any experience using Net::FTP going against IBM TSO? As
usual, IBM (just like Microsloth) have their own way of doing things,
unfortunately. It would *appear* from using debug in a real FTP client
and turning on Net::Cmd debugging that the real FTP client sends an
'EPSV' command just before each 'get' on a TSO dataset. Trying to
simulate this behavior using Net::FTP doesn't seem to do the job.
Sending or not sending EPSV results in the same thing using Net::FTP --
a download timeout. So now I'm faced with the possibility of doing an
external 'ftp -s' hack... :-(

Here is the main part of the script with all the Net::FTP commands I am
using, the XML file I am using, and the debug output from each
(Net::FTP and real ftp client running on Linux). The hostname,
username, password and the gets I want to do are defined in an XML
file. So don't worry about those details in the script too much. It's
irrelevant. All the information from the XML file (handled by
XML::Simple) is fine -- I've verified the data I need to drive this is
being picked up and used just fine. It's the actual $ftp->get() that
isn't working with or without the $ftp->quot( 'EPSV' ) command:

__PERL__

## Get hostname, username, and password, decrypt password if encrypted,
and login to
## host via FTP.

my $dlConfig = $config->{Download};
my $lnConfig = $dlConfig->{Login};
$lnConfig->{Password} = decrypt( $lnConfig->{Password} )
if $lnConfig->{PasswordEncrypted};
my $ftp = Net::FTP->new( $lnConfig->{Host}, Debug => $lnConfig->{Debug}
|| 0 ) ||
error( "Can't open session to host /$lnConfig->{Host}/!" );
$ftp->login( $lnConfig->{Username}, $lnConfig->{Password} ) ||
error( "Can't log in to host /$lnConfig->{Host}/ as username
/$lnConfig->{Username}/!" );

## Get all files designated in get section of download section.

my $gets = getCollection( $dlConfig->{Gets}->{Get} );
for my $get (@{$gets}) {
puts( " Getting: Remote( $get->{Remote} ) -> Local( $get->{Local}
)" );
$ftp->quot( 'EPSV' ); # Doesn't seem to matter if I use this or
not...
$ftp->get( $get->{Remote}, $get->{Local} ); # Line 69 in Perl debug
session below
}
$ftp->quit(); # Line 71 in Perl debug session below

sub puts { for (@_) { print "$_\n" } }
sub getCollection { ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ] }

__END__

__XML__

<?xml version="1.0" encoding="utf-8"?>
<config>
<Download>
<Login>
<Host>ftphost.some.private.net</Host>
<Username>foouser</Username>
<Password>foo.password</Password>
<PasswordEncrypted>0</PasswordEncrypted>
<Debug>1</Debug>
</Login>
<Gets>
<Get Remote="'FOO.SOX.COBOL(GEN1)'" Local="/tmp/foouser.cob"
/>
<Get Remote="'FOO.SOX.COBOL(GEN1)'" Local="/tmp/a.cob" />
<Get Remote="'FOO.SOX.COBOL(GEN1)'" Local="/tmp/b.cob" />
</Gets>
</Download>
</config>

__END__

Real ftp client debug session:

chris@mylinux:~/work/foo> ftp ftphost.some.private.net
Connected to ftphost.some.private.net.
220-FTPHOST IBM FTP CS V1R6 at FTPHOST.SOME.PRIVATE.NET, 21:52:49 on
2006-03-14.
220 Connection will close if idle for more than 5 minutes.
Name (ftphost.some.private.net:chris): foouser
331 Send password please.
Password:
230 FOOUSER is logged on. Working directory is "FOOUSER.".
Remote system type is MVS.
ftp> debug
Debugging on (debug=1).
ftp> get 'FOO.SOX.COBOL(GEN1)' x.cob
local: x.cob remote: 'FOO.SOX.COBOL(GEN1)'
ftp: setsockopt (ignored): Permission denied
---> EPSV
229 Entering Extended Passive Mode (|||1874|)
---> RETR 'FOO.SOX.COBOL(GEN1)'
125 Sending data set FOO.SOX.COBOL(GEN1) FIXrecfm 80
250 Transfer completed successfully.
114308 bytes received in 00:00 (281.03 KB/s)
ftp> quit
---> QUIT
221 Quit command received. Goodbye.


Net::FTP debug session:

colive@mylinux:~/work/foo> ./download-dbload-foo.pl -i
download-dbload-foo.xml download-dbload-foo.pl (v1.00-000) - Download &
DB Load Foo Feed
(c) Copyright 2006, My Big Company - All rights reserved

Net::FTP>>> Net::FTP(2.75)
Net::FTP>>> Exporter(5.58)
Net::FTP>>> Net::Cmd(2.26)
Net::FTP>>> IO::Socket::INET(1.28)
Net::FTP>>> IO::Socket(1.28)
Net::FTP>>> IO::Handle(1.24)
Net::FTP=GLOB(0x87846d0)<<< 220-FTPHOST IBM FTP CS V1R6 at
FTPHOST.SOME.PRIVATE.NET, 21:57:41 on 2006-03-14.
Net::FTP=GLOB(0x87846d0)<<< 220 Connection will close if idle for more
than 5 minutes.
Net::FTP=GLOB(0x87846d0)>>> user foouser
Net::FTP=GLOB(0x87846d0)<<< 331 Send password please.
Net::FTP=GLOB(0x87846d0)>>> PASS ....
Net::FTP=GLOB(0x87846d0)<<< 230 FOOUSER is logged on. Working
directory is "FOOUSER.".
Getting: Remote( 'FOO.SOX.COBOL(GEN1)' ) -> Local( /tmp/foouser.cob
)
Net::FTP=GLOB(0x87846d0)>>> EPSV
Net::FTP=GLOB(0x87846d0)<<< 229 Entering Extended Passive Mode
(|||1875|)
Net::FTP=GLOB(0x87846d0)>>> PORT 172,17,4,47,4,80
Net::FTP=GLOB(0x87846d0)<<< 200 Port request OK.
Net::FTP=GLOB(0x87846d0)>>> RETR 'FOO.SOX.COBOL(GEN1)'
Net::FTP=GLOB(0x87846d0): Timeout at ./download-dbload-foo.pl line 69
Net::FTP=GLOB(0x87846d0)>>> QUIT
Net::FTP=GLOB(0x87846d0): Timeout at ./download-dbload-foo.pl line 71


The "get" on a.cob and b.cob in the XML file above is never done
because I took them out for the debug session above. I left them in
the XML file above to demonstrate the intent of the getCollection() and
for loop for the gets.

If I take the $ftp->quot( 'EPSV' ) line out, I get the exact same
results as the above Net::FTP session save for seeing EPSV sent to the
host, of course.

If anyone has any ideas, I've love to hear about them. I grow weary of
IBM and Microsoft and their "implementation" of TCP/IP standards. :-/
Sorry if anyone here works for them, but those are my candid
sentiments.

-ceo

 

[usenet]

Anyone have any experience using Net::FTP going against IBM TSO?

Yeah, I've got a nightly cron script at work that uses Net::FTP from an
AIX box to an MVS TSO big-iron host, but it does a put(), not a get().
It's a very old script, but I don't remember having any problems with
it - pretty straightforward. But, like I say, I put, not get.

Have you tried a put as a test?

I can post a bit of relevant code when I get in the office tomorrow.

 

[Chris]

Yeah, I've got a nightly cron script at work that uses Net::FTP from an
AIX box to an MVS TSO big-iron host, but it does a put(), not a get().
It's a very old script, but I don't remember having any problems with
it - pretty straightforward. But, like I say, I put, not get.

Have you tried a put as a test?

No, I suppose I could try this. Maybe it would shed some light on the
situation.

I can post a bit of relevant code when I get in the office tomorrow.

Thanks. I'll look forward to it.

Meanwhile, I'm digging out and updating a Net:suedoFTP I wrote
somewhere in 2000 that mimics most of the regular Net::FTP commands,
but on $ftp->quit() executes an ftp script. It's a dropin replacement
practically for Net::FTP. If there is any interest in it, I can see
about posting it to CPAN.

-ceo

 

[usenet]

Thanks. I'll look forward to it.

I did one better - I created a simple script (adapted from my nightly
cron script) to get() the file that I usually put() to the host in my
nightly cron. This works perfectly from my AIX workstation connecting
to my MVS TSO host:

#!/usr/bin/perl

use Net::FTP;
my %ftp = qw{ host 123.456.78.90
userid MYUSER
password MYPASS
remote_dir DEPT.TFOO.BAR
remote_file INDEX
local_file foo.txt };

$ftp = Net::FTP->new($ftp{'host'}) || die $!;
$ftp->login ($ftp{'userid'}, $ftp{'password'}) || die $!;
$ftp->cwd ("..") || die $!;
$ftp->cwd ($ftp{'remote_dir'}) || die $!;
$ftp->ascii () || die $!;
$ftp->get ($ftp{'remote_file'}, $ftp{'local_file'}) || die $!;
$ftp->quit ();

__END__

 

[Chris]

I did one better - I created a simple script (adapted from my nightly
cron script) to get() the file that I usually put() to the host in my
nightly cron. This works perfectly from my AIX workstation connecting
to my MVS TSO host:

#!/usr/bin/perl

use Net::FTP;
my %ftp = qw{ host 123.456.78.90
userid MYUSER
password MYPASS
remote_dir DEPT.TFOO.BAR
remote_file INDEX
local_file foo.txt };

$ftp = Net::FTP->new($ftp{'host'}) || die $!;
$ftp->login ($ftp{'userid'}, $ftp{'password'}) || die $!;
$ftp->cwd ("..") || die $!;
$ftp->cwd ($ftp{'remote_dir'}) || die $!;
$ftp->ascii () || die $!;
$ftp->get ($ftp{'remote_file'}, $ftp{'local_file'}) || die $!;
$ftp->quit ();

__END__

Hmmm, I see you are doing some cwd's and maybe positioning yourself
better than I am before doing the get. My get requires the tick marks
(') around the dataset name, but it's only because I'm getting it
"fully qualified" for lack of any other term I know to use because I
know precious little about TSO (and I'd like to keep it that way).

I'm beginning to wonder if I can finagle this using something closer to
your approach. I'll get a hold of one of our TSO guys and learn enough
TSO to get this to work, hopefully. This obviously is not the place to
get TSO help...!

Thanks much!
-ceo

 

[usenet]

Hmmm, I see you are doing some cwd's and maybe positioning yourself

Ya know, that does ring a faint, distant bell (but it was a long time
ago). I think I tried to first process the fully-qualified
fileset/member name but that didn't work. So I cwd'ed to the fileset
and processed the member as a plain name, which worked. If I
understood TSO better I might know why... but, as you say, this isn't a
TSO group (and most folks here probably have no idea what TSO is, and
those folks are lucky!)

 

[DJ Stunks]

Hmmm, I see you are doing some cwd's and maybe positioning yourself
better than I am before doing the get. My get requires the tick marks
(') around the dataset name, but it's only because I'm getting it
"fully qualified" for lack of any other term I know to use because I
know precious little about TSO (and I'd like to keep it that way).

I have no idea what either of you are talking about, but remember that
if you want to pass 's in a string, using q// is probably a good way to
do so.

I'm sure both of you already know that, but I hadn't posted in a while
and I heard Paul was going to start keeping stats...

HTH!
-jp

 

[Chris]

I have no idea what either of you are talking about,

Are you insinuting that we *do* know what we are talking about when
we're trying our best to indicate we don't (insofar as TSO is
concerned)...!??!!! ;-)

but remember that
if you want to pass 's in a string, using q// is probably a good way to
do so.

Yeah, I'm grabbing the filename string from an XML file and the ' is in
the filename, essentially. If you check out my debug session (the one
trying to use Net::FTP) in the OP, you'll see that the 'FILENAME' is
being passed literally, as it should be. So no worries there.

This is a TSO nuance is my understanding. I think IBM TSO stands for:

(I)'ll (B)et (M)oney this will (T)ick (S)omeone (O)ff.

So that will get you up to speed on what IBM TSO means. And yep --
they were right.

I'm sure both of you already know that, but I hadn't posted in a while
and I heard Paul was going to start keeping stats...

Tally 'em up!

-ceo

 

[l v]

Yeah, I'm grabbing the filename string from an XML file and the ' is in
the filename, essentially. If you check out my debug session (the one
trying to use Net::FTP) in the OP, you'll see that the 'FILENAME' is
being passed literally, as it should be. So no worries there.

The OP may want to verify the version of their Net::FTP is the highest
avaiable. I once had some issues with Net::FTP, upgrading it corrected
my problem.

Mainframe file names are quite simple IMO.

Here are the file name translations to unix (using
(e-mail address removed)'s example program):

1) a . (dot) is a /
so DEPT.TFOO.BAR is DEPT/TFOO/BAR

2) if the file name is surrounded by single quotes, then that is the
full file name.
'DEPT.TFOO.BAR' is equlivant to /DEPT/TFOO/BAR

3) if the file name is *not* surrounded by single quotes, then the
login user name is prepended.
DEPT.TFOO.BAR is equlivant to 'MYUSER.DEPT.TFOO.BAR' which is
equlivant to /home/MYUSER/DEPT/TFOO/BAR

4) A mainframe file name can not begin with a digit. I think there are
some other restrictions.

Where it get confusing is that a file (dataset) on the mainframe can be
of two types, 1) a sequential dataset, or 2) a partitioned dataset.
You need to know which type you are dealing with.

A sequential dataset is a normal every day file that you would use OPEN
and CLOSE on. You use DEPT.TFOO.BAR to retrieve

A partition dataset (PDS) contains many files within it. Consider this
a directory. You retrieve the file using a file name like
'MYUSER.DEPT.TFOO.BAR(filename)' and need to do this for every file
within the PDS you want to ftp.

The following is an edited version of what I run, I hope I did not snip
too much.

use Net::FTP;

my $jclLib = 'sysx.p.jcllib';

### logon and ftp spool file to mainframe.
my $ftp = Net::FTP->new("$host", Debug => 0);
$ftp->login("$acf2Id", "$acf2Pw") || fail("Could not login - $@");
$ftp->site("lrecl=132 recfm=fb") || fail("Could not send site commands
- $@");
$ftp->put("$mvsFile.data", "'$mfFile'") || fail("Could not put
$mvsFile.data, '$mfFile' - $@");

# now get jcl
$ftp->get("'$jclLib(sp12)'", "$fileName.bjcl.in") || fail("Could not
get $jclLib(sp12 - $@");

# now put the edited jcl on the mainframe.
$ftp->site("lrecl=80 recfm=fb") || fail("Could not send site commands 2
- $@");
$ftp->put("$fileName.bjcl", "'tmp.foo.$mvsFile.bjcl'") || fail("Could
not put $fileName.bjcl - $@");

# now send the edited NDM jcl to the mainframe's JES queue to execute
the JCL.
$ftp->site("filetype=jes lrecl=80 recfm=fb") || fail("Could not send
site commands to send JCL to JES - $@");
$ftp->put("$fileName.jcl") || fail("Could not put $fileName.jcl - $@");

# now end the ftp session.
$ftp->quit || fail("Could not quit ftp session - $@");

Len

 

[Rick Scott]

This is a TSO nuance is my understanding. I think IBM TSO stands for:

(I)'ll (B)et (M)oney this will (T)ick (S)omeone (O)ff.

So that will get you up to speed on what IBM TSO means. And yep --
they were right.

OH! So you mean that the entertaining interactions I had with IBM's
software while writing an AIX package-handling script weren't a one-off
occurrence, then? I thought I was the only one! =)

=head1 BUGS

B<update.fixget> does not automatically recognize and repair
memos that are misformatted, mangled, confusing,
self-contradictory, or inutile. At least, not yet.




Rick

 

[/usr/ceo]

The OP may want to verify the version of their Net::FTP is the highest
avaiable. I once had some issues with Net::FTP, upgrading it corrected
my problem.

Mainframe file names are quite simple IMO.

Here are the file name translations to unix (using
(e-mail address removed)'s example program):

1) a . (dot) is a /
so DEPT.TFOO.BAR is DEPT/TFOO/BAR

2) if the file name is surrounded by single quotes, then that is the
full file name.
'DEPT.TFOO.BAR' is equlivant to /DEPT/TFOO/BAR

3) if the file name is *not* surrounded by single quotes, then the
login user name is prepended.
DEPT.TFOO.BAR is equlivant to 'MYUSER.DEPT.TFOO.BAR' which is
equlivant to /home/MYUSER/DEPT/TFOO/BAR

4) A mainframe file name can not begin with a digit. I think there are
some other restrictions.

Where it get confusing is that a file (dataset) on the mainframe can be
of two types, 1) a sequential dataset, or 2) a partitioned dataset.
You need to know which type you are dealing with.

A sequential dataset is a normal every day file that you would use OPEN
and CLOSE on. You use DEPT.TFOO.BAR to retrieve

A partition dataset (PDS) contains many files within it. Consider this
a directory. You retrieve the file using a file name like
'MYUSER.DEPT.TFOO.BAR(filename)' and need to do this for every file
within the PDS you want to ftp.

The following is an edited version of what I run, I hope I did not snip
too much.

use Net::FTP;

my $jclLib = 'sysx.p.jcllib';

### logon and ftp spool file to mainframe.
my $ftp = Net::FTP->new("$host", Debug => 0);
$ftp->login("$acf2Id", "$acf2Pw") || fail("Could not login - $@");
$ftp->site("lrecl=132 recfm=fb") || fail("Could not send site commands
- $@");
$ftp->put("$mvsFile.data", "'$mfFile'") || fail("Could not put
$mvsFile.data, '$mfFile' - $@");

# now get jcl
$ftp->get("'$jclLib(sp12)'", "$fileName.bjcl.in") || fail("Could not
get $jclLib(sp12 - $@");

# now put the edited jcl on the mainframe.
$ftp->site("lrecl=80 recfm=fb") || fail("Could not send site commands 2
- $@");
$ftp->put("$fileName.bjcl", "'tmp.foo.$mvsFile.bjcl'") || fail("Could
not put $fileName.bjcl - $@");

# now send the edited NDM jcl to the mainframe's JES queue to execute
the JCL.
$ftp->site("filetype=jes lrecl=80 recfm=fb") || fail("Could not send
site commands to send JCL to JES - $@");
$ftp->put("$fileName.jcl") || fail("Could not put $fileName.jcl - $@");

# now end the ftp session.
$ftp->quit || fail("Could not quit ftp session - $@");

Very handy and informative. I didn't expect to get that kind of help
here and this may help me resolve what issues I have.

-ceo

 

[usenet]

[ an informative discussion of mainframe filenaming ]

Very handy and informative.

I'll second that! I usually just guess around until something works!
It's good to have some actual understanding of what's going on.
Thanks!

 

[Big and Blue]

It would *appear* from using debug in a real FTP client
and turning on Net::Cmd debugging that the real FTP client sends an
'EPSV' command just before each 'get' on a TSO dataset.

As it says, that says the connexion is going int to passive mode. So
instead of the server sending back the data on the ftp-data port it opens a
port, tells you what it is, waits for you to call it then sends the data
back on that. This setup came about to allow access through firewalls,
which wouldn't allow the external servers to open a connexion coming in, but
you (on the inside) can connect to what they've said is open.

Trying to
simulate this behavior using Net::FTP doesn't seem to do the job.

That's because all you ar edoing is sending is sending "PSV", which
isn't sufficient. The application has to handle what the response to this is.

$ftp->quot( 'EPSV' ); # Doesn't seem to matter if I use this or not...

No - the local client needs to work on the response.

Net::FTP=GLOB(0x87846d0)>>> EPSV
Net::FTP=GLOB(0x87846d0)<<< 229 Entering Extended Passive Mode (|||1875|)

IIRC the rmeote side has now opened port 1875 and is waiting for you to
call it...

Net::FTP=GLOB(0x87846d0)>>> PORT 172,17,4,47,4,80

...but you just tell it that it should contact port 80 (!!???) locally.

Net::FTP=GLOB(0x87846d0)<<< 200 Port request OK.
Net::FTP=GLOB(0x87846d0)>>> RETR 'FOO.SOX.COBOL(GEN1)'
Net::FTP=GLOB(0x87846d0): Timeout at ./download-dbload-foo.pl line 69

Presumably there is a firewall in the way

If anyone has any ideas, I've love to hear about them. I grow weary of
IBM and Microsoft and their "implementation" of TCP/IP standards.

This isn't TCP/IP - it's FTP (totally different RFC and sits above TCP).
From what I can see its implemented perfectly - just that you aren't using
it correctly.

And it's not Perl either.

 

[/usr/ceo]

As it says, that says the connexion is going int to passive mode. So
instead of the server sending back the data on the ftp-data port it opens a
port, tells you what it is, waits for you to call it then sends the data
back on that. This setup came about to allow access through firewalls,
which wouldn't allow the external servers to open a connexion coming in, but
you (on the inside) can connect to what they've said is open.


That's because all you ar edoing is sending is sending "PSV", which
isn't sufficient. The application has to handle what the response to this is.



No - the local client needs to work on the response.



IIRC the rmeote side has now opened port 1875 and is waiting for you to
call it...


...but you just tell it that it should contact port 80 (!!???) locally.


Presumably there is a firewall in the way


This isn't TCP/IP - it's FTP (totally different RFC and sits above TCP).
From what I can see its implemented perfectly - just that you aren't using
it correctly.

And it's not Perl either.

I beg to differ. I just tried this from a Windows machine, and IT
WORKED using Net::FTP. Same script, same everything. I did not get a
timeout. The details of what you shared above may be true, but I
expect when I use Net::FTP that it works the same way everywhere.
Either it should timeout in both places or it should work in both
places. And there is no firewall inbetween.

I'll have to look into more of what you've discussed above. I'm aware
of FTP's place in the TCP/IP scheme of things. "TCP/IP" is often used
to discuss those things which utilize that stack. Microsoft and IBM's
implementation of TCP/IP standards be it the underlying protocols (TCP,
UDP, etc), or the general suite of the various RFC implementations as
well as HTTP standards has been suspect at best.

-ceo

 

[Big and Blue]

I beg to differ.

That's your right, but you are the one complaining about IBM and
Microsoft's implementation of IP, TCP and FTP (not sure which) and that is
not Perl.

I just tried this from a Windows machine, and IT
WORKED using Net::FTP. Same script, same everything.

No - a totally different environment, with different environment
variables set.

I did not get a
timeout.

Then I suggest you get some network snooping software (ethereal?) to
find out what is (not) going on at the network level.

I'll have to look into more of what you've discussed above.

In which case bear in mind that this comment:

is wrong - it's actually port 1104 (4*256 + 80).

Microsoft and IBM's
implementation of TCP/IP standards be it the underlying protocols (TCP,
UDP, etc), or the general suite of the various RFC implementations as
well as HTTP standards has been suspect at best.

I've never hgad a problem with Microsoft's TCP/IP (apart from speed). I
agree that its HTTP implementation is pretty awful though. e.g., use IE to
connect to Web host my1.server.com:1234, which redirects to a URI on
my1.server.com:4321 and you'll find that the Host header on the redirect
call says my1.server.com:1234 (ie: still the original one before redirect).
If the server hostname changes all is OK, so I hate to think what the code
looks like that gets this wrong. But you come to expect that of Microsoft.

 

[ednotover]

It would *appear* from using debug in a real FTP client
and turning on Net::Cmd debugging that the real FTP client sends an
'EPSV' command just before each 'get' on a TSO dataset. Trying to
simulate this behavior using Net::FTP doesn't seem to do the job.

Net::FTP states:

Net::FTP is a class implementing a simple FTP client in Perl as
described in RFC959. It provides wrappers for a subset of the RFC959
commands.

EPSV and EPRT came along in RFC2428. So I'd assume they're not
supported in Net::FTP. And that seems to be true based on your debug
log:

Net::FTP=GLOB(0x87846d0)>>> EPSV
Net::FTP=GLOB(0x87846d0)<<< 229 Entering Extended Passive Mode (|||1875|)
Net::FTP=GLOB(0x87846d0)>>> PORT 172,17,4,47,4,80
Net::FTP=GLOB(0x87846d0)<<< 200 Port request OK.

Note that Net::FTP is following your EPSV request with a PORT, not
EPRT.

I just tried this from a Windows machine, and IT WORKED using Net::FTP.

Are you using the same version Net::FTP in both places? What does the
Net::FTP debug show from Windows?

Ed