Version Control for VHDL Project

[MikeWhy]

3. Have the new design reference the new widget as in #2. But when you
get to the point of releasing something and want to be able to create a
tag that you can use as a real archive so that you go back and pull the
design as it really existed at some prior time, but still have the
benefits of referencing a single widget design when you're in development
mode, then you need to use 'svncopy.pl'
(http://svn.collab.net/repos/svn/trunk/contrib/client-side/svncopy/svncopy.README).

This is one particular area of source control where MS Source Safe beats
Subversion hands down. The fact that you can't reference a widget in some
repository and be able to later tag it down to a specific revision at
design complete without a lot of effort (#2) or having to use scripts to
help along the source control tool's weakness (#3) is a major weakness,
'specially considering that this is after all a source control tool.

Unless I missed the full import of the context, for it to work in VSS, it
has to all reside in the one repository, and so works even better in SVN.
SVN doesn't need tags or labels. A single revision number refers to the
entire state of the repository, including branches. You run into trouble
only if your work spans different branches, but that's a problematic
workflow issue regardless of the version control tool. For example, your
project is in the trunk, but you branched the libraries to support your
local work. The better solution is to branch lower so they have a common
root. If they span repositories, ... you might want to reconsider the
repository schema.

 

[MikeWhy]

Unless I missed the full import of the context, for it to work in VSS, it
has to all reside in the one repository, and so works even better in SVN.
SVN doesn't need tags or labels. A single revision number refers to the
entire state of the repository, including branches. You run into trouble
only if your work spans different branches, but that's a problematic
workflow issue regardless of the version control tool. For example, your
project is in the trunk, but you branched the libraries to support your
local work. The better solution is to branch lower so they have a common
root. If they span repositories, ... you might want to reconsider the
repository schema.

[ps] Oops. Scratch that. I've successfully avoided needing SVN externals
altogether, but they solve the problem of cross branches or repositories.
Thought I'd save everyone the trouble of correcting me. Once the externals
are setup, the project can always be reverted to the right state for a given
revision.

 

[MikeWhy]

Creating tags using 'svn copy' though doesn't lock down the revision of
externals when creating the tag so when you go back later to get the tag
and check it out, anything from an 'svn_external' will return the 'head'
revision of the referenced folder, not the revision that existed at the
time that you created the tag...

The externals property takes an optional revision number:
fooPath -rNnnn URL

Since properies are versioned, the externals will track properly when you
manually track the external revision.

 

[KJ]

The externals property takes an optional revision number:
fooPath -rNnnn URL

Since properies are versioned, the externals will track properly when you
manually track the external revision.

That's good if you really like *manually* setting revisions on all
svn_externals. When you're getting ready to release and really lock
down on a final version, the last thing you want is some manual
process like this to be in the loop.

From a 'two years down the road' support perspective of being able to
exactly recreate all source files as they really were at the time of
release, the last thing you want is to have to hope and pray that
someone manually did this to each and every svn_external (on each and
every release) when you know that they were under the gun to get the
thing released and out the door.

Neither scenario is a good one, which is why if you use svn_external
you really need to use something like svncopy.pl to create the tags
for release and not rely on 'svn copy' or TortoiseSVN or the methods
discussed by the Subversion folks to create a tag.

KJ

 

[KJ]

But the labels in VSS actually allow one to easily mark everything so that
you can come back years later and recover it exactly. I'm not pushing VSS,
actually I like Subversion and Tortoise better than VSS, I just happen to
use both.

Once the externals are setup, the project can always be reverted to the
right state for a given revision.

Try the following experiment (or just read along and trust that it really
turns out the way that I say it does...the ending has quite the twist for
those who think that Subversion tags actually do anything useful from an
archival standpoint)

1. Setup a new repository
2. Add new folders called 'Project1' and 'Project2' to the repo.
3. Export 'Project1' to some folder
4. Add the svn_external property to the working copy of 'Project1' to
reference 'Project2'.
5. Do an update to get Project2 into the working copy.
6. Create a new text file called 'Project2_File.txt' in the Project2 folder.
7. Edit Project2_File.txt so it has the single line of text 'This is line
#1'
8. Add Project2_File.txt into 'Project2' and commit the changes to both
'Project1' and 'Project2'.

At this point, what you've emulated is the creation of a new design called
'Project1' that refers to some presumably reusable module called 'Project2'.
Consider for the moment that reusable modules always begin life somewhere
and many times this begining is during the development of some bigger thing.
So in this case 'Project1' is the *bigger* thing, and 'Project2' is the
newly created (but intended to be reusable) module.

In this scenario both projects are likely both being developed concurrently
and, for the most part, up until towards the end of the development cycle
'Project1' will most likely always want to be referring to the
latest/greatest (i.e. the 'head' revision in Subversion-speak). For that
scenario then, one would most likely want to leave the revision off of the
'svn_external' reference thereby defaulting to the 'head'. You might not
want to, but indulge me. Anyway, so far, so good.

9. Create your tag of 'Project1' using 'svn copy' or TortoiseSVN so that you
get the warm fuzzy feeling that you've successfully archived away the
current state of 'Project1' at this moment in time confident that you can
come back at any time to recover this entire project exactly as it was left
after step #8. To do so you will of course be specifying the exact revision
of 'Project1' to use to create the tag. Lets say this tagged version of
'Project1' is put into the archive at 'Tags/Project1_1.0'

Step 9 emulates what one would be doing when you've completed development
and are getting ready to release. The purpose of the tag
'Tags/Project1_1.0' is to permanently archive away the 'Project1' design and
any of its subprojects for posterity so that they can be recreated at any
point in the future regardless of any future changes to either 'Project1' or
'Project2'.

10. Checkout 'Tags/Project1_1.0' to a folder 'Project1_1.0_Step10'
11. Open up the file ''Project1_1.0_Step10\Project2\Project2_File.txt' and
it will of course have one line of text "This is line #1"...again all is
well and good

12. Go back to the original working copy where you created 'Project1', or
better yet delete all of the folders so there are no working copies. Then
check out 'Project1' (not the archived tag, but the original) to create a
working copy in 'Project1-NEW'
13. Edit the file 'Project1-NEW\Project2\Project2_File.txt' by adding a new
line "This is line #2" and commit it to the repository.

Steps 12 and 13 emulate entering design changes into the 'trunk', perhaps to
add new functionality, bug fixes, documenation, whatever. You've dutifully
avoided changing the 'tag' branch, you've made modifications only to the
trunk as per all the Subversion recommended procedures (actually I've
skipped over the creating of a branch and the merge back into the trunk just
for simplicity).

14. Now let's checkout 'Tags/Project1_1.0' again, this time to the folder
'Project1_1.0_Step_14'

Step 14 is emulating the attempted recovery of the tag of 'Project1' as it
was archived back on step 9. This would be a normal support activity, the
re-creation of all of the source files as they existed at some previous time
to investigate some odd customer reported activity.

15. Open the file 'Project1_1.0_Step_14\Project2\Project2_File.txt'

16. Pick your jaw up, wipe the sweat off your brow, or do whatever it is
that you think is appropriate when you see that this file (checked out from
the safely and dutifully archived 'tag' remember) now has two lines of text
This is line #1
This is line #2

17. Explain to me how you think you've archived the actual source files and
can re-create any of your released builds because you've 'tagged' them.

---
Now this story could have had a happier ending. You could vehemently object
and say that in step #4 when the 'svn_external' was first added that by
golly you would never have let this go, you would always have specified a
particular revision. Fair enough, and if 'Project2' is always some locked
down, well debugged, never ever going to change thing and that's the only
type of thing you would ever reference via 'svn_external' you might have a
point. But remember the scenario I was painting, both 'Project1' and
'Project2' were being developed simultaneously, both subject to change.
I'll boldly make the following claims about such an approach
- Very, very little is ever likely to meet the standard of "locked down,
well debugged, never ever going to change"
- You're creating a lot more needless work for yourself by manually
specifying revisions for every 'svn_external'...and when you think about it,
this extra work is because the version control system you're using has
trouble recreating a specific revision.
- You haven't considered that others might not be quite as diligent as you
about always manually adding revisions, so then what?

A few questions come to mind: How much effort do you think is worth
expending to cover the omissions of others? Ask yourself why a highly
regarded version control system should have such a glaring deficiency in
actually being able to recover archived versions properly? Don't work
yourself into a dither though, help may be on the way.

The method to recover from all the bad things that you realized in step #16
might be are (but beware the bogey on step 22):
18. Google for svncopy.pl. Download it, it's a PERL script
19. Google for and download and install some version of PERL if necessary.
20. Go to your repository log and figure out which revision of your
repository created the tag that you did in Step #9. I'm sure you added a
log message entry during your commit to facilitate this
21. Type in the following (replacing xxxx with the revision you determined
in #20, and of course replacing the repo links as appropriate for the source
and destination of the copy
perl svncopy.pl --tag --revision xxxx file:///C:/Mysvn/Project1
file:///C:/Mysvn/Tags/Project1_1.0-Svncopy
22. Hope that you don't get the error message:
Can't call method "temp_dir" on an undefined value at \program
files\subversion\svncopy.pl line 221.
23. If you're lucky with step #22, you should now have a correctly tagged
'Project1' that has really archived the entire revision. Repeat steps 12
and 13 and see if it really sticks. If so, then exhale.

If you're not so lucky on step #22 and get the error message, please post
the solution to this problem. Even Google failed to come up with the
solution although it appears many have run across this problem. If you're
stuck having this problem, exhale anyway, holding your breath doesn't really
help.

Kevin Jennings

P.S. I am looking for the solution to the error message in a Windows
environment if anyone can provide it.
P.P.S. Obviously this post has more to do with Subversion then VHDL so I
apologize but there do seem to be some misconceptions among this group about
just how useful a 'tag' really is from an archival perspective when using
'svn_external' to reference reusable widgets.

 

[MikeWhy]

I wish I had sent the <AOL>Me too!</AOL> earlier rather than deleting it. It
might have saved you a lot of typing. Fundamentally, I agree with your
observation, that svn:externals have archival problems. But I come to a
different conclusion. If you need to use them, the externals must all have
explicit revision references. Otherwise, they are little more useful than
`ln -s` soft links to a local working copy of the foreign repository. In
fact, they are worse in some respects. You get the worst of both worlds:
archival issues, AND continual integration issues as you blindly track their
updates. The soft-link at least allows you control over when you update the
foreign source.

My day job is software. The revision id is post-processed into the binaries
and debug symbols so the toolchain can pull the right source versions for
autopsies. With unrestrained externals in the tree, traceability is already
compromised without talking about archiving. It isn't completely broken, as
the externals version can be narrowed _manually_ by approximate time and
date. But you should know that anything that interferes with a smooth
workflow on a 02:00 server outage brings painful wrath immediately.

The problem with SVN externals is not so much their implementation, but
their visibility. They're essentially hidden without tools that show clearly
the state, that they're external, and to which revision they are locked if
it is or isn't. I don't believe that Tortoise does this, but I have no
experience with externals. It was a wise rule: NO FREAKING EXTERNALS. I've
reaffirmed it with the thought exercise.

An additional comment below...


....

21. Type in the following (replacing xxxx with the revision you determined
in #20, and of course replacing the repo links as appropriate for the
source and destination of the copy
perl svncopy.pl --tag --revision xxxx file:///C:/Mysvn/Project1
file:///C:/Mysvn/Tags/Project1_1.0-Svncopy
22. Hope that you don't get the error message:
Can't call method "temp_dir" on an undefined value at \program
files\subversion\svncopy.pl line 221. ....

P.S. I am looking for the solution to the error message in a Windows
environment if anyone can provide it.

Many tools have a problem with spaces in the path name. Have you tried
simply moving the script to a directory with a simpler name? E:\tools and
E:\apps became my local convention after being bitten too many times by
cross-platform packages.

 

[Mike Treseler]

I wish I had sent the <AOL>Me too!</AOL> earlier rather than deleting
it. It might have saved you a lot of typing.

Yes, thanks to KJ for sharing the hard knocks.

... You get the worst
of both worlds: archival issues, AND continual integration issues as you
blindly track their updates. The soft-link at least allows you control
over when you update the foreign source.

Since an svn copy is essentially free,
and since almost anything might be reused,
what is the downside to keeping all of
an organization's vhdl code in the same repository?

It was a wise rule: NO FREAKING
EXTERNALS. I've reaffirmed it with the thought exercise.

*Me too*, at least until I understand their upside.

-- Mike Treseler

 

[MikeWhy]

Yes, thanks to KJ for sharing the hard knocks.


Since an svn copy is essentially free,
and since almost anything might be reused,
what is the downside to keeping all of
an organization's vhdl code in the same repository?


*Me too*, at least until I understand their upside.

So... what are the benefits? If it's a single repository, I can see building
a virtual view of the tree, with just the parts I want, and arranged more
meaningfully in my local project. It makes sense sometimes to lift
libraries, say, so they are more accessible and visible, below the project
itself. That's not a problem for archiving, since they are snapped from the
same repository, with the same revision ids. In this case, you wouldn't lock
down the revisions unless you had a specific need for a particular revision.
Referencing a branch with the external probably makes as much or more sense
than explicitly tagging the external revision.

What about foreign repositories? The choices there are to import them into
your own repository. Leaving it as a foreign external, I would think you
want to lock down the revision.

 

[KJ]

*Me too*, at least until I understand their upside.

One upside is that you don't have to try to remember on which project you
used 'widget' where you had made some improvement over the last time you
used 'widget'. All the reusable components would be in one place, a new
design project would simply svn_external reference them to get them into the
new design and then add new code for whatever is unique to the new design.
That way you don't have multiple similar (or the same) versions of 'widget'
around, just one, presumably the 'best' in that it has been refined over
time with various improvements.

The huge archive downside though makes svn_external something to be
extremely careful about ever using. The headache of manual revision
specification (and the potential to overlook at least one of them) is just
stupid.

KJ

 

[Marcus Harnisch]

Hi KJ

Your "how-things-can-go-wrong" example is purely based on a work flow
that the inventors of SVN explicitly advise against.[1] It isn't as
hard as you make it look like.

Here's my interpretation of what you really want: Rather than having
to explicitly specifying revisions, Subversion ought to automatically
remember the current(?) state of any externally referenced
workspace. All metadata for doing this is available.

How's the system supposed to behave in different scenarios and how do
you specify the different behavior in each case?


Project1 -+- src
|
+- data
|
`-- Project2 [external reference]

1. Update Project1 workspace without also updating Project2 workspace,
because you really want to stick to one version of Project2 for the
time being.

2. Update Project1 workspace including updating Project2 workspace to
HEAD.

3. Update Project1 workspace including updating Project2 workspace to
a specific revision.

4. When you checkout a new working copy of Project1, which version of
Project2 will be used? HEAD, the one that was in the Project1
workspace that was last checked in or a specific revision? How much
work is involved tracking the actual changes (not just looking at
revision numbers) in Project2 since you last touched Project1? This
would be necessary to gain confidence you actually want that release.

Neither approach saves you from having to do this, but the argument
that having to explicitly specify a revision number makes things
considerably worse just doesn't hold any water.

Staying on top of all this involves much more work than adding an
explicit revision number to some text entry field. OIW, the extra
effort you complain about is neglectible in projects big enough to
make it worthwhile thinking about externals to begin with.

At least the Subversion externals mechanism lets everyone have her
way. I prefer that over tools that are trying to outsmart developers.

Lastly, you had doubts about your fellow co-workers being able to
properly maintain an externals property with revision numbers:

How much effort do you think is worth expending to cover the
omissions of others?

How about a trivial pre-commit hook checking that all externals have
explicit revision numbers. Which, BTW, could be -rHEAD, too. Which
shows another benefit: When I check out a working copy and get
Project2@HEAD I know that this is because sbdy explicitly wanted me to
get this and not because the tool happens to behave this way by
default.

Best regards
Marcus


Footnotes:
[1] "You should seriously consider using explicit revision numbers in all
of your externals definitions. [...]"
http://svnbook.red-bean.com/en/1.5/svn.advanced.externals.html

 

[KJ]

Hi KJ

Your "how-things-can-go-wrong" example is purely based on a work flow
that the inventors of SVN explicitly advise against.[1]

Marcus,

The workflow I suggested is an appropriate workflow if you have the
following goals in mind:
- You create modules intended to be reused, and...
- Those reusable modules get enhanced or changed over time, and...
- You need to be able to accurately reproduce all of the source files for a
particular build sometime in the future, and...
- You don't want to burden developers with a task that is easy to get wrong
or be incomplete

That SVN explicitly advises against it, simply indicates that those
developers do not have the above list of criteria in mind when they use SVN.
That's fine, they don't have to, others do though and other tools do support
that workflow.

One simple reason a supposedly 'reusable' module would change would simply
be because when it was first created and used, not necessarily every
optimization (or perhaps assertion checking) was put in. When you go to
reuse the module in some later design, you might improve it. That
improvement should propogate to all new users (i.e. checkouts from 'head')
but should never propogate to archival tags, ever.

Many developers don't meet the above criteria because
- They don't create reusable modules, they start from scratch or copy paste
snippets from here and there for every new design
- They create perfect reusable modules that can not possibly be improved on
- Never have to go back to investigate a problem peculiar to a particular
design that is not the current head revision.
- They have the time to burn having developers type in specific revision
numbers for each and every module (hoping they get every one of them) THEN
produce the final build for qualification testing. And have even more time
to do this whole exercise again for the next build that occurs the next
day...

It isn't as hard as you make it look like.

In what way did I make it look harder than it is? I think I described the
steps accurately as well as describing the scenario and pressures that the
developers would be under at the particular time and the repercussions down
the road for making a bad move.

Here's my interpretation of what you really want: Rather than having
to explicitly specifying revisions, Subversion ought to automatically
remember the current(?) state of any externally referenced
workspace.

No, Subversion doesn't need to remember anything. It just needs to be able
to have a way to create a real archival tag. A reader of the Subversion
manual can be led to believe that doing the 'svn copy' thing to create a tag
does this, but in fact it doesn't if there is an svn_external with an
unspecified revision anywhere in the design. I'm not saying that what
Subversion has it wrong, just saying that it is missing a key piece if you
meet the criteria I outlined above. Google for svncopy.pl, find their
readme for more info on why you should be using that to create your archival
tags and not the native 'svn copy'

All metadata for doing this is available.

That's possibly how svncopy.pl can perform it's magic.

How's the system supposed to behave in different scenarios and how do
you specify the different behavior in each case?

I think I described the different scenarios adequately in the previous
postings.

Neither approach saves you from having to do this, but the argument
that having to explicitly specify a revision number makes things
considerably worse just doesn't hold any water.

Unless there is some very good reason (and there can be), I don't think
there should be any specific revision numbers called out for any
svn_externals that are referenced in the trunk or any code intended to be
merged back into the trunk in the future (i.e. any time you create new
code). The time when specific revision numbers should be added is when
you're creating an archival tag (i.e. a new build that will be tested). The
adding of those specific revision numbers during the creation of the tag
should be an automated process that replaces all unspecified revisions with
the current (head) revision of whatever thing is being referenced by that
svn_external (regardless of whether that thing is in the same physical repo
or not). It should not be manual and subject to errors.

If you have only one svn_external reference in your design, you can probably
handle this without much trouble. If you're really reusing modules though,
I'd wager that you have a lot more than that. Whether you think that is
'considerably worse' or not probably depends directly on whether or not
you've ever been on the back end and had to recreate the source to
investigate some problem on an old design.

Staying on top of all this involves much more work than adding an
explicit revision number to some text entry field. OIW, the extra
effort you complain about is neglectible in projects big enough to
make it worthwhile thinking about externals to begin with.

We'll just have to disagree on this point, 'specially when you're talking
about big projects.

At least the Subversion externals mechanism lets everyone have her
way. I prefer that over tools that are trying to outsmart developers.

Again, I never asked SVN to outsmart anyone. I'm just saying that it
doesn't natively provide a way to handle the workflow I mentioned (which is
OK) but it also tends to mislead one into thinking that it is correctly
archiving (which is not OK). The fact that svncopy.pl picks up the slack is
great...now if someone could fix svncopy.pl so that IT always worked.

Lastly, you had doubts about your fellow co-workers being able to
properly maintain an externals property with revision numbers:


How about a trivial pre-commit hook checking that all externals have
explicit revision numbers. Which, BTW, could be -rHEAD, too.

As I said, specific revision numbers for the most part should not be
commited. It's when you want to create an archival copy (i.e. a new build
to test) that they are needed. As for the pre-commit hook...so, if the tool
can't do what it should be capable of doing, then create a nag message? We
think differently on that point I guess.

Kevin Jennings

 

[MikeWhy]

Many developers don't meet the above criteria because
- They don't create reusable modules, they start from scratch or copy
paste snippets from here and there for every new design
- They create perfect reusable modules that can not possibly be improved
on
- Never have to go back to investigate a problem peculiar to a particular
design that is not the current head revision.
- They have the time to burn having developers type in specific revision
numbers for each and every module (hoping they get every one of them) THEN
produce the final build for qualification testing. And have even more
time to do this whole exercise again for the next build that occurs the
next day...

Wow. That's a mouthful of disdain. How about: I don't have time for
externals and have successfully avoided needing them. Not because we don't
reuse modules. The opposite is true. We've learned to reference them in
place.

 

[Mike Treseler]

One upside is that you don't have to try to remember on which project you
used 'widget' where you had made some improvement over the last time you
used 'widget'. All the reusable components would be in one place, a new
design project would simply svn_external reference them to get them into the
new design and then add new code for whatever is unique to the new design.
That way you don't have multiple similar (or the same) versions of 'widget'
around, just one, presumably the 'best' in that it has been refined over
time with various improvements.

The huge archive downside though makes svn_external something to be
extremely careful about ever using. The headache of manual revision
specification (and the potential to overlook at least one of them) is just
stupid.

I will stick with tagging my own reusable
vhdl library units and keeping all vhdl code in one repos.

The svn users know not to update a tags directory
even if they don't bother with formal reuse.
If a tag update does happen, there is an audit trail.

Upsides:
-If I update a library unit, I just copy a new tag.
-All changes are observable.
-No fussing with externals.
-Disinterested developers are not burdened by my reuse.

Downsides:
-No formal reuse enforcement or marketing.
-Interested developers might reinvent the wheel.

-- Mike Treseler

 

[Marcus Harnisch]

Your "how-things-can-go-wrong" example is purely based on a work flow
that the inventors of SVN explicitly advise against.[1]

That SVN explicitly advises against it, simply indicates that those
developers do not have the above list of criteria in mind when they use SVN.
That's fine, they don't have to, others do though and other tools do support
that workflow.

Sorry, my mistake. What I meant was not your workflow per se, but the
fact that externals references be used without explicit version
number.

In what way did I make it look harder than it is? I think I described the
steps accurately as well as describing the scenario and pressures that the
developers would be under at the particular time and the repercussions down
the road for making a bad move.

Sure you did. But as you wrote in that same message, you chose not to
put a revision number in the externals property, despite the fact that
you were aware of the issue. I guess you could say that your provoked
the result.

No, Subversion doesn't need to remember anything. It just needs to be able
to have a way to create a real archival tag.

But it would help you, I think. First of all, in SVN there is no such
thing as an archival tag. *Every* revision is one implicitly. I know
that other tools treat archiving specially but SVN doesn't. That's one
reason why it is so simple. If Subversion automatically remembered the
current revision of a checked out sub-project Project2 when you commit
Project1, you would be able to restore the exact state of Project1 at
any time, no?


Thinking about it, I suppose you could come up with a pre-commit (or
perhaps start-commit) script, that scans your externals property and
updates revision tags for all sub-projects.

A reader of the Subversion manual can be led to believe that doing
the 'svn copy' thing to create a tag does this.
Hardly.

Google for svncopy.pl, find their readme for more info on why you
should be using that to create your archival tags and not the native
'svn copy'.

Hmm, half of the job of svncopy.pl (absolute paths) has been obsoleted
by changes in SVN's externals syntax. This wouldn't apply to me
personally anyway, since I prefer using separate repositories for
projects.

The other half (maintaining the externals revisions) could be handled
by a hook (see above), so that users don't even have to worry about this.

Unless there is some very good reason (and there can be), I don't think
there should be any specific revision numbers called out for any
svn_externals that are referenced in the trunk or any code intended to be
merged back into the trunk in the future (i.e. any time you create new
code). The time when specific revision numbers should be added is when
you're creating an archival tag (i.e. a new build that will be tested). The
adding of those specific revision numbers during the creation of the tag
should be an automated process that replaces all unspecified revisions with
the current (head) revision of whatever thing is being referenced by that
svn_external (regardless of whether that thing is in the same physical repo
or not). It should not be manual and subject to errors.

Since "archival tags" are not part of the SVN concept, here is how it
might work. Given a hook script as described above, you may settle on
a repository structure similar to the quasi standard "trunk, branches,
tags". The proposed hook could fix the externals revisions whenever
you are about to commit something into "tags". Commiting to other
directories would not touch your revision-less externals property.

As I said, specific revision numbers for the most part should not be
commited.

The revisions information about your "archival tag" has to be stored
somwhere. Whether automatically managed or not, it *has* to be
committed.

Regards
Marcus

 

[Andy Peters]

I wish I had sent the <AOL>Me too!</AOL> earlier rather than deleting it. It
might have saved you a lot of typing. Fundamentally, I agree with your
observation, that svn:externals have archival problems. But I come to a
different conclusion. If you need to use them, the externals must all have
explicit revision references. Otherwise, they are little more useful than
`ln -s` soft links to a local working copy of the foreign repository. In
fact, they are worse in some respects. You get the worst of both worlds:
archival issues, AND continual integration issues as you blindly track their
updates. The soft-link at least allows you control over when you update the
foreign source.

If you consider that your subproject modules -- the things you would
include in a larger design using externals -- can be tagged, then
simply adopt the paradigm that tags are immutable. For released
designs you should always use a tagged version of the module, never
the trunk. It's really pretty simple, and it works. There's no need to
keep track of which revision of the submodule (no need to deal with
pinned revisions), and as noted, the svn copy you do to make the tag
is "cheap."

Really -- it works.

-a

 

[KJ]

If you consider that your subproject modules -- the things you would
include in a larger design using externals -- can be tagged, then
simply adopt the paradigm that tags are immutable.

I'm not sure I follow what you're saying. Creating a tag does not
make the tag immutable, nor do I get how one would "adopt a pardigm"
that it is (although I absolutely agree that the thing that is tagged,
and all of the things it references, if any, 'should' not be
changable). The tag simply indicates the revision level of the
repository of a particular project (i.e. the top level of the design)

For released
designs you should always use a tagged version of the module, never
the trunk.

I agree, create the tag and build from that. But if you have an
svn_external with an unspecified revision anywhere then when you go to
retrieve the tagged item one year from now, you won't get the same
thing that you get if you retrieve it today if the thing that is
referenced via the svn_external happens to change between those two
times.

It's really pretty simple, and it works. There's no need to
keep track of which revision of the submodule (no need to deal with
pinned revisions), and as noted, the svn copy you do to make the tag
is "cheap."

Really -- it works.

If so, then you should be able to complete steps 1 thru 16 from my May
20 post and then have an explanation as requested for step #17. So
far nobody has.

If there is some method to use Subversion to meet the following goals
(from the May 22 post) I'd like to hear it.
- You create modules intended to be reused, and...
- Those reusable modules get enhanced or changed over time, and...
- You need to be able to accurately reproduce all of the source files
for a
particular build sometime in the future, and...
- You don't want to burden developers with a task that is easy to get
wrong
or be incomplete

KJ

 

[Charles Gardiner]

On May 26, 4:02 pm, Andy Peters <[email protected]> wrote:

If so, then you should be able to complete steps 1 thru 16 from my May
20 post and then have an explanation as requested for step #17. So
far nobody has.

If there is some method to use Subversion to meet the following goals
(from the May 22 post) I'd like to hear it.
- You create modules intended to be reused, and...
- Those reusable modules get enhanced or changed over time, and...
- You need to be able to accurately reproduce all of the source files
for a
particular build sometime in the future, and...
- You don't want to burden developers with a task that is easy to get
wrong
or be incomplete

KJ

Hi KJ,
I completely agree with Andy. It does work. The 'paradigm change' is in
step 4 of your May 20 post.

The steps are
1. Setup a new repository
2. Add new folders called 'Project1' and 'Project2' to the repo.
3. Export 'Project1' to some folder
4. Add the svn_external property to the working copy of 'Project1' to
reference 'Project2'.

4+ Do not use the working copy of Project2. Tag (svn copy) Project2
first. Reference this tagged version of Project2 in Project1.


Any further development to Project2 is done on the trunk. If you ever
find you have to change the version of Project2 in Project1, make a new
tagged version of Project2 first and change the svn:externals to
reference this.
Never ever import the _trunk_ of a sub project into a super-project. The
trunk is for development, tags are for releases.

I extracted one of my archived projects (about a year old) a few days
ago and it worked just as expected.

Regards,
Charles

 

[KJ]

1. Setup a new repository
2. Add new folders called 'Project1' and 'Project2' to the repo.
3. Export 'Project1' to some folder
4. Add the svn_external property to the working copy of 'Project1' to
reference 'Project2'.

4+ Do not use the working copy of Project2. Tag (svn copy) Project2
first. Reference this tagged version of Project2 in Project1.

Step 4+ is essentially the same thing as referencing a specific revision of
Project2 in Project1. Either way, in order to prepare for a release
candidate build of Project1, one would have to manually make sure that each
and every revision reference points to the latest/greatest (or per your
suggestion, make sure that each sub-project tag is a copy of the
latest/greatest). I'm not saying that this method doesn't work, but (in my
opinion) violates the goal of

Compare this with VSS, in which the equivalent to the busy work of manual
revision checking is to simply label Project1, and then export it. Maybe we
just don't agree that the manual task of making sure that each and every
revision is up to date at the time when you need to create a release
candidate build is a burden, or that it could ever be done incorrectly (like
missing the last update to some module)

I had listed several goals, all of which *should* be achievable by a
revision control tool, but do not appear to be possible using SVN but can
with other tools. I'm not a big fan of creating extra manual tasks that can
be done wrong when working concurrently on a project.

Thanks for your input on how you do things.

KJ

 

[Andy Peters]

I'm not sure I follow what you're saying.  Creating a tag does not
make the tag immutable, nor do I get how one would "adopt a pardigm"
that it is (although I absolutely agree that the thing that is tagged,
and all of the things it references, if any, 'should' not be
changable).  The tag simply indicates the revision level of the
repository of a particular project (i.e. the top level of the design)

By "paradigm" I suppose I really mean "common practice." And the
common practice with Subversion is that you never modify tags. If you
use TortoiseSVN, and you try to commit a change to a tag, TSVN will
complain. It's a kind of gentle reminder.

I agree, create the tag and build from that.  But if you have an
svn_external with an unspecified revision anywhere then when you go to
retrieve the tagged item one year from now, you won't get the same
thing that you get if you retrieve it today if the thing that is
referenced via the svn_external happens to change between those two
times.

That is NOT true. If no changes are committed to the tagged branch
(sorry for mixing svn metaphors here), and the whole point of keeping
tags is that you do NOT change them -- then you WILL get exactly what
you expect. Think of the tag as a mnemonic that refers to the state of
the repository at the time at which the tag was made. Why remember -
r45677 when instead you can remember tag\Release-v1.0.0?

If so, then you should be able to complete steps 1 thru 16 from my May
20 post and then have an explanation as requested for step #17.  So
far nobody has.

If there is some method to use Subversion to meet the following goals
(from the May 22 post) I'd like to hear it.
- You create modules intended to be reused, and...
- Those reusable modules get enhanced or changed over time, and...
- You need to be able to accurately reproduce all of the source files
for a
particular build sometime in the future, and...
- You don't want to burden developers with a task that is easy to get
wrong
or be incomplete

Tags do that! The changes to the reusable modules get made to those
modules' TRUNKS, never the tags.

And if you decide that you like a newer version of your foo module,
tag it as tags\Release-v2.0.0. If you check out any previous designs
that depended on Release-v1.0.0, you will in fact get the state of all
of that module's files as of the Release-v1.0.0 tag! The old tag is
never removed and never changed, and you can use either version as you
see fit.

I am not sure if I can explain this any better.

-a

 

[KJ]

By "paradigm" I suppose I really mean "common practice." And the
common practice with Subversion is that you never modify tags.

I agree, never modify tags...and if you look at my simple example of a usage
of svn_externals in my posting from May 20, in step 9 I said to create a
tag. In steps 10 thru 15 I made a change to the 'trunk' of 'Project2' and
when I re-exported the tag in step #15, I didn't get what was originally
tagged, 'Project2' had stuff from the trunk.

It's not totally clear to me why somebody would prefer such behaviour, and I
don't begrudge them having it if they so choose, but saying that you can
recover exactly what you originally tagged, just because you created a tag
and did not modify the tag is not really correct. You can only make that
statement if there are either no externals or every external is explicitly
revision specified.

That is NOT true. If no changes are committed to the tagged branch
(sorry for mixing svn metaphors here), and the whole point of keeping
tags is that you do NOT change them -- then you WILL get exactly what
you expect.

If you really believe that then you should be able to explain why in my May
20 posting, I got different results from exporting the tag after modifying
only the trunk. The answer, quite simply, (and I knew this going in and you
probably do too) is that the different results from exporting an unmodified
tag are a result of the lack of a revision being specified...OK, so what
mechanism exists to lock down revisions at the time of tag creation?
svncopy.pl is the only mechanism I've run across, but it has problems as
well.

The other mechanism is to explicitly declare revision levels. If done
during concurrent development this task will hinder development since every
user of a widget will have to manually change the revision level in order to
get the latest/greatest from something that has already been tested and
commited by the developer. When it gets near time for product release, this
manual step is something again that can get in the way at a time when
developers are under pressure to get their last fixes in. Charles, in his
reply, suggested another step that gets around the problem, but is
essentially the same as simply locking down a revision (actually a slight
bit more work). Whether or not somebody thinks this cost is 'worth it' or
'not a big deal', is totally up to that person. It certainly appears though
to be extra effort needed so that you can use SVN, but with no real upside
(other than so you can continue to use SVN, or Tortoise...I do by the way).

I don't mind that SVN has 'cheap copies' that cut down on disk usage, but I
think it's an oversight on SVN's part that it doesn't also support a
mechanism to simply take a snapshot of a project (and it's sub-projects) and
lock down any revisions that are not specified. When I do a checkout or an
export, it tells me the revision of each external so it's not like it can't
be done. The fact that svncopy.pl exists at all suggests I'm not alone in
seeing this as something important (although I seem to be almost alone in
this particular newsgroup). I've posted to the Subversion group a (as have
others) and either nothing comes back, or in one case the repliers got
somewhat hostile for suggesting that SVN might be missing something. The
Tortoise newsgroup though seems to agree that this would be a good thing,
but basically thought that it is up to SVN to first provide the capability
so that Tortoise can take advantage of it.

Think of the tag as a mnemonic that refers to the state of
the repository at the time at which the tag was made. Why remember -
r45677 when instead you can remember tag\Release-v1.0.0?

I agree.

Tags do that! The changes to the reusable modules get made to those
modules' TRUNKS, never the tags.

I guess we disagree on this point. But if you can list a process that
really does meet all of the above mentioned goals I'd like to hear it.

And if you decide that you like a newer version of your foo module,
tag it as tags\Release-v2.0.0. If you check out any previous designs
that depended on Release-v1.0.0, you will in fact get the state of all
of that module's files as of the Release-v1.0.0 tag! The old tag is
never removed and never changed, and you can use either version as you
see fit.

I understand that and agree that the v1.0.0 tag will be retrieved under the
conditions you describe...but only if module 'foo' doesn't happen to have
any svn_externals in it. Now stand back for a moment and view your process
from the point of view of doing concurrent development where both the new
widget is being developed as well as the widget being integrated into a
design and evaluate it against these two particular points (which you've
already stated are handled by tags)

- Those reusable modules get enhanced or changed over time, and...
- You don't want to burden developers with a task that is easy to get wrong
or be incomplete

The time period over which the module changes can be short (it's under
development remember, we're not talking about the integration of a stable,
static design). What happens is the integrator is burdened with having to
update his links to point to a new tag, just so he can get the
latest/greatest. Before updating though, he is also burdened with simply
discovering that there is an updated version even available. Multiply that
effort by say 10 modules and again by 4 designs and several changes to
modules and the whole mess has to be released at the end of the month. Is
that really how you want your integrator spending their time? How about the
delay that is created because the integrator has to do this process for each
build...or does this get shortcut because the integrator 'knows' what things
have changed? How do you 'know' that he/she 'knows' and didn't miss
something? The integrator is also burdened with insuring that there are no
unspecified revisions on any externals that may be lurking in all modules
(on every build). The developer has the additional burden of creating a new
tag, as opposed to simply committing his change. How do you insure that
happens every time? How do you check to see that the repo is in that state
prior to a build?

I'm not necessarily expecting answers here to all this, just potential holes
that I see in your process that can only be fixed by manual
intervention...all because SVN doesn't natively support a method of
capturing a snapshot of the repo and being able to truly recreate it later
except under special conditions...conditions that cause more work.

From a more fundamental standpoint, you should always be able to answer
'yes' to the following questions:
- Is your build built from an unchangable tag? (I agree that your method
gets a yes)
- Is your tag built from a snapshot of the trunk? Given your process, you
would have to manually verify this by checking that the tag for each module
reflects what is in the trunk and then verify that the integration of each
module also points to this latest tag. Maybe you can automate this...if
it's not, you can't really guarantee this...you can only hope that mistakes
don't get made by mere mortals.

With my method of unspecified versions on externals, it exposes me only on
the archival front and none of the others. If some Perl wizard would fix
svncopy.pl, the archival hole could be closed as well (or of course, if SVN
added a new form of copy that locked down revisions to whatever they are at
the time that the copy is performed).

I am not sure if I can explain this any better.

You don't have to, thanks for your explanation. I think you're overstating
a bit what tags really do because of the way you happen to choose to use
them. But because of the way you use them, you're exposed on a couple of
the points. You're maybe aware of them anyway, but it was a good
discussion...even if it really had no place in comp.lang.vhdl in the first
place. Apologies to those clv readers that found this boring and off topic.

Kevin Jennings