Current status of cvs to git migrartion

[Alexey Shvetsov]

Hi all!
What is current status of cvs to git migrartion?

[Robin H. Johnson]

Post by Alexey Shvetsov
Hi all!
What is current status of cvs to git migrartion?

There has been ZERO progress since my last status email 'Progress
summary, 2009/08/27'.

A couple of people have stepped up to offer help from items on that
list, and I've asked/assigned them to a particular task, but there
hasn't been much forward momentum.

cvs2svn work, Betelgeuse+dberkholz:
-----------------------------------
pass9 of cvs2svn (and also pass8 to a lesser degree). Mostly
parallelization but I also believe optimizations are possible.
dberkholz most recently was going to try while he was at the GSoC summit
this past weekend.

pre-upload-hook, ford_prefect:
------------------------------
Nothing done yet.

Commit signing:
---------------
Was offered to rbu after his interest in the previous threads, didn't
hear back at all.

Sparse trees:
-------------
Upstream work. See the upstream mailing list, this has made perhaps the
most progress thanks to pclouds, but isn't quite ready yet.

Testing git-cvsserver:
----------------------
Nobody took it up at all.


Alexxy: If you want to test git-cvsserver using the existing snapshot
migration, that would be great. Basically clone to locally, run
git-cvsserver, then cvs checkout from it, and try all normal developer
CVS actions (update, repoman etc) on that "CVS" repo.

[Arun Raghavan]

2009/10/26 Robin H. Johnson <robbat2-aBrp7R+bbdUdnm+***@public.gmane.org>:
[...]

Post by Robin H. Johnson
------------------------------
Nothing done yet.

Spent a short while looking at this. As Robin pointed out, there's
post-upload support in upstream git so basing this off that should be
reasonably straightforward.

I got a little lost with regards to exactly why we we're doing this,
so will figure things out and get on this (hoping to get some time to
attack this in the next 2 weeks).

Cheers,

[Robin H. Johnson]

Post by Arun Raghavan
Spent a short while looking at this. As Robin pointed out, there's
post-upload support in upstream git so basing this off that should be
reasonably straightforward.
I got a little lost with regards to exactly why we we're doing this,
so will figure things out and get on this (hoping to get some time to
attack this in the next 2 weeks).

The body of the pre-upload hook is basically going to function like
this:
====
if(pull is an update) then
// allow
elseif(pull is an initial clone) then
// deny, instruct to use the initial Git bundle, give URL.
end
====

Because doing an initial clone:
- is not restartable (bundles via HTTP/rsync are)
- causes a lot of server load
- is not always optimally packed
- is a huge download.

We fully intend to deny initial full clones. Shallow clones we will
probably allow (they look different in the pre-upload stage).

[Donnie Berkholz]

Post by Robin H. Johnson

Post by Alexey Shvetsov
Hi all!
What is current status of cvs to git migrartion?

There has been ZERO progress since my last status email 'Progress
summary, 2009/08/27'.
A couple of people have stepped up to offer help from items on that
list, and I've asked/assigned them to a particular task, but there
hasn't been much forward momentum.
-----------------------------------
pass9 of cvs2svn (and also pass8 to a lesser degree). Mostly
parallelization but I also believe optimizations are possible.
dberkholz most recently was going to try while he was at the GSoC summit
this past weekend.

...

I must have been living in a fantasy land when I thought I could get any
hacking done at the summit. I was way too busy talking to people to
huddle in a corner.

Still consider it something I want to work on, but please don't let that
stand in anyone else's way if you also want to give it a shot.

[Robin H. Johnson]

So, it's been a couple of months since the last one, and this time I can
report substantial progress that has all taken place within the last 3
weeks.

Executive summary:
------------------
- We can do a conversion pass of CVS to Git in just under 3 hours!
- pre-upload-hook is shaping up, as upstream commit-notes and sparse
checkouts.
- Work in feature development / Git conversions of existing
infrastructure is needed now.

cvs2svn work:
-------------
ferringb stepped up, and kicked the living crap out of the cvs2svn code.
I don't have a complete diff of his changes on hand, but by using
snakeoil, unladen swallow and fixing up some parts of the cvs2svn code,
he's dropped our cvs2svn time to an amazing 110 minutes. (When we
started this process, it was 19 hours, then 9 hours, now sub 2 hours).

I don't think it's going to get much better than this :-).
Time breakdown for conversion, approximate values:
cvs2svn = 110 minutes.
fast-import = 25 minutes
initial repack = 25 minutes

Validation of converted tree:
-----------------------------
Still needs work, hope to work on it in February month myself.

pre-upload-hook, ford_prefect:
-------------------------------
The hook code has gone through two rounds of review between ford_prefect and
myself, and should hopefully be posted to the upstream Git mailing list this
week.

commit signing:
---------------
No news. rbu? Volunteers needed. New ypstream 1.7.x branch DOES have
commit notes support now.

Thin manifest:
--------------
No news.

sparse trees:
-------------
Now available in upstream 1.7.0-rc0, which has been tagged, but no
tarballs released yet.

Testing git-cvsserver:
----------------------
No news. Volunteers needed.

commit hooks/portage to rsync conversion scripts:
-------------------------------------------------
No news. Volunteers needed.

cvs2svn Statistics:
------------------
Total CVS Files: 378069
Total CVS Revisions: 2436256
Total CVS Branches: 120165
Total CVS Tags: 23298
Total Unique Tags: 5
Total Unique Branches: 2
CVS Repos Size in KB: 1632386
Total SVN Commits: 710891
First Revision Date: Fri Jul 28 00:35:42 2000
Last Revision Date: Sun Jan 17 01:25:36 2010
------------------
Timings (seconds):
------------------
905 pass1 CollectRevsPass
94 pass2 CleanMetadataPass
0 pass3 CollateSymbolsPass
425 pass4 FilterSymbolsPass
7 pass5 SortRevisionSummaryPass
0 pass6 SortSymbolSummaryPass
594 pass7 InitializeChangesetsPass
677 pass8 BreakRevisionChangesetCyclesPass
631 pass9 RevisionTopologicalSortPass
194 pass10 BreakSymbolChangesetCyclesPass
607 pass11 BreakAllChangesetCyclesPass
452 pass12 TopologicalSortPass
758 pass13 CreateRevsPass
13 pass14 SortSymbolsPass
13 pass15 IndexSymbolsPass
1254 pass16 OutputPass
6624 total

[Rémi Cardona]

Post by Robin H. Johnson
-------------
ferringb stepped up, and kicked the living crap out of the cvs2svn code.
I don't have a complete diff of his changes on hand, but by using
snakeoil, unladen swallow and fixing up some parts of the cvs2svn code,
he's dropped our cvs2svn time to an amazing 110 minutes. (When we
started this process, it was 19 hours, then 9 hours, now sub 2 hours).

That's fantastic news, and an awesome improvement!

Post by Robin H. Johnson
----------------------
No news. Volunteers needed.

Was there a lot of interest in this? I had the impression fewer and
fewer people actually wanted/needed this...

In any case, thanks to all for this impressive work and the status update.

Cheers,

Rémi

[Robin H. Johnson]

Post by Rémi Cardona

Post by Robin H. Johnson
----------------------
No news. Volunteers needed.

Was there a lot of interest in this? I had the impression fewer and
fewer people actually wanted/needed this...

We wanted it for people that didn't want to change all of their workflow
at once.
1. Clone repo to local system.
2. Do a CVS checkout of what they wanted from the git-cvsserver (running locally).
3. After CVS commit, they have to do Git push themselves still.

It's mainly testing that it works and writing up a set of instructions
for doing so.

Not proposed at this time: Providing a central git-cvsserver instance,
as users would be forced to update very often I think (but testing this
assumption IS needed).

[Fabian Groffen]

Post by Robin H. Johnson

Post by Rémi Cardona

Post by Robin H. Johnson
----------------------
No news. Volunteers needed.

Was there a lot of interest in this? I had the impression fewer and
fewer people actually wanted/needed this...

We wanted it for people that didn't want to change all of their workflow
at once.
1. Clone repo to local system.
2. Do a CVS checkout of what they wanted from the git-cvsserver (running locally).
3. After CVS commit, they have to do Git push themselves still.
It's mainly testing that it works and writing up a set of instructions
for doing so.
Not proposed at this time: Providing a central git-cvsserver instance,
as users would be forced to update very often I think (but testing this
assumption IS needed).

...

I toyed with this myself recently.

I don't see any benefit of having users locally running cvs against
their own bare git repo checkout. With our current unmasked versions of
CVS in tree we need to do hacks with scripts via CVS_RSH to work with
it. I would say that it should be offered (via ssh as normal) by the
main git repo to be useful.

I can't say anything else but that git's cvsserver works extremely well.
It does "atomic multi directory commits" fine, and keeps a revision
number for files like CVS does. All common operations (add, remove,
update, diff, commit, status) work fine, be it with slightly different
messages returned by the server. Keyword expansion doesn't work, so our
Ids wouldn't be updated.

Interaction with the same repo being committed to with CVS and pushed to
with git goes well, conflicts are handled for CVS commits the usual way
by requiring to be uptodate.

I believe (and that is how I am planning to use it initially for one of
the other projects I participate in) that the git-cvsserver makes it
very easy for users in the switch. You only need to recheckout a CVS
working copy, but then you don't have to change any of your habits.
Once you're ready for git, you can switch, and even that you can do
gradually with a cvs and git checkout side by side.

[Robin H. Johnson]

Post by Robin H. Johnson
-------------
Now available in upstream 1.7.0-rc0, which has been tagged, but no
tarballs released yet.

You mean sparse checkouts? This will still clone everything from upstream.

I should clarify yes. The sparseness is merely in the checked out files.
It can however be combined with a shallow clone.

[Jeremy Olexa]

On Thu, 28 Jan 2010 03:37:52 +0000, "Robin H. Johnson"

Post by Robin H. Johnson
So, it's been a couple of months since the last one, and this time I can
report substantial progress that has all taken place within the last 3
weeks.
------------------
- We can do a conversion pass of CVS to Git in just under 3 hours!
- pre-upload-hook is shaping up, as upstream commit-notes and sparse
checkouts.
- Work in feature development / Git conversions of existing
infrastructure is needed now.
-------------
ferringb stepped up, and kicked the living crap out of the cvs2svn code.
I don't have a complete diff of his changes on hand, but by using
snakeoil, unladen swallow and fixing up some parts of the cvs2svn code,
he's dropped our cvs2svn time to an amazing 110 minutes. (When we
started this process, it was 19 hours, then 9 hours, now sub 2 hours).
I don't think it's going to get much better than this :-).
cvs2svn = 110 minutes.
fast-import = 25 minutes
initial repack = 25 minutes
-----------------------------
Still needs work, hope to work on it in February month myself.
-------------------------------
The hook code has gone through two rounds of review between ford_prefect and
myself, and should hopefully be posted to the upstream Git mailing list this
week.
---------------
No news. rbu? Volunteers needed. New ypstream 1.7.x branch DOES have
commit notes support now.
--------------
No news.
-------------
Now available in upstream 1.7.0-rc0, which has been tagged, but no
tarballs released yet.
----------------------
No news. Volunteers needed.
-------------------------------------------------
No news. Volunteers needed.

...

First, thanks for the status report.

I would like to volunteer for this, mainly the "portage to rsync
conversion scripts". We will be in touch off-list to discuss details.
-Jeremy

Post by Robin H. Johnson
------------------
Total CVS Files: 378069
Total CVS Revisions: 2436256
Total CVS Branches: 120165
Total CVS Tags: 23298
Total Unique Tags: 5
Total Unique Branches: 2
CVS Repos Size in KB: 1632386
Total SVN Commits: 710891
First Revision Date: Fri Jul 28 00:35:42 2000
Last Revision Date: Sun Jan 17 01:25:36 2010
------------------
------------------
905 pass1 CollectRevsPass
94 pass2 CleanMetadataPass
0 pass3 CollateSymbolsPass
425 pass4 FilterSymbolsPass
7 pass5 SortRevisionSummaryPass
0 pass6 SortSymbolSummaryPass
594 pass7 InitializeChangesetsPass
677 pass8 BreakRevisionChangesetCyclesPass
631 pass9 RevisionTopologicalSortPass
194 pass10 BreakSymbolChangesetCyclesPass
607 pass11 BreakAllChangesetCyclesPass
452 pass12 TopologicalSortPass
758 pass13 CreateRevsPass
13 pass14 SortSymbolsPass
13 pass15 IndexSymbolsPass
1254 pass16 OutputPass
6624 total

...

[Donnie Berkholz]

Post by Robin H. Johnson
So, it's been a couple of months since the last one, and this time I can
report substantial progress that has all taken place within the last 3
weeks.
------------------
- We can do a conversion pass of CVS to Git in just under 3 hours!
- pre-upload-hook is shaping up, as upstream commit-notes and sparse
checkouts.
- Work in feature development / Git conversions of existing
infrastructure is needed now.
-------------
ferringb stepped up, and kicked the living crap out of the cvs2svn code.
I don't have a complete diff of his changes on hand, but by using
snakeoil, unladen swallow and fixing up some parts of the cvs2svn code,
he's dropped our cvs2svn time to an amazing 110 minutes. (When we
started this process, it was 19 hours, then 9 hours, now sub 2 hours).

...

Robin and Brian,

Thanks so much for your work on this! I'm thrilled that you've found the
time to make things happen and really excited about what you've done.