path: root/docs/chapters/import.sgml

<chapter id="gbp.import">
    <title>Importing Sources</title>

    <sect1 id="gbp.import.existing">
    <title>Importing already existing &debian; packages</title>
    <para>Importing an already existing &debian; package into a &git; repository is as easy as:
<programlisting>
  &gbp-import-dsc; package_0.1-1.dsc
</programlisting>
    This will create a new &git; repository named after the imported package, put
    the upstream sources onto the <option>upstream-branch</option> and the
    &debian; patch on the <option>debian-branch</option>. In case of a &debian;
    native package, only the <option>debian-branch</option> is being used.
    You can specify alternative branch names via the
    <option>--upstream-branch</option> and <option>--debian-branch</option>
    options, or via the <option>upstream-branch</option> and
    <option>debian-branch</option> options in the configuration file.
    </para>
    <para>
    If you want to be able to exactly recreate the original tarball
    (orig.tar.gz) from &git;, you should also specify the
    <option>--pristine-tar</option> option.  This is recommended.
    </para>
    <para>
    If you want to import further versions, you can change into your shiny new
    &git; repository and just continue with the same command:
<programlisting>
  cd package/
  &gbp-import-dsc; package_0.1-2.dsc
  &gbp-import-dsc; package_0.1-3.dsc
  &gbp-import-dsc; package_0.2-1.dsc
</programlisting>
    </para>
    <para>
Or you can import all versions at once using &gbp-import-dscs;:
<programlisting>
  &gbp-import-dscs; /path/to/history/package_*.dsc
</programlisting>
This will create a &git; repository if necessary and import all versions sorted
by version number.
    </para>
    <para>You can also import all versions of a package known from the
    <ulink url="http://snapshot.debian.org/">snapshot.debian.org</ulink> service
    using the <option>--debsnap</option> option of &gbp-import-dscs;:
<programlisting>
  &gbp-import-dscs; --debsnap package
</programlisting>
    </para>
    </sect1>

    <sect1 id="gbp.import.new.upstream">
    <title>Importing a new upstream version</title>
    <para>Change into your &git; repository (which can be empty), make sure it
    has all local modifications committed, and run either of:
<programlisting>
  &gbp-import-orig; <filename>/path/to/package_0.2.orig.tar.gz</filename>
  &gbp-import-orig; <filename>/path/to/package_0.2.tar.bz2</filename>
  &gbp-import-orig; <filename>/path/to/package-0.2/</filename>
</programlisting>
    This puts the upstream sources onto the <option>upstream-branch</option> and
    tags them accordingly (the default tag format is
    <replaceable>upstream/%(version)s</replaceable>).
    The result is then merged onto the <option>debian-branch</option>,
    and a new &debian; changelog entry is created. You can again specify
    different branch names via the <option>--upstream-branch</option> and
    <option>--debian-branch</option> options.
    </para>
    <para>If you are using <filename>debian/watch</filename> to keep track of how to retrieve upstream sources, you can also simply use the <option>--uscan</option> option:
<programlisting>
  &gbp-import-orig; --uscan
</programlisting>
    </para>
    <para>You can also filter out content
    you don't want imported:
<programlisting>
  &gbp-import-orig; <option>--filter</option>=<replaceable>'CVS/*'</replaceable> <filename>/path/to/package_0.2.orig.tar.gz</filename>
</programlisting>
    The <option>--filter</option> option can be used multiple times for more
    complex filtering.
    </para>
    <para>
    If you expect a merge conflict, you can delay the merge to the
    <option>debian-branch</option> via the <option>--no-merge</option> option and pull in
    the changes from the <option>upstream-branch</option> later.
    </para>
    <para>
    If you want to be able to exactly recreate the original tarball
    (orig.tar.gz) from &git;, you should also specify the
    <option>--pristine-tar</option> option.  This is recommended.
    </para>
    <para>To customize the commit message used by &gbp-import-orig;, use
    the <option>--import-msg</option> option. This string is a standard
    Python format string, into which the
    <replaceable>version</replaceable> variable is interpolated. (i.e.,
    use <replaceable>%(version)s</replaceable> in your message to get
    the imported upstream version).
    </para>

    <sect2>
      <title>Recovering from errors</title>
      <para>In case of an error &gbp-import-orig; will by default
	rollback (undo) all changes it has done to the repository so far:
      </para>
      <programlisting>
	$ &gbp; import-orig --verbose --filter='CVS/*' --filter='.bzr/*' --filter='.hg/*' --filter='.svn/*' --upstream-version=1.9 ../gif2apng-1.9.tar.gz
	gbp:info: Importing '../gif2apng-1.9.tar.gz' to branch 'upstream' (filtering out ["'CVS/*'", "'.bzr/*'", "'.hg/*'", "'.svn/*'"])...
	gbp:info: Source package is gif2apng
	gbp:info: Upstream version is 1.9
	gbp:info: Merging to 'master'
	gbp:error: Automatic merge failed.
	gbp:error: Error detected, Will roll back changes.
	gbp:info: Rolling back branch upstream by resetting it to a6733c1f1e190ac0ed4774abc3466e9986a6df5e
	gbp:info: Rolling back branch pristine-tar by resetting it to 0ee24ac614c920e30af82d602882c2ee841c88e5
	gbp:info: Rolling back tag upstream/1.9 by deleting it
	gbp:info: Rolling back branch master by resetting it to ce99782336e83a56e8e579b3cdadf93b0c19e1a8
	gbp:info: Rolling back failed merge of upstream/1.9
	gbp:error: Rolled back changes after import error.
      </programlisting>
      <para>
      In this case the import failed due to a merge conflict. Other
      reasons are running out of disk space, problems when generating
      the pristine-tar delta. If you don't want &gbp-import-orig; to
      undo changes made to the repository use
      the <option>--no-rollback</option>.
      </para>
    </sect2>
    </sect1>

    <sect1 id="gbp.import.convert">
    <title>Converting an existing &git; repository</title>
    <para>
    If the &git; repository wasn't created with &gbp-import-dsc;, you have to
    tell &gbp-buildpackage; and friends where to find the upstream sources.
    </para>
    <sect2>
    <title>Upstream sources on a branch</title>
    <para>
    If the upstream sources are already on a separate branch, things are pretty
    simple. You can either rename that branch to the default
    <option>upstream-branch</option> name <emphasis>upstream</emphasis> with:
<programlisting>
  &gitcmd; branch upstream theupstream-branch
  &gitcmd; branch <option>-D</option> theupstream-branch
</programlisting>
    or you can tell &gbp-buildpackage; the name of the branch to use as
    <option>upstream-branch</option>:
<programlisting>
<command>cat</command> &lt;&lt;EOF &gt; <filename>.git/gbp.conf</filename>
[DEFAULT]
# this is the upstream-branch:
upstream-branch=theupstream-branch
EOF
</programlisting>
    If you then use &gbp-import-orig; to import new upstream sources, they will
    from now on end up on <emphasis>theupstream-branch</emphasis> and
    merged to the <option>debian-branch</option>.
    </para>
    </sect2>
    <sect2>
    <title>Upstream sources not on a branch</title>
    <para>
    If you don't have an upstream branch but started your repository with only
    the upstream sources (not the &debian; patch), you can simply branch from that
    point. So use &gitkcmd; or &gitcmd;-log to locate the commit-id of that commit
    and create the upstream branch from there, e.g.:
<programlisting>
    COMMIT_ID=`&gitcmd; log --pretty=oneline | tail -1 | awk '{ print $1 }'`
    &gitcmd; branch upstream $COMMIT_ID
</programlisting>
    The important thing here is that the <envar>COMMIT_ID</envar> specifies a
    point on the master branch that carried <emphasis>only</emphasis> the
    upstream sources and not the &debian; modifications. The above example
    assumes that this was the first commit to that repository.
    </para>
    <warning><para>There's currently no <emphasis>easy</emphasis> way to create the
    <option>upstream-branch</option> if you never had the upstream sources
    as a single commit. Using &gbp-import-orig; on such repositories might lead
    to unexpected merge results.</para></warning>
    <para>In order to fix this you can prepend the upstream sources as a
    single commit to your tree using &git;'s <ulink
    url="http://git.or.cz/gitwiki/GraftPoint">grafts</ulink>. Afterwards you
    can simply create a branch as explained above and &gbp-import-orig; will
    work as expected.</para>
    <para>Alternatively, if you are only importing source from original tarballs
    (for instance when converting from a Subversion repository where the
    mergeWithUpstream was set for svn-buildpackage), you can create an empty
    upstream branch with the following commands:
<programlisting>
  <command>git checkout</command> <option>--orphan</option> <replaceable>upstream</replaceable>
  <command>git rm</command> <option>-rf</option> <replaceable>.</replaceable>
  <command>git commit</command> <option>--allow-empty</option> <option>-m</option> <replaceable>'Initial upstream branch.'</replaceable>
  <command>git checkout</command> <option>-f</option> <replaceable>master</replaceable>
</programlisting>
    With Git versions lower than 1.7.2.3, the commands are slightly more complicated:
<programlisting>
  <command>git symbolic-ref</command> <replaceable>HEAD</replaceable> <replaceable>refs/heads/upstream</replaceable>
  <command>git rm</command> <option>--cached</option> <option>-r</option> <replaceable>.</replaceable>
  <command>git commit</command> <option>--allow-empty</option> <option>-m</option> <replaceable>'Initial upstream branch.'</replaceable>
  <command>git checkout</command> <option>-f</option> <replaceable>master</replaceable>
</programlisting>
    </para>
    </sect2>
    </sect1>

    <sect1 id="gbp.import.fromscratch">
    <title>Starting a &debian; package from scratch</title>
    <para>
    So far, we assumed you already have a &debian; package to start with, but
    what if you want to start a new package? First, create an empty repository:
    </para>
<programlisting>
  <command>mkdir</command> package-0.1
  <command>cd</command> package-0.1
  <command>git init</command>
</programlisting>
    <para>Then, you import the upstream sources, branch off the
    <option>upstream-branch</option> branch and add the &debian; files (e.g. via dh_make):
<programlisting>
  &gbp-import-orig; <option>-u</option> <replaceable>0.1</replaceable> <filename>../package-0.1.tar.gz</filename>
  <command>dh_make</command>
</programlisting>
    That's it, you're done. If you want to publish your new repository, you can use &gbp-create-remote-repo;.
    </para>
    </sect1>

    <sect1 id="gbp.import.upstream-git">
    <title>When upstream uses Git</title>
    <para>
      If upstream already uses git for packaging, there are several ways to handle packaging. Two of them will
      be described in a bit detail here:
    </para>

      <sect2 id="gbp.import.upstream.git.notarball">
	<title>No upstream tarballs</title>
	<para>If upstream doesn't build upstream tarballs, or you don't care about them, the simplest
	  way is to clone upstream's repository and create a separate packaging branch in there.
	</para>

	<para>
	  In order to help &gbp-buildpackage; to find upstream tags, you need to specify the format
	  using the <option>--git-upstream-tag</option> command line option or the the <option>upstream-tag</option>
	  configuration variable.
	</para>

	<para>
	  A common upstream format is to put a <replaceable>v</replaceable> in front of the version number.
	  In this case, the configuration option would look like:
	</para>
<programlisting>
[git-buildpackage]
upstream-tag = v%(version)s
</programlisting>
	<para>
	  <replaceable>version</replaceable> will be replaced with the upstream version number as read from
	  <filename>debian/changelog</filename>.
	</para>
	<para>If you're using &pristine-tar;, you can make &gbp-buildpackage; commit the generated tarball back to the
	  pristine-tar branch by using the <option>--git-pristine-tar-commit</option> option. This will make sure
	  others building your package can regenerate the tarball you generated for building the &debian; package.
	</para>

	<sect3>
	  <title>Upstream tag version mangling</title>
	<para>
	  Since version 0.8.7, &gbp-buildpackage; supports basic version mangling for upstream tags via
	  substitution. The substitution syntax is as follows:
	</para>
<programlisting>
[git-buildpackage]
upstream-tag = v%(version%A%B)s
</programlisting>
	<para>
	  In this example, each occurrence of <replaceable>A</replaceable> will be replaced by <replaceable>B</replaceable>
	  in the upstream version number. An occasional upstream format is to use tag <replaceable>v1_2_3</replaceable>
	  for release <replaceable>1.2.3</replaceable>. Use <replaceable>v%(version%.%_)s</replaceable> to transform
	  <replaceable>1.2.3</replaceable> into <replaceable>v1_2_3</replaceable>.
	</para>
	<para>
	  If you need to use the <replaceable>%</replaceable> character in either of the substitution strings,
	  you have to escape it. E.g. <replaceable>%(version%-%\%)s</replaceable> will replace <replaceable>-</replaceable> with
	  <replaceable>%</replaceable>, transforming <replaceable>1-A.B.C</replaceable> to <replaceable>1%A.B.C</replaceable>.
	  <replaceable>%(version%\%%.)s</replaceable> on the other hand will replace <replaceable>%</replaceable> with
	  <replaceable>.</replaceable>, turning <replaceable>X%Y%Z</replaceable> into <replaceable>X.Y.Z</replaceable>.
	</para>
	</sect3>

	<sect3>
	  <title>Step by step</title>
	<para>To not make any assumptions about &gbp;'s configuration, the following steps have all options given
	  in its long versions on the command line. You can add these
	  to &gbp.conf; to save lots of typing.
	</para>

	<para>First, we clone the upstream repository. To avoid any ambiguities between the &debian; packaging repository
	  and the upstream repository, we name the upstream repository <replaceable>upstream</replaceable> instead of the
	  default <replaceable>origin</replaceable>.
<programlisting>
  <command>git clone</command> --no-checkout -o upstream git://git.example.com/libgbp.git
  <command>cd</command> libgbp
  <command>git checkout</command> -b debian/sid v1.0
</programlisting>
	  The above makes sure we have <replaceable>debian/sid</replaceable> for the &debian; packaging. We didn't create
	  any <replaceable>upstream/*</replaceable> branches; they're not needed for the packaging and only need to be
	  kept up to date. After adding the &debian; packaging, we build the package. This assumes you're using &pristine-tar;
	  and upstream uses a version number format as described above:
<programlisting>
  <command>gbp buildpackage</command> --git-pristine-tar --git-pristine-tar-commit --git-upstream-tag='v%(version)s' --git-debian-branch=debian/sid
</programlisting>
	  When updating to a new upstream version, we simply fetch from upstream and merge in the new tag. Afterwards, we
	  update the changelog and build the package:
<programlisting>
  <command>git fetch</command> upstream
  <command>git merge</command> v1.1
  <command>gbp dch</command> --debian-branch=debian/sid --snapshot --auto debian/
  &gbp-buildpackage; --git-ignore-new --git-pristine-tar --git-pristine-tar-commit --git-upstream-tag='v%(version)s'
</programlisting>
	  Note that the above &gbp-dch; call makes sure we only pickup changes in the <filename>debian/</filename>
	  directory. Since we told it to build a snapshot changelog entry and we hadn't commit the changelog yet,
	  we need to tell &gbp-buildpackage; that the working directory is unclean via the <option>--git-ignore-new</option> option.
	  Once everything looks good, commit the changelog and build a release version:
<programlisting>
  <command>gbp dch</command> --release --auto --git-debian-branch=debian/sid
  <command>git commit</command> -m"Release 1.1-1" debian/changelog
  &gbp-buildpackage; --git-upstream-tag='v%(version)s' --git-debian-branch=debian/sid
</programlisting>
	  If you want to share your repository with others, you can use &gbp-create-remote-repo; and &gbp-pull; as usual.
	</para>
	</sect3>
      </sect2>

      <sect2 id="gbp.import.upstream.git.tarball">
	<title>Upstream tarballs</title>
	<para>If you want to track upstream's &git; but continue to import the upstream tarballs,
	  e.g. to make sure the tarball uploaded
	  to &debian; has the same checksum as upstream's, you can use the <option>--upstream-vcs-tag</option> option
	  when importing new tarballs with &gbp-import-orig;. Assuming you have the upstream source in your
	  repository with a tag <replaceable>v0.0.1</replaceable>, you can use:
<programlisting>
  &gbp-import-orig; --upstream-vcs-tag=v0.0.1 foo_0.0.1.orig.tar.gz
</programlisting>
	  to add upstream's tag as additional parent to the merge commit.
	  See <ulink url="http://bugs.debian.org/664771">#664771</ulink> for more details.
	</para>
      </sect2>

    </sect1>
    <sect1 id="gbp.branch.naming">
      <title>Branch layout</title>
      <para>
	By default, &gbp; uses one branch to keep the &debian; packaging called <emphasis>master</emphasis>
	and a branch to keep the upstream packaging called <emphasis>upstream</emphasis>.
      </para>
      <para>
	This layout is simple to get started but falls short if one needs to maintain several versions of
	the package at the same time. Therefore the following layout is recommended:
      </para>

      <variablelist>
	<varlistentry>
	  <term>
	    debian/&lt;release&gt;
	  </term>
	  <listitem>
	    <para>
	      the &debian; packaging for a release <emphasis>jessie</emphasis>,
	      <emphasis>stretch</emphasis>, <emphasis>sid</emphasis>, <emphasis>jessie</emphasis>, <emphasis>jessie-backports</emphasis>, <emphasis>jessie-security</emphasis>
	      or <emphasis>experimental</emphasis>.
	    </para>
	  </listitem>
	</varlistentry>
	<varlistentry>
	  <term>
	    upstream/&lt;release&gt;
	  </term>
	  <listitem>
	    <para>
	      the upstream sources for a release matching one of the above
	    </para>
	  </listitem>
	</varlistentry>
	<varlistentry>
	  <term>
	    dfsg/&lt;release&gt;
	  </term>
	  <listitem>
	    <para>
	      the DFSG-clean upstream sources in case the cleanup is done via a &git;
	      merge from upstream to this branch.
	    </para>
	  </listitem>
	</varlistentry>
      </variablelist>
      <para>
	In case &pristine-tar; is being used, there will be a single <emphasis>pristine-tar</emphasis>
	branch that keeps all binary deltas.
      </para>
    </sect1>
</chapter>

<!--  LocalWords:  tarballs
 -->